From 79b6f390dd4b644256e3547ff5c6a0cfb44dcb54 Mon Sep 17 00:00:00 2001 From: qian Date: Sun, 10 May 2026 13:00:48 +0800 Subject: [PATCH] =?UTF-8?q?feat(doc):=E8=A1=A5=E5=85=85=E5=8F=82=E8=80=83?= =?UTF-8?q?=E6=96=87=E6=A1=A3=EF=BC=8C=E4=BE=BF=E4=BA=8EAI=E5=88=86?= =?UTF-8?q?=E6=9E=90?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/reference/api-format-reference.md | 261 ++ docs/reference/goframe-conventions.md | 257 ++ docs/协议文档/API参考-Google-Gemini.md | 1506 +++++++ .../Anthropic-Claude-Messages-API-完整文档.md | 1731 ++++++++ docs/协议文档/Claude协议文档.md | 1036 +++++ docs/协议文档/FLUX及其他图像生成API文档.md | 992 +++++ docs/协议文档/Gemini协议文档.md | 1903 +++++++++ docs/协议文档/Gemini系列/banana介绍.md | 2260 +++++++++++ docs/协议文档/Gemini系列/gemini3介绍.md | 3594 +++++++++++++++++ docs/协议文档/Gemini系列/模型.md | 292 ++ docs/协议文档/Gemini系列/生成内容.md | 3594 +++++++++++++++++ .../OpenAI/OpenAI-Responses-API-完整文档.md | 1752 ++++++++ .../OpenAI/OpenAI-Responses协议文档.md | 1777 ++++++++ docs/协议文档/OpenAI/OpenAI协议文档.md | 1072 +++++ docs/协议文档/OpenAI/OpenAI图像生成文档.md | 637 +++ docs/协议文档/Stability-AI图像生成API文档.md | 745 ++++ docs/协议文档/智谱GLM/GLM图像生成（异步）.md | 167 + docs/协议文档/智谱GLM/GLM对话补全.md | 1700 ++++++++ docs/协议文档/智谱GLM/GLM对话补全（异步）.md | 1276 ++++++ docs/协议文档/智谱GLM/GLM查询异步结果.md | 528 +++ docs/协议文档/豆包系列/豆包图像生成API文档.md | 1209 ++++++ .../阿里系列/通义万相图像生成API文档.md | 1586 ++++++++ 22 files changed, 29875 insertions(+) create mode 100644 docs/reference/api-format-reference.md create mode 100644 docs/reference/goframe-conventions.md create mode 100644 docs/协议文档/API参考-Google-Gemini.md create mode 100644 docs/协议文档/Anthropic-Claude-Messages-API-完整文档.md create mode 100644 docs/协议文档/Claude协议文档.md create mode 100644 docs/协议文档/FLUX及其他图像生成API文档.md create mode 100644 docs/协议文档/Gemini协议文档.md create mode 100644 docs/协议文档/Gemini系列/banana介绍.md create mode 100644 docs/协议文档/Gemini系列/gemini3介绍.md create mode 100644 docs/协议文档/Gemini系列/模型.md create mode 100644 docs/协议文档/Gemini系列/生成内容.md create mode 100644 docs/协议文档/OpenAI/OpenAI-Responses-API-完整文档.md create mode 100644 docs/协议文档/OpenAI/OpenAI-Responses协议文档.md create mode 100644 docs/协议文档/OpenAI/OpenAI协议文档.md create mode 100644 docs/协议文档/OpenAI/OpenAI图像生成文档.md create mode 100644 docs/协议文档/Stability-AI图像生成API文档.md create mode 100644 docs/协议文档/智谱GLM/GLM图像生成（异步）.md create mode 100644 docs/协议文档/智谱GLM/GLM对话补全.md create mode 100644 docs/协议文档/智谱GLM/GLM对话补全（异步）.md create mode 100644 docs/协议文档/智谱GLM/GLM查询异步结果.md create mode 100644 docs/协议文档/豆包系列/豆包图像生成API文档.md create mode 100644 docs/协议文档/阿里系列/通义万相图像生成API文档.md diff --git a/docs/reference/api-format-reference.md b/docs/reference/api-format-reference.md new file mode 100644 index 0000000..72e2d4a --- /dev/null +++ b/docs/reference/api-format-reference.md @@ -0,0 +1,261 @@ +# API 接口格式参考 + +本文档是 CLAUDE.md 中 API 接口规范的详细参考资料，包含完整的 JSON 示例、响应格式、错误类型映射等。开发时按需查阅。 + +## 统一响应格式详细示例 + +**适用范围**：`/api/admin/*`、`/api/tenant/*`、`/api/payment/*`、`/api/open/*`、`/api/status`。 + +### 字段说明 + +| 字段 | 类型 | 说明 | +|------|------|------| +| `code` | int | `0` = 成功，非 `0` = 错误。标准 HTTP 错误直接用状态码（400/401/403/404/409/500），业务错误用 >= 10000 的自定义码 | +| `message` | string | 用户可读的中文提示。成功时固定 `"ok"`，错误时描述具体原因，禁止暴露技术细节 | +| `data` | any | 成功时为业务数据（对象/数组/字符串），错误时为 `null` | +| `request_id` | string | 每个请求的唯一标识，贯穿全链路，用于日志追踪 | + +### 响应示例 + +```json +// 成功（HTTP 200） +{"code": 0, "message": "ok", "data": {"id": 1, "name": "张三"}, "request_id": "req_abc123"} + +// 参数错误（HTTP 400） +{"code": 400, "message": "用户名不能为空", "data": null, "request_id": "req_def456"} + +// 业务错误（HTTP 422） +{"code": 10001, "message": "余额不足", "data": null, "request_id": "req_ghi789"} +``` + +### data 字段格式规范 + +| 接口类型 | data 结构 | 示例 | +|----------|----------|------| +| 分页列表 | `{list, total, page, page_size}` | `{"list": [...], "total": 100, "page": 1, "page_size": 20}` | +| 不分页列表 | `{list}` | `{"list": [...]}` | +| 创建资源 | `{id}` | `{"id": 42}` | +| 资源详情 | 直接返回对象 | `{"id": 1, "name": "张三", ...}` | +| 更新/删除 | `null` | `null` | + +**规则**： +- 列表数据的数组字段名统一用 `list`，禁止使用 `data`、`items`、`records` 等其他名称 +- 分页列表必须返回 `total`（总条数）和 `page`（当前页码），可选返回 `page_size` +- 创建操作只返回新资源的 `id`，不返回完整对象 +- 更新/删除操作返回 `null`，前端通过 HTTP 200 + `code: 0` 判断成功 +- 如果 list 内部可以为空，但是不能有 null 出现 + +### HTTP 状态码映射规则 + +| 业务场景 | HTTP 状态码 | code 值 | 说明 | +|----------|------------|---------|------| +| 成功 | 200 | `0` | 请求处理成功 | +| 参数校验失败 | 400 | `400` | 请求体格式错误、必填字段缺失、值不合法 | +| 未认证 | 401 | `401` | Token 缺失、过期、无效 | +| 无权限 | 403 | `403` | 已认证但无权访问该资源 | +| 资源不存在 | 404 | `404` | 查询的对象不存在 | +| 请求频率超限 | 429 | `429` | 触发限流 | +| 业务规则错误 | 422 | `>= 10000` | 业务逻辑不满足，使用自定义错误码（见 `consts.go`） | +| 服务器内部错误 | 500 | `500` | 未预期的异常 | + +### 业务错误码定义（>= 10000） + +业务错误码定义在 `internal/consts/consts.go` 中，每个错误码有对应的中文消息常量。新增业务错误时必须同时在 `consts.go` 中添加 `Code` 和 `Msg` 常量。 + +| 错误码 | 常量名 | 默认消息 | +|--------|--------|---------| +| 10001 | `CodeInsufficientBalance` | 余额不足 | +| 10002 | `CodeQuotaExceeded` | 额度已用完 | +| 10003 | `CodeChannelUnavailable` | 没有可用的渠道 | +| ... | 更多见 `consts.go` | ... | + +## 大模型代理接口格式（`/v1/*`、`/v1beta/*`、`/suno/*`） + +### 已注册的代理端点 + +| 方法 | 路径 | 功能 | 请求格式 | 流式支持 | +|------|------|------|---------|---------| +| POST | `/v1/chat/completions` | Chat Completions | OpenAI | SSE | +| POST | `/v1/completions` | Text Completions | OpenAI | SSE | +| POST | `/v1/embeddings` | 文本向量 | OpenAI | 否 | +| POST | `/v1/images/generations` | 图像生成 | OpenAI | 否 | +| POST | `/v1/images/edits` | 图像编辑 | OpenAI | 否 | +| POST | `/v1/messages` | Claude Messages | Claude | SSE | +| POST | `/v1/responses` | OpenAI Responses | OpenAI | SSE | +| POST | `/v1/audio/speech` | 语音合成 | OpenAI | 否 | +| POST | `/v1/audio/transcriptions` | 语音转文字 | OpenAI | 否 | +| POST | `/v1/audio/translations` | 语音翻译 | OpenAI | 否 | +| POST | `/v1/rerank` | 重排序 | OpenAI | 否 | +| POST | `/v1/moderations` | 内容审核 | OpenAI | 否 | +| GET | `/v1/models` | 模型列表 | — | 否 | +| GET | `/v1/models/{model_id}` | 模型详情 | — | 否 | +| GET | `/v1/realtime` | 实时对话（WebSocket） | OpenAI | WebSocket | +| POST | `/v1/video/generations` | 视频生成（异步任务） | 自定义 | 否 | +| GET | `/v1/video/generations/{task_id}` | 视频生成任务查询 | — | 否 | +| GET | `/v1beta/models` | Gemini 模型列表 | Gemini | 否 | +| GET | `/v1beta/models/{model}` | Gemini 模型详情 | Gemini | 否 | +| POST | `/v1beta/models/{model}` | Gemini 内容生成 | Gemini | SSE | +| POST | `/suno/submit/{action}` | Suno 音乐生成提交 | 自定义 | 否 | +| POST | `/suno/fetch` | Suno 批量查询 | 自定义 | 否 | +| GET | `/suno/fetch/{task_id}` | Suno 任务查询 | — | 否 | + +### OpenAI 格式端点响应 + +**非流式响应**直接透传上游 JSON，典型结构： + +```json +// Chat Completions 响应 +{ + "id": "chatcmpl-abc123", + "object": "chat.completion", + "created": 1677652288, + "model": "gpt-4o", + "choices": [ + { + "index": 0, + "message": {"role": "assistant", "content": "Hello!"}, + "finish_reason": "stop" + } + ], + "usage": { + "prompt_tokens": 9, + "completion_tokens": 12, + "total_tokens": 21 + } +} +``` + +```json +// Embeddings 响应 +{ + "object": "list", + "data": [ + {"object": "embedding", "embedding": [0.1, 0.2, ...], "index": 0} + ], + "model": "text-embedding-3-small", + "usage": {"prompt_tokens": 5, "total_tokens": 5} +} +``` + +**流式响应**使用 SSE（Server-Sent Events），`Content-Type: text/event-stream`： + +``` +data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1677652288,"model":"gpt-4o","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]} + +data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1677652288,"model":"gpt-4o","choices":[{"index":0,"delta":{"content":"Hello"},"finish_reason":null}]} + +data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1677652288,"model":"gpt-4o","choices":[{"index":0,"delta":{},"finish_reason":"stop"}],"usage":{"prompt_tokens":9,"completion_tokens":12,"total_tokens":21}} + +data: [DONE] +``` + +**OpenAI 格式错误响应**： + +```json +{ + "error": { + "type": "invalid_request_error", + "message": "余额不足，请联系管理员充值", + "param": null, + "code": null + } +} +``` + +**错误类型映射**： + +| 平台错误场景 | error.type | HTTP 状态码 | +|-------------|-----------|------------| +| 认证失败（API Key 无效） | `authentication_error` | 401 | +| 权限不足（Key 无权访问模型） | `permission_error` | 403 | +| 余额不足 / 额度耗尽 | `insufficient_quota` | 402 | +| 模型不存在 / 参数错误 | `invalid_request_error` | 400 | +| 请求频率超限 | `rate_limit_error` | 429 | +| 没有可用渠道 | `server_error` | 503 | +| 上游供应商错误 | 原样透传上游错误类型 | 上游状态码 | +| 平台内部错误 | `internal_error` | 500 | + +### Claude 格式端点响应（`/messages`） + +**非流式响应**： + +```json +{ + "id": "msg_abc123", + "type": "message", + "role": "assistant", + "content": [{"type": "text", "text": "Hello!"}], + "model": "claude-sonnet-4-20250514", + "stop_reason": "end_turn", + "usage": { + "input_tokens": 25, + "output_tokens": 10 + } +} +``` + +**流式响应（SSE）**： + +``` +event: message_start +data: {"type":"message_start","message":{"id":"msg_abc123","type":"message","role":"assistant","content":[],"model":"claude-sonnet-4-20250514","usage":{"input_tokens":25,"output_tokens":0}}} + +event: content_block_delta +data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"Hello!"}} + +event: message_delta +data: {"type":"message_delta","delta":{"stop_reason":"end_turn"},"usage":{"output_tokens":10}} + +event: message_stop +data: {"type":"message_stop"} +``` + +**Claude 格式错误响应**： + +```json +{ + "type": "error", + "error": { + "type": "authentication_error", + "message": "余额不足，请联系管理员充值" + } +} +``` + +### Gemini 格式端点响应（`/v1beta/models/*`） + +使用 Google Gemini API 原生格式透传，错误格式： + +```json +{ + "error": { + "code": 400, + "message": "请求参数错误", + "status": "INVALID_ARGUMENT" + } +} +``` + +### 错误处理实现规范 + +- `WriteRelayError(w, err)` — 写入 OpenAI 格式错误，用于 `/v1/chat/completions`、`/v1/embeddings` 等 OpenAI 格式端点 +- `WriteClaudeRelayError(w, err)` — 写入 Claude 格式错误，用于 `/v1/messages` 端点 +- 平台级错误（余额不足、渠道不可用、频率限制）转换为供应商原生格式的错误类型，消息使用中文 +- 上游供应商错误原样透传，不做二次包装 +- 限流错误额外设置 `X-RateLimit-Limit`、`X-RateLimit-Remaining`、`X-RateLimit-Reset` 响应头 + +## 中间件配置差异 + +| 中间件 | 管理接口 | 代理接口 | +|--------|---------|---------| +| AdminAuth / TenantAuth（JWT） | 使用 | 不使用 | +| ApiKeyAuth（Bearer Token） | 不使用 | 使用 | +| MiddlewareHandlerResponse（统一响应） | 使用 | 不使用 | +| ErrorHandler | 使用（支付回调） | 不使用 | +| MaintenanceMode（维护模式） | 使用（租户） | 使用 | +| ContentFilter（内容过滤） | 不使用 | 使用 | +| OperationLog（操作日志） | 使用（管理后台） | 不使用 | +| OpenPlatformAuth（HMAC-SHA256） | 仅开放平台 | 不使用 | +| RBAC（权限校验） | 使用 | 不使用 | +| Idempotency（幂等性） | 按需使用 | 不使用 | +| RequestID 注入 | 使用 | 使用 | diff --git a/docs/reference/goframe-conventions.md b/docs/reference/goframe-conventions.md new file mode 100644 index 0000000..e1ede6f --- /dev/null +++ b/docs/reference/goframe-conventions.md @@ -0,0 +1,257 @@ +# GoFrame 框架使用规范 + +本文档记录项目中 GoFrame v2 的使用规范、正确模式和已修复的框架使用错误。开发时按需查阅，发现新的框架使用 bug 修复后同步更新本文档。 + +> **提示**：遇到不确定的 GoFrame 用法时，可使用 `/goframe-v2` skill 查询框架最新规范和最佳实践。 + +## 时间处理 + +### 规则 + +- **ORM 写入**：使用 `gtime.Now()`，与 GoFrame ORM 的时间类型自然兼容 +- **纯计算/计时**：使用标准库 `time.Now()`（如 `time.Since(start)` 计算耗时、`time.Now().Unix()` 生成时间戳） +- **time → gtime 转换**：使用 `gtime.NewFromTime(t)` 包装后再写入 ORM + +### 正确示例 + +```go +// ORM 写入场景 — 用 gtime +dao.XxxTable.Ctx(ctx).Data(g.Map{ + "updated_at": gtime.Now(), +}).Update() + +// 耗时计算 — 用标准 time +start := time.Now() +doSomething() +elapsed := time.Since(start) + +// time.Time 转 gtime 写入 ORM +nextRetry := time.Now().Add(delay) +dao.XxxTable.Ctx(ctx).Data(g.Map{ + "next_retry_at": gtime.NewFromTime(nextRetry), +}).Update() +``` + +### 常见错误 + +- 在 ORM `.Data()` 中直接传 `time.Now()` — 应使用 `gtime.Now()` +- 在 `time.Since()` 中使用 `gtime.Now()` — 应使用 `time.Now()` + +## 错误处理 + +### 规则 + +- 业务错误使用 `gerror.NewCode(gcode.New(code, msg, nil), msg)` 创建预定义错误常量 +- 预定义错误常量放在 `internal/consts/consts.go` +- 包装底层错误时使用 `gerror.Wrapf(err, "context message")` +- 临时业务错误使用 `gerror.Newf("中文提示 %s", param)` +- Controller 层不处理错误，直接 `return nil, err` 交给中间件统一处理 + +### 错误码分级 + +| 码段 | 用途 | 示例 | +|------|------|------| +| 400-499 | 标准 HTTP 客户端错误 | 401 未认证、403 无权限、404 不存在 | +| 500 | 服务器内部错误 | 未预期的异常 | +| >= 10000 | 业务规则错误（HTTP 422） | 10001 余额不足、10002 额度用完 | + +### 正确示例 + +```go +// consts.go — 预定义业务错误 +var ErrInsufficientBalance = gerror.NewCode( + gcode.New(CodeInsufficientBalance, MsgInsufficientBalance, nil), + MsgInsufficientBalance, +) + +// logic 层 — 返回预定义错误 +if balance < amount { + return nil, consts.ErrInsufficientBalance +} + +// logic 层 — 包装底层错误 +result, err := dao.XxxTable.Ctx(ctx).Insert(data) +if err != nil { + return nil, gerror.Wrapf(err, "insert xxx") +} + +// logic 层 — 临时业务错误 +return nil, gerror.Newf("兑换码状态为%s", redemption.Status) +``` + +### 错误安全规则 + +`internal/response` 包自动执行错误脱敏： +- 业务错误（>= 10000）和 HTTP 4xx：原始消息透传给客户端 +- 其他错误（数据库、网络等）：替换为 `"服务器内部错误"`，原始错误记日志 +- 禁止在错误消息中暴露 SQL、堆栈、内部路径等技术细节 + +## 日志记录 + +### 规则 + +- 统一使用 `g.Log()` 全局日志，不要创建自定义 Logger 实例 +- **必须传 ctx**：`g.Log().Errorf(ctx, "message: %v", err)`，确保日志携带 request_id +- 日志级别：`Error`（需要修复的异常）、`Warning`（降级处理、已脱敏的错误）、`Info`（关键业务事件） +- 后台 goroutine 中如果没有请求 ctx，使用 `context.TODO()` + +### 正确示例 + +```go +g.Log().Errorf(ctx, "update task %d to failed: %v", taskID, err) +g.Log().Warningf(ctx, "webhook: config %d auto-disabled after %d failures", configID, count) +g.Log().Infof(ctx, "API key %d disabled, cache invalidation for prefix %s", keyID, prefix) +``` + +### 常见错误 + +- `g.Log().Error("message")` — 缺少 ctx，日志中不会有 request_id +- 在可恢复的业务场景使用 Error 级别 — 应使用 Warning + +## ORM 操作 + +### DAO 模式（优先） + +```go +// 查询 +var user entity.TntUsers +err := dao.TntUsers.Ctx(ctx).Where("id", id).Where("tenant_id", tenantID).Scan(&user) + +// 插入（使用 DO 结构体） +id, err := dao.ChnChannels.Ctx(ctx).InsertAndGetId(do.ChnChannels{ + Name: req.Name, + Provider: req.Provider, +}) + +// 更新（使用 DO 结构体） +_, err := dao.TntUsers.Ctx(ctx).Where("id", id).Data(do.TntUsers{ + Status: "disabled", +}).Update() + +// 删除 +_, err := dao.ApiKeys.Ctx(ctx).Where("id", id).Where("tenant_id", tenantID).Delete() + +// 分页 +m := dao.TntUsers.Ctx(ctx).Where("tenant_id", tenantID) +total, _ := m.Count() +var list []entity.TntUsers +err := m.Page(page, pageSize).OrderDesc("id").Scan(&list) +``` + +### 原生 SQL（仅用于复杂查询和批量操作） + +```go +// 聚合查询、分析报表等无法用 ORM 表达的场景 +result, err := g.DB().Ctx(ctx).Raw(` + SELECT date_trunc('hour', created_at) AS hour, count(*) + FROM aud_request_logs WHERE ... +`, args...).All() + +// 原子更新（如余额操作） +_, err = g.DB().Exec(ctx, + `UPDATE bil_wallets SET balance = balance - $1 WHERE id = $2 AND balance >= $1`, + amount, walletID, +) +``` + +### 事务 + +```go +err := g.DB().Transaction(ctx, func(ctx context.Context, tx gdb.TX) error { + // tx.Model() 替代 dao.Xxx.Ctx()，确保操作在同一事务内 + _, err := tx.Model("tnt_tenants").Ctx(ctx).Data(do.TntTenants{...}).Insert() + if err != nil { + return err // 自动回滚 + } + _, err = tx.Model("tnt_users").Ctx(ctx).Data(do.TntUsers{...}).Insert() + return err +}) +``` + +### g.Map vs DO 结构体 + +- **新增/全量更新**：优先使用 `do.XxxTable{}` 结构体，类型安全 +- **部分更新**：可使用 `g.Map{"field": value}`，灵活但需注意字段名拼写 +- **混用场景**：changelog 等直接操作表名的场景可用 `g.DB().Model("table").Data(g.Map{...})` + +## 配置读取 + +```go +// 使用 g.Cfg() 读取 manifest/config/ 下的配置 +secret := g.Cfg().MustGet(ctx, "jwt.secret").String() +maxSessions := g.Cfg().MustGet(ctx, "jwt.adminMaxSessions").Int() +hexKey := g.Cfg().MustGet(ctx, "crypto.encryptionKey").String() +``` + +## 参数校验 + +API 结构体使用 `v` tag 声明校验规则，GoFrame 自动执行校验： + +```go +type CreateAdminUserReq struct { + g.Meta `path:"/admin-users" method:"post" tags:"管理员" summary:"创建管理员"` + Username string `json:"username" v:"required|length:3,50#请输入用户名|用户名长度为3-50位"` + Password string `json:"password" v:"required|length:8,64#请输入密码|密码长度为8-64位"` + Status string `json:"status" v:"required|in:active,disabled#请选择状态|状态值无效"` +} +``` + +- `#` 后面是自定义中文错误消息，多个规则的消息用 `|` 分隔 +- 校验失败自动返回 400 错误，无需在 logic 层重复校验 + +## 类型转换 + +使用 `gconv` 包进行安全类型转换： + +```go +name := gconv.String(record["name"]) +ids := gconv.Int64s(record["user_ids"]) +rate := gconv.Float64(record["threshold"]) +methods := gconv.Strings(record["notification_methods"]) +``` + +## 缓存 + +项目使用 `gcache` 作为 L1 内存缓存，封装在 `internal/logic/common/cache.go`： + +```go +// 通过 common 包的缓存封装使用，不要直接调用 gcache +common.CacheSet(ctx, "key", value, ttl) +val, err := common.CacheGet(ctx, "key") +common.CacheRemove(ctx, "key") +``` + +## Context 传递 + +```go +// 从 Context 读取请求级变量（中间件注入） +requestID := r.GetCtxVar("RequestId").String() +scope := r.GetCtxVar("ApiKeyScope").String() + +// 获取当前请求对象 +r := g.RequestFromCtx(ctx) +``` + +--- + +## 已修复的框架使用错误记录 + +> 每次修复 GoFrame 框架使用相关的 bug 后，在此处记录修复内容、原因和正确做法，防止同类问题再次出现。 + +### 2026-04-16：GoFrame gcode 非标准码用作 HTTP 状态码导致 panic + +**问题**：`response.Error()` 中直接将 `gerror.Code().Code()` 作为 HTTP 状态码传给 `WriteHeader()`，但 GoFrame 内置的 `gcode`（如 `CodeNotModified=68`）不是合法 HTTP 状态码，导致 `WriteHeader` panic。 + +**修复**：在 `response.go` 中增加范围检查，非 100-599 的 code 回退为 500。 + +### 2026-04-16：MiddlewareHandlerResponse 未过滤文件下载响应 + +**问题**：导出 CSV/Excel 时，控制器直接写入 `r.Response.Writer`，但中间件仍然追加 `{"code":0,...}` JSON 到响应体末尾，导致文件损坏。 + +**修复**：在 `handler_response.go` 中检查 `Content-Type`，匹配 `text/csv`、`application/vnd.openxmlformats` 等下载类型时跳过 JSON 包装。 + +### 2026-04-16：isSystemError 日志污染 + +**问题**：所有错误（包括 400 参数校验、401 认证失败等正常业务流）都以 Warning 级别记录日志，大量无意义日志淹没真正的异常。 + +**修复**：增加 `isSystemError()` 函数，4xx 和 >= 10000 的业务错误不记日志，只有 5xx 和未知错误才记录。 diff --git a/docs/协议文档/API参考-Google-Gemini.md b/docs/协议文档/API参考-Google-Gemini.md new file mode 100644 index 0000000..9a851fd --- /dev/null +++ b/docs/协议文档/API参考-Google-Gemini.md @@ -0,0 +1,1506 @@ +# Google Gemini API 完整参考文档 + +> 基于 Google AI for Developers 官方文档整理（最后更新: 2026-04-19） +> 本文档用于 team-api 网关的协议转换模块开发参考 + +--- + +## 目录 + +1. [API 端点概览](#1-api-端点概览) +2. [generateContent 请求体](#2-generatecontent-请求体) +3. [Content 与 Part 类型](#3-content-与-part-类型) +4. [GenerationConfig 完整字段](#4-generationconfig-完整字段) +5. [Tool 系统](#5-tool-系统) +6. [ThinkingConfig](#6-thinkingconfig) +7. [SafetySettings](#7-safetysettings) +8. [GenerateContentResponse 响应格式](#8-generatecontentresponse-响应格式) +9. [流式响应格式 (streamGenerateContent)](#9-流式响应格式-streamgeneratecontent) +10. [Context Caching API](#10-context-caching-api) +11. [File API](#11-file-api) +12. [Token 计数端点](#12-token-计数端点) +13. [Embeddings 端点](#13-embeddings-端点) +14. [OpenAI 兼容端点](#14-openai-兼容端点) +15. [错误格式](#15-错误格式) +16. [可用模型列表](#16-可用模型列表) +17. [枚举值速查表](#17-枚举值速查表) + +--- + +## 1. API 端点概览 + +### 核心生成端点 + +| 方法 | 端点 | 说明 | +|------|------|------| +| POST | `https://generativelanguage.googleapis.com/v1beta/{model=models/*}:generateContent` | 非流式生成 | +| POST | `https://generativelanguage.googleapis.com/v1beta/{model=models/*}:streamGenerateContent` | 流式生成（SSE） | + +### Token 计数端点 + +| 方法 | 端点 | 说明 | +|------|------|------| +| POST | `https://generativelanguage.googleapis.com/v1beta/{model=models/*}:countTokens` | Token 计数 | + +### Embeddings 端点 + +| 方法 | 端点 | 说明 | +|------|------|------| +| POST | `https://generativelanguage.googleapis.com/v1beta/{model=models/*}:embedContent` | 单条嵌入 | +| POST | `https://generativelanguage.googleapis.com/v1beta/{model=models/*}:batchEmbedContents` | 批量嵌入 | +| POST | `https://generativelanguage.googleapis.com/v1beta/{batch.model=models/*}:asyncBatchEmbedContent` | 异步批量嵌入 | + +### Context Caching 端点 + +| 方法 | 端点 | 说明 | +|------|------|------| +| POST | `https://generativelanguage.googleapis.com/v1beta/cachedContents` | 创建缓存 | +| GET | `https://generativelanguage.googleapis.com/v1beta/cachedContents` | 列出缓存 | +| GET | `https://generativelanguage.googleapis.com/v1beta/{name=cachedContents/*}` | 获取缓存 | +| PATCH | `https://generativelanguage.googleapis.com/v1beta/{cachedContent.name=cachedContents/*}` | 更新缓存（仅过期时间） | +| DELETE | `https://generativelanguage.googleapis.com/v1beta/{name=cachedContents/*}` | 删除缓存 | + +### File API 端点 + +| 方法 | 端点 | 说明 | +|------|------|------| +| POST | `https://generativelanguage.googleapis.com/upload/v1beta/files` | 上传文件（resumable） | +| POST | `https://generativelanguage.googleapis.com/v1beta/files` | 上传文件（metadata only） | +| GET | `https://generativelanguage.googleapis.com/v1beta/{name=files/*}` | 获取文件信息 | +| GET | `https://generativelanguage.googleapis.com/v1beta/files` | 列出文件 | +| DELETE | `https://generativelanguage.googleapis.com/v1beta/{name=files/*}` | 删除文件 | +| POST | `https://generativelanguage.googleapis.com/v1beta/files:register` | 注册 GCS 文件 | + +### OpenAI 兼容端点 + +| 方法 | 端点 | 说明 | +|------|------|------| +| POST | `https://generativelanguage.googleapis.com/v1beta/openai/chat/completions` | Chat Completions | +| POST | `https://generativelanguage.googleapis.com/v1beta/openai/images/generations` | 图像生成 | +| POST | `https://generativelanguage.googleapis.com/v1beta/openai/videos` | 视频生成 | +| GET | `https://generativelanguage.googleapis.com/v1beta/openai/videos/{id}` | 查询视频状态 | +| POST | `https://generativelanguage.googleapis.com/v1beta/openai/embeddings` | 嵌入 | +| GET | `https://generativelanguage.googleapis.com/v1beta/openai/models` | 模型列表 | +| GET | `https://generativelanguage.googleapis.com/v1beta/openai/models/{model}` | 模型详情 | + +### Stable v1 端点 + +v1 端点使用稳定模型，格式为 `https://generativelanguage.googleapis.com/v1/...`，其余路径结构与 v1beta 相同。 + +--- + +## 2. generateContent 请求体 + +```json +{ + "contents": [{ "role": "user", "parts": [{ "text": "Hello" }] }], + "tools": [], + "toolConfig": {}, + "safetySettings": [], + "systemInstruction": { "role": "user", "parts": [{ "text": "You are a helpful assistant." }] }, + "generationConfig": {}, + "cachedContent": "cachedContents/{id}", + "serviceTier": "STANDARD", + "store": false +} +``` + +### 字段说明 + +| 字段 | 类型 | 必填 | 说明 | +|------|------|------|------| +| `contents[]` | `object(Content)` | **是** | 对话内容数组。单轮查询为单个实例，多轮对话为完整历史 | +| `tools[]` | `object(Tool)` | 否 | 可用工具列表（函数声明、代码执行、搜索等） | +| `toolConfig` | `object(ToolConfig)` | 否 | 工具配置 | +| `safetySettings[]` | `object(SafetySetting)` | 否 | 安全过滤设置，每个类别最多一个 | +| `systemInstruction` | `object(Content)` | 否 | 系统指令（developer prompt） | +| `generationConfig` | `object(GenerationConfig)` | 否 | 生成参数配置 | +| `cachedContent` | `string` | 否 | 已缓存的上下文名称，格式: `cachedContents/{id}` | +| `serviceTier` | `enum(ServiceTier)` | 否 | 服务层级 | +| `store` | `boolean` | 否 | 配置请求的日志记录行为 | + +--- + +## 3. Content 与 Part 类型 + +### Content 结构 + +```json +{ + "role": "user", + "parts": [ + { "text": "Hello" }, + { "inlineData": { "mimeType": "image/png", "data": "base64..." } }, + { "functionCall": { "name": "get_weather", "args": { "city": "NYC" } } } + ] +} +``` + +| 字段 | 类型 | 说明 | +|------|------|------| +| `role` | `string` | `"user"` 或 `"model"` | +| `parts[]` | `object(Part)` | 消息各部分，可混合多种 MIME 类型 | + +### Part 类型（联合类型，只能包含以下之一） + +#### text — 文本内容 +```json +{ "text": "Hello, how are you?" } +``` + +#### inlineData (Blob) — 内联二进制媒体 +```json +{ + "inlineData": { + "mimeType": "image/png", + "data": "iVBORw0KGgo..." // base64 编码 + } +} +``` + +#### functionCall — 模型预测的函数调用 +```json +{ + "functionCall": { + "id": "call_abc123", // 可选，唯一标识符 + "name": "get_weather", // 必填，函数名 + "args": { "city": "NYC" } // 可选，JSON 对象参数 + } +} +``` + +#### functionResponse — 函数调用结果 +```json +{ + "functionResponse": { + "id": "call_abc123", + "name": "get_weather", + "response": { "temperature": 72, "condition": "sunny" }, + "parts": [], + "willContinue": false, + "scheduling": "WHEN_IDLE" + } +} +``` + +| 字段 | 类型 | 说明 | +|------|------|------| +| `id` | `string` | 可选，匹配对应 functionCall 的 id | +| `name` | `string` | 必填，函数名 | +| `response` | `object(Struct)` | 必填，JSON 对象格式的函数响应 | +| `parts[]` | `FunctionResponsePart[]` | 可选，函数响应的多媒体部分 | +| `willContinue` | `boolean` | 仅 NON_BLOCKING 函数适用，表示后续还有更多响应 | +| `scheduling` | `enum(Scheduling)` | 调度方式：`SILENT`/`WHEN_IDLE`/`INTERRUPT` | + +#### fileData — URI 引用的文件 +```json +{ + "fileData": { + "mimeType": "video/mp4", + "fileUri": "https://generativelanguage.googleapis.com/v1beta/files/abc-123" + } +} +``` + +#### executableCode — 模型生成的可执行代码 +```json +{ + "executableCode": { + "id": "code_1", + "language": "PYTHON", + "code": "print('Hello, World!')" + } +} +``` + +#### codeExecutionResult — 代码执行结果 +```json +{ + "codeExecutionResult": { + "id": "code_1", + "outcome": "OUTCOME_OK", + "output": "Hello, World!\n" + } +} +``` + +| outcome 值 | 说明 | +|------------|------| +| `OUTCOME_OK` | 执行成功，output 包含 stdout | +| `OUTCOME_FAILED` | 执行失败，output 包含 stderr 和 stdout | +| `OUTCOME_DEADLINE_EXCEEDED` | 超时被取消，可能有部分 output | + +#### toolCall — 服务端工具调用 +```json +{ + "toolCall": { + "id": "tc_001", + "toolType": "GOOGLE_SEARCH_WEB", + "args": { "query": "latest AI news" } + } +} +``` + +ToolType 枚举值： +- `TOOL_TYPE_UNSPECIFIED` +- `GOOGLE_SEARCH_WEB` — Google 网页搜索 +- `GOOGLE_SEARCH_IMAGE` — Google 图片搜索 +- `URL_CONTEXT` — URL 上下文检索 +- `GOOGLE_MAPS` — Google 地图 +- `FILE_SEARCH` — 文件搜索 + +#### toolResponse — 服务端工具响应 +```json +{ + "toolResponse": { + "id": "tc_001", + "toolType": "GOOGLE_SEARCH_WEB", + "response": { "results": [...] } + } +} +``` + +### Part 的附加元数据字段 + +| 字段 | 类型 | 说明 | +|------|------|------| +| `thought` | `boolean` | 是否为思考内容 | +| `thoughtSignature` | `string(bytes)` | 思考签名，用于后续请求复用（base64） | +| `partMetadata` | `object(Struct)` | 自定义元数据 | +| `mediaResolution` | `object(MediaResolution)` | 输入媒体的分辨率设置 | +| `videoMetadata` | `object(VideoMetadata)` | 视频元数据（startOffset, endOffset, fps） | + +--- + +## 4. GenerationConfig 完整字段 + +```json +{ + "stopSequences": ["\n"], + "responseMimeType": "application/json", + "responseSchema": {}, + "responseJsonSchema": {}, + "responseModalities": ["TEXT"], + "candidateCount": 1, + "maxOutputTokens": 8192, + "temperature": 0.7, + "topP": 0.95, + "topK": 40, + "seed": 42, + "presencePenalty": 0.0, + "frequencyPenalty": 0.0, + "responseLogprobs": false, + "logprobs": 0, + "enableEnhancedCivicAnswers": false, + "speechConfig": {}, + "thinkingConfig": {}, + "imageConfig": {}, + "mediaResolution": "MEDIA_RESOLUTION_UNSPECIFIED" +} +``` + +| 字段 | 类型 | 默认值 | 说明 | +|------|------|--------|------| +| `stopSequences[]` | `string[]` | — | 最多 5 个停止序列 | +| `responseMimeType` | `string` | `"text/plain"` | 输出 MIME 类型：`text/plain` / `application/json` / `text/x.enum` | +| `responseSchema` | `object(Schema)` | — | OpenAPI 子集的输出 schema，需配合 `application/json` | +| `responseJsonSchema` | `Value` | — | JSON Schema 格式的输出 schema，与 responseSchema 互斥 | +| `responseModalities[]` | `enum(Modality)[]` | `["TEXT"]` | 请求的响应模态：`TEXT`/`IMAGE`/`AUDIO` | +| `candidateCount` | `integer` | `1` | 候选响应数量 | +| `maxOutputTokens` | `integer` | 因模型而异 | 最大输出 token 数 | +| `temperature` | `number` | 因模型而异 | 随机性控制，范围 [0.0, 2.0] | +| `topP` | `number` | 因模型而异 | 核采样概率阈值 | +| `topK` | `integer` | 因模型而异 | 最大候选 token 数 | +| `seed` | `integer` | 随机 | 解码种子 | +| `presencePenalty` | `number` | — | 存在惩罚（二值开关） | +| `frequencyPenalty` | `number` | — | 频率惩罚（按使用次数累加） | +| `responseLogprobs` | `boolean` | `false` | 是否返回 logprobs | +| `logprobs` | `integer` | `0` | 返回 top logprobs 数量，范围 [0, 20] | +| `enableEnhancedCivicAnswers` | `boolean` | `false` | 增强公民答案 | +| `speechConfig` | `object(SpeechConfig)` | — | 语音生成配置 | +| `thinkingConfig` | `object(ThinkingConfig)` | — | 思考功能配置 | +| `imageConfig` | `object(ImageConfig)` | — | 图像生成配置 | +| `mediaResolution` | `enum(MediaResolution)` | — | 输入媒体分辨率 | + +### responseJsonSchema 支持的 JSON Schema 属性 + +`$id`, `$defs`, `$ref`, `$anchor`, `type`, `format`, `title`, `description`, `enum`, `items`, `prefixItems`, `minItems`, `maxItems`, `minimum`, `maximum`, `anyOf`, `oneOf`（等同于 anyOf）, `properties`, `additionalProperties`, `required`。还支持非标准的 `propertyOrdering`。 + +### SpeechConfig + +```json +{ + "voiceConfig": { + "prebuiltVoiceConfig": { "voiceName": "Aoede" } + }, + "multiSpeakerVoiceConfig": { + "speakerVoiceConfigs": [ + { "speaker": "Speaker1", "voiceConfig": { "prebuiltVoiceConfig": { "voiceName": "Aoede" } } } + ] + }, + "languageCode": "en-US" +} +``` + +| 字段 | 类型 | 说明 | +|------|------|------| +| `voiceConfig` | `object(VoiceConfig)` | 单声音配置 | +| `multiSpeakerVoiceConfig` | `object(MultiSpeakerVoiceConfig)` | 多说话人配置，与 voiceConfig 互斥 | +| `languageCode` | `string` | BCP-47 语言代码 | + +支持的语言代码：`de-DE`, `en-AU`, `en-GB`, `en-IN`, `en-US`, `es-US`, `fr-FR`, `hi-IN`, `pt-BR`, `ar-XA`, `es-ES`, `fr-CA`, `id-ID`, `it-IT`, `ja-JP`, `tr-TR`, `vi-VN`, `bn-IN`, `gu-IN`, `kn-IN`, `ml-IN`, `mr-IN`, `ta-IN`, `te-IN`, `nl-NL`, `ko-KR`, `cmn-CN`, `pl-PL`, `ru-RU`, `th-TH` + +### ImageConfig + +```json +{ + "aspectRatio": "16:9", + "imageSize": "1K" +} +``` + +| 字段 | 说明 | +|------|------| +| `aspectRatio` | 支持：`1:1`, `1:4`, `4:1`, `1:8`, `8:1`, `2:3`, `3:2`, `3:4`, `4:3`, `4:5`, `5:4`, `9:16`, `16:9`, `21:9` | +| `imageSize` | 支持：`512`, `1K`, `2K`, `4K`，默认 `1K` | + +### MediaResolution 枚举 + +| 值 | 说明 | +|---|------| +| `MEDIA_RESOLUTION_UNSPECIFIED` | 未设置 | +| `MEDIA_RESOLUTION_LOW` | 低分辨率（64 tokens） | +| `MEDIA_RESOLUTION_MEDIUM` | 中分辨率（256 tokens） | +| `MEDIA_RESOLUTION_HIGH` | 高分辨率（zoomed reframing with 256 tokens） | + +--- + +## 5. Tool 系统 + +### Tool 类型概览 + +```json +{ + "tools": [ + { "functionDeclarations": [...] }, + { "googleSearchRetrieval": { ... } }, + { "codeExecution": {} }, + { "googleSearch": { ... } }, + { "computerUse": { ... } }, + { "urlContext": {} }, + { "fileSearch": { ... } }, + { "mcpServers": [...] }, + { "googleMaps": { ... } } + ] +} +``` + +每个 Tool 对象可包含以下字段之一（互斥）： + +### 5.1 functionDeclarations — 自定义函数调用 + +```json +{ + "functionDeclarations": [ + { + "name": "get_weather", + "description": "Get the current weather in a given location", + "behavior": "BLOCKING", + "parameters": { + "type": "OBJECT", + "properties": { + "location": { "type": "STRING", "description": "The city and state" }, + "unit": { "type": "STRING", "enum": ["celsius", "fahrenheit"] } + }, + "required": ["location"] + }, + "parametersJsonSchema": { ... }, + "response": { "type": "OBJECT", "properties": { ... } }, + "responseJsonSchema": { ... } + } + ] +} +``` + +| 字段 | 类型 | 说明 | +|------|------|------| +| `name` | `string` | 必填，函数名，最长 128 字符 | +| `description` | `string` | 必填，函数描述 | +| `behavior` | `enum(Behavior)` | `BLOCKING`（默认）/ `NON_BLOCKING` | +| `parameters` | `object(Schema)` | OpenAPI 3.03 格式的参数定义 | +| `parametersJsonSchema` | `Value` | JSON Schema 格式的参数定义，与 parameters 互斥 | +| `response` | `object(Schema)` | 函数输出 schema | +| `responseJsonSchema` | `Value` | JSON Schema 格式的输出定义，与 response 互斥 | + +### Schema 结构（OpenAPI 子集） + +```json +{ + "type": "OBJECT", + "format": "", + "title": "", + "description": "", + "nullable": false, + "enum": ["EAST", "NORTH"], + "maxItems": "10", + "minItems": "1", + "properties": { ... }, + "required": ["name"], + "minProperties": "0", + "maxProperties": "10", + "minLength": "1", + "maxLength": "100", + "pattern": "^[a-z]+$", + "example": {}, + "anyOf": [ ... ], + "propertyOrdering": ["name", "age"], + "default": {}, + "items": { ... }, + "minimum": 0, + "maximum": 100 +} +``` + +Schema Type 枚举：`TYPE_UNSPECIFIED` / `STRING` / `NUMBER` / `INTEGER` / `BOOLEAN` / `ARRAY` / `OBJECT` / `NULL` + +### 5.2 codeExecution — 代码执行工具 + +```json +{ "codeExecution": {} } +``` + +无额外字段。启用后模型自动生成 `ExecutableCode`（PYTHON >= 3.10）和 `CodeExecutionResult`。 + +### 5.3 googleSearch — Google 搜索工具（新版） + +```json +{ + "googleSearch": { + "timeRangeFilter": { + "startTime": "2025-01-01T00:00:00Z", + "endTime": "2025-12-31T23:59:59Z" + }, + "searchTypes": { + "webSearch": {}, + "imageSearch": {} + } + } +} +``` + +| 字段 | 类型 | 说明 | +|------|------|------| +| `timeRangeFilter` | `object(Interval)` | 时间范围过滤，start 和 end 必须同时设置 | +| `searchTypes` | `object(SearchTypes)` | 搜索类型：`webSearch` / `imageSearch`，不设置则默认 web 搜索 | + +### 5.4 googleSearchRetrieval — Google 搜索检索（旧版） + +```json +{ + "googleSearchRetrieval": { + "dynamicRetrievalConfig": { + "mode": "MODE_DYNAMIC", + "dynamicThreshold": 0.3 + } + } +} +``` + +| 字段 | 说明 | +|------|------| +| `mode` | `MODE_UNSPECIFIED`（始终触发）/ `MODE_DYNAMIC`（系统判断） | +| `dynamicThreshold` | 动态检索阈值 [0, 1] | + +### 5.5 computerUse — 计算机使用工具 + +```json +{ + "computerUse": { + "environment": "ENVIRONMENT_BROWSER", + "excludedPredefinedFunctions": ["screenshot"] + } +} +``` + +| 字段 | 说明 | +|------|------| +| `environment` | 目前仅支持 `ENVIRONMENT_BROWSER` | +| `excludedPredefinedFunctions[]` | 排除的预定义函数列表 | + +### 5.6 urlContext — URL 上下文检索 + +```json +{ "urlContext": {} } +``` + +无额外字段。 + +### 5.7 fileSearch — 文件搜索工具 + +```json +{ + "fileSearch": { + "fileSearchStoreNames": ["fileSearchStores/my-store-123"], + "metadataFilter": "category = 'tech'", + "topK": 5 + } +} +``` + +### 5.8 mcpServers — MCP 服务器集成 + +```json +{ + "mcpServers": [ + { + "name": "my-mcp-server", + "streamableHttpTransport": { + "url": "https://api.example.com/mcp", + "headers": { "Authorization": "Bearer token" }, + "timeout": "30s", + "sseReadTimeout": "60s", + "terminateOnClose": true + } + } + ] +} +``` + +### 5.9 googleMaps — Google 地图工具 + +```json +{ + "googleMaps": { + "enableWidget": true + } +} +``` + +### ToolConfig + +```json +{ + "toolConfig": { + "functionCallingConfig": { + "mode": "AUTO", + "allowedFunctionNames": ["get_weather"] + }, + "retrievalConfig": { + "latLng": { "latitude": 40.7128, "longitude": -74.0060 }, + "languageCode": "en" + }, + "includeServerSideToolInvocations": false + } +} +``` + +#### FunctionCallingConfig Mode 枚举 + +| 值 | 说明 | +|---|------| +| `AUTO` | 默认，模型自行决定是否调用函数 | +| `ANY` | 强制模型调用函数，限制在 allowedFunctionNames 内 | +| `NONE` | 禁止函数调用 | +| `VALIDATED` | 模型决定是否调用，但使用约束解码验证函数调用 | + +--- + +## 6. ThinkingConfig + +```json +{ + "generationConfig": { + "thinkingConfig": { + "includeThoughts": true, + "thinkingBudget": 8192, + "thinkingLevel": "MEDIUM" + } + } +} +``` + +| 字段 | 类型 | 说明 | +|------|------|------| +| `includeThoughts` | `boolean` | 是否在响应中包含思考内容 | +| `thinkingBudget` | `integer` | 思考 token 数量上限 | +| `thinkingLevel` | `enum(ThinkingLevel)` | 思考深度等级 | + +### ThinkingLevel 枚举 + +| 值 | 说明 | +|---|------| +| `THINKING_LEVEL_UNSPECIFIED` | 默认值 | +| `MINIMAL` | 几乎不思考 | +| `LOW` | 低度思考 | +| `MEDIUM` | 中度思考 | +| `HIGH` | 高度思考（默认） | + +### OpenAI reasoning_effort 到 Gemini 的映射 + +| reasoning_effort (OpenAI) | Gemini 3.1 Pro thinking_level | Gemini 3.1 Flash-Lite thinking_level | Gemini 3 Flash thinking_level | Gemini 2.5 thinking_budget | +|---|---|---|---|---| +| `minimal` | `low` | `minimal` | `minimal` | `1,024` | +| `low` | `low` | `low` | `low` | `1,024` | +| `medium` | `medium` | `medium` | `medium` | `8,192` | +| `high` | `high` | `high` | `high` | `24,576` | + +注意：`reasoning_effort` 和 `thinking_level`/`thinking_budget` 不能同时使用。Gemini 2.5 Pro 和 Gemini 3 系列无法关闭 thinking。 + +--- + +## 7. SafetySettings + +### SafetySetting 结构 + +```json +{ + "safetySettings": [ + { + "category": "HARM_CATEGORY_HARASSMENT", + "threshold": "BLOCK_MEDIUM_AND_ABOVE" + } + ] +} +``` + +### HarmCategory 枚举 + +#### Gemini 系列（推荐使用） + +| 值 | 说明 | +|---|------| +| `HARM_CATEGORY_HARASSMENT` | 骚扰内容 | +| `HARM_CATEGORY_HATE_SPEECH` | 仇恨言论 | +| `HARM_CATEGORY_SEXUALLY_EXPLICIT` | 色情内容 | +| `HARM_CATEGORY_DANGEROUS_CONTENT` | 危险内容 | +| `HARM_CATEGORY_CIVIC_INTEGRITY` | 危害公民诚信的内容（已弃用，用 enableEnhancedCivicAnswers 替代） | + +#### PaLM 系列（旧版） + +| 值 | 说明 | +|---|------| +| `HARM_CATEGORY_DEROGATORY` | 针对身份/受保护属性的负面评论 | +| `HARM_CATEGORY_TOXICITY` | 粗鲁、不敬或亵渎内容 | +| `HARM_CATEGORY_VIOLENCE` | 暴力描述 | +| `HARM_CATEGORY_SEXUAL` | 性行为引用或淫秽内容 | +| `HARM_CATEGORY_MEDICAL` | 未经核实的医疗建议 | +| `HARM_CATEGORY_DANGEROUS` | 有害行为推广 | + +### HarmBlockThreshold 枚举 + +| 值 | 说明 | +|---|------| +| `HARM_BLOCK_THRESHOLD_UNSPECIFIED` | 未指定 | +| `BLOCK_LOW_AND_ABOVE` | 允许 NEGLIGIBLE | +| `BLOCK_MEDIUM_AND_ABOVE` | 允许 NEGLIGIBLE 和 LOW | +| `BLOCK_ONLY_HIGH` | 允许 NEGLIGIBLE、LOW 和 MEDIUM | +| `BLOCK_NONE` | 允许所有内容 | +| `OFF` | 关闭安全过滤 | + +### HarmProbability 枚举 + +| 值 | 说明 | +|---|------| +| `HARM_PROBABILITY_UNSPECIFIED` | 未指定 | +| `NEGLIGIBLE` | 安全可能性极低 | +| `LOW` | 安全可能性低 | +| `MEDIUM` | 安全可能性中等 | +| `HIGH` | 安全可能性高 | + +--- + +## 8. GenerateContentResponse 响应格式 + +```json +{ + "candidates": [ + { + "content": { + "role": "model", + "parts": [{ "text": "Hello! How can I help you?" }] + }, + "finishReason": "STOP", + "safetyRatings": [ + { "category": "HARM_CATEGORY_HARASSMENT", "probability": "NEGLIGIBLE", "blocked": false } + ], + "citationMetadata": { + "citationSources": [ + { "startIndex": 0, "endIndex": 100, "uri": "https://example.com", "license": "MIT" } + ] + }, + "tokenCount": 42, + "groundingAttributions": [], + "groundingMetadata": {}, + "avgLogprobs": -0.123, + "logprobsResult": {}, + "urlContextMetadata": {}, + "index": 0, + "finishMessage": "" + } + ], + "promptFeedback": { + "blockReason": "BLOCK_REASON_UNSPECIFIED", + "safetyRatings": [] + }, + "usageMetadata": {}, + "modelVersion": "gemini-3-flash-preview-2025-09-01", + "responseId": "resp_abc123", + "modelStatus": { + "modelStage": "STABLE", + "retirementTime": "", + "message": "" + } +} +``` + +### Candidate 字段 + +| 字段 | 类型 | 说明 | +|------|------|------| +| `content` | `object(Content)` | 模型生成的内容 | +| `finishReason` | `enum(FinishReason)` | 停止生成的原因 | +| `safetyRatings[]` | `object(SafetyRating)` | 安全评级列表 | +| `citationMetadata` | `object(CitationMetadata)` | 引用信息 | +| `tokenCount` | `integer` | 候选 token 数 | +| `groundingAttributions[]` | `object(GroundingAttribution)` | 归因信息 | +| `groundingMetadata` | `object(GroundingMetadata)` | Grounding 元数据 | +| `avgLogprobs` | `number` | 平均 log 概率 | +| `logprobsResult` | `object(LogprobsResult)` | Logprobs 结果 | +| `urlContextMetadata` | `object(UrlContextMetadata)` | URL 上下文元数据 | +| `index` | `integer` | 候选索引 | +| `finishMessage` | `string` | 停止原因详情 | + +### FinishReason 枚举 + +| 值 | 说明 | +|---|------| +| `FINISH_REASON_UNSPECIFIED` | 默认值 | +| `STOP` | 自然停止或命中停止序列 | +| `MAX_TOKENS` | 达到最大 token 限制 | +| `SAFETY` | 安全原因被标记 | +| `RECITATION` | 引用（版权）原因被标记 | +| `LANGUAGE` | 不支持的语言被标记 | +| `OTHER` | 未知原因 | +| `BLOCKLIST` | 包含禁止术语 | +| `PROHIBITED_CONTENT` | 包含禁止内容 | +| `SPII` | 可能包含敏感个人信息 | +| `MALFORMED_FUNCTION_CALL` | 函数调用格式无效 | +| `IMAGE_SAFETY` | 生成的图片包含安全违规 | +| `IMAGE_PROHIBITED_CONTENT` | 图片包含其他禁止内容 | +| `IMAGE_OTHER` | 图片生成的其他问题 | +| `NO_IMAGE` | 预期生成图片但未生成 | +| `IMAGE_RECITATION` | 图片引用问题 | +| `UNEXPECTED_TOOL_CALL` | 模型生成了工具调用但未启用工具 | +| `TOO_MANY_TOOL_CALLS` | 连续工具调用过多 | +| `MISSING_THOUGHT_SIGNATURE` | 缺少思考签名 | +| `MALFORMED_RESPONSE` | 响应格式错误 | + +### UsageMetadata + +```json +{ + "promptTokenCount": 100, + "cachedContentTokenCount": 50, + "candidatesTokenCount": 200, + "toolUsePromptTokenCount": 30, + "thoughtsTokenCount": 500, + "totalTokenCount": 880, + "promptTokensDetails": [ + { "modality": "TEXT", "tokenCount": 80 }, + { "modality": "IMAGE", "tokenCount": 20 } + ], + "cacheTokensDetails": [], + "candidatesTokensDetails": [], + "toolUsePromptTokensDetails": [] +} +``` + +| 字段 | 类型 | 说明 | +|------|------|------| +| `promptTokenCount` | `integer` | 提示 token 数（含缓存） | +| `cachedContentTokenCount` | `integer` | 缓存部分的 token 数 | +| `candidatesTokenCount` | `integer` | 所有候选的总 token 数 | +| `toolUsePromptTokenCount` | `integer` | 工具使用提示的 token 数 | +| `thoughtsTokenCount` | `integer` | 思考 token 数（thinking models） | +| `totalTokenCount` | `integer` | 总 token 数（提示 + 响应） | +| `promptTokensDetails[]` | `ModalityTokenCount[]` | 按模态的输入 token 详情 | +| `cacheTokensDetails[]` | `ModalityTokenCount[]` | 按模态的缓存 token 详情 | +| `candidatesTokensDetails[]` | `ModalityTokenCount[]` | 按模态的输出 token 详情 | +| `toolUsePromptTokensDetails[]` | `ModalityTokenCount[]` | 按模态的工具使用 token 详情 | + +ModalityTokenCount 结构：`{ "modality": "TEXT", "tokenCount": 100 }` + +### GroundingMetadata + +```json +{ + "groundingChunks": [ + { "web": { "uri": "https://example.com", "title": "Example" } }, + { "image": { "sourceUri": "https://...", "imageUri": "https://...", "title": "...", "domain": "example.com" } }, + { "retrievedContext": { "uri": "...", "title": "...", "text": "...", "fileSearchStore": "fileSearchStores/123" } }, + { "maps": { "uri": "...", "title": "...", "text": "...", "placeId": "places/ChIJ...", "placeAnswerSources": { "reviewSnippets": [...] } } } + ], + "groundingSupports": [ + { + "groundingChunkIndices": [0, 2], + "confidenceScores": [0.95, 0.88], + "renderedParts": [0], + "segment": { "partIndex": 0, "startIndex": 0, "endIndex": 100, "text": "..." } + } + ], + "webSearchQueries": ["latest AI news"], + "imageSearchQueries": ["cute cats"], + "searchEntryPoint": { + "renderedContent": "
...
", + "sdkBlob": "base64..." + }, + "retrievalMetadata": { + "googleSearchDynamicRetrievalScore": 0.85 + }, + "googleMapsWidgetContextToken": "token_string" +} +``` + +### LogprobsResult + +```json +{ + "topCandidates": [ + { "candidates": [ + { "token": "Hello", "tokenId": 1234, "logProbability": -0.01 }, + { "token": "Hi", "tokenId": 5678, "logProbability": -3.5 } + ]} + ], + "chosenCandidates": [ + { "token": "Hello", "tokenId": 1234, "logProbability": -0.01 } + ], + "logProbabilitySum": -15.3 +} +``` + +### PromptFeedback + +```json +{ + "blockReason": "SAFETY", + "safetyRatings": [ + { "category": "HARM_CATEGORY_HARASSMENT", "probability": "NEGLIGIBLE", "blocked": false } + ] +} +``` + +### BlockReason 枚举 + +| 值 | 说明 | +|---|------| +| `BLOCK_REASON_UNSPECIFIED` | 默认值 | +| `SAFETY` | 安全原因被阻止 | +| `OTHER` | 未知原因 | +| `BLOCKLIST` | 术语黑名单 | +| `PROHIBITED_CONTENT` | 禁止内容 | +| `IMAGE_SAFETY` | 不安全的图片生成内容 | + +--- + +## 9. 流式响应格式 (streamGenerateContent) + +### 请求方式 + +``` +POST https://generativelanguage.googleapis.com/v1beta/{model=models/*}:streamGenerateContent?alt=sse +``` + +使用 `?alt=sse` 查询参数启用 SSE 流式输出。 + +### SSE 响应格式 + +每个事件是一个完整的 `GenerateContentResponse` JSON： + +``` +data: {"candidates":[{"content":{"role":"model","parts":[{"text":"Hello"}]},"finishReason":"STOP"}],"usageMetadata":{"promptTokenCount":10,"candidatesTokenCount":1,"totalTokenCount":11}} + +data: {"candidates":[{"content":{"role":"model","parts":[{"text":"!"}]},"finishReason":"STOP"}],"usageMetadata":{"promptTokenCount":10,"candidatesTokenCount":2,"totalTokenCount":12}} + +data: {"candidates":[{"content":{"role":"model","parts":[{"text":""}]},"finishReason":"STOP","index":0}],"usageMetadata":{"promptTokenCount":10,"candidatesTokenCount":2,"totalTokenCount":12}} +``` + +注意：Gemini 流式响应**没有** `[DONE]` 标记（不同于 OpenAI 的 SSE 格式）。流在 HTTP 连接关闭时结束。 + +### Grounding 流式注意 + +当使用 grounding 时，`groundingChunks` 仅包含**尚未出现在先前响应中的** grounding 元数据块。客户端需要自行累积所有响应中的 grounding chunks。 + +--- + +## 10. Context Caching API + +### CachedContent 资源 + +```json +{ + "name": "cachedContents/abc123", + "displayName": "My Cache", + "model": "models/gemini-2.5-flash", + "contents": [{ "role": "user", "parts": [{ "text": "Long document..." }] }], + "tools": [], + "systemInstruction": { "role": "user", "parts": [{ "text": "System prompt" }] }, + "toolConfig": {}, + "createTime": "2025-01-01T00:00:00Z", + "updateTime": "2025-01-01T00:00:00Z", + "usageMetadata": { "totalTokenCount": 50000 }, + "expireTime": "2025-01-08T00:00:00Z", + "ttl": "604800s" +} +``` + +### 创建缓存 + +``` +POST /v1beta/cachedContents +``` + +请求体为 CachedContent 对象，必填字段： +- `model` — 必填，格式 `models/{model}` +- `contents` — 可选，要缓存的内容 +- `expiration` — 必须设置 `expireTime` 或 `ttl` 之一 + +### 更新缓存 + +``` +PATCH /v1beta/{cachedContent.name=cachedContents/*}?updateMask=expireTime,ttl +``` + +仅可更新过期时间（`expireTime` 或 `ttl`）。 + +### 使用缓存 + +在 generateContent 请求中设置 `cachedContent` 字段： +```json +{ + "cachedContent": "cachedContents/abc123", + "contents": [{ "role": "user", "parts": [{ "text": "Follow-up question" }] }] +} +``` + +--- + +## 11. File API + +### 上传文件 + +使用 resumable upload： +``` +POST https://generativelanguage.googleapis.com/upload/v1beta/files +``` + +或 metadata only： +``` +POST https://generativelanguage.googleapis.com/v1beta/files +``` + +### File 资源 + +```json +{ + "name": "files/abc-123", + "displayName": "My Image", + "mimeType": "image/png", + "sizeBytes": "123456", + "createTime": "2025-01-01T00:00:00Z", + "updateTime": "2025-01-01T00:00:00Z", + "expirationTime": "2025-01-08T00:00:00Z", + "sha256Hash": "base64...", + "uri": "https://generativelanguage.googleapis.com/v1beta/files/abc-123", + "downloadUri": "https://...", + "state": "ACTIVE", + "source": "UPLOADED", + "error": { "code": 0, "message": "" }, + "videoMetadata": { "videoDuration": "120s" } +} +``` + +### State 枚举 + +| 值 | 说明 | +|---|------| +| `STATE_UNSPECIFIED` | 默认 | +| `PROCESSING` | 正在处理 | +| `ACTIVE` | 可用 | +| `FAILED` | 处理失败 | + +### Source 枚举 + +| 值 | 说明 | +|---|------| +| `SOURCE_UNSPECIFIED` | 未指定 | +| `UPLOADED` | 用户上传 | +| `GENERATED` | Google 生成 | +| `REGISTERED` | 注册的 GCS 文件 | + +### 列出文件 + +``` +GET /v1beta/files?pageSize=10&pageToken=xxx +``` + +响应：`{ "files": [...], "nextPageToken": "xxx" }` + +### 注册 GCS 文件 + +``` +POST /v1beta/files:register +{ "uris": ["gs://bucket/object"] } +``` + +响应：`{ "files": [...] }` + +--- + +## 12. Token 计数端点 + +``` +POST /v1beta/{model=models/*}:countTokens +``` + +### 请求体 + +```json +{ + "contents": [{ "role": "user", "parts": [{ "text": "Hello" }] }] +} +``` + +或使用完整 generateContent 请求（两者互斥）： + +```json +{ + "generateContentRequest": { + "contents": [...], + "systemInstruction": {...}, + "tools": [...], + "generationConfig": {...} + } +} +``` + +### 响应体 + +```json +{ + "totalTokens": 42, + "cachedContentTokenCount": 10, + "promptTokensDetails": [ + { "modality": "TEXT", "tokenCount": 32 }, + { "modality": "IMAGE", "tokenCount": 10 } + ], + "cacheTokensDetails": [] +} +``` + +--- + +## 13. Embeddings 端点 + +### 单条嵌入 + +``` +POST /v1beta/{model=models/*}:embedContent +``` + +```json +{ + "content": { "parts": [{ "text": "Hello world" }] }, + "taskType": "RETRIEVAL_DOCUMENT", + "title": "My Document", + "outputDimensionality": 256 +} +``` + +| 字段 | 类型 | 说明 | +|------|------|------| +| `content` | `object(Content)` | 必填，要嵌入的内容 | +| `taskType` | `enum(TaskType)` | 可选，任务类型 | +| `title` | `string` | 可选，仅适用于 RETRIEVAL_DOCUMENT | +| `outputDimensionality` | `integer` | 可选，输出维度截断 | + +响应： +```json +{ + "embedding": { + "values": [0.1, 0.2, ...], + "shape": [768] + } +} +``` + +### 批量嵌入 + +``` +POST /v1beta/{model=models/*}:batchEmbedContents +``` + +```json +{ + "requests": [ + { + "model": "models/gemini-embedding-001", + "content": { "parts": [{ "text": "Hello" }] }, + "taskType": "RETRIEVAL_QUERY" + } + ] +} +``` + +响应：`{ "embeddings": [{ "values": [...], "shape": [768] }] }` + +### TaskType 枚举 + +| 值 | 说明 | +|---|------| +| `TASK_TYPE_UNSPECIFIED` | 默认 | +| `RETRIEVAL_QUERY` | 搜索/检索场景的查询 | +| `RETRIEVAL_DOCUMENT` | 被搜索的文档 | +| `SEMANTIC_SIMILARITY` | 语义相似度 | +| `CLASSIFICATION` | 文本分类 | +| `CLUSTERING` | 聚类 | +| `QUESTION_ANSWERING` | 问答 | +| `FACT_VERIFICATION` | 事实验证 | +| `CODE_RETRIEVAL_QUERY` | 代码检索 | + +--- + +## 14. OpenAI 兼容端点 + +Base URL: `https://generativelanguage.googleapis.com/v1beta/openai/` + +认证方式: `Authorization: Bearer $GEMINI_API_KEY` + +### Chat Completions + +``` +POST /v1beta/openai/chat/completions +``` + +请求/响应格式与 OpenAI Chat Completions API 完全兼容。支持： +- Thinking（`reasoning_effort`） +- Streaming +- Function calling +- Image understanding +- Audio understanding +- Structured output +- Flex/Priority inference（`service_tier`） + +### Images + +``` +POST /v1beta/openai/images/generations +``` + +支持的参数：`prompt`, `model`, `n`, `size`, `response_format`。 +可用模型：`gemini-2.5-flash-image`, `gemini-3-pro-image-preview`。 + +### Videos + +``` +POST /v1beta/openai/videos +``` + +创建视频生成任务（长时运行操作），返回 operation ID。 +可用模型：`veo-3.1-generate-preview`。 + +``` +GET /v1beta/openai/videos/{id} +``` + +轮询视频生成状态。 + +### Embeddings + +``` +POST /v1beta/openai/embeddings +``` + +可用模型：`gemini-embedding-2-preview`, `gemini-embedding-001`。 + +### Models + +``` +GET /v1beta/openai/models +GET /v1beta/openai/models/{model} +``` + +### extra_body 字段（Gemini 专有参数） + +通过 `extra_body` 传递 Gemini 特有的参数： + +| 参数 | 类型 | 适用端点 | 说明 | +|------|------|---------|------| +| `cached_content` | Text | Chat | 内容缓存 | +| `thinking_config` | Object | Chat | ThinkingConfig 配置 | +| `aspect_ratio` | Text | Images | 输出宽高比 | +| `generation_config` | Object | Images | Gemini 生成配置 | +| `safety_settings` | List | Images | 安全过滤设置 | +| `tools` | List | Images | 启用 grounding | +| `aspect_ratio` | Text | Video | 视频宽高比 | +| `resolution` | Text | Video | 输出分辨率（720p/1080p/4K） | +| `duration_seconds` | Integer | Video | 生成时长（4/6/8） | +| `frame_rate` | Text | Video | 帧率 | +| `input_reference` | Text | Video | 参考输入 | +| `extend_video_id` | Text | Video | 视频扩展 ID | +| `negative_prompt` | Text | Video | 排除项 | +| `seed` | Integer | Video | 确定性生成 | +| `style` | Text | Video | 视觉风格（cinematic/creative） | +| `person_generation` | Text | Video | 人物生成控制 | +| `reference_images` | List | Video | 参考图片（最多 3 张） | +| `image` | Text | Video | 首帧图片 | +| `last_frame` | Object | Video | 末帧图片 | + +### Flex 和 Priority 推理 + +通过 `service_tier` 参数指定： +- `"standard"` — 标准层级（默认） +- `"flex"` — 灵活层级（更低成本，可能更高延迟） +- `"priority"` — 优先层级（更低延迟） + +--- + +## 15. 错误格式 + +### REST API 错误（Google RPC Status 格式） + +所有 Gemini REST API 端点使用 Google RPC Status 错误格式： + +```json +{ + "error": { + "code": 400, + "message": "API key not valid. Please pass a valid API key.", + "status": "INVALID_ARGUMENT", + "details": [ + { + "@type": "type.googleapis.com/google.rpc.ErrorInfo", + "reason": "API_KEY_INVALID", + "domain": "googleapis.com", + "metadata": { "service": "generativelanguage.googleapis.com" } + } + ] + } +} +``` + +| 字段 | 类型 | 说明 | +|------|------|------| +| `code` | `integer` | HTTP 状态码 | +| `message` | `string` | 面向开发者的英文错误信息 | +| `status` | `string` | gRPC 状态码字符串 | +| `details[]` | `object[]` | 错误详情列表 | + +### 常见 HTTP 状态码 + +| 状态码 | 含义 | +|--------|------| +| 400 | 请求参数错误（INVALID_ARGUMENT） | +| 401 | 认证失败（UNAUTHENTICATED） | +| 403 | 权限不足（PERMISSION_DENIED） | +| 404 | 资源不存在（NOT_FOUND） | +| 429 | 频率限制（RESOURCE_EXHAUSTED） | +| 500 | 内部错误（INTERNAL） | +| 503 | 服务不可用（UNAVAILABLE） | + +### 内容过滤错误（promptFeedback.blockReason） + +当提示被阻止时，响应中 `promptFeedback.blockReason` 不为空，且 `candidates` 为空数组。此时应检查 `promptFeedback.safetyRatings` 了解具体被阻止的类别。 + +--- + +## 16. 可用模型列表 + +### 当前活跃模型 + +#### Gemini 3 系列 + +| 模型 ID | 说明 | +|---------|------| +| `gemini-3-flash-preview` | 最佳性价比推理模型 | +| `gemini-3-pro-image-preview` | 带图像生成的专业模型 | + +#### Gemini 2.5 系列 + +| 模型 ID | 说明 | 输入限制 | 输出限制 | +|---------|------|---------|---------| +| `gemini-2.5-flash` | 高性能推理模型 | 1,048,576 tokens | 65,536 tokens | +| `gemini-2.5-flash-lite` | 轻量级推理模型 | 1,048,576 tokens | 65,536 tokens | +| `gemini-2.5-pro` | 最先进的复杂任务模型 | 1,048,576 tokens | 65,536 tokens | +| `gemini-2.5-flash-image` | 图像生成模型 | — | — | +| `gemini-2.5-pro-tts-preview` | 高保真语音合成 | — | — | + +#### Nano Banana 系列（图像生成） + +| 模型 ID | 说明 | +|---------|------| +| `nano-banana` | 原生图像生成与编辑 | +| `nano-banana-2-preview` | 高效率大规模视觉创作 | +| `nano-banana-pro-preview` | 专业 4K 视觉设计引擎 | + +#### Veo 系列（视频生成） + +| 模型 ID | 说明 | +|---------|------| +| `veo-3.1-generate-preview` | 电影级视频生成 | +| `veo-3.1-lite-preview` | 低成本视频生成 | + +#### Lyria 系列（音乐生成） + +| 模型 ID | 说明 | +|---------|------| +| `lyria-3-pro-preview` | 全长歌曲生成 | +| `lyria-3-clip-preview` | 短音乐片段生成（30s） | +| `lyria-realtime-experimental` | 实时音乐生成 | + +#### Imagen 系列 + +| 模型 ID | 说明 | +|---------|------| +| `imagen-4` | 文本到图像，支持 2K 分辨率 | + +#### 特殊用途模型 + +| 模型 ID | 说明 | +|---------|------| +| `computer-use-preview` | 计算机使用（浏览器自动化） | +| `gemini-deep-research-preview` | 深度研究代理 | +| `gemini-embedding-2-preview` | 多模态嵌入模型 | +| `gemini-embedding-001` | 文本嵌入（768 维） | +| `text-embedding-004` | 文本嵌入（768 维） | +| `gemini-robotics-er-1.6-preview` | 机器人具身推理 | + +#### 已弃用模型 + +| 模型 ID | 说明 | +|---------|------| +| `gemini-2.0-flash` | 已弃用 | +| `gemini-1.5-flash` | 输入 1,048,576 / 输出 8,192 | +| `gemini-1.5-flash-8b` | 输入 1,048,576 / 输出 8,192 | +| `gemini-1.5-pro` | 输入 2,097,152 / 输出 8,192 | +| `gemini-1.0-pro` | 已弃用 | +| `aqa` | 输入 7,168 / 输出 1,024 | + +### 模型版本命名规则 + +| 后缀 | 说明 | 示例 | +|------|------|------| +| 无后缀 | 稳定版 | `gemini-2.5-flash` | +| `-preview-*` | 预览版，可用于生产 | `gemini-2.5-flash-preview-09-2025` | +| `-latest` | 最新版本别名 | `gemini-flash-latest` | +| `-experimental-*` | 实验版，不推荐生产使用 | — | + +### ModelStage 枚举 + +| 值 | 说明 | +|---|------| +| `MODEL_STAGE_UNSPECIFIED` | 未指定 | +| `UNSTABLE_EXPERIMENTAL` | 频繁调整的实验阶段 | +| `EXPERIMENTAL` | 实验阶段 | +| `PREVIEW` | 预览阶段（比实验更成熟） | +| `STABLE` | 稳定阶段（可用于生产） | +| `LEGACY` | 遗留阶段（即将弃用） | +| `DEPRECATED` | 已弃用（不可使用） | +| `RETIRED` | 已退役（不可使用） | + +### ServiceTier 枚举 + +| 值 | 说明 | +|---|------| +| `unspecified` | 默认，等同于 standard | +| `standard` | 标准层级 | +| `flex` | 灵活层级（低成本，高延迟） | +| `priority` | 优先层级（低延迟） | + +--- + +## 17. 枚举值速查表 + +### FinishReason 完整列表（21 值） + +| Gemini | OpenAI 映射 | Claude 映射 | +|--------|------------|-------------| +| `STOP` | `stop` | `end_turn` | +| `MAX_TOKENS` | `length` | `max_tokens` | +| `SAFETY` | `content_filter` | — | +| `RECITATION` | — | — | +| `LANGUAGE` | — | — | +| `OTHER` | — | — | +| `BLOCKLIST` | `content_filter` | — | +| `PROHIBITED_CONTENT` | `content_filter` | — | +| `SPII` | — | — | +| `MALFORMED_FUNCTION_CALL` | — | — | +| `IMAGE_SAFETY` | `content_filter` | — | +| `IMAGE_PROHIBITED_CONTENT` | `content_filter` | — | +| `IMAGE_OTHER` | — | — | +| `NO_IMAGE` | — | — | +| `IMAGE_RECITATION` | — | — | +| `UNEXPECTED_TOOL_CALL` | — | — | +| `TOO_MANY_TOOL_CALLS` | — | — | +| `MISSING_THOUGHT_SIGNATURE` | — | — | +| `MALFORMED_RESPONSE` | — | — | + +### FunctionCallingConfig Mode + +| Gemini | OpenAI tool_choice 映射 | +|--------|------------------------| +| `AUTO` | `"auto"` | +| `ANY` | `"required"` 或 `{"type": "function", "function": {"name": "..."}}` | +| `NONE` | `"none"` | +| `VALIDATED` | `"auto"`（带约束验证） | + +### Content Part Modality（用于 ModalityTokenCount） + +| 值 | 说明 | +|---|------| +| `TEXT` | 纯文本 | +| `IMAGE` | 图像 | +| `VIDEO` | 视频 | +| `AUDIO` | 音频 | +| `DOCUMENT` | 文档（如 PDF） | + +### Response Modality（用于 responseModalities） + +| 值 | 说明 | +|---|------| +| `TEXT` | 返回文本 | +| `IMAGE` | 返回图像 | +| `AUDIO` | 返回音频 | + +### UrlRetrievalStatus + +| 值 | 说明 | +|---|------| +| `URL_RETRIEVAL_STATUS_UNSPECIFIED` | 默认 | +| `URL_RETRIEVAL_STATUS_SUCCESS` | 成功 | +| `URL_RETRIEVAL_STATUS_ERROR` | 失败 | +| `URL_RETRIEVAL_STATUS_PAYWALL` | 内容付费墙 | +| `URL_RETRIEVAL_STATUS_UNSAFE` | 内容不安全 | + +### Code Language + +| 值 | 说明 | +|---|------| +| `LANGUAGE_UNSPECIFIED` | 未指定 | +| `PYTHON` | Python >= 3.10 | + +### Function Behavior + +| 值 | 说明 | +|---|------| +| `UNSPECIFIED` | 未指定 | +| `BLOCKING` | 阻塞等待响应 | +| `NON_BLOCKING` | 非阻塞，异步响应 | + +### Scheduling + +| 值 | 说明 | +|---|------| +| `SCHEDULING_UNSPECIFIED` | 默认 | +| `SILENT` | 仅添加到上下文，不触发生成 | +| `WHEN_IDLE` | 空闲时触发生成 | +| `INTERRUPT` | 中断当前生成并触发 | diff --git a/docs/协议文档/Anthropic-Claude-Messages-API-完整文档.md b/docs/协议文档/Anthropic-Claude-Messages-API-完整文档.md new file mode 100644 index 0000000..59f7ba1 --- /dev/null +++ b/docs/协议文档/Anthropic-Claude-Messages-API-完整文档.md @@ -0,0 +1,1731 @@ +# Anthropic Claude Messages API 完整文档 + +> 最后更新: 2026-04-19 | API 版本: 2023-06-01 +> 来源: https://docs.anthropic.com/en/api/messages + +--- + +## 目录 + +1. [端点概览](#1-端点概览) +2. [Messages API 请求参数](#2-messages-api-请求参数) +3. [消息内容类型](#3-消息内容类型) +4. [响应格式](#4-响应格式) +5. [响应内容块类型](#5-响应内容块类型) +6. [流式响应 (SSE)](#6-流式响应-sse) +7. [扩展思维 (Extended Thinking)](#7-扩展思维-extended-thinking) +8. [工具使用 (Tool Use)](#8-工具使用-tool-use) +9. [视觉/图像支持 (Vision)](#9-视觉图像支持-vision) +10. [提示缓存 (Prompt Caching)](#10-提示缓存-prompt-caching) +11. [错误格式](#11-错误格式) +12. [模型 ID 与定价](#12-模型-id-与定价) +13. [Token 计数端点](#13-token-计数端点) +14. [Batch API](#14-batch-api) +15. [MCP 服务器](#15-mcp-服务器) + +--- + +## 1. 端点概览 + +### Messages API + +| 属性 | 值 | +|------|-----| +| **端点** | `POST https://api.anthropic.com/v1/messages` | +| **Content-Type** | `application/json` | +| **认证** | `x-api-key` 请求头 | +| **API 版本** | `anthropic-version: 2023-06-01` | +| **最大请求体** | 32 MB | + +### 必需请求头 + +| 请求头 | 类型 | 必需 | 说明 | +|--------|------|------|------| +| `anthropic-version` | string | **是** | API 版本号，当前为 `2023-06-01` | +| `x-api-key` | string | **是** | API 密钥，每个密钥绑定到一个 Workspace | +| `anthropic-beta` | string[] | 否 | Beta 功能标识，多个用逗号分隔，如 `beta1,beta2` | +| `content-type` | string | **是** | `application/json` | + +--- + +## 2. Messages API 请求参数 + +### 请求体 (Request Body) + +| 参数 | 类型 | 必需 | 默认值 | 说明 | +|------|------|------|--------|------| +| `model` | string | **是** | - | 模型 ID，长度 1-256 字符 | +| `messages` | object[] | **是** | - | 输入消息数组，最多 100,000 条 | +| `max_tokens` | integer | **是** | - | 最大生成 token 数，>= 1 | +| `system` | string \| object[] | 否 | - | 系统提示词 | +| `temperature` | number | 否 | `1.0` | 随机性控制，范围 0-1 | +| `top_p` | number | 否 | - | 核采样，范围 0-1 | +| `top_k` | integer | 否 | - | Top-K 采样，>= 0 | +| `stop_sequences` | string[] | 否 | - | 自定义停止序列 | +| `stream` | boolean | 否 | `false` | 是否使用 SSE 流式响应 | +| `tools` | object[] | 否 | - | 工具定义数组 | +| `tool_choice` | object | 否 | - | 工具选择策略 | +| `thinking` | object | 否 | - | 扩展思维配置 | +| `metadata` | object | 否 | - | 请求元数据 | +| `service_tier` | enum | 否 | - | 服务层级：`auto`、`standard_only` | +| `container` | string \| null | 否 | - | 容器标识符，用于跨请求复用 | +| `mcp_servers` | object[] | 否 | - | MCP 服务器配置 | + +### 参数详细说明 + +#### `model` (string, 必需) + +要使用的模型 ID。 + +```json +"model": "claude-sonnet-4-20250514" +``` + +#### `messages` (object[], 必需) + +输入消息数组。模型训练在交替的 `user` 和 `assistant` 对话轮次上运行。连续的 `user` 或 `assistant` 消息会被合并为一个轮次。 + +**消息对象:** + +| 字段 | 类型 | 必需 | 说明 | +|------|------|------|------| +| `role` | enum | **是** | `user` 或 `assistant` | +| `content` | string \| object[] | **是** | 消息内容，可以是字符串或内容块数组 | + +**示例:** + +```json +// 单条用户消息 +[{"role": "user", "content": "Hello, Claude"}] + +// 多轮对话 +[ + {"role": "user", "content": "Hello there."}, + {"role": "assistant", "content": "Hi, I'm Claude. How can I help you?"}, + {"role": "user", "content": "Can you explain LLMs in plain English?"} +] + +// 预填充助手响应 +[ + {"role": "user", "content": "What's the Greek name for Sun? (A) Sol (B) Helios (C) Sun"}, + {"role": "assistant", "content": "The best answer is ("} +] + +// 等价写法 +{"role": "user", "content": "Hello, Claude"} +// 等价于 +{"role": "user", "content": [{"type": "text", "text": "Hello, Claude"}]} +``` + +**注意:** 系统提示词使用顶层 `system` 参数，消息中没有 `system` 角色。 + +#### `max_tokens` (integer, 必需) + +最大生成 token 数。模型可能在此之前停止。不同模型有不同的最大值。 + +```json +"max_tokens": 1024 +``` + +#### `system` (string | object[], 可选) + +系统提示词，为 Claude 提供上下文和指令。 + +```json +// 字符串格式 +"system": "Today's date is 2023-01-01." + +// 数组格式（支持缓存控制） +"system": [ + { + "text": "Today's date is 2024-06-01.", + "type": "text" + } +] +``` + +#### `temperature` (number, 可选, 默认 1.0) + +控制响应随机性。接近 0.0 适合分析/多选题，接近 1.0 适合创意/生成任务。即使设为 0.0 也不会完全确定性。 + +```json +"temperature": 1.0 +``` + +#### `top_p` (number, 可选) + +核采样参数。应只修改 `temperature` 或 `top_p` 其中一个，不要同时修改。 + +```json +"top_p": 0.7 +``` + +#### `top_k` (integer, 可选) + +只从概率最高的 K 个选项中采样，用于去除"长尾"低概率响应。仅推荐高级场景使用。 + +```json +"top_k": 5 +``` + +#### `stop_sequences` (string[], 可选) + +自定义停止序列。当模型遇到这些字符串时会停止生成。 + +```json +"stop_sequences": ["\n\nHuman:", "\n\nAssistant:"] +``` + +#### `stream` (boolean, 可选, 默认 false) + +是否使用 Server-Sent Events (SSE) 增量流式传输响应。**当 `max_tokens` > 21,333 时必须启用流式传输。** + +```json +"stream": true +``` + +#### `metadata` (object, 可选) + +请求元数据。 + +| 字段 | 类型 | 说明 | +|------|------|------| +| `user_id` | string \| null | 用户外部标识符（UUID/哈希），最长 256 字符。不应包含姓名、邮箱等身份信息 | + +```json +"metadata": { + "user_id": "13803d75-b4b5-4c3e-b2a2-6f21399b021b" +} +``` + +#### `service_tier` (enum, 可选) + +服务层级选择。 + +| 值 | 说明 | +|-----|------| +| `auto` | 自动选择优先或标准容量 | +| `standard_only` | 仅使用标准容量 | + +#### `container` (string | null, 可选) + +容器标识符，用于跨请求复用。 + +--- + +## 3. 消息内容类型 + +### 输入消息内容块 + +消息的 `content` 字段可以是字符串或内容块数组。支持以下内容块类型: + +#### 3.1 文本内容块 + +```json +{ + "type": "text", + "text": "描述这张图片的内容。" +} +``` + +#### 3.2 图像内容块 + +支持三种图像源类型: + +**Base64 编码图像:** + +```json +{ + "type": "image", + "source": { + "type": "base64", + "media_type": "image/jpeg", + "data": "base64编码的图像数据..." + } +} +``` + +**URL 图像:** + +```json +{ + "type": "image", + "source": { + "type": "url", + "url": "https://example.com/image.jpg" + } +} +``` + +**Files API 图像 (需要 `anthropic-beta: files-api-2025-04-14`):** + +```json +{ + "type": "image", + "source": { + "type": "file", + "file_id": "file_abc123" + } +} +``` + +#### 3.3 工具结果内容块 + +用于将工具执行结果返回给模型: + +```json +{ + "type": "tool_result", + "tool_use_id": "toolu_01D7FLrfh4GYq7yT1ULFeyMV", + "content": "259.75 USD" +} +``` + +`content` 可以是字符串或内容块数组（用于返回图像等复杂结果）。 + +#### 3.4 思维内容块 (Thinking) + +扩展思维启用时，需要将上一轮的 thinking 块原样传回: + +```json +{ + "type": "thinking", + "thinking": "Let me solve this step by step...", + "signature": "EqQBCgIYAhIM1gbcDa9GJwZA2b3hGgxBdjrkzLoky3dl1pkiMOYds..." +} +``` + +#### 3.5 脱敏思维内容块 (Redacted Thinking) + +```json +{ + "type": "redacted_thinking", + "thinking": "加密内容...", + "signature": "签名数据..." +} +``` + +#### 3.6 带缓存控制的内容块 + +任何内容块都可以附加 `cache_control`: + +```json +{ + "type": "text", + "text": "大段静态文本内容...", + "cache_control": { + "type": "ephemeral", + "ttl": "5m" + } +} +``` + +--- + +## 4. 响应格式 + +### 非流式响应 (HTTP 200) + +```json +{ + "id": "msg_013Zva2CMHLNnXjNJJKqJ2EF", + "type": "message", + "role": "assistant", + "content": [ + { + "type": "text", + "text": "Hi! My name is Claude." + } + ], + "model": "claude-sonnet-4-20250514", + "stop_reason": "end_turn", + "stop_sequence": null, + "usage": { + "input_tokens": 2095, + "output_tokens": 503, + "cache_creation_input_tokens": 0, + "cache_read_input_tokens": 0, + "cache_creation": { + "ephemeral_5m_input_tokens": 0, + "ephemeral_1h_input_tokens": 0 + }, + "server_tool_use": { + "web_search_requests": 0 + }, + "service_tier": "standard" + }, + "container": null +} +``` + +### 响应字段详解 + +| 字段 | 类型 | 必需 | 说明 | +|------|------|------|------| +| `id` | string | **是** | 唯一对象标识符，如 `msg_013Zva2CMHLNnXjNJJKqJ2EF` | +| `type` | enum | **是** | 固定为 `"message"` | +| `role` | enum | **是** | 固定为 `"assistant"` | +| `content` | object[] | **是** | 内容块数组，见第 5 节 | +| `model` | string | **是** | 处理请求的模型 ID | +| `stop_reason` | enum \| null | **是** | 停止原因，见下表 | +| `stop_sequence` | string \| null | **是** | 匹配到的自定义停止序列 | +| `usage` | object | **是** | Token 使用量，见下文 | +| `container` | object \| null | **是** | 容器信息（使用代码执行等工具时非空） | + +### stop_reason 枚举值 + +| 值 | 说明 | +|-----|------| +| `end_turn` | 模型自然停止 | +| `max_tokens` | 超过请求的 `max_tokens` 或模型最大值 | +| `stop_sequence` | 遇到自定义停止序列 | +| `tool_use` | 模型调用了一个或多个工具 | +| `pause_turn` | 暂停长运行的轮次，可在后续请求中原样传回继续 | +| `refusal` | 流式分类器干预处理潜在的策略违规 | + +### usage 对象 + +| 字段 | 类型 | 默认值 | 说明 | +|------|------|--------|------| +| `input_tokens` | integer | - | 使用的输入 token 数 | +| `output_tokens` | integer | - | 使用的输出 token 数 | +| `cache_creation_input_tokens` | integer \| null | - | 创建缓存条目使用的 token 数 | +| `cache_read_input_tokens` | integer \| null | - | 从缓存读取的 token 数 | +| `cache_creation` | object \| null | - | 按 TTL 分组的缓存创建明细 | +| `cache_creation.ephemeral_5m_input_tokens` | integer | `0` | 5 分钟缓存写入的 token 数 | +| `cache_creation.ephemeral_1h_input_tokens` | integer | `0` | 1 小时缓存写入的 token 数 | +| `server_tool_use` | object \| null | - | 服务器端工具使用统计 | +| `server_tool_use.web_search_requests` | integer | `0` | Web 搜索工具请求次数 | +| `service_tier` | enum \| null | - | 服务层级：`standard`、`priority`、`batch` | + +**总输入 token 计算:** `input_tokens` + `cache_creation_input_tokens` + `cache_read_input_tokens` + +### container 对象 + +当使用容器工具（如代码执行）时非空。 + +| 字段 | 类型 | 说明 | +|------|------|------| +| `id` | string | 容器标识符 | +| `expires_at` | string | 容器过期时间 | + +--- + +## 5. 响应内容块类型 + +### 5.1 text - 文本内容块 + +```json +{ + "type": "text", + "text": "Hi! My name is Claude." +} +``` + +### 5.2 thinking - 思维内容块 + +扩展思维启用时返回。包含模型的内部推理过程。 + +```json +{ + "type": "thinking", + "thinking": "Let me solve this step by step:\n\n1. First break down 27 * 453...", + "signature": "EqQBCgIYAhIM1gbcDa9GJwZA2b3hGgxBdjrkzLoky3dl1pkiMOYds..." +} +``` + +| 字段 | 类型 | 说明 | +|------|------|------| +| `type` | string | 固定 `"thinking"` | +| `thinking` | string | 思维内容文本 | +| `signature` | string | 签名，用于验证 thinking 块的完整性，传回 API 时必须原样保留 | + +**Claude 4 模型返回的是思维摘要 (Summarized Thinking)，不是完整思维。** + +### 5.3 redacted_thinking - 脱敏思维内容块 + +安全系统标记时返回加密的思维内容。 + +```json +{ + "type": "redacted_thinking", + "thinking": "加密的不可读内容...", + "signature": "签名数据..." +} +``` + +### 5.4 tool_use - 工具使用内容块 + +```json +{ + "type": "tool_use", + "id": "toolu_01D7FLrfh4GYq7yT1ULFeyMV", + "name": "get_stock_price", + "input": { + "ticker": "^GSPC" + } +} +``` + +| 字段 | 类型 | 说明 | +|------|------|------| +| `type` | string | 固定 `"tool_use"` | +| `id` | string | 工具调用唯一标识，如 `toolu_01...` | +| `name` | string | 调用的工具名称 | +| `input` | object | 工具输入参数（JSON 对象） | + +### 5.5 server_tool_use - 服务器端工具使用 + +```json +{ + "type": "server_tool_use", + "id": "server_tool_id", + "name": "web_search", + "input": { ... } +} +``` + +### 5.6 web_search_tool_result - Web 搜索工具结果 + +```json +{ + "type": "web_search_tool_result", + "tool_use_id": "server_tool_id", + "content": [ ... ] +} +``` + +### 5.7 code_execution_tool_result - 代码执行工具结果 + +```json +{ + "type": "code_execution_tool_result", + "tool_use_id": "...", + "content": [ ... ] +} +``` + +### 5.8 mcp_tool_use - MCP 工具使用 + +```json +{ + "type": "mcp_tool_use", + "id": "...", + "name": "...", + "input": { ... }, + "server_name": "..." +} +``` + +### 5.9 mcp_tool_result - MCP 工具结果 + +```json +{ + "type": "mcp_tool_result", + "tool_use_id": "...", + "content": [ ... ] +} +``` + +### 5.10 container_upload - 容器上传 + +```json +{ + "type": "container_upload", + "file_id": "file_abc123" +} +``` + +--- + +## 6. 流式响应 (SSE) + +当设置 `"stream": true` 时，响应使用 Server-Sent Events (SSE) 格式，`Content-Type: text/event-stream`。 + +### 6.1 事件流顺序 + +``` +1. message_start → 包含空 content 的 Message 对象 +2. 内容块系列（每个）: + a. content_block_start → 内容块开始 + b. content_block_delta → 一个或多个增量事件 + c. content_block_stop → 内容块结束 +3. message_delta → 顶层 Message 对象变更 +4. message_stop → 消息结束 +``` + +可能穿插 `ping` 事件。 + +### 6.2 事件类型 + +#### message_start + +```json +event: message_start +data: { + "type": "message_start", + "message": { + "id": "msg_1nZdL29xx5MUA1yADyHTEsnR8uuvGzszyY", + "type": "message", + "role": "assistant", + "content": [], + "model": "claude-3-7-sonnet-20250219", + "stop_reason": null, + "stop_sequence": null, + "usage": { + "input_tokens": 25, + "output_tokens": 1 + } + } +} +``` + +#### content_block_start + +```json +event: content_block_start +data: { + "type": "content_block_start", + "index": 0, + "content_block": { + "type": "text", + "text": "" + } +} +``` + +#### content_block_delta + +见下文 Delta 类型。 + +#### content_block_stop + +```json +event: content_block_stop +data: { + "type": "content_block_stop", + "index": 0 +} +``` + +#### message_delta + +```json +event: message_delta +data: { + "type": "message_delta", + "delta": { + "stop_reason": "end_turn", + "stop_sequence": null + }, + "usage": { + "output_tokens": 15 + } +} +``` + +#### message_stop + +```json +event: message_stop +data: { + "type": "message_stop" +} +``` + +#### ping + +```json +event: ping +data: {"type": "ping"} +``` + +#### error + +```json +event: error +data: { + "type": "error", + "error": { + "type": "overloaded_error", + "message": "Overloaded" + } +} +``` + +### 6.3 Delta 类型 + +#### text_delta - 文本增量 + +```json +event: content_block_delta +data: { + "type": "content_block_delta", + "index": 0, + "delta": { + "type": "text_delta", + "text": "ello frien" + } +} +``` + +#### input_json_delta - 工具输入 JSON 增量 + +增量是**部分 JSON 字符串**，需要累积后解析。最终 `tool_use.input` 总是一个 JSON 对象。 + +```json +event: content_block_delta +data: { + "type": "content_block_delta", + "index": 1, + "delta": { + "type": "input_json_delta", + "partial_json": "{\"location\": \"San Fra" + } +} +``` + +**注意:** 当前模型一次只支持输出一个完整的 key-value 属性，因此工具使用时可能有事件间隔。 + +#### thinking_delta - 思维增量 + +```json +event: content_block_delta +data: { + "type": "content_block_delta", + "index": 0, + "delta": { + "type": "thinking_delta", + "thinking": "Let me solve this step by step:\n\n1. First break down 27 * 453" + } +} +``` + +#### signature_delta - 签名增量 + +在 `content_block_stop` 之前发送，用于验证 thinking 块的完整性。 + +```json +event: content_block_delta +data: { + "type": "content_block_delta", + "index": 0, + "delta": { + "type": "signature_delta", + "signature": "EqQBCgIYAhIM1gbcDa9GJwZA2b3hGgxBdjrkzLoky3dl1pkiMOYds..." + } +} +``` + +### 6.4 完整流式示例 + +#### 基本流式请求 + +``` +请求: +POST /v1/messages +{ + "model": "claude-3-7-sonnet-20250219", + "messages": [{"role": "user", "content": "Hello"}], + "max_tokens": 256, + "stream": true +} + +响应: +event: message_start +data: {"type": "message_start", "message": {"id": "msg_1nZ...", "type": "message", "role": "assistant", "content": [], "model": "claude-3-7-sonnet-20250219", "stop_reason": null, "stop_sequence": null, "usage": {"input_tokens": 25, "output_tokens": 1}}} + +event: content_block_start +data: {"type": "content_block_start", "index": 0, "content_block": {"type": "text", "text": ""}} + +event: ping +data: {"type": "ping"} + +event: content_block_delta +data: {"type": "content_block_delta", "index": 0, "delta": {"type": "text_delta", "text": "Hello"}} + +event: content_block_delta +data: {"type": "content_block_delta", "index": 0, "delta": {"type": "text_delta", "text": "!"}} + +event: content_block_stop +data: {"type": "content_block_stop", "index": 0} + +event: message_delta +data: {"type": "message_delta", "delta": {"stop_reason": "end_turn", "stop_sequence": null}, "usage": {"output_tokens": 15}} + +event: message_stop +data: {"type": "message_stop"} +``` + +#### 工具使用流式请求 + +``` +请求: +POST /v1/messages +{ + "model": "claude-3-7-sonnet-20250219", + "max_tokens": 1024, + "tools": [{ + "name": "get_weather", + "description": "Get the current weather in a given location", + "input_schema": { + "type": "object", + "properties": { + "location": {"type": "string", "description": "The city and state, e.g. San Francisco, CA"} + }, + "required": ["location"] + } + }], + "tool_choice": {"type": "any"}, + "messages": [{"role": "user", "content": "What is the weather like in San Francisco?"}], + "stream": true +} + +响应: +event: message_start +data: {"type":"message_start","message":{"id":"msg_014p7gG3wDgGV9EUtLvnow3U","type":"message","role":"assistant","model":"claude-3-haiku-20240307","stop_sequence":null,"usage":{"input_tokens":472,"output_tokens":2},"content":[],"stop_reason":null}} + +event: content_block_start +data: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""}} + +event: content_block_delta +data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"Okay"}} +... + +event: content_block_stop +data: {"type":"content_block_stop","index":0} + +event: content_block_start +data: {"type":"content_block_start","index":1,"content_block":{"type":"tool_use","id":"toolu_01T1x1fJ34qAmk2tNTrN7Up6","name":"get_weather","input":{}}} + +event: content_block_delta +data: {"type":"content_block_delta","index":1,"delta":{"type":"input_json_delta","partial_json":""}} + +event: content_block_delta +data: {"type":"content_block_delta","index":1,"delta":{"type":"input_json_delta","partial_json":"{\"location\":"}} + +event: content_block_delta +data: {"type":"content_block_delta","index":1,"delta":{"type":"input_json_delta","partial_json":" \"San Francisc"}} + +event: content_block_delta +data: {"type":"content_block_delta","index":1,"delta":{"type":"input_json_delta","partial_json":"o, CA\"}"}} + +event: content_block_stop +data: {"type":"content_block_stop","index":1} + +event: message_delta +data: {"type":"message_delta","delta":{"stop_reason":"tool_use","stop_sequence":null},"usage":{"output_tokens":89}} + +event: message_stop +data: {"type":"message_stop"} +``` + +#### 扩展思维流式请求 + +``` +请求: +POST /v1/messages +{ + "model": "claude-3-7-sonnet-20250219", + "max_tokens": 20000, + "stream": true, + "thinking": { + "type": "enabled", + "budget_tokens": 16000 + }, + "messages": [{"role": "user", "content": "What is 27 * 453?"}] +} + +响应: +event: message_start +data: {"type": "message_start", "message": {"id": "msg_01...", ...}} + +event: content_block_start +data: {"type": "content_block_start", "index": 0, "content_block": {"type": "thinking", "thinking": ""}} + +event: content_block_delta +data: {"type": "content_block_delta", "index": 0, "delta": {"type": "thinking_delta", "thinking": "Let me solve this step by step:\n\n1. First break down 27 * 453"}} + +event: content_block_delta +data: {"type": "content_block_delta", "index": 0, "delta": {"type": "thinking_delta", "thinking": "\n2. 453 = 400 + 50 + 3"}} + +... + +event: content_block_delta +data: {"type": "content_block_delta", "index": 0, "delta": {"type": "signature_delta", "signature": "EqQBCgIYAhIM1gbcDa9GJwZA2b3hGgxBdjrkzLoky3dl1pkiMOYds..."}} + +event: content_block_stop +data: {"type": "content_block_stop", "index": 0} + +event: content_block_start +data: {"type": "content_block_start", "index": 1, "content_block": {"type": "text", "text": ""}} + +event: content_block_delta +data: {"type": "content_block_delta", "index": 1, "delta": {"type": "text_delta", "text": "27 * 453 = 12,231"}} + +event: content_block_stop +data: {"type": "content_block_stop", "index": 1} + +event: message_delta +data: {"type": "message_delta", "delta": {"stop_reason": "end_turn", "stop_sequence": null}} + +event: message_stop +data: {"type": "message_stop"} +``` + +--- + +## 7. 扩展思维 (Extended Thinking) + +### 支持的模型 + +- `claude-opus-4-1-20250805` (Claude Opus 4.1) +- `claude-opus-4-20250514` (Claude Opus 4) +- `claude-sonnet-4-20250514` (Claude Sonnet 4) +- `claude-3-7-sonnet-20250219` (Claude Sonnet 3.7) + +### 启用方式 + +```json +{ + "model": "claude-sonnet-4-20250514", + "max_tokens": 16000, + "thinking": { + "type": "enabled", + "budget_tokens": 10000 + }, + "messages": [{"role": "user", "content": "Solve this complex problem..."}] +} +``` + +### thinking 对象参数 + +| 字段 | 类型 | 必需 | 说明 | +|------|------|------|------| +| `type` | enum | **是** | 固定 `"enabled"` | +| `budget_tokens` | integer | **是** | 思维 token 预算，>= 1024 且 < `max_tokens` | + +### 响应格式 + +启用扩展思维后，响应按顺序包含: +1. `thinking` 内容块 - 模型的内部推理 +2. `text` 内容块 - 最终回答 + +```json +{ + "content": [ + { + "type": "thinking", + "thinking": "Let me analyze this step by step...", + "signature": "EqQBCgIYAhIM..." + }, + { + "type": "text", + "text": "The answer is 42." + } + ] +} +``` + +### Claude 4 模型的思维摘要 (Summarized Thinking) + +Claude 4 模型（Opus 4.1、Opus 4、Sonnet 4）返回思维**摘要**，不是完整思维。 + +| 特性 | Claude Sonnet 3.7 | Claude 4 模型 | +|------|-------------------|---------------| +| 思维输出 | 返回完整思维 | 返回思维摘要 | +| 交错思维 | 不支持 | 支持（需 `interleaved-thinking-2025-05-14` beta header） | + +**计费说明:** +- **输入 token:** 原始请求中的 token（不含之前轮次的思维 token） +- **输出 token（计费）:** Claude 内部生成的原始思维 token +- **输出 token（可见）:** 响应中的摘要思维 token +- **不收费:** 生成摘要的 token + +### 思维加密 (Thinking Encryption) + +- 完整思维内容在 `signature` 字段中以加密形式返回 +- 用于验证 thinking 块由 Claude 生成 +- 流式响应中，签名通过 `signature_delta` 事件在 `content_block_stop` 之前发送 +- `signature` 是不透明字段，不应解释或解析 +- Claude 4 的签名值比之前模型显著更长 +- 签名值跨平台兼容（Anthropic API、Amazon Bedrock、Vertex AI） + +### 思维脱敏 (Thinking Redaction) + +当安全系统标记部分思维内容时: +- 被标记的部分变为 `redacted_thinking` 块 +- 内容被加密，不可人工阅读 +- 传回 API 时会被解密，Claude 可继续推理 +- **必须**原样传回所有 thinking 和 redacted_thinking 块 + +### 交错思维 (Interleaved Thinking) + +仅 Claude 4 模型支持。需要 beta header: `interleaved-thinking-2025-05-14` + +允许 Claude 在工具调用之间进行思维: +- 分析工具调用结果后再决定下一步 +- 在推理步骤之间链式调用多个工具 +- 基于中间结果做出更细致的决策 + +**注意事项:** +- 启用交错思维时，`budget_tokens` 可以超过 `max_tokens`（代表一个助手轮次内所有思维块的总预算） +- 仅支持通过 Messages API 使用的工具 +- 在非 Claude 4 模型上传递此 header 无效果 + +### 工具使用中的思维保留 + +**关键规则:** 工具使用时，必须将上一轮助手的 thinking 块**完整、原样**传回 API。 + +原因: +1. **推理连续性:** thinking 块包含导致工具请求的逐步推理 +2. **上下文维护:** 工具结果在 API 结构中表现为用户消息，但属于连续推理流 + +**重要:** 连续的 thinking 块序列必须与模型原始输出完全匹配，不能重新排列或修改。 + +### 扩展思维的限制 + +- 不兼容 `temperature` 或 `top_k` 修改 +- 不兼容强制工具使用 (`tool_choice: {"type": "any"}` 或 `tool_choice: {"type": "tool", "name": "..."}`) +- 启用思维时可设置 `top_p` 为 1 到 0.95 之间的值 +- 启用思维时不能预填充响应 +- `max_tokens` 严格执行，超出上下文窗口会返回验证错误 + +### 性能建议 + +| 建议 | 说明 | +|------|------| +| 预算优化 | 从 1,024 开始，逐步增加。预算 > 32k 时建议使用批处理 | +| 起始值 | 复杂任务从 16k+ 开始 | +| 大预算 | 超过 32k 可能导致系统超时和连接限制 | +| 流式要求 | `max_tokens` > 21,333 时**必须**启用流式 | + +--- + +## 8. 工具使用 (Tool Use) + +### 工具定义 (tools) + +```json +{ + "tools": [ + { + "name": "get_stock_price", + "description": "Get the current stock price for a given ticker symbol.", + "input_schema": { + "type": "object", + "properties": { + "ticker": { + "type": "string", + "description": "The stock ticker symbol, e.g. AAPL for Apple Inc." + } + }, + "required": ["ticker"] + } + } + ] +} +``` + +### tools 数组中每个对象的字段 + +| 字段 | 类型 | 必需 | 说明 | +|------|------|------|------| +| `name` | string | **是** | 工具名称，1-128 字符 | +| `input_schema` | object | **是** | 工具输入的 JSON Schema | +| `input_schema.type` | enum | **是** | 固定 `"object"` | +| `input_schema.properties` | object \| null | 否 | 属性定义 | +| `input_schema.required` | string[] \| null | 否 | 必需属性列表 | +| `type` | enum \| null | 否 | 工具类型，`"custom"` | +| `description` | string | 否 | 工具描述（强烈建议提供） | +| `cache_control` | object \| null | 否 | 缓存控制断点 | + +### cache_control 对象 + +| 字段 | 类型 | 必需 | 说明 | +|------|------|------|------| +| `type` | enum | **是** | 固定 `"ephemeral"` | +| `ttl` | enum | 否 | `"5m"` (默认) 或 `"1h"` | + +### tool_choice 对象 + +控制模型如何使用提供的工具。 + +#### auto (默认) + +模型自行决定是否使用工具。 + +```json +"tool_choice": {"type": "auto"} +``` + +#### any + +模型必须使用某个工具（但不指定哪个）。 + +```json +"tool_choice": {"type": "any"} +``` + +#### tool + +指定使用特定工具。 + +```json +"tool_choice": { + "type": "tool", + "name": "get_weather" +} +``` + +#### none + +禁止使用工具。 + +```json +"tool_choice": {"type": "none"} +``` + +#### disable_parallel_tool_use + +适用于 `auto`、`any`、`tool` 类型。设为 `true` 时，模型最多输出一个工具使用。 + +```json +"tool_choice": { + "type": "auto", + "disable_parallel_tool_use": true +} +``` + +### 工具类型列表 + +| 工具类型 | Beta Header | 说明 | +|----------|-------------|------| +| `custom` | - | 自定义客户端工具 | +| Bash tool (2024-10-22) | `computer-use-2024-10-22` | Bash 工具 | +| Bash tool (2025-01-24) | `computer-use-2025-01-24` | Bash 工具（更新版） | +| Code execution (2025-05-22) | `code-execution-2025-05-22` | 代码执行工具 | +| Computer use (2024-01-22) | `computer-use-2024-01-22` | 计算机使用工具 | +| Computer use (2025-01-24) | `computer-use-2025-01-24` | 计算机使用工具（更新版） | +| Text editor (2024-10-22) | `computer-use-2024-10-22` | 文本编辑器工具 | +| Text editor (2025-01-24) | `computer-use-2025-01-24` | 文本编辑器工具（更新版） | +| Text editor (2025-04-29) | `computer-use-2025-04-29` | 文本编辑器工具（最新版） | +| TextEditor_20250728 | - | 文本编辑器工具 | +| Web search (2025-03-05) | `web-search-2025-03-05` | Web 搜索工具 | + +### 工具使用完整流程 + +``` +1. 用户发送请求（带 tools 定义） +2. 模型返回 tool_use 内容块（stop_reason = "tool_use"） +3. 客户端执行工具 +4. 客户端将 tool_result 作为 user 消息发回 +5. 模型继续生成（可能再次调用工具或给出最终回答） +``` + +**tool_result 内容块:** + +```json +{ + "type": "tool_result", + "tool_use_id": "toolu_01D7FLrfh4GYq7yT1ULFeyMV", + "content": "259.75 USD" +} +``` + +`content` 可以是字符串或内容块数组，支持返回图像等复杂结果。 + +--- + +## 9. 视觉/图像支持 (Vision) + +### 支持的图像格式 + +| Media Type | 格式 | +|------------|------| +| `image/jpeg` | JPEG | +| `image/png` | PNG | +| `image/gif` | GIF | +| `image/webp` | WebP | + +### 图像限制 + +| 限制 | 值 | +|------|-----| +| 单张最大尺寸 | 8000 x 8000 px | +| 单张最大文件大小 | 5 MB (API), 10 MB (claude.ai) | +| 每次请求最大图片数 | 100 张 (API), 20 张 (claude.ai) | +| 超过 20 张时尺寸限制 | 2000 x 2000 px | + +### 图像源类型 + +#### Base64 编码 + +```json +{ + "type": "image", + "source": { + "type": "base64", + "media_type": "image/jpeg", + "data": "base64编码数据..." + } +} +``` + +#### URL 引用 + +```json +{ + "type": "image", + "source": { + "type": "url", + "url": "https://example.com/image.jpg" + } +} +``` + +#### Files API (需要 `anthropic-beta: files-api-2025-04-14`) + +```json +{ + "type": "image", + "source": { + "type": "file", + "file_id": "file_abc123" + } +} +``` + +### Token 估算 + +**公式:** `tokens = (width * height) / 750` + +**推荐尺寸:** 长边不超过 1568 像素（约 1.15 兆像素），超过会自动缩放。 + +| 图像尺寸 | 像素 | 估算 Token | 每张成本 (Sonnet) | +|----------|------|------------|-------------------| +| 200x200 | 0.04 MP | ~54 | ~$0.00016 | +| 1000x1000 | 1 MP | ~1,334 | ~$0.004 | +| 1092x1092 | 1.19 MP | ~1,590 | ~$0.0048 | + +### 不超过缩放的最大尺寸 + +| 宽高比 | 图像尺寸 | +|--------|----------| +| 1:1 | 1092x1092 px | +| 3:4 | 951x1268 px | +| 2:3 | 896x1344 px | +| 9:16 | 819x1456 px | +| 1:2 | 784x1568 px | + +### 最佳实践 + +- 图像放在问题/指令之前效果最佳（image-then-text 结构） +- 多张图像时用 "Image 1:"、"Image 2:" 标注 +- 不读取图像元数据 +- 不用于识别人物 +- 对计数、空间推理等任务有一定局限性 + +--- + +## 10. 提示缓存 (Prompt Caching) + +### 工作原理 + +1. 系统检查是否存在已缓存的提示前缀 +2. 如果命中，使用缓存版本，减少处理时间和成本 +3. 如果未命中，处理完整提示并缓存前缀 + +缓存默认 5 分钟生命周期，每次使用时免费刷新。 + +### 支持的模型 + +所有当前模型均支持，包括 Claude Opus 4.1、Opus 4、Sonnet 4、Sonnet 3.7、Haiku 3.5、Haiku 3 等。 + +### 实现 + +在任何内容块上添加 `cache_control`: + +```json +{ + "type": "text", + "text": "大段静态文本...", + "cache_control": { + "type": "ephemeral", + "ttl": "5m" + } +} +``` + +| 字段 | 类型 | 必需 | 说明 | +|------|------|------|------| +| `type` | enum | **是** | 固定 `"ephemeral"` | +| `ttl` | enum | 否 | `"5m"` (默认，5分钟) 或 `"1h"` (1小时) | + +### 缓存层级 + +前缀按以下顺序创建: `tools` -> `system` -> `messages` + +### 缓存限制 + +| 模型 | 最小可缓存 token 数 | +|------|---------------------| +| Claude Opus 4, Sonnet 4, Sonnet 3.7, Opus 3 | 1,024 | +| Claude Haiku 3.5, Haiku 3 | 2,048 | + +- 最多 4 个缓存断点 +- 并发请求中，缓存条目在首次响应开始后才可用 + +### 可缓存内容 + +| 内容类型 | 可缓存 | +|----------|--------| +| Tools 工具定义 | 是 | +| System 系统消息 | 是 | +| Text 文本消息 | 是 | +| Images 图像 | 是 | +| Documents 文档 | 是 | +| Tool use / tool results | 是 | +| Thinking 块 | 间接缓存（随其他内容一起） | +| Citations 子内容块 | 间接缓存（缓存顶级块） | +| 空文本块 | 否 | + +### 缓存失效规则 + +| 变更内容 | Tools 缓存 | System 缓存 | Messages 缓存 | +|----------|-----------|-------------|---------------| +| 工具定义变更 | 失效 | 失效 | 失效 | +| Web 搜索开关 | 有效 | 失效 | 失效 | +| Citations 开关 | 有效 | 失效 | 失效 | +| tool_choice 变更 | 有效 | 有效 | 失效 | +| 图像变更 | 有效 | 有效 | 失效 | +| Thinking 参数变更 | 有效 | 有效 | 失效 | +| 非工具结果传入 thinking 请求 | 有效 | 有效 | 失效 | + +### 缓存性能追踪 + +响应 `usage` 中的关键字段: + +| 字段 | 说明 | +|------|------| +| `cache_creation_input_tokens` | 新建缓存条目写入的 token 数 | +| `cache_read_input_tokens` | 从缓存读取的 token 数 | +| `input_tokens` | 未缓存也未创建缓存的 token 数 | + +### 混合 TTL 计费 + +使用 1h 和 5m 混合 TTL 时: +- 1h 缓存必须出现在 5m 缓存之前 +- 确定三个计费位置: + - **位置 A:** 最高缓存命中的 token 数 + - **位置 B:** A 之后最高 1h cache_control 的 token 数 + - **位置 C:** 最后一个 cache_control 的 token 数 + +**费用计算:** +1. 缓存读取 token = A +2. 1h 缓存写入 token = (B - A) +3. 5m 缓存写入 token = (C - B) + +### 缓存隔离 + +- **组织隔离:** 不同组织的缓存互不共享 +- **精确匹配:** 缓存命中要求 100% 完全相同的提示段（包括所有文本和图像） +- **输出不受影响:** 缓存不影响输出 token 生成 + +### 定价 + +| 模型 | 基础输入 | 5m 缓存写入 | 1h 缓存写入 | 缓存命中/刷新 | 输出 | +|------|----------|-------------|-------------|---------------|------| +| Claude Opus 4.1 | $15/MTok | $18.75/MTok | $30/MTok | $1.50/MTok | $75/MTok | +| Claude Opus 4 | $15/MTok | $18.75/MTok | $30/MTok | $1.50/MTok | $75/MTok | +| Claude Sonnet 4 | $3/MTok | $3.75/MTok | $6/MTok | $0.30/MTok | $15/MTok | +| Claude Sonnet 3.7 | $3/MTok | $3.75/MTok | $6/MTok | $0.30/MTok | $15/MTok | +| Claude Haiku 3.5 | $0.80/MTok | $1/MTok | $1.6/MTok | $0.08/MTok | $4/MTok | +| Claude Haiku 3 | $0.25/MTok | $0.30/MTok | $0.50/MTok | $0.03/MTok | $1.25/MTok | + +--- + +## 11. 错误格式 + +### 错误响应结构 + +```json +{ + "type": "error", + "error": { + "type": "invalid_request_error", + "message": "具体错误描述" + } +} +``` + +### HTTP 状态码与错误类型 + +| HTTP 状态码 | error.type | 说明 | +|-------------|-----------|------| +| 400 | `invalid_request_error` | 请求格式错误、参数无效 | +| 401 | `authentication_error` | API Key 无效 | +| 403 | `permission_error` | 无权限访问 | +| 404 | `not_found_error` | 资源不存在 | +| 413 | `request_too_large` | 请求体过大 | +| 429 | `rate_limit_error` | 请求频率超限 | +| 500 | `api_error` | API 内部错误 | +| 529 | `overloaded_error` | API 过载 | + +### 请求大小限制 + +| API | 最大请求体大小 | +|-----|---------------| +| Messages API | 32 MB | +| Token Counting API | 32 MB | +| Batch API | 256 MB | +| Files API | 500 MB | + +### 流式错误 + +在 SSE 流中可能发送错误事件: + +``` +event: error +data: {"type": "error", "error": {"type": "overloaded_error", "message": "Overloaded"}} +``` + +--- + +## 12. 模型 ID 与定价 + +### 当前模型 + +| 模型 ID | 名称 | 上下文窗口 | 最大输出 | 输入价格 | 输出价格 | +|---------|------|-----------|----------|----------|----------| +| `claude-opus-4-1-20250805` | Claude Opus 4.1 | 200K | - | $15/MTok | $75/MTok | +| `claude-opus-4-20250514` | Claude Opus 4 | 200K | - | $15/MTok | $75/MTok | +| `claude-sonnet-4-20250514` | Claude Sonnet 4 | 200K | - | $3/MTok | $15/MTok | +| `claude-3-7-sonnet-20250219` | Claude Sonnet 3.7 | 200K | - | $3/MTok | $15/MTok | + +### 已弃用模型 + +| 模型 ID | 状态 | +|---------|------| +| `claude-3-5-sonnet-20241022` | 已弃用 | +| `claude-3-opus-20240229` | 已弃用 | + +### 模型选择字符串长度 + +- `model` 字段要求 1-256 字符 + +### 扩展思维支持 + +| 模型 | 扩展思维 | 摘要思维 | 交错思维 | +|------|----------|----------|----------| +| Claude Opus 4.1 | 是 | 是 | 是 | +| Claude Opus 4 | 是 | 是 | 是 | +| Claude Sonnet 4 | 是 | 是 | 是 | +| Claude Sonnet 3.7 | 是 | 否 | 否 | + +--- + +## 13. Token 计数端点 + +### 端点 + +| 属性 | 值 | +|------|-----| +| **端点** | `POST https://api.anthropic.com/v1/messages/count_tokens` | +| **Content-Type** | `application/json` | +| **认证** | `x-api-key` 请求头 | +| **API 版本** | `anthropic-version: 2023-06-01` | + +### 请求参数 + +与 Messages API 共享部分参数: + +| 参数 | 类型 | 必需 | 说明 | +|------|------|------|------| +| `model` | string | **是** | 模型 ID，1-256 字符 | +| `messages` | object[] | **是** | 输入消息数组（格式同 Messages API） | +| `system` | string \| object[] | 否 | 系统提示词 | +| `tools` | object[] | 否 | 工具定义 | +| `tool_choice` | object | 否 | 工具选择策略 | +| `thinking` | object | 否 | 扩展思维配置 | +| `mcp_servers` | object[] | 否 | MCP 服务器配置 | + +**注意:** 不需要 `max_tokens` 参数。 + +### 请求示例 + +```bash +curl https://api.anthropic.com/v1/messages/count_tokens \ + --header "x-api-key: $ANTHROPIC_API_KEY" \ + --header "anthropic-version: 2023-06-01" \ + --header "content-type: application/json" \ + --data '{ + "model": "claude-3-7-sonnet-20250219", + "messages": [ + {"role": "user", "content": "Hello, world"} + ] + }' +``` + +### 响应 + +```json +{ + "input_tokens": 2095 +} +``` + +| 字段 | 类型 | 说明 | +|------|------|------| +| `input_tokens` | integer | 所有提供的消息、系统提示词和工具的 token 总数 | + +--- + +## 14. Batch API + +### 端点 + +| 属性 | 值 | +|------|-----| +| **端点** | `POST https://api.anthropic.com/v1/messages/batches` | +| **最大请求体** | 256 MB | + +### 概述 + +Batch API 允许异步处理大量消息请求。创建批处理后，系统异步执行，完成后可获取结果。 + +### 主要端点 + +| 操作 | 方法 | 端点 | +|------|------|------| +| 创建批处理 | POST | `/v1/messages/batches` | +| 获取批处理状态 | GET | `/v1/messages/batches/{batch_id}` | +| 列出批处理 | GET | `/v1/messages/batches` | +| 取消批处理 | POST | `/v1/messages/batches/{batch_id}/cancel` | +| 获取批处理结果 | GET | `/v1/messages/batches/{batch_id}/results` | + +### 批处理请求格式 + +每个批处理请求包含多个子请求，每个子请求是一个完整的 Messages API 请求: + +```json +{ + "requests": [ + { + "custom_id": "request-1", + "params": { + "model": "claude-sonnet-4-20250514", + "max_tokens": 1024, + "messages": [{"role": "user", "content": "Hello"}] + } + }, + { + "custom_id": "request-2", + "params": { + "model": "claude-sonnet-4-20250514", + "max_tokens": 1024, + "messages": [{"role": "user", "content": "World"}] + } + } + ] +} +``` + +### 批处理定价 + +批处理请求使用标准 token 定价的 50%（`service_tier` 返回 `"batch"`）。 + +### 状态值 + +| 状态 | 说明 | +|------|------| +| `in_progress` | 正在处理 | +| `completed` | 已完成 | +| `failed` | 失败 | +| `expired` | 已过期 | +| `cancelled` | 已取消 | + +> **注意:** Batch API 页面为 JavaScript 渲染，上述信息基于 API 参考文档整合。详细字段请参阅 Anthropic 官方文档。 + +--- + +## 15. MCP 服务器 + +### 概述 + +Messages API 支持通过 MCP（Model Context Protocol）连接远程 MCP 服务器，使 Claude 能够使用这些服务器提供的工具。 + +### mcp_servers 参数 + +```json +{ + "mcp_servers": [ + { + "name": "my-mcp-server", + "type": "url", + "url": "https://mcp.example.com/sse", + "authorization_token": "Bearer token...", + "tool_configuration": { + "allowed_tools": ["tool1", "tool2"], + "enabled": true + } + } + ] +} +``` + +### mcp_servers 对象字段 + +| 字段 | 类型 | 必需 | 说明 | +|------|------|------|------| +| `name` | string | **是** | MCP 服务器名称 | +| `type` | enum | **是** | 固定 `"url"` | +| `url` | string | **是** | MCP 服务器 URL | +| `authorization_token` | string \| null | 否 | 授权 token | +| `tool_configuration` | object \| null | 否 | 工具配置 | + +### tool_configuration 对象 + +| 字段 | 类型 | 说明 | +|------|------|------| +| `allowed_tools` | string[] \| null | 允许使用的工具列表，null 表示允许所有 | +| `enabled` | boolean \| null | 是否启用，null 表示启用 | + +### MCP 相关的响应内容块 + +- `mcp_tool_use` - MCP 工具调用 +- `mcp_tool_result` - MCP 工具结果 + +--- + +## 附录 A: 完整请求示例 + +### 基本文本请求 + +```bash +curl https://api.anthropic.com/v1/messages \ + --header "x-api-key: $ANTHROPIC_API_KEY" \ + --header "anthropic-version: 2023-06-01" \ + --header "content-type: application/json" \ + --data '{ + "model": "claude-sonnet-4-20250514", + "max_tokens": 1024, + "messages": [ + {"role": "user", "content": "Hello, world"} + ] + }' +``` + +### 带系统提示词和工具的请求 + +```bash +curl https://api.anthropic.com/v1/messages \ + --header "x-api-key: $ANTHROPIC_API_KEY" \ + --header "anthropic-version: 2023-06-01" \ + --header "content-type: application/json" \ + --data '{ + "model": "claude-sonnet-4-20250514", + "max_tokens": 1024, + "system": "You are a helpful weather assistant.", + "tools": [ + { + "name": "get_weather", + "description": "Get the current weather in a given location", + "input_schema": { + "type": "object", + "properties": { + "location": { + "type": "string", + "description": "The city and state, e.g. San Francisco, CA" + } + }, + "required": ["location"] + } + } + ], + "messages": [ + {"role": "user", "content": "What is the weather in SF?"} + ] + }' +``` + +### 带扩展思维和图像的请求 + +```bash +curl https://api.anthropic.com/v1/messages \ + --header "x-api-key: $ANTHROPIC_API_KEY" \ + --header "anthropic-version: 2023-06-01" \ + --header "content-type: application/json" \ + --data '{ + "model": "claude-sonnet-4-20250514", + "max_tokens": 16000, + "thinking": { + "type": "enabled", + "budget_tokens": 10000 + }, + "messages": [ + { + "role": "user", + "content": [ + { + "type": "image", + "source": { + "type": "url", + "url": "https://example.com/chart.png" + } + }, + { + "type": "text", + "text": "Analyze this chart in detail." + } + ] + } + ] + }' +``` + +### 带提示缓存的请求 + +```bash +curl https://api.anthropic.com/v1/messages \ + --header "x-api-key: $ANTHROPIC_API_KEY" \ + --header "anthropic-version: 2023-06-01" \ + --header "content-type: application/json" \ + --data '{ + "model": "claude-sonnet-4-20250514", + "max_tokens": 1024, + "system": [ + { + "type": "text", + "text": "你是一个专业的客服助手，以下是公司的产品手册全文...(大段静态文本)", + "cache_control": { + "type": "ephemeral", + "ttl": "1h" + } + } + ], + "messages": [ + {"role": "user", "content": "请介绍一下你们的产品"} + ] + }' +``` + +--- + +## 附录 B: 关键限制汇总 + +| 限制项 | 值 | +|--------|-----| +| 消息数组最大长度 | 100,000 条 | +| 单次请求最大图像数 | 100 张 | +| 单张图像最大尺寸 | 8000 x 8000 px | +| 单张图像最大文件大小 | 5 MB | +| model 字段长度 | 1-256 字符 | +| 工具名称长度 | 1-128 字符 | +| user_id 长度 | 最大 256 字符 | +| 扩展思维最小 budget | 1,024 tokens | +| 缓存断点最大数 | 4 个 | +| 流式必需条件 | max_tokens > 21,333 | +| Messages API 最大请求体 | 32 MB | +| Batch API 最大请求体 | 256 MB | +| Files API 最大文件大小 | 500 MB | diff --git a/docs/协议文档/Claude协议文档.md b/docs/协议文档/Claude协议文档.md new file mode 100644 index 0000000..b1405a3 --- /dev/null +++ b/docs/协议文档/Claude协议文档.md @@ -0,0 +1,1036 @@ +# Anthropic Claude Messages API 协议文档 + +> 基于 Anthropic 官方 API 文档整理，适用于 team-api 项目 Relay 层的 Claude 协议适配开发。 +> +> 官方文档地址：https://docs.anthropic.com/en/api/messages + +--- + +## 目录 + +- [1. API 概览](#1-api-概览) +- [2. 请求头 (Headers)](#2-请求头-headers) +- [3. 请求体参数 (Request Body)](#3-请求体参数-request-body) +- [4. 消息内容块类型 (Content Block Types)](#4-消息内容块类型-content-block-types) +- [5. 工具定义 (Tools)](#5-工具定义-tools) +- [6. tool_choice 配置](#6-tool_choice-配置) +- [7. 响应格式 (Response)](#7-响应格式-response) +- [8. stop_reason 值](#8-stop_reason-值) +- [9. 流式响应 (SSE Streaming)](#9-流式响应-sse-streaming) +- [10. 错误格式](#10-错误格式) +- [11. Prompt Caching (cache_control)](#11-prompt-caching-cache_control) + +--- + +## 1. API 概览 + +| 属性 | 说明 | +|------|------| +| **端点** | `POST https://api.anthropic.com/v1/messages` | +| **功能** | 发送结构化的输入消息列表（文本和/或图像），模型生成对话中的下一条消息 | +| **用途** | 支持单轮查询和无状态多轮对话 | +| **Content-Type** | `application/json` | +| **当前 API 版本** | `2023-06-01` | + +### 基本请求示例 + +```bash +curl https://api.anthropic.com/v1/messages \ + --header "x-api-key: $ANTHROPIC_API_KEY" \ + --header "anthropic-version: 2023-06-01" \ + --header "content-type: application/json" \ + --data '{ + "model": "claude-sonnet-4-20250514", + "max_tokens": 1024, + "messages": [ + {"role": "user", "content": "Hello, world"} + ] + }' +``` + +### 基本响应示例 + +```json +{ + "content": [ + { + "text": "Hi! My name is Claude.", + "type": "text" + } + ], + "id": "msg_013Zva2CMHLNnXjNJJKqJ2EF", + "model": "claude-sonnet-4-20250514", + "role": "assistant", + "stop_reason": "end_turn", + "stop_sequence": null, + "type": "message", + "usage": { + "input_tokens": 2095, + "output_tokens": 503 + } +} +``` + +--- + +## 2. 请求头 (Headers) + +| 请求头 | 是否必填 | 类型 | 说明 | +|--------|---------|------|------| +| `x-api-key` | **必填** | string | API 密钥，用于身份认证。通过 Anthropic Console 获取，每个密钥绑定一个 Workspace | +| `anthropic-version` | **必填** | string | API 版本号，当前为 `2023-06-01` | +| `anthropic-beta` | 可选 | string[] | Beta 功能标识。多个 Beta 用逗号分隔（如 `beta1,beta2`），或多次指定该请求头 | +| `content-type` | **必填** | string | 固定为 `application/json` | + +--- + +## 3. 请求体参数 (Request Body) + +### 完整参数一览 + +| 参数 | 是否必填 | 类型 | 默认值 | 说明 | +|------|---------|------|--------|------| +| `model` | **必填** | string | — | 模型 ID，如 `claude-sonnet-4-20250514`。长度限制：1-256 字符 | +| `messages` | **必填** | object[] | — | 输入消息列表。最大 100,000 条消息 | +| `max_tokens` | **必填** | integer | — | 最大生成 token 数，必须 >= 1 | +| `system` | 可选 | string / object[] | — | 系统提示词。可以是纯字符串或 content_block 数组 | +| `temperature` | 可选 | number | `1` | 随机性控制，范围 `0.0` ~ `1.0`。接近 0 适合分析/选择题，接近 1 适合创意/生成任务 | +| `top_p` | 可选 | number | — | 核采样 (Nucleus Sampling)，范围 `0` ~ `1`。与 `temperature` 二选一调整 | +| `top_k` | 可选 | integer | — | 仅从概率最高的 K 个 token 中采样，>= 0。用于去除低概率"长尾"响应 | +| `stop_sequences` | 可选 | string[] | — | 自定义停止序列列表。模型匹配到任一序列时停止生成 | +| `stream` | 可选 | boolean | `false` | 是否使用 SSE 流式响应 | +| `tools` | 可选 | object[] | — | 工具定义列表。模型可返回 `tool_use` 内容块 | +| `tool_choice` | 可选 | object | — | 工具调用策略：auto / any / tool / none | +| `thinking` | 可选 | object | — | 扩展思考配置 `{ type: "enabled", budget_tokens: >=1024 }` | +| `metadata` | 可选 | object | — | 请求元数据，含 `user_id` 字段 | +| `service_tier` | 可选 | string | — | 服务层级：`auto` 或 `standard_only` | +| `mcp_servers` | 可选 | object[] | — | MCP 服务器配置 | +| `container` | 可选 | string / null | — | 容器标识，用于跨请求复用容器 | + +### messages 参数详解 + +每条消息包含以下字段： + +| 字段 | 类型 | 说明 | +|------|------|------| +| `role` | enum | `user` 或 `assistant` | +| `content` | string / object[] | 消息内容。字符串等价于 `[{"type": "text", "text": "..."}]` | + +**关键规则：** + +- 模型按交替的 `user` / `assistant` 轮次训练。连续相同角色的消息会被合并。 +- 没有独立的 `system` 角色，系统提示词通过顶层 `system` 参数传入。 +- 若最后一条消息为 `assistant` 角色，响应内容会从该消息末尾继续，可用于约束模型输出。 + +**示例：** + +```json +// 单轮对话 +[{"role": "user", "content": "Hello, Claude"}] + +// 多轮对话 +[ + {"role": "user", "content": "Hello there."}, + {"role": "assistant", "content": "Hi, I'm Claude. How can I help you?"}, + {"role": "user", "content": "Can you explain LLMs in plain English?"} +] + +// 约束模型输出（prefilled response） +[ + {"role": "user", "content": "What's the Greek name for Sun? (A) Sol (B) Helios (C) Sun"}, + {"role": "assistant", "content": "The best answer is ("} +] +``` + +### thinking 参数详解 + +启用扩展思考后，响应将包含 `thinking` 内容块，展示模型的思考过程。 + +| 字段 | 类型 | 必填 | 说明 | +|------|------|------|------| +| `type` | string | 是 | 固定为 `"enabled"` | +| `budget_tokens` | integer | 是 | 思考 token 预算，必须 >= 1024 且小于 `max_tokens` | + +```json +{ + "thinking": { + "type": "enabled", + "budget_tokens": 16000 + }, + "max_tokens": 20000 +} +``` + +### metadata 参数详解 + +| 字段 | 类型 | 说明 | +|------|------|------| +| `user_id` | string / null | 用户外部标识，建议使用 UUID 或哈希值。最大长度 256 字符。不要包含姓名、邮箱等个人信息 | + +### mcp_servers 参数详解 + +| 字段 | 类型 | 必填 | 说明 | +|------|------|------|------| +| `name` | string | 是 | 服务器名称 | +| `type` | string | 是 | 固定为 `"url"` | +| `url` | string | 是 | MCP 服务器 URL | +| `authorization_token` | string / null | 否 | 授权令牌 | +| `tool_configuration` | object / null | 否 | 工具配置 | +| `tool_configuration.enabled` | boolean / null | 否 | 是否启用 | +| `tool_configuration.allowed_tools` | string[] / null | 否 | 允许的工具列表 | + +--- + +## 4. 消息内容块类型 (Content Block Types) + +### 4.1 输入内容块 (Input Content Blocks) + +#### text — 文本 + +```json +{ + "type": "text", + "text": "描述内容..." +} +``` + +#### image — 图像 + +图像支持两种来源方式： + +**Base64 编码：** + +```json +{ + "type": "image", + "source": { + "type": "base64", + "media_type": "image/jpeg", + "data": "/9j/4AAQSkZJRg..." + } +} +``` + +**URL 引用：** + +```json +{ + "type": "image", + "source": { + "type": "url", + "url": "https://example.com/image.jpg" + } +} +``` + +支持的 `media_type`：`image/jpeg`、`image/png`、`image/gif`、`image/webp`。 + +#### tool_use — 工具调用（assistant 消息中） + +```json +{ + "type": "tool_use", + "id": "toolu_01D7FLrfh4GYq7yT1ULFeyMV", + "name": "get_weather", + "input": { + "location": "San Francisco, CA" + } +} +``` + +#### tool_result — 工具结果（user 消息中） + +```json +{ + "type": "tool_result", + "tool_use_id": "toolu_01D7FLrfh4GYq7yT1ULFeyMV", + "content": "259.75 USD", + "is_error": false +} +``` + +`content` 可以是字符串或内容块数组。`is_error` 为可选布尔值，标记是否为错误结果。 + +#### thinking — 思考过程（assistant 消息中） + +```json +{ + "type": "thinking", + "thinking": "Let me solve this step by step...", + "signature": "EqQBCgIYAhIM1gbcDa9GJwZA2b3hGg..." +} +``` + +#### redacted_thinking — 脱敏思考 + +```json +{ + "type": "redacted_thinking", + "signature": "EqQBCgIYAhIM1gbcDa9GJwZA2b3hGg..." +} +``` + +### 4.2 输出内容块 (Output Content Blocks) + +| 类型 | 说明 | 结构 | +|------|------|------| +| `text` | 文本内容 | `{ type, text }` | +| `tool_use` | 工具调用 | `{ type, id, name, input }` | +| `thinking` | 扩展思考 | `{ type, thinking, signature }` | +| `redacted_thinking` | 脱敏思考 | `{ type, signature }` | +| `server_tool_use` | 服务器工具调用 | `{ type, id, name, input }` | +| `web_search_tool_result` | 网页搜索结果 | `{ type, tool_use_id, content, status_code }` | +| `code_execution_tool_result` | 代码执行结果 | `{ type, tool_use_id, content }` | +| `mcp_tool_use` | MCP 工具调用 | `{ type, id, name, input, server_name }` | +| `mcp_tool_result` | MCP 工具结果 | `{ type, tool_use_id, content, is_error }` | +| `container_upload` | 容器上传 | `{ type, ... }` | + +--- + +## 5. 工具定义 (Tools) + +### 自定义工具 (Custom Tool) + +每个自定义工具定义包含以下字段： + +| 字段 | 是否必填 | 类型 | 说明 | +|------|---------|------|------| +| `name` | **必填** | string | 工具名称，长度 1-128 字符。模型通过此名称调用工具 | +| `input_schema` | **必填** | object | JSON Schema 对象，定义工具的输入结构。必须包含 `type: "object"` | +| `type` | 可选 | string | 工具类型，固定为 `"custom"` | +| `description` | 可选 | string | 工具描述。建议尽可能详细，帮助模型更好地理解和使用工具 | +| `cache_control` | 可选 | object | 缓存控制断点 `{ type: "ephemeral", ttl: "5m" / "1h" }` | + +**input_schema 结构：** + +```json +{ + "type": "object", + "properties": { + "location": { + "type": "string", + "description": "The city and state, e.g. San Francisco, CA" + }, + "unit": { + "type": "string", + "description": "Unit for the output - one of (celsius, fahrenheit)" + } + }, + "required": ["location"] +} +``` + +**完整工具定义示例：** + +```json +[ + { + "name": "get_stock_price", + "description": "Get the current stock price for a given ticker symbol.", + "input_schema": { + "type": "object", + "properties": { + "ticker": { + "type": "string", + "description": "The stock ticker symbol, e.g. AAPL for Apple Inc." + } + }, + "required": ["ticker"] + } + } +] +``` + +### 内置工具类型 + +以下为 Anthropic 提供的服务器端内置工具： + +| 工具类型 | Beta 版本标识 | 说明 | +|----------|--------------|------| +| `bash` | `2024-10-22` / `2025-01-24` | Bash 命令执行 | +| `code_execution` | `2025-05-22` | 代码沙箱执行 | +| `computer_use` | `2024-01-22` / `2025-01-24` | 计算机使用（屏幕操作） | +| `text_editor` | `2024-10-22` / `2025-01-24` / `2025-04-29` | 文本编辑器操作 | +| `web_search` | `2025-03-05` | 网页搜索 | + +使用内置工具时需要在 `anthropic-beta` 请求头中指定对应的 Beta 版本标识。 + +### 工具调用完整流程示例 + +**1. 定义工具并发送请求：** + +```json +{ + "model": "claude-sonnet-4-20250514", + "max_tokens": 1024, + "tools": [ + { + "name": "get_weather", + "description": "Get the current weather in a given location", + "input_schema": { + "type": "object", + "properties": { + "location": { + "type": "string", + "description": "The city and state, e.g. San Francisco, CA" + } + }, + "required": ["location"] + } + } + ], + "messages": [ + {"role": "user", "content": "What is the weather in San Francisco?"} + ] +} +``` + +**2. 模型返回 tool_use：** + +```json +{ + "content": [ + { + "type": "text", + "text": "Let me check the weather for you." + }, + { + "type": "tool_use", + "id": "toolu_01D7FLrfh4GYq7yT1ULFeyMV", + "name": "get_weather", + "input": {"location": "San Francisco, CA"} + } + ], + "stop_reason": "tool_use" +} +``` + +**3. 将工具结果传回模型（下一条 user 消息）：** + +```json +{ + "messages": [ + {"role": "user", "content": "What is the weather in San Francisco?"}, + {"role": "assistant", "content": [ + {"type": "text", "text": "Let me check the weather for you."}, + {"type": "tool_use", "id": "toolu_01D7FLrfh4GYq7yT1ULFeyMV", "name": "get_weather", "input": {"location": "San Francisco, CA"}} + ]}, + {"role": "user", "content": [ + {"type": "tool_result", "tool_use_id": "toolu_01D7FLrfh4GYq7yT1ULFeyMV", "content": "72°F, Sunny"} + ]} + ] +} +``` + +--- + +## 6. tool_choice 配置 + +控制模型如何使用提供的工具。 + +### 选项一览 + +| 类型 | 结构 | 说明 | +|------|------|------| +| **auto** | `{ "type": "auto" }` | 自动决定是否使用工具（默认行为） | +| **any** | `{ "type": "any" }` | 必须调用任意一个工具，但不指定具体哪个 | +| **tool** | `{ "type": "tool", "name": "get_weather" }` | 必须调用指定名称的工具 | +| **none** | `{ "type": "none" }` | 禁止使用任何工具 | + +### disable_parallel_tool_use + +每种 `tool_choice` 类型都可以附加 `disable_parallel_tool_use` 布尔字段： + +```json +{ + "type": "auto", + "disable_parallel_tool_use": true +} +``` + +| 字段 | 类型 | 默认值 | 说明 | +|------|------|--------|------| +| `disable_parallel_tool_use` | boolean | `false` | 设为 `true` 时，模型最多只输出一个工具调用 | + +--- + +## 7. 响应格式 (Response) + +### 完整响应结构 + +```json +{ + "id": "msg_013Zva2CMHLNnXjNJJKqJ2EF", + "type": "message", + "role": "assistant", + "content": [ + { + "type": "text", + "text": "Hi! My name is Claude." + }, + { + "type": "tool_use", + "id": "toolu_01D7FLrfh4GYq7yT1ULFeyMV", + "name": "get_weather", + "input": {"location": "San Francisco, CA"} + } + ], + "model": "claude-sonnet-4-20250514", + "stop_reason": "end_turn", + "stop_sequence": null, + "usage": { + "input_tokens": 2095, + "output_tokens": 503, + "cache_creation_input_tokens": 0, + "cache_read_input_tokens": 0, + "cache_creation": { + "ephemeral_5m_input_tokens": 0, + "ephemeral_1h_input_tokens": 0 + }, + "server_tool_use": { + "web_search_requests": 0 + }, + "service_tier": "standard" + }, + "container": { + "id": "container_xxx", + "expires_at": "2024-01-01T00:00:00Z" + } +} +``` + +### 响应字段详解 + +| 字段 | 类型 | 必返回 | 说明 | +|------|------|--------|------| +| `id` | string | 是 | 唯一消息标识符，格式如 `msg_013Zva2CMHLNnXjNJJKqJ2EF` | +| `type` | string | 是 | 固定为 `"message"` | +| `role` | string | 是 | 固定为 `"assistant"` | +| `content` | object[] | 是 | 内容块数组 | +| `model` | string | 是 | 实际处理请求的模型 ID | +| `stop_reason` | string / null | 是 | 停止原因（见下一节） | +| `stop_sequence` | string / null | 是 | 匹配到的自定义停止序列，未匹配时为 null | +| `usage` | object | 是 | Token 用量统计 | +| `container` | object / null | 是 | 容器信息，使用容器工具时非 null | + +### usage 字段详解 + +| 字段 | 类型 | 说明 | +|------|------|------| +| `input_tokens` | integer | 输入 token 数（不含缓存部分） | +| `output_tokens` | integer | 输出 token 数 | +| `cache_creation_input_tokens` | integer / null | 写入缓存的 token 数 | +| `cache_read_input_tokens` | integer / null | 从缓存读取的 token 数 | +| `cache_creation` | object / null | 按 TTL 细分的缓存创建统计 | +| `cache_creation.ephemeral_5m_input_tokens` | integer | 5 分钟 TTL 缓存写入 token 数 | +| `cache_creation.ephemeral_1h_input_tokens` | integer | 1 小时 TTL 缓存写入 token 数 | +| `server_tool_use` | object / null | 服务器工具使用统计 | +| `server_tool_use.web_search_requests` | integer | 网页搜索请求次数 | +| `service_tier` | string / null | 使用的服务层级：`standard` / `priority` / `batch` | + +> **总输入 token 数** = `input_tokens` + `cache_creation_input_tokens` + `cache_read_input_tokens` + +### container 字段详解 + +| 字段 | 类型 | 说明 | +|------|------|------| +| `id` | string | 容器标识符 | +| `expires_at` | string | 容器过期时间（ISO 8601 格式） | + +--- + +## 8. stop_reason 值 + +| 值 | 说明 | +|-----|------| +| `end_turn` | 模型自然完成回复 | +| `max_tokens` | 达到 `max_tokens` 上限或模型最大 token 限制 | +| `stop_sequence` | 匹配到自定义停止序列（`stop_sequences` 参数中定义的） | +| `tool_use` | 模型调用了工具 | +| `pause_turn` | 暂停长轮次。可将响应原样回传，让模型继续 | +| `refusal` | 流式分类器介入处理潜在的政策违规 | + +> **注意：** 非流式模式下 `stop_reason` 一定不为 null。流式模式下，在 `message_start` 事件中为 null，其他事件中不为 null。 + +--- + +## 9. 流式响应 (SSE Streaming) + +设置 `"stream": true` 后，响应通过 Server-Sent Events (SSE) 逐步返回。 + +### 事件流顺序 + +``` +1. message_start → 消息开始，包含完整 Message 对象（content 为空数组） +2. [多个内容块，每个包含]： + a. content_block_start → 内容块开始 + b. content_block_delta → 内容增量（一个或多个） + c. content_block_stop → 内容块结束 +3. message_delta → 消息级更新（stop_reason、output_tokens） +4. message_stop → 消息结束 +``` + +期间可能穿插 `ping` 心跳事件和 `error` 错误事件。 + +### 事件类型详解 + +#### message_start — 消息开始 + +包含一个完整的 `message` 对象，`content` 为空数组，`stop_reason` 为 null。 + +```json +event: message_start +data: {"type": "message_start", "message": {"id": "msg_1nZdL29xx5MUA1yADyHTEsnR8uuvGzszyY", "type": "message", "role": "assistant", "content": [], "model": "claude-sonnet-4-20250514", "stop_reason": null, "stop_sequence": null, "usage": {"input_tokens": 25, "output_tokens": 1}}} +``` + +#### content_block_start — 内容块开始 + +```json +event: content_block_start +data: {"type": "content_block_start", "index": 0, "content_block": {"type": "text", "text": ""}} +``` + +`index` 对应最终 `content` 数组中的下标位置。 + +对于 tool_use 类型： + +```json +event: content_block_start +data: {"type": "content_block_start", "index": 1, "content_block": {"type": "tool_use", "id": "toolu_01T1x1fJ34qAmk2tNTrN7Up6", "name": "get_weather", "input": {}}} +``` + +对于 thinking 类型： + +```json +event: content_block_start +data: {"type": "content_block_start", "index": 0, "content_block": {"type": "thinking", "thinking": ""}} +``` + +#### content_block_delta — 内容增量 + +每个 delta 事件包含 `index` 和 `delta` 对象，delta 的 `type` 决定了增量数据的格式： + +| delta 类型 | 适用内容块 | 结构 | +|-----------|-----------|------| +| `text_delta` | text | `{ "type": "text_delta", "text": "..." }` | +| `input_json_delta` | tool_use | `{ "type": "input_json_delta", "partial_json": "..." }` | +| `thinking_delta` | thinking | `{ "type": "thinking_delta", "thinking": "..." }` | +| `signature_delta` | thinking | `{ "type": "signature_delta", "signature": "..." }` | + +**text_delta 示例：** + +```json +event: content_block_delta +data: {"type": "content_block_delta", "index": 0, "delta": {"type": "text_delta", "text": "Hello"}} +``` + +**input_json_delta 示例：** + +`partial_json` 是部分 JSON 字符串，需累积后在 `content_block_stop` 时解析为完整 JSON 对象。 + +```json +event: content_block_delta +data: {"type": "content_block_delta", "index": 1, "delta": {"type": "input_json_delta", "partial_json": "{\"location\": \"San Fra"}} +``` + +**thinking_delta 示例：** + +```json +event: content_block_delta +data: {"type": "content_block_delta", "index": 0, "delta": {"type": "thinking_delta", "thinking": "Let me solve this step by step:\n\n1. First break down 27 * 453"}} +``` + +**signature_delta 示例：** + +在 thinking 内容块的 `content_block_stop` 事件之前发送，用于验证思考块的完整性。 + +```json +event: content_block_delta +data: {"type": "content_block_delta", "index": 0, "delta": {"type": "signature_delta", "signature": "EqQBCgIYAhIM1gbcDa9GJwZA2b3hGgxBdjrkzLoky3dl1pkiMOYds..."}} +``` + +#### content_block_stop — 内容块结束 + +```json +event: content_block_stop +data: {"type": "content_block_stop", "index": 0} +``` + +#### message_delta — 消息级更新 + +包含 `stop_reason`、`stop_sequence` 和 `output_tokens`： + +```json +event: message_delta +data: {"type": "message_delta", "delta": {"stop_reason": "end_turn", "stop_sequence": null}, "usage": {"output_tokens": 15}} +``` + +#### message_stop — 消息结束 + +```json +event: message_stop +data: {"type": "message_stop"} +``` + +#### ping — 心跳 + +```json +event: ping +data: {"type": "ping"} +``` + +#### error — 错误 + +```json +event: error +data: {"type": "error", "error": {"type": "overloaded_error", "message": "Overloaded"}} +``` + +### 完整流式响应示例 + +#### 基本文本生成 + +**请求：** + +```bash +curl https://api.anthropic.com/v1/messages \ + --header "anthropic-version: 2023-06-01" \ + --header "content-type: application/json" \ + --header "x-api-key: $ANTHROPIC_API_KEY" \ + --data '{ + "model": "claude-sonnet-4-20250514", + "messages": [{"role": "user", "content": "Hello"}], + "max_tokens": 256, + "stream": true + }' +``` + +**响应：** + +``` +event: message_start +data: {"type": "message_start", "message": {"id": "msg_1nZdL29xx5MUA1yADyHTEsnR8uuvGzszyY", "type": "message", "role": "assistant", "content": [], "model": "claude-sonnet-4-20250514", "stop_reason": null, "stop_sequence": null, "usage": {"input_tokens": 25, "output_tokens": 1}}} + +event: content_block_start +data: {"type": "content_block_start", "index": 0, "content_block": {"type": "text", "text": ""}} + +event: ping +data: {"type": "ping"} + +event: content_block_delta +data: {"type": "content_block_delta", "index": 0, "delta": {"type": "text_delta", "text": "Hello"}} + +event: content_block_delta +data: {"type": "content_block_delta", "index": 0, "delta": {"type": "text_delta", "text": "!"}} + +event: content_block_stop +data: {"type": "content_block_stop", "index": 0} + +event: message_delta +data: {"type": "message_delta", "delta": {"stop_reason": "end_turn", "stop_sequence": null}, "usage": {"output_tokens": 15}} + +event: message_stop +data: {"type": "message_stop"} +``` + +#### 工具调用流式响应 + +**请求：** + +```bash +curl https://api.anthropic.com/v1/messages \ + -H "content-type: application/json" \ + -H "x-api-key: $ANTHROPIC_API_KEY" \ + -H "anthropic-version: 2023-06-01" \ + -d '{ + "model": "claude-sonnet-4-20250514", + "max_tokens": 1024, + "tools": [ + { + "name": "get_weather", + "description": "Get the current weather in a given location", + "input_schema": { + "type": "object", + "properties": { + "location": { + "type": "string", + "description": "The city and state, e.g. San Francisco, CA" + } + }, + "required": ["location"] + } + } + ], + "tool_choice": {"type": "any"}, + "messages": [ + {"role": "user", "content": "What is the weather like in San Francisco?"} + ], + "stream": true + }' +``` + +**响应：** + +``` +event: message_start +data: {"type":"message_start","message":{"id":"msg_014p7gG3wDgGV9EUtLvnow3U","type":"message","role":"assistant","model":"claude-sonnet-4-20250514","stop_sequence":null,"usage":{"input_tokens":472,"output_tokens":2},"content":[],"stop_reason":null}} + +event: content_block_start +data: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""}} + +event: ping +data: {"type": "ping"} + +event: content_block_delta +data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"Okay, let's check the weather for San Francisco, CA:"}} + +event: content_block_stop +data: {"type":"content_block_stop","index":0} + +event: content_block_start +data: {"type":"content_block_start","index":1,"content_block":{"type":"tool_use","id":"toolu_01T1x1fJ34qAmk2tNTrN7Up6","name":"get_weather","input":{}}} + +event: content_block_delta +data: {"type":"content_block_delta","index":1,"delta":{"type":"input_json_delta","partial_json":""}} + +event: content_block_delta +data: {"type":"content_block_delta","index":1,"delta":{"type":"input_json_delta","partial_json":"{\"location\":"}} + +event: content_block_delta +data: {"type":"content_block_delta","index":1,"delta":{"type":"input_json_delta","partial_json":" \"San Francisc"}} + +event: content_block_delta +data: {"type":"content_block_delta","index":1,"delta":{"type":"input_json_delta","partial_json":"o, CA\"}"}} + +event: content_block_stop +data: {"type":"content_block_stop","index":1} + +event: message_delta +data: {"type":"message_delta","delta":{"stop_reason":"tool_use","stop_sequence":null},"usage":{"output_tokens":89}} + +event: message_stop +data: {"type":"message_stop"} +``` + +#### 扩展思考流式响应 + +**请求：** + +```bash +curl https://api.anthropic.com/v1/messages \ + --header "x-api-key: $ANTHROPIC_API_KEY" \ + --header "anthropic-version: 2023-06-01" \ + --header "content-type: application/json" \ + --data '{ + "model": "claude-sonnet-4-20250514", + "max_tokens": 20000, + "stream": true, + "thinking": { + "type": "enabled", + "budget_tokens": 16000 + }, + "messages": [ + {"role": "user", "content": "What is 27 * 453?"} + ] + }' +``` + +**响应：** + +``` +event: message_start +data: {"type": "message_start", "message": {"id": "msg_01...", "type": "message", "role": "assistant", "content": [], "model": "claude-sonnet-4-20250514", "stop_reason": null, "stop_sequence": null}} + +event: content_block_start +data: {"type": "content_block_start", "index": 0, "content_block": {"type": "thinking", "thinking": ""}} + +event: content_block_delta +data: {"type": "content_block_delta", "index": 0, "delta": {"type": "thinking_delta", "thinking": "Let me solve this step by step:\n\n1. First break down 27 * 453"}} + +event: content_block_delta +data: {"type": "content_block_delta", "index": 0, "delta": {"type": "thinking_delta", "thinking": "\n2. 453 = 400 + 50 + 3"}} + +event: content_block_delta +data: {"type": "content_block_delta", "index": 0, "delta": {"type": "thinking_delta", "thinking": "\n3. 27 * 400 = 10,800"}} + +event: content_block_delta +data: {"type": "content_block_delta", "index": 0, "delta": {"type": "thinking_delta", "thinking": "\n4. 27 * 50 = 1,350"}} + +event: content_block_delta +data: {"type": "content_block_delta", "index": 0, "delta": {"type": "thinking_delta", "thinking": "\n5. 27 * 3 = 81"}} + +event: content_block_delta +data: {"type": "content_block_delta", "index": 0, "delta": {"type": "thinking_delta", "thinking": "\n6. 10,800 + 1,350 + 81 = 12,231"}} + +event: content_block_delta +data: {"type": "content_block_delta", "index": 0, "delta": {"type": "signature_delta", "signature": "EqQBCgIYAhIM1gbcDa9GJwZA2b3hGgxBdjrkzLoky3dl1pkiMOYds..."}} + +event: content_block_stop +data: {"type": "content_block_stop", "index": 0} + +event: content_block_start +data: {"type": "content_block_start", "index": 1, "content_block": {"type": "text", "text": ""}} + +event: content_block_delta +data: {"type": "content_block_delta", "index": 1, "delta": {"type": "text_delta", "text": "27 * 453 = 12,231"}} + +event: content_block_stop +data: {"type": "content_block_stop", "index": 1} + +event: message_delta +data: {"type": "message_delta", "delta": {"stop_reason": "end_turn", "stop_sequence": null}} + +event: message_stop +data: {"type": "message_stop"} +``` + +> **注意：** 根据 Anthropic 的版本策略，未来可能新增未知事件类型，客户端代码应优雅处理未知事件。 + +--- + +## 10. 错误格式 + +### 错误响应结构 + +```json +{ + "type": "error", + "error": { + "type": "authentication_error", + "message": "invalid x-api-key" + } +} +``` + +每个 API 响应都包含一个唯一的 `request-id` 响应头（如 `req_018EeWyXxfu5pfWkrYcMdjWG`），用于问题追踪。 + +### 错误类型与 HTTP 状态码映射 + +| HTTP 状态码 | 错误类型 | 说明 | +|------------|---------|------| +| 400 | `invalid_request_error` | 请求格式或内容有问题。其他未列出的 4XX 错误也使用此类型 | +| 401 | `authentication_error` | API 密钥有问题 | +| 403 | `permission_error` | API 密钥没有权限访问指定资源 | +| 404 | `not_found_error` | 请求的资源不存在 | +| 413 | `request_too_large` | 请求超过最大字节数限制（Messages API 最大 32MB） | +| 429 | `rate_limit_error` | 触发速率限制 | +| 500 | `api_error` | Anthropic 内部系统出现意外错误 | +| 529 | `overloaded_error` | Anthropic API 临时过载 | + +### 请求大小限制 + +| 端点类型 | 最大请求大小 | +|---------|------------| +| Messages API | 32 MB | +| Token Counting API | 32 MB | +| Batch API | 256 MB | +| Files API | 500 MB | + +### 流式响应中的错误 + +在 SSE 流式响应中，错误可能在 200 响应已返回后发生，此时错误不遵循标准 HTTP 错误机制，而是以 `error` 事件发送： + +``` +event: error +data: {"type": "error", "error": {"type": "overloaded_error", "message": "Overloaded"}} +``` + +--- + +## 11. Prompt Caching (cache_control) + +### 概述 + +Prompt Caching 允许从提示词的特定前缀处恢复处理，显著减少重复任务或具有固定内容的提示的处理时间和成本。 + +### 工作原理 + +1. 系统检查提示词前缀（到指定缓存断点）是否已被缓存。 +2. 如找到匹配，使用缓存版本，减少处理时间和成本。 +3. 如未找到，处理完整提示词并在响应开始时缓存前缀。 + +### cache_control 参数 + +| 字段 | 是否必填 | 类型 | 说明 | +|------|---------|------|------| +| `type` | **必填** | string | 固定为 `"ephemeral"` | +| `ttl` | 可选 | string | 缓存有效期：`"5m"`（5 分钟，默认）或 `"1h"`（1 小时） | + +```json +{ + "cache_control": { + "type": "ephemeral", + "ttl": "5m" + } +} +``` + +### 使用方式 + +`cache_control` 可以添加在以下位置： + +| 位置 | 示例 | +|------|------| +| `system` 内容块 | `[{"type": "text", "text": "...", "cache_control": {"type": "ephemeral"}}]` | +| `messages.content` 内容块 | `[{"type": "text", "text": "...", "cache_control": {"type": "ephemeral"}}]` | +| `tools` 定义 | `[{"name": "...", "cache_control": {"type": "ephemeral"}, ...}]` | + +**缓存前缀顺序：** `tools` → `system` → `messages`，形成层级结构。 + +### 缓存最小 token 数 + +| 模型 | 最小可缓存 token 数 | +|------|-------------------| +| Claude Opus 4 / Sonnet 4 / Sonnet 3.7 / Opus 3 | 1024 tokens | +| Claude Haiku 3.5 / Haiku 3 | 2048 tokens | + +### 缓存断点数量 + +- 最多可定义 **4 个**缓存断点。 +- 只需一个断点即可——系统会自动检查之前约 20 个内容块边界处的缓存命中。 +- 超过 20 个内容块的提示词，需要额外的 `cache_control` 参数确保内容可缓存。 + +### 计费 + +| 类型 | 说明 | +|------|------| +| 常规输入 token | 未缓存、未命中部分 | +| 缓存写入 (Cache Write) | 新写入缓存的 token，5 分钟 TTL 为基础价格的 125%，1 小时 TTL 为 200% | +| 缓存读取 (Cache Hit / Refresh) | 命中缓存，为基础价格的 10% | + +缓存断点本身不产生费用。 + +### 缓存命中监控 + +通过响应 `usage` 字段追踪缓存性能： + +| 字段 | 说明 | +|------|------| +| `cache_creation_input_tokens` | 写入缓存的 token 总数 | +| `cache_read_input_tokens` | 从缓存读取的 token 总数 | +| `cache_creation.ephemeral_5m_input_tokens` | 5 分钟 TTL 缓存写入 token 数 | +| `cache_creation.ephemeral_1h_input_tokens` | 1 小时 TTL 缓存写入 token 数 | +| `input_tokens` | 未使用缓存的输入 token 数 | + +### 缓存失效规则 + +缓存层级为 `tools` → `system` → `messages`，某层级变更会使该层级及之后所有层级的缓存失效： + +| 变更内容 | Tools 缓存 | System 缓存 | Messages 缓存 | +|---------|-----------|------------|--------------| +| 工具定义变更 | 失效 | 失效 | 失效 | +| 网页搜索开关 | 有效 | 失效 | 失效 | +| 引用开关 | 有效 | 失效 | 失效 | +| tool_choice 变更 | 有效 | 有效 | 失效 | +| 图像变更 | 有效 | 有效 | 失效 | +| thinking 参数变更 | 有效 | 有效 | 失效 | + +### 混合 TTL 使用 + +可以在同一请求中使用 5 分钟和 1 小时 TTL，但约束条件：**较长 TTL 的缓存断点必须出现在较短 TTL 之前**（即 1 小时断点必须在 5 分钟断点之前）。 + +### 不可缓存的内容 + +- thinking 内容块不能直接标记 `cache_control`（但会随其他内容一起被缓存） +- 子内容块（如引用 citations）不能直接缓存，需缓存顶层块 +- 空文本块不能被缓存 diff --git a/docs/协议文档/FLUX及其他图像生成API文档.md b/docs/协议文档/FLUX及其他图像生成API文档.md new file mode 100644 index 0000000..f8daeab --- /dev/null +++ b/docs/协议文档/FLUX及其他图像生成API文档.md @@ -0,0 +1,992 @@ +# FLUX 及其他图像生成 API 文档 + +## 概述 + +本文档涵盖 FLUX 模型系列及通过第三方平台（Together AI、Replicate、Fireworks AI、Black Forest Labs）提供的图像生成 API。 + +--- + +## 一、Black Forest Labs（BFL）— FLUX 模型官方 API + +### 概述 + +Black Forest Labs（BFL）是 FLUX 模型的创建者，由 Stable Diffusion 原始团队成员创立。FLUX 系列是目前开源图像生成领域最先进的模型之一。 + +- **官网**：https://blackforestlabs.ai +- **API 文档**：https://docs.bfl.ml +- **API Base URL**：`https://api.bfl.ml` +- **认证方式**：Bearer Token（API Key） +- **协议格式**：REST + JSON + +### 可用模型 + +| 模型 ID | 说明 | 特点 | +|---------|------|------| +| `flux-pro-1.1` | FLUX.1.1 Pro | 高质量，6 步生成 | +| `flux-pro-1.0` | FLUX.1 Pro | 旗舰级质量 | +| `flux-dev-1.0` | FLUX.1 Dev | 开发者版本，可自部署 | +| `flux-schnell-1.0` | FLUX.1 Schnell | 快速版，4 步生成，Apache 2.0 开源 | +| `flux-kontext-1.0` | FLUX.1 Kontext | 图像编辑（inpainting/outpainting） | +| `flux-canny-1.0` | FLUX.1 Canny | Canny 边缘引导生成 | +| `flux-depth-1.0` | FLUX.1 Depth | 深度图引导生成 | +| `flux-fill-1.0` | FLUX.1 Fill | 局部重绘/填充 | + +### 认证 + +``` +X-Key: sk-xxxxxxxxxxxxxxxxxxxxxxxx +``` + +在请求头中使用 `X-Key` 字段传递 API Key。 + +### API 端点 + +#### 1. 文生图（Text to Image） + +**请求** + +``` +POST https://api.bfl.ml/v1/flux-pro-1.1 +``` + +**Headers** + +| Header | 值 | 必填 | +|--------|---|------| +| `Content-Type` | `application/json` | 是 | +| `X-Key` | `sk-xxx` | 是 | + +**请求体** + +```json +{ + "prompt": "A beautiful sunset over a calm ocean with vibrant colors", + "width": 1024, + "height": 1024, + "steps": 25, + "guidance": 3.5, + "seed": 42, + "safety_tolerance": 2, + "output_format": "jpeg", + "output_quality": 80 +} +``` + +| 参数 | 类型 | 必填 | 默认值 | 说明 | +|------|------|------|--------|------| +| `prompt` | string | 是 | — | 正向提示词 | +| `width` | integer | 否 | `1024` | 图像宽度。推荐值：512 ~ 1536，必须是 16 的倍数 | +| `height` | integer | 否 | `1024` | 图像高度。推荐值：512 ~ 1536，必须是 16 的倍数 | +| `steps` | integer | 否 | `25` | 推理步数。范围：1 ~ 50（Schnell 推荐 4） | +| `guidance` | float | 否 | `3.0` | CFG 引导尺度。范围：1.0 ~ 20.0 | +| `seed` | integer | 否 | 随机 | 随机种子。范围：0 ~ 4294967295 | +| `safety_tolerance` | integer | 否 | `2` | 安全过滤级别。范围：1 ~ 6，1=最严格，6=最宽松 | +| `output_format` | string | 否 | `"jpeg"` | 输出格式。可选值：`"jpeg"`, `"png"` | +| `output_quality` | integer | 否 | `80` | JPEG 输出质量。范围：1 ~ 100（仅 JPEG 有效） | + +#### 2. 图生图（Image to Image） + +**请求** + +``` +POST https://api.bfl.ml/v1/flux-pro-1.1-ultra +``` + +**请求体** + +```json +{ + "prompt": "Transform into a watercolor painting", + "image": "base64_encoded_image_data", + "image_prompt": "watercolor style artwork", + "strength": 0.7, + "width": 1024, + "height": 1024, + "steps": 25, + "guidance": 3.5, + "seed": 42, + "output_format": "png" +} +``` + +| 参数 | 类型 | 必填 | 默认值 | 说明 | +|------|------|------|--------|------| +| `prompt` | string | 是 | — | 提示词 | +| `image` | string | 是 | — | Base64 编码的输入图像 | +| `image_prompt` | string | 否 | `""` | 图像提示词（额外引导） | +| `strength` | float | 否 | `0.5` | 图像变化强度。范围：0.0 ~ 1.0 | + +#### 3. 局部重绘（Inpainting / Fill） + +``` +POST https://api.bfl.ml/v1/flux-fill-1.0 +``` + +```json +{ + "prompt": "A beautiful garden with flowers", + "image": "base64_encoded_image", + "mask": "base64_encoded_mask", + "width": 1024, + "height": 1024, + "steps": 25, + "guidance": 3.5, + "seed": 42 +} +``` + +| 参数 | 类型 | 必填 | 说明 | +|------|------|------|------| +| `prompt` | string | 是 | 重绘区域描述 | +| `image` | string | 是 | Base64 编码的原始图像 | +| `mask` | string | 是 | Base64 编码的遮罩图像（白色区域为重绘区域） | + +#### 4. Canny 边缘引导生成 + +``` +POST https://api.bfl.ml/v1/flux-canny-1.0 +``` + +```json +{ + "prompt": "A futuristic city skyline", + "control_image": "base64_encoded_control_image", + "guidance": 3.5, + "steps": 25, + "seed": 42 +} +``` + +#### 5. 深度图引导生成 + +``` +POST https://api.bfl.ml/v1/flux-depth-1.0 +``` + +```json +{ + "prompt": "A cozy living room", + "control_image": "base64_encoded_depth_map", + "guidance": 3.5, + "steps": 25, + "seed": 42 +} +``` + +### 异步响应 + +BFL API 默认返回异步任务： + +```json +{ + "id": "abc123-def456", + "status": "Pending", + "result": null +} +``` + +**轮询结果** + +``` +GET https://api.bfl.ml/v1/get_result?id=abc123-def456 +``` + +**Headers** + +``` +X-Key: sk-xxx +``` + +**完成响应** + +```json +{ + "id": "abc123-def456", + "status": "Ready", + "result": { + "sample": "https://delivery.bfl.ml/xxx/image.jpeg", + "prompt": "A beautiful sunset..." + } +} +``` + +| 字段 | 类型 | 说明 | +|------|------|------| +| `id` | string | 任务 ID | +| `status` | string | 任务状态：`"Pending"`、`"Ready"`、`"Failed"` | +| `result.sample` | string | 生成图像的 URL | +| `result.prompt` | string | 使用的提示词 | + +### 错误响应 + +```json +{ + "status_code": 400, + "errors": [ + { + "message": "Invalid request: prompt is required" + } + ] +} +``` + +### 定价（参考） + +| 模型 | 每张价格 | 说明 | +|------|---------|------| +| FLUX.1.1 Pro | ~$0.04 | 高质量 | +| FLUX.1 Pro | ~$0.05 | 旗舰级 | +| FLUX.1 Schnell | ~$0.003 | 快速/低成本 | + +--- + +## 二、Together AI — FLUX/SDXL 图像生成 + +### 概述 + +Together AI 提供多种开源图像模型的托管推理服务，包括 FLUX 系列、Stable Diffusion XL 等。API 兼容 OpenAI Images 格式。 + +- **官网**：https://together.ai +- **API 文档**：https://docs.together.ai +- **API Base URL**：`https://api.together.xyz` +- **认证方式**：Bearer Token +- **协议格式**：兼容 OpenAI Images API + +### 认证 + +``` +Authorization: Bearer {TOGETHER_API_KEY} +``` + +### 可用图像模型 + +| 模型 ID | 说明 | 每步价格 | 质量 | +|---------|------|---------|------| +| `black-forest-labs/FLUX.1-schnell` | FLUX.1 Schnell（免费） | $0.00 | 快速 | +| `black-forest-labs/FLUX.1-schnell-Free` | FLUX.1 Schnell 免费版 | $0.00 | 快速（限速） | +| `black-forest-labs/FLUX.1.1-pro` | FLUX.1.1 Pro | 按量计费 | 高质量 | +| `black-forest-labs/FLUX.1-pro` | FLUX.1 Pro | 按量计费 | 旗舰级 | +| `stabilityai/stable-diffusion-xl-base-1.0` | SDXL 1.0 Base | 按量计费 | 中等质量 | + +### API 端点 + +#### 图像生成 + +``` +POST https://api.together.xyz/v1/images/generations +``` + +**请求体** + +```json +{ + "model": "black-forest-labs/FLUX.1-schnell", + "prompt": "A cat sitting on a windowsill looking at the moon", + "n": 1, + "size": "1024x1024", + "steps": 4, + "seed": 42, + "response_format": "b64_json", + "guidance_scale": 7.5 +} +``` + +| 参数 | 类型 | 必填 | 默认值 | 说明 | +|------|------|------|--------|------| +| `model` | string | 是 | — | 模型 ID（见上方模型列表） | +| `prompt` | string | 是 | — | 提示词 | +| `n` | integer | 否 | `1` | 生成图像数量（最大 8） | +| `size` | string | 否 | `"1024x1024"` | 图像尺寸。支持：`"256x256"`, `"512x512"`, `"1024x1024"`, `"1024x768"`, `"768x1024"`, `"1792x1024"`, `"1024x1792"` | +| `steps` | integer | 否 | `20` | 推理步数。FLUX Schnell 推荐 4 | +| `seed` | integer | 否 | 随机 | 随机种子 | +| `response_format` | string | 否 | `"url"` | 响应格式。可选值：`"url"`, `"b64_json"` | +| `guidance_scale` | float | 否 | `7.5` | CFG 引导尺度 | +| `negative_prompt` | string | 否 | `""` | 反向提示词（部分模型支持） | +| `width` | integer | 否 | 1024 | 图像宽度（可替代 size） | +| `height` | integer | 否 | 1024 | 图像高度（可替代 size） | + +**响应** + +```json +{ + "id": "gen-abc123", + "object": "list", + "created": 1677652288, + "data": [ + { + "url": "https://api.together.xyz/images/abc123.png", + "b64_json": null + } + ] +} +``` + +或（`response_format: "b64_json"`）： + +```json +{ + "id": "gen-abc123", + "object": "list", + "created": 1677652288, + "data": [ + { + "url": null, + "b64_json": "iVBORw0KGgoAAAANSUhEUgAA..." + } + ] +} +``` + +**示例请求（cURL）** + +```bash +curl -X POST https://api.together.xyz/v1/images/generations \ + -H "Authorization: Bearer $TOGETHER_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{ + "model": "black-forest-labs/FLUX.1-schnell", + "prompt": "A serene Japanese garden with cherry blossoms", + "n": 1, + "size": "1024x1024", + "steps": 4, + "response_format": "url" + }' +``` + +**示例请求（Python）** + +```python +from openai import OpenAI + +client = OpenAI( + api_key="your_together_api_key", + base_url="https://api.together.xyz/v1" +) + +response = client.images.generate( + model="black-forest-labs/FLUX.1-schnell", + prompt="A serene Japanese garden with cherry blossoms", + n=1, + size="1024x1024", + response_format="url" +) + +print(response.data[0].url) +``` + +### 错误响应 + +与 OpenAI 格式一致： + +```json +{ + "error": { + "type": "invalid_request_error", + "message": "Invalid model ID", + "param": "model", + "code": null + } +} +``` + +### 定价 + +| 模型 | 计费方式 | +|------|---------| +| FLUX.1 Schnell Free | 免费（有限速率） | +| FLUX.1 Schnell | $0.002/张 | +| FLUX.1.1 Pro | ~$0.03/张 | +| FLUX.1 Pro | ~$0.04/张 | +| SDXL 1.0 | ~$0.002/张 | + +--- + +## 三、Replicate — 通用模型推理平台 + +### 概述 + +Replicate 提供云端模型推理服务，支持数千种模型（图像、视频、音频、文本）。图像生成是其核心功能之一，支持 FLUX、SDXL、Kandinsky、Playground 等多种模型。 + +- **官网**：https://replicate.com +- **API 文档**：https://replicate.com/docs +- **API Base URL**：`https://api.replicate.com` +- **认证方式**：Bearer Token +- **协议格式**：REST + JSON + +### 认证 + +``` +Authorization: Bearer r8_xxxxxxxxxxxxxxxxxxxxxxxx +``` + +在 https://replicate.com/account/api-tokens 获取 API Token。 + +### API 端点 + +#### 1. 创建预测（同步） + +``` +POST https://api.replicate.com/v1/predictions +``` + +**Headers** + +| Header | 值 | 必填 | +|--------|---|------| +| `Authorization` | `Bearer r8_xxx` | 是 | +| `Content-Type` | `application/json` | 是 | +| `Prefer` | `wait` | 否 | 设置为 `wait` 表示同步等待结果（最长 60s） | + +**请求体** + +```json +{ + "version": "ac732df83cea7fff18b947a8b9a204a33cd3f2f1e5e6b1e5f5f5f5f5f5f5f5f5", + "input": { + "prompt": "A beautiful sunset over mountains", + "width": 1024, + "height": 1024, + "num_outputs": 1, + "guidance_scale": 7.5, + "num_inference_steps": 28, + "seed": 42 + } +} +``` + +| 参数 | 类型 | 必填 | 说明 | +|------|------|------|------| +| `version` | string | 否 | 模型版本 ID（与 `model` 二选一） | +| `model` | string | 否 | 模型标识（如 `"black-forest-labs/flux-schnell"`） | +| `input` | object | 是 | 模型输入参数（因模型而异） | +| `webhook` | string | 否 | 完成/失败时的回调 URL | +| `webhook_events_filter` | array | 否 | 触发回调的事件：`["start", "output", "logs", "completed"]` | +| `Cancel-After` | string | 否 | 超时时间（ISO 8601 duration，如 `"PT60S"`） | + +#### 2. 创建预测（使用模型最新版本） + +``` +POST https://api.replicate.com/v1/models/{owner}/{model}/predictions +``` + +**请求体** + +```json +{ + "input": { + "prompt": "A cat wearing a hat", + "width": 1024, + "height": 768, + "num_inference_steps": 28, + "guidance_scale": 3.5, + "seed": 42 + } +} +``` + +#### 3. 获取预测结果 + +``` +GET https://api.replicate.com/v1/predictions/{prediction_id} +``` + +#### 4. 取消预测 + +``` +POST https://api.replicate.com/v1/predictions/{prediction_id}/cancel +``` + +#### 5. 列出预测 + +``` +GET https://api.replicate.com/v1/predictions +``` + +### 预测响应 + +```json +{ + "id": "abc123-def456-ghi789", + "model": "black-forest-labs/flux-schnell", + "version": "ac732df83cea...", + "input": { + "prompt": "A beautiful sunset" + }, + "output": [ + "https://replicate.delivery/xxx/image1.png" + ], + "status": "succeeded", + "created_at": "2024-01-15T10:30:00.000Z", + "started_at": "2024-01-15T10:30:01.000Z", + "completed_at": "2024-01-15T10:30:15.000Z", + "logs": "Using seed: 42\nGenerating...", + "error": null, + "metrics": { + "predict_time": 14.5 + } +} +``` + +| 字段 | 类型 | 说明 | +|------|------|------| +| `id` | string | 预测唯一 ID | +| `model` | string | 模型标识 | +| `version` | string | 模型版本 ID | +| `input` | object | 输入参数 | +| `output` | array | 输出结果（通常是图像 URL 数组） | +| `status` | string | 状态：`"starting"`, `"processing"`, `"succeeded"`, `"failed"`, `"canceled"` | +| `created_at` | string | 创建时间（ISO 8601） | +| `started_at` | string | 开始处理时间 | +| `completed_at` | string | 完成时间 | +| `logs` | string | 执行日志 | +| `error` | string | 错误信息（失败时） | +| `metrics` | object | 性能指标 | + +### 状态流转 + +``` +starting → processing → succeeded + → failed + → canceled（手动取消） +``` + +### Webhook 回调 + +当预测完成时，Replicate 向指定 URL 发送 POST 请求： + +```json +{ + "id": "abc123", + "event": "completed", + "status": "succeeded", + "output": ["https://replicate.delivery/xxx/image.png"], + "error": null +} +``` + +### 常用图像模型及输入参数 + +#### FLUX.1 Schnell + +```json +{ + "model": "black-forest-labs/flux-schnell", + "input": { + "prompt": "description of the image", + "num_outputs": 1, + "aspect_ratio": "1:1", + "output_format": "webp", + "output_quality": 80, + "seed": 42 + } +} +``` + +| 参数 | 类型 | 默认值 | 说明 | +|------|------|--------|------| +| `prompt` | string | — | 提示词 | +| `num_outputs` | integer | 1 | 输出图像数量（1 ~ 4） | +| `aspect_ratio` | string | `"1:1"` | 宽高比：`"1:1"`, `"16:9"`, `"9:16"`, `"4:3"`, `"3:4"`, `"3:2"`, `"2:3"` | +| `output_format` | string | `"webp"` | 输出格式：`"webp"`, `"jpg"`, `"png"` | +| `output_quality` | integer | 80 | 输出质量（1 ~ 100） | +| `seed` | integer | 随机 | 随机种子 | + +#### FLUX.1 Pro + +```json +{ + "model": "black-forest-labs/flux-pro", + "input": { + "prompt": "description", + "width": 1024, + "height": 1024, + "steps": 28, + "guidance": 3.5, + "interval": 0.5, + "seed": 42 + } +} +``` + +| 参数 | 类型 | 默认值 | 说明 | +|------|------|--------|------| +| `prompt` | string | — | 提示词 | +| `width` | integer | 1024 | 图像宽度 | +| `height` | integer | 1024 | 图像高度 | +| `steps` | integer | 28 | 推理步数 | +| `guidance` | float | 3.5 | CFG 引导尺度 | +| `interval` | float | 0.5 | 引导间隔 | +| `seed` | integer | 随机 | 随机种子 | + +#### SDXL + +```json +{ + "model": "stability-ai/sdxl", + "input": { + "prompt": "description", + "negative_prompt": "blurry, bad quality", + "width": 1024, + "height": 1024, + "num_inference_steps": 30, + "guidance_scale": 7.5, + "scheduler": "K_EULER", + "seed": 42, + "num_outputs": 1 + } +} +``` + +### 错误响应 + +```json +{ + "status": 422, + "detail": "Input validation failed: prompt is required" +} +``` + +### 定价 + +Replicate 按推理时间计费，不同模型价格不同： + +| 模型 | 每秒价格 | 典型生成时间 | 每张价格 | +|------|---------|-------------|---------| +| FLUX.1 Schnell | ~$0.0003/s | ~2s | ~$0.003 | +| FLUX.1 Pro | ~$0.0003/s | ~10s | ~$0.03 | +| SDXL | ~$0.0003/s | ~5s | ~$0.01 | + +### Python SDK + +```python +import replicate + +# 方式1：使用模型标识 +output = replicate.run( + "black-forest-labs/flux-schnell", + input={ + "prompt": "A cat in a garden", + "aspect_ratio": "16:9", + "output_format": "png" + } +) + +# output 是图像 URL 列表 +print(output[0]) # https://replicate.delivery/xxx/image.png + +# 方式2：异步预测 +prediction = replicate.predictions.create( + model="black-forest-labs/flux-schnell", + input={"prompt": "A cat in a garden"}, + webhook="https://your-app.com/webhook", + webhook_events_filter=["completed"] +) + +# 轮询 +prediction.reload() +while prediction.status not in ["succeeded", "failed", "canceled"]: + time.sleep(1) + prediction.reload() +``` + +### Go HTTP 调用示例 + +```go +package main + +import ( + "bytes" + "encoding/json" + "fmt" + "io" + "net/http" + "time" +) + +type PredictionRequest struct { + Model string `json:"model"` + Input map[string]interface{} `json:"input"` +} + +type Prediction struct { + ID string `json:"id"` + Status string `json:"status"` + Output []string `json:"output"` + Error string `json:"error"` +} + +func createPrediction(apiKey string, req PredictionRequest) (*Prediction, error) { + body, _ := json.Marshal(req) + httpReq, _ := http.NewRequest("POST", + "https://api.replicate.com/v1/predictions", + bytes.NewReader(body)) + httpReq.Header.Set("Authorization", "Bearer "+apiKey) + httpReq.Header.Set("Content-Type", "application/json") + httpReq.Header.Set("Prefer", "wait") + + resp, err := http.DefaultClient.Do(httpReq) + if err != nil { + return nil, err + } + defer resp.Body.Close() + + var pred Prediction + json.NewDecoder(resp.Body).Decode(&pred) + return &pred, nil +} +``` + +--- + +## 四、Fireworks AI — 图像生成 + +### 概述 + +Fireworks AI 提供快速推理服务，支持 FLUX、Stable Diffusion 等多种图像生成模型，以低延迟和高吞吐著称。 + +- **官网**：https://fireworks.ai +- **API 文档**：https://docs.fireworks.ai +- **API Base URL**：`https://api.fireworks.ai` +- **认证方式**：Bearer Token +- **协议格式**：兼容 OpenAI Images API + +### 认证 + +``` +Authorization: Bearer {FIREWORKS_API_KEY} +``` + +### 可用模型 + +| 模型 ID | 说明 | +|---------|------| +| `stable-diffusion-xl-1024-v1-0` | SDXL 1.0 | +| `FLUX.1-schnell` | FLUX Schnell 快速版 | +| `FLUX.1-dev` | FLUX Dev 开发者版 | + +### API 端点 + +#### 图像生成 + +``` +POST https://api.fireworks.ai/v1/images/generations +``` + +**请求体** + +```json +{ + "model": "FLUX.1-schnell", + "prompt": "A dragon flying over a medieval castle", + "n": 1, + "size": "1024x1024", + "cfg_scale": 7.5, + "steps": 4, + "seed": 42, + "sampler": "DDIM" +} +``` + +| 参数 | 类型 | 必填 | 默认值 | 说明 | +|------|------|------|--------|------| +| `model` | string | 是 | — | 模型 ID | +| `prompt` | string | 是 | — | 提示词 | +| `n` | integer | 否 | `1` | 生成数量 | +| `size` | string | 否 | `"1024x1024"` | 图像尺寸 | +| `cfg_scale` | float | 否 | `7.0` | CFG 引导尺度 | +| `steps` | integer | 否 | `20` | 推理步数 | +| `seed` | integer | 否 | 随机 | 随机种子 | +| `sampler` | string | 否 | 自动 | 采样器类型 | +| `negative_prompt` | string | 否 | `""` | 反向提示词 | + +**响应** + +兼容 OpenAI Images API 格式： + +```json +{ + "created": 1677652288, + "data": [ + { + "url": "https://fireworks-api-images.s3.amazonaws.com/xxx/image.png" + } + ] +} +``` + +### 示例请求 + +```bash +curl -X POST https://api.fireworks.ai/v1/images/generations \ + -H "Authorization: Bearer $FIREWORKS_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{ + "model": "FLUX.1-schnell", + "prompt": "A majestic phoenix rising from flames", + "n": 1, + "size": "1024x1024", + "steps": 4 + }' +``` + +### Python 示例（兼容 OpenAI SDK） + +```python +from openai import OpenAI + +client = OpenAI( + api_key="your_fireworks_api_key", + base_url="https://api.fireworks.ai/v1" +) + +response = client.images.generate( + model="FLUX.1-schnell", + prompt="A majestic phoenix rising from flames", + n=1, + size="1024x1024" +) + +print(response.data[0].url) +``` + +--- + +## 五、各平台对比总结 + +### 功能对比 + +| 特性 | BFL 官方 | Together AI | Replicate | Fireworks AI | +|------|---------|-------------|-----------|-------------| +| 认证方式 | `X-Key` Header | Bearer Token | Bearer Token | Bearer Token | +| 请求格式 | JSON | JSON（OpenAI 兼容） | JSON | JSON（OpenAI 兼容） | +| 响应格式 | 异步 + URL | OpenAI 格式 | 异步 + URL | OpenAI 格式 | +| OpenAI SDK 兼容 | 否 | 是 | 否 | 是 | +| 文生图 | ✓ | ✓ | ✓ | ✓ | +| 图生图 | ✓ | 部分支持 | ✓ | 部分支持 | +| 局部重绘 | ✓（Fill 模型） | 部分支持 | ✓ | 部分支持 | +| Canny/Depth 引导 | ✓ | — | ✓ | — | +| Webhook | — | — | ✓ | — | +| 免费额度 | 无 | Schnell 免费 | 无 | 无 | +| FLUX 模型 | 全系列 | Schnell + Pro | 全系列 | Schnell + Dev | +| SDXL | — | ✓ | ✓ | ✓ | + +### 价格对比（每张 1024x1024 图像，参考价格） + +| 模型 | BFL 官方 | Together AI | Replicate | Fireworks AI | +|------|---------|-------------|-----------|-------------| +| FLUX.1 Schnell | ~$0.003 | 免费 / $0.002 | ~$0.003 | ~$0.002 | +| FLUX.1 Pro | ~$0.05 | ~$0.04 | ~$0.03 | — | +| FLUX.1.1 Pro | ~$0.04 | ~$0.03 | ~$0.035 | — | +| SDXL | — | ~$0.002 | ~$0.01 | ~$0.002 | + +### 延迟对比 + +| 模型 | BFL 官方 | Together AI | Replicate | Fireworks AI | +|------|---------|-------------|-----------|-------------| +| FLUX Schnell | ~2s | ~2s | ~3s | ~1s | +| FLUX Pro | ~10s | ~10s | ~12s | — | +| SDXL | — | ~5s | ~6s | ~3s | + +--- + +## 六、team-api 适配器实现参考 + +### 端点映射 + +对于 OpenAI 兼容的供应商（Together AI、Fireworks AI），可以直接透传 `/v1/images/generations` 请求，只需替换 base URL 和 API Key。 + +| team-api 入口 | Together AI | Fireworks AI | Replicate | +|--------------|-------------|-------------|-----------| +| `POST /v1/images/generations` | `POST /v1/images/generations` | `POST /v1/images/generations` | `POST /v1/predictions` | +| `POST /v1/images/edits` | 不支持 | 不支持 | `POST /v1/predictions` | +| `POST /v1/images/variations` | 不支持 | 不支持 | `POST /v1/predictions` | + +### 请求转换（OpenAI 兼容供应商） + +Together AI 和 Fireworks AI 的请求格式与 OpenAI 基本一致，适配器只需： + +1. 替换 `base_url`（`api.openai.com` → `api.together.xyz` / `api.fireworks.ai`） +2. 替换 `model` 字段（`dall-e-3` → `black-forest-labs/FLUX.1-schnell`） +3. 透传其余参数 + +### 请求转换（Replicate） + +Replicate 的请求格式与 OpenAI 差异较大，需要专用适配器： + +```go +// OpenAI Images 请求 → Replicate 预测请求 +func ConvertToReplicateRequest(openaiReq *OpenAIImageRequest) *ReplicateRequest { + input := map[string]interface{}{ + "prompt": openaiReq.Prompt, + "aspect_ratio": sizeToAspectRatio(openaiReq.Size), + "output_format": "png", + "num_outputs": openaiReq.N, + } + if openaiReq.Seed > 0 { + input["seed"] = openaiReq.Seed + } + + return &ReplicateRequest{ + Model: mapModelToReplicate(openaiReq.Model), + Input: input, + } +} + +// Replicate 响应 → OpenAI Images 响应 +func ConvertFromReplicateResponse(resp *ReplicatePrediction) *OpenAIImageResponse { + data := make([]ImageData, len(resp.Output)) + for i, url := range resp.Output { + data[i] = ImageData{URL: url} + } + return &OpenAIImageResponse{ + Created: time.Now().Unix(), + Data: data, + } +} + +// 尺寸映射 +func sizeToAspectRatio(size string) string { + switch size { + case "1024x1024": return "1:1" + case "1792x1024": return "16:9" + case "1024x1792": return "9:16" + case "1536x1024": return "3:2" + case "1024x1536": return "2:3" + default: return "1:1" + } +} + +// 模型映射 +func mapModelToReplicate(model string) string { + switch model { + case "flux-schnell": return "black-forest-labs/flux-schnell" + case "flux-pro": return "black-forest-labs/flux-pro" + case "sdxl": return "stability-ai/sdxl" + default: return model + } +} +``` + +### Replicate 异步轮询 + +```go +func PollReplicatePrediction(apiKey, predictionID string, timeout time.Duration) (*ReplicatePrediction, error) { + deadline := time.Now().Add(timeout) + for time.Now().Before(deadline) { + pred, err := GetPrediction(apiKey, predictionID) + if err != nil { + return nil, err + } + switch pred.Status { + case "succeeded": + return pred, nil + case "failed": + return nil, fmt.Errorf("prediction failed: %s", pred.Error) + case "canceled": + return nil, fmt.Errorf("prediction was canceled") + } + time.Sleep(1 * time.Second) + } + return nil, fmt.Errorf("prediction timed out after %v", timeout) +} +``` diff --git a/docs/协议文档/Gemini协议文档.md b/docs/协议文档/Gemini协议文档.md new file mode 100644 index 0000000..0641e6f --- /dev/null +++ b/docs/协议文档/Gemini协议文档.md @@ -0,0 +1,1903 @@ +# Google Gemini API 协议文档 + +> 基于官方 API Reference (https://ai.google.dev/api/generate-content) 整理 +> +> 本文档用于 team-api 协议转换模块的开发参考，涵盖 Gemini GenerateContent API 的完整请求/响应规范、流式传输、上下文缓存、向量嵌入、Token 计数等。 + +--- + +## 目录 + +- [1. API 概览](#1-api-概览) +- [2. 认证方式](#2-认证方式) +- [3. 端点列表](#3-端点列表) +- [4. 请求体参数 (Request Body)](#4-请求体参数-request-body) +- [5. Content 结构](#5-content-结构) +- [6. Part 类型详解](#6-part-类型详解) +- [7. GenerationConfig 生成配置](#7-generationconfig-生成配置) +- [8. ThinkingConfig 思考配置](#8-thinkingconfig-思考配置) +- [9. SpeechConfig 语音配置](#9-speechconfig-语音配置) +- [10. Tool 工具系统](#10-tool-工具系统) +- [11. ToolConfig 工具配置](#11-toolconfig-工具配置) +- [12. SafetySettings 安全设置](#12-safetysettings-安全设置) +- [13. 响应格式 (Response)](#13-响应格式-response) +- [14. FinishReason 枚举](#14-finishreason-枚举) +- [15. 流式响应 (SSE Streaming)](#15-流式响应-sse-streaming) +- [16. 错误格式](#16-错误格式) +- [17. 上下文缓存 (Context Caching)](#17-上下文缓存-context-caching) +- [18. 向量嵌入 (Embeddings)](#18-向量嵌入-embeddings) +- [19. Token 计数 API](#19-token-计数-api) +- [20. 结构化输出 (Structured Output)](#20-结构化输出-structured-output) +- [21. 服务层级 (Service Tier)](#21-服务层级-service-tier) +- [22. 模型阶段 (Model Stage)](#22-模型阶段-model-stage) +- [23. 协议转换注意事项](#23-协议转换注意事项) + +--- + +## 1. API 概览 + +| 属性 | 说明 | +|------|------| +| **协议类型** | REST (HTTP/JSON) | +| **基础 URL** | `https://generativelanguage.googleapis.com` | +| **API 版本** | `v1beta`（最新功能）/ `v1`（稳定版） | +| **核心功能** | 根据文本、图片、音频、视频等输入生成模型响应 | +| **Content-Type** | `application/json` | +| **特色能力** | 原生多模态输入/输出、思考模式(Thinking)、代码执行、搜索集成、MCP Server 集成、上下文缓存 | + +### 基本请求示例 + +```bash +curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.5-flash:generateContent?key=$GEMINI_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{ + "contents": [ + { + "role": "user", + "parts": [ + {"text": "Hello, world!"} + ] + } + ] + }' +``` + +### 基本响应示例 + +```json +{ + "candidates": [ + { + "content": { + "parts": [ + {"text": "Hello! How can I help you today?"} + ], + "role": "model" + }, + "finishReason": "STOP", + "index": 0 + } + ], + "usageMetadata": { + "promptTokenCount": 7, + "candidatesTokenCount": 9, + "totalTokenCount": 16 + }, + "modelVersion": "gemini-2.5-flash-001" +} +``` + +--- + +## 2. 认证方式 + +Gemini API 支持以下认证方式： + +| 认证方式 | 说明 | 适用场景 | +|---------|------|---------| +| API Key (查询参数) | `?key=YOUR_API_KEY` | 开发调试、简单调用 | +| API Key (请求头) | `x-goog-api-key: YOUR_API_KEY` | 生产环境推荐 | +| OAuth 2.0 | `Authorization: Bearer YOUR_ACCESS_TOKEN` | Google Cloud 项目集成 | +| Service Account | 通过 Google Cloud IAM 认证 | 服务端应用 | + +### 请求头示例 + +```http +# 方式一：请求头传递 API Key（推荐） +x-goog-api-key: AIzaSy... + +# 方式二：查询参数传递 API Key +GET /v1beta/models/gemini-2.5-flash:generateContent?key=AIzaSy... + +# 方式三：OAuth 2.0 Bearer Token +Authorization: Bearer ya29.a0AfH6... +``` + +--- + +## 3. 端点列表 + +### 3.1 内容生成端点 + +| 方法 | 路径 | 功能 | 流式支持 | +|------|------|------|---------| +| POST | `/v1beta/models/{model}:generateContent` | 同步生成内容 | 否 | +| POST | `/v1beta/models/{model}:streamGenerateContent?alt=sse` | 流式生成内容 | 是 (SSE) | +| POST | `/v1beta/models/{model}:countTokens` | 计算 Token 数量 | 否 | +| POST | `/v1beta/models/{model}:countTextTokens` | 计算文本 Token 数量 | 否 | +| POST | `/v1beta/models/{model}:computeTokens` | 计算 Tokens（通用） | 否 | +| POST | `/v1beta/models/{model}:embedContent` | 单条文本向量化 | 否 | +| POST | `/v1beta/models/{model}:batchEmbedContents` | 批量文本向量化 | 否 | + +### 3.2 模型管理端点 + +| 方法 | 路径 | 功能 | +|------|------|------| +| GET | `/v1beta/models` | 列出可用模型 | +| GET | `/v1beta/models/{model}` | 获取模型详情 | + +### 3.3 上下文缓存端点 + +| 方法 | 路径 | 功能 | +|------|------|------| +| POST | `/v1beta/cachedContents` | 创建缓存内容 | +| GET | `/v1beta/cachedContents/{name}` | 获取缓存内容 | +| PATCH | `/v1beta/cachedContents/{name}` | 更新缓存内容 | +| DELETE | `/v1beta/cachedContents/{name}` | 删除缓存内容 | +| GET | `/v1beta/cachedContents` | 列出缓存内容 | + +### 3.4 URL 格式 + +``` +# 同步生成 +POST https://generativelanguage.googleapis.com/v1beta/models/{model}:generateContent + +# 流式生成（注意 alt=sse 参数） +POST https://generativelanguage.googleapis.com/v1beta/models/{model}:streamGenerateContent?alt=sse + +# 使用缓存 +POST https://generativelanguage.googleapis.com/v1beta/models/{model}:generateContent +# 请求体中通过 cachedContent 字段引用缓存 +``` + +--- + +## 4. 请求体参数 (Request Body) + +### 4.1 完整参数一览 + +请求体是一个 JSON 对象，`GenerateContentRequest` 结构： + +| 参数 | 类型 | 必填 | 默认值 | 说明 | +|------|------|------|--------|------| +| `contents` | Content[] | **是** | — | 对话内容列表，按时间顺序排列 | +| `tools` | Tool[] | 否 | — | 工具声明列表，如 Function Calling、Google Search 等 | +| `toolConfig` | ToolConfig | 否 | — | 工具配置，控制工具行为 | +| `safetySettings` | SafetySetting[] | 否 | — | 安全过滤设置 | +| `systemInstruction` | Content | 否 | — | 系统指令（开发者提示词） | +| `generationConfig` | GenerationConfig | 否 | — | 生成参数配置 | +| `cachedContent` | string | 否 | — | 引用已缓存内容的资源名称，如 `cachedContents/xxx` | + +### 4.2 请求体示例 + +```json +{ + "contents": [ + { + "role": "user", + "parts": [ + {"text": "请分析这张图片"}, + {"inlineData": {"mimeType": "image/jpeg", "data": "base64EncodedImage..."}} + ] + }, + { + "role": "model", + "parts": [ + {"text": "这是一张风景照片..."} + ] + }, + { + "role": "user", + "parts": [ + {"text": "能更详细描述吗？"} + ] + } + ], + "systemInstruction": { + "parts": [ + {"text": "你是一个专业的图像分析师。"} + ] + }, + "generationConfig": { + "temperature": 0.7, + "topP": 0.95, + "topK": 40, + "maxOutputTokens": 8192, + "thinkingConfig": { + "includeThoughts": true, + "thinkingBudget": 8192 + } + }, + "tools": [ + {"googleSearch": {}} + ], + "safetySettings": [ + { + "category": "HARM_CATEGORY_HARASSMENT", + "threshold": "BLOCK_MEDIUM_AND_ABOVE" + } + ] +} +``` + +--- + +## 5. Content 结构 + +Content 是 Gemini 消息的基本单元，与 OpenAI 的 message 和 Claude 的 content block 对应。 + +### 5.1 Content 对象 + +| 字段 | 类型 | 说明 | +|------|------|------| +| `parts` | Part[] | 内容部件列表，按顺序排列。每个 Part 可以是文本、图片、函数调用等 | +| `role` | string | 消息角色：`"user"`（用户）、`"model"`（模型） | + +### 5.2 与其他协议的角色映射 + +| Gemini role | OpenAI role | Claude role | 说明 | +|-------------|-------------|-------------|------| +| `user` | `user` | `user` | 用户消息 | +| `model` | `assistant` | `assistant` | 模型回复 | +| — | `system` | — | Gemini 使用独立的 `systemInstruction` 字段 | +| `user`（function 回复） | `tool` | `user`（tool_result） | 函数执行结果 | + +### 5.3 system 消息处理差异 + +Gemini **不使用** `contents` 数组中的 `role: "system"`，而是通过请求体顶层独立的 `systemInstruction` 字段传递系统提示词： + +```json +{ + "systemInstruction": { + "parts": [ + {"text": "你是一个有用的AI助手。"} + ] + }, + "contents": [ + {"role": "user", "parts": [{"text": "你好"}]} + ] +} +``` + +--- + +## 6. Part 类型详解 + +Part 是 Gemini 内容的最小单元，支持多种类型。每个 Part 对象是一个**联合类型（union type）**，同一时间只应设置一个类型字段。 + +### 6.1 Part 类型总览 + +| 字段名 | 类型 | 说明 | +|--------|------|------| +| `text` | string | 纯文本内容 | +| `inlineData` | Blob | 内联的二进制数据（图片、音频、视频等） | +| `functionCall` | FunctionCall | 模型发起的函数调用 | +| `functionResponse` | FunctionResponse | 返回给模型的函数执行结果 | +| `fileData` | FileData | 引用已上传文件的引用 | +| `executableCode` | ExecutableCode | 可执行代码（代码执行工具生成） | +| `codeExecutionResult` | CodeExecutionResult | 代码执行结果 | +| `videoMetadata` | VideoMetadata | 视频元数据（配合 inlineData 或 fileData 使用） | +| `thought` | string | 思考内容（Thinking 模式） | +| `thoughtSignature` | string | 思考签名（用于验证思考过程完整性） | +| `toolCall` | ToolCall | 服务端工具调用（如 Google Search、代码执行） | +| `toolResponse` | ToolResponse | 服务端工具响应 | + +### 6.2 文本 Part + +```json +{ + "text": "这是一段文本内容" +} +``` + +### 6.3 内联数据 Part (inlineData / Blob) + +用于发送图片、音频、视频等二进制数据，需 Base64 编码。 + +```json +{ + "inlineData": { + "mimeType": "image/jpeg", + "data": "/9j/4AAQSkZJRgABAQ..." + } +} +``` + +**Blob 对象字段**： + +| 字段 | 类型 | 必填 | 说明 | +|------|------|------|------| +| `mimeType` | string | **是** | MIME 类型 | +| `data` | string | **是** | Base64 编码的数据 | + +**支持的 MIME 类型**： + +| 类别 | MIME 类型示例 | +|------|-------------| +| 图片 | `image/jpeg`, `image/png`, `image/webp`, `image/gif`, `image/bmp`, `image/tiff` | +| 音频 | `audio/wav`, `audio/mp3`, `audio/aiff`, `audio/aac`, `audio/ogg`, `audio/flac` | +| 视频 | `video/mp4`, `video/mpeg`, `video/webm`, `video/x-matroska`, `video/x-flv` | +| 文档 | `application/pdf` | +| 文本 | `text/plain`, `text/html`, `text/css`, `text/csv`, `text/xml` | +| 代码 | `text/x-python`, `text/x-java`, `text/x-c`, `text/x-go`, `text/x-javascript` | + +### 6.4 文件引用 Part (fileData) + +引用通过 File API 上传的文件或 Google Cloud Storage URI。 + +```json +{ + "fileData": { + "mimeType": "video/mp4", + "fileUri": "gs://bucket-name/file.mp4" + } +} +``` + +**FileData 对象字段**： + +| 字段 | 类型 | 必填 | 说明 | +|------|------|------|------| +| `mimeType` | string | 否 | 文件 MIME 类型 | +| `fileUri` | string | **是** | 文件 URI，支持 `gs://` 格式或 File API 返回的 URI | + +### 6.5 视频元数据 Part (videoMetadata) + +配合 inlineData 或 fileData 使用，提供视频特定信息。 + +```json +{ + "videoMetadata": { + "startOffset": "10s", + "endOffset": "60s" + } +} +``` + +**VideoMetadata 对象字段**： + +| 字段 | 类型 | 说明 | +|------|------|------| +| `startOffset` | string (Duration) | 视频起始偏移量，格式如 `"10s"` | +| `endOffset` | string (Duration) | 视频结束偏移量，格式如 `"60s"` | + +### 6.6 函数调用 Part (functionCall) + +模型发起的函数调用请求。 + +```json +{ + "functionCall": { + "name": "get_weather", + "args": { + "location": "Beijing", + "unit": "celsius" + } + } +} +``` + +**FunctionCall 对象字段**： + +| 字段 | 类型 | 说明 | +|------|------|------| +| `name` | string | 函数名称 | +| `args` | object | 函数参数（键值对） | + +### 6.7 函数响应 Part (functionResponse) + +返回函数执行结果给模型。 + +```json +{ + "functionResponse": { + "name": "get_weather", + "response": { + "temperature": 22, + "condition": "sunny", + "unit": "celsius" + } + } +} +``` + +**FunctionResponse 对象字段**： + +| 字段 | 类型 | 说明 | +|------|------|------| +| `name` | string | 函数名称（与 functionCall 的 name 对应） | +| `response` | object | 函数执行结果（任意 JSON 结构） | + +### 6.8 可执行代码 Part (executableCode) + +代码执行工具生成的可执行代码。 + +```json +{ + "executableCode": { + "language": "PYTHON", + "code": "print('Hello, World!')" + } +} +``` + +**ExecutableCode 对象字段**： + +| 字段 | 类型 | 说明 | +|------|------|------| +| `language` | string (enum) | 编程语言：`PYTHON` | +| `code` | string | 代码内容 | + +### 6.9 代码执行结果 Part (codeExecutionResult) + +代码执行的输出结果。 + +```json +{ + "codeExecutionResult": { + "outcome": "OUTCOME_OK", + "output": "Hello, World!" + } +} +``` + +**CodeExecutionResult 对象字段**： + +| 字段 | 类型 | 说明 | +|------|------|------| +| `outcome` | string (enum) | 执行结果状态：`OUTCOME_OK`、`OUTCOME_FAILED`、`OUTCOME_DEADLINE_EXCEEDED` | +| `output` | string | 执行输出 | + +### 6.10 思考 Part (thought) + +当启用 Thinking 模式（`thinkingConfig.includeThoughts: true`）时，模型内部推理过程。 + +```json +{ + "thought": "让我分析这个问题...首先需要考虑..." +} +``` + +### 6.11 思考签名 Part (thoughtSignature) + +用于验证思考过程完整性的签名。某些模型在流式响应中会附带。 + +```json +{ + "thoughtSignature": "AQIDBA..." +} +``` + +### 6.12 服务端工具调用 Part (toolCall) + +服务端工具（如 Google Search、代码执行）的调用信息。 + +```json +{ + "toolCall": { + "toolType": "GOOGLE_SEARCH", + "id": "tool-call-id-123" + } +} +``` + +**ToolCall 对象字段**： + +| 字段 | 类型 | 说明 | +|------|------|------| +| `toolType` | string (ToolType enum) | 工具类型 | +| `id` | string | 工具调用标识 | + +### 6.13 服务端工具响应 Part (toolResponse) + +服务端工具的执行结果。 + +```json +{ + "toolResponse": { + "id": "tool-call-id-123", + "result": "搜索结果内容..." + } +} +``` + +**ToolResponse 对象字段**： + +| 字段 | 类型 | 说明 | +|------|------|------| +| `id` | string | 与 toolCall 的 id 对应 | +| `result` | string | 工具执行结果 | + +--- + +## 7. GenerationConfig 生成配置 + +GenerationConfig 控制模型的生成行为，是 Gemini 中最重要的配置对象之一。 + +### 7.1 完整字段一览 + +| 字段 | 类型 | 默认值 | 说明 | +|------|------|--------|------| +| `temperature` | number | 模型默认 | 采样温度。控制输出随机性，范围 0.0-2.0。值越低越确定，越高越随机 | +| `topP` | number | 模型默认 | 核采样（Nucleus Sampling）概率阈值。范围 0.0-1.0 | +| `topK` | integer | 模型默认 | Top-K 采样。仅从概率最高的 K 个 token 中采样 | +| `candidateCount` | integer | `1` | 生成的候选响应数量（目前仅支持 1） | +| `maxOutputTokens` | integer | 模型默认 | 最大输出 Token 数 | +| `stopSequences` | string[] | — | 停止序列列表，模型遇到任一序列时停止生成 | +| `responseMimeType` | string | `text/plain` | 响应 MIME 类型：`text/plain`、`application/json`、`text/x.enum` | +| `responseSchema` | Schema | — | 结构化输出的 JSON Schema（仅当 `responseMimeType` 为 `application/json` 时有效） | +| `responseJsonSchema` | JsonSchema | — | 增强的 JSON Schema 定义（含 ref 支持和严格模式） | +| `presencePenalty` | number | `0` | 存在惩罚。正值增加讨论新话题的概率，范围 -2.0 ~ 2.0 | +| `frequencyPenalty` | number | `0` | 频率惩罚。正值降低逐字重复的概率，范围 -2.0 ~ 2.0 | +| `seed` | integer | — | 随机种子。设置后可尽量保证相同输入产生相同输出 | +| `thinkingConfig` | ThinkingConfig | — | 思考模式配置（详见第 8 节） | +| `speechConfig` | SpeechConfig | — | 语音输出配置（详见第 9 节） | +| `audioConfig` | AudioConfig | — | 音频输出配置 | +| `imageConfig` | ImageConfig | — | 图片输出配置 | +| `mediaResolution` | string (enum) | — | 媒体分辨率：`MEDIA_RESOLUTION_LOW`、`MEDIA_RESOLUTION_MEDIUM`、`MEDIA_RESOLUTION_HIGH` | +| `routingConfig` | GenerationConfigRoutingConfig | — | 路由配置，控制请求在模型变体间的分发策略 | + +### 7.2 与 OpenAI/Claude 参数映射 + +| Gemini 参数 | OpenAI Chat Completions | Claude Messages | 说明 | +|-------------|------------------------|----------------|------| +| `temperature` | `temperature` | `temperature` | 语义相同 | +| `topP` | `top_p` | `top_p` | 语义相同 | +| `topK` | — | `top_k` | OpenAI 不支持 | +| `maxOutputTokens` | `max_completion_tokens` | `max_tokens` | 语义相同 | +| `stopSequences` | `stop` | `stop_sequences` | 语义相同 | +| `candidateCount` | `n` | — | Gemini 目前仅支持 1 | +| `presencePenalty` | `presence_penalty` | — | Claude 不支持 | +| `frequencyPenalty` | `frequency_penalty` | — | Claude 不支持 | +| `seed` | `seed` | — | Claude 不支持 | +| `responseMimeType` | `response_format.type` | — | 结构化输出 | +| `responseSchema` | `response_format.json_schema` | — | Schema 定义 | +| `thinkingConfig` | — | `thinking.budget_tokens` | 思考模式 | + +### 7.3 AudioConfig 对象 + +| 字段 | 类型 | 说明 | +|------|------|------| +| `audioConfig` | object | 音频输出配置（用于音频生成模型） | + +### 7.4 ImageConfig 对象 + +| 字段 | 类型 | 说明 | +|------|------|------| +| `imageConfig` | object | 图片输出配置（用于图片生成模型） | + +--- + +## 8. ThinkingConfig 思考配置 + +ThinkingConfig 控制模型的内部推理（思考）行为，是 Gemini 2.5 系列模型的核心特性。 + +### 8.1 ThinkingConfig 对象 + +| 字段 | 类型 | 必填 | 默认值 | 说明 | +|------|------|------|--------|------| +| `includeThoughts` | boolean | 否 | `false` | 是否在响应中包含思考过程 | +| `thinkingBudget` | integer | 否 | 模型默认 | 思考 Token 预算，控制模型在思考上花费的最大 Token 数 | +| `thinkingLevel` | string (enum) | 否 | 模型默认 | 思考级别枚举 | + +### 8.2 thinkingLevel 枚举值 + +| 值 | 说明 | +|----|------| +| `THINKING_LEVEL_UNSPECIFIED` | 未指定，使用模型默认 | +| `THINKING_LEVEL_LOW` | 低级别思考，快速但简略 | +| `THINKING_LEVEL_MEDIUM` | 中级别思考，平衡速度与深度 | +| `THINKING_LEVEL_HIGH` | 高级别思考，深度推理 | +| `THINKING_LEVEL_NONE` | 禁用思考 | + +### 8.3 配置示例 + +```json +{ + "generationConfig": { + "thinkingConfig": { + "includeThoughts": true, + "thinkingBudget": 8192 + } + } +} +``` + +### 8.4 思考模式响应示例 + +启用思考后，响应中会包含 `thought` 类型的 Part： + +```json +{ + "candidates": [ + { + "content": { + "parts": [ + {"thought": "让我分析这个问题...\n首先需要考虑数学公式..."}, + {"text": "答案是 42。"} + ], + "role": "model" + } + } + ] +} +``` + +--- + +## 9. SpeechConfig 语音配置 + +用于配置文本转语音（TTS）输出的语音参数。 + +### 9.1 SpeechConfig 对象 + +| 字段 | 类型 | 说明 | +|------|------|------| +| `voiceConfig` | VoiceConfig | 单说话人语音配置 | +| `multiSpeakerVoiceConfig` | MultiSpeakerVoiceConfig | 多说话人语音配置 | +| `languageCode` | string | 语言代码，如 `"en-US"`、`"zh-CN"` | + +### 9.2 VoiceConfig 对象 + +| 字段 | 类型 | 说明 | +|------|------|------| +| `prebuiltVoiceConfig` | PrebuiltVoiceConfig | 预置语音配置 | + +### 9.3 PrebuiltVoiceConfig 对象 + +| 字段 | 类型 | 说明 | +|------|------|------| +| `voiceName` | string | 语音名称，如 `"Aoede"`、`"Charon"`、`"Puck"` 等 | + +### 9.4 MultiSpeakerVoiceConfig 对象 + +| 字段 | 类型 | 说明 | +|------|------|------| +| `speakerVoiceConfigs` | SpeakerVoiceConfig[] | 各说话人的语音配置 | + +### 9.5 配置示例 + +```json +{ + "generationConfig": { + "speechConfig": { + "voiceConfig": { + "prebuiltVoiceConfig": { + "voiceName": "Aoede" + } + }, + "languageCode": "zh-CN" + } + } +} +``` + +--- + +## 10. Tool 工具系统 + +Gemini 的 Tool 系统支持多种工具类型，包括自定义函数、Google 搜索、代码执行等。 + +### 10.1 Tool 对象 + +每个 Tool 是一个联合类型，可以包含以下任一类型： + +| 字段 | 类型 | 说明 | +|------|------|------| +| `functionDeclarations` | FunctionDeclaration[] | 函数声明列表（Function Calling） | +| `googleSearch` | GoogleSearch | Google 搜索工具（即 Grounding） | +| `googleSearchRetrieval` | GoogleSearchRetrieval | Google 搜索检索（旧版，推荐使用 googleSearch） | +| `codeExecution` | CodeExecution | 代码执行工具 | +| `computerUse` | ComputerUse | 计算机操控工具 | +| `urlContext` | UrlContext | URL 上下文工具 | +| `fileSearch` | FileSearch | 文件搜索工具 | +| `mcpServers` | MCPServer[] | MCP（Model Context Protocol）服务端集成 | +| `googleMaps` | GoogleMaps | Google Maps 工具 | + +### 10.2 FunctionDeclaration 函数声明 + +定义可供模型调用的函数。 + +```json +{ + "functionDeclarations": [ + { + "name": "get_weather", + "description": "获取指定城市的天气信息", + "parameters": { + "type": "OBJECT", + "properties": { + "location": { + "type": "STRING", + "description": "城市名称" + }, + "unit": { + "type": "STRING", + "description": "温度单位", + "enum": ["celsius", "fahrenheit"] + } + }, + "required": ["location"] + } + } + ] +} +``` + +**FunctionDeclaration 字段**： + +| 字段 | 类型 | 必填 | 说明 | +|------|------|------|------| +| `name` | string | **是** | 函数名称，必须唯一 | +| `description` | string | **是** | 函数描述，模型据此决定何时调用 | +| `parameters` | Schema | 否 | 函数参数的 JSON Schema | +| `response` | Schema | 否 | 函数返回值的 Schema（部分模型支持） | + +### 10.3 Schema 类型系统 + +Gemini 使用自定义 Schema 格式而非标准的 JSON Schema： + +| 字段 | 类型 | 说明 | +|------|------|------| +| `type` | string (enum) | 类型枚举 | +| `format` | string | 格式说明（如 `"int64"`、`"double"`） | +| `description` | string | 字段描述 | +| `nullable` | boolean | 是否可为空 | +| `enum` | string[] | 枚举值列表 | +| `maxItems` | string (int64) | 数组最大元素数 | +| `minItems` | string (int64) | 数组最小元素数 | +| `properties` | map | 对象属性 | +| `required` | string[] | 必填字段列表 | +| `items` | Schema | 数组元素类型 | +| `anyOf` | Schema[] | 联合类型 | + +**type 枚举值**： + +| 值 | 说明 | +|----|------| +| `TYPE_UNSPECIFIED` | 未指定 | +| `STRING` | 字符串 | +| `NUMBER` | 数字（浮点） | +| `INTEGER` | 整数 | +| `BOOLEAN` | 布尔值 | +| `ARRAY` | 数组 | +| `OBJECT` | 对象 | + +### 10.4 GoogleSearch 工具 + +```json +{ + "tools": [ + { + "googleSearch": {} + } + ] +} +``` + +GoogleSearch 目前为空对象，启用即表示使用默认配置。模型会自动在需要时触发搜索。 + +### 10.5 GoogleSearchRetrieval 工具（旧版） + +```json +{ + "tools": [ + { + "googleSearchRetrieval": { + "disableAttribution": false + } + } + ] +} +``` + +| 字段 | 类型 | 说明 | +|------|------|------| +| `disableAttribution` | boolean | 是否禁用搜索来源归因 | + +### 10.6 CodeExecution 工具 + +```json +{ + "tools": [ + { + "codeExecution": {} + } + ] +} +``` + +CodeExecution 目前为空对象。启用后模型会生成并执行代码。 + +### 10.7 ComputerUse 工具 + +```json +{ + "tools": [ + { + "computerUse": {} + } + ] +} +``` + +用于计算机操控能力（类似 Claude 的 Computer Use）。 + +### 10.8 UrlContext 工具 + +```json +{ + "tools": [ + { + "urlContext": {} + } + ] +} +``` + +允许模型获取和引用 URL 内容。 + +### 10.9 FileSearch 工具 + +```json +{ + "tools": [ + { + "fileSearch": {} + } + ] +} +``` + +允许模型在用户上传的文件中进行搜索。 + +### 10.10 MCP Server 集成 + +```json +{ + "tools": [ + { + "mcpServers": [ + { + "name": "my-mcp-server", + "streamableHttpTransport": { + "url": "https://example.com/mcp" + } + } + ] + } + ] +} +``` + +**MCPServer 对象字段**： + +| 字段 | 类型 | 说明 | +|------|------|------| +| `name` | string | MCP 服务器名称 | +| `streamableHttpTransport` | StreamableHttpTransport | HTTP 传输配置 | + +**StreamableHttpTransport 对象字段**： + +| 字段 | 类型 | 说明 | +|------|------|------| +| `url` | string | MCP 服务器 URL | + +### 10.11 ToolType 枚举 + +服务端工具的类型标识： + +| 值 | 说明 | +|----|------| +| `TOOL_TYPE_UNSPECIFIED` | 未指定 | +| `GOOGLE_SEARCH` | Google 搜索 | +| `CODE_EXECUTION` | 代码执行 | +| `GOOGLE_MAPS` | Google Maps | + +--- + +## 11. ToolConfig 工具配置 + +ToolConfig 控制工具的全局行为。 + +### 11.1 ToolConfig 对象 + +| 字段 | 类型 | 说明 | +|------|------|------| +| `functionCallingConfig` | FunctionCallingConfig | 函数调用配置 | +| `retrievalConfig` | RetrievalConfig | 检索配置（旧版搜索工具） | + +### 11.2 FunctionCallingConfig 对象 + +| 字段 | 类型 | 说明 | +|------|------|------| +| `allowedFunctionNames` | string[] | 允许调用的函数名称列表。不设置则允许所有已声明的函数 | +| `mode` | string (enum) | 函数调用模式 | + +**mode 枚举值**： + +| 值 | 说明 | +|----|------| +| `AUTO` | 自动模式：模型自行决定是否调用函数（默认） | +| `ANY` | 任意模式：模型必须调用至少一个函数 | +| `NONE` | 禁用模式：模型不得调用任何函数 | +| `VALIDATED` | 验证模式：仅调用已验证可用的函数 | + +### 11.3 配置示例 + +```json +{ + "toolConfig": { + "functionCallingConfig": { + "allowedFunctionNames": ["get_weather", "get_news"], + "mode": "AUTO" + } + } +} +``` + +--- + +## 12. SafetySettings 安全设置 + +控制模型的内容安全过滤行为。 + +### 12.1 SafetySetting 对象 + +| 字段 | 类型 | 说明 | +|------|------|------| +| `category` | string (HarmCategory enum) | 安全类别 | +| `threshold` | string (HarmBlockThreshold enum) | 阻断阈值 | +| `method` | string (HarmBlockMethod enum) | 阻断方法 | + +### 12.2 HarmCategory 安全类别 + +| 枚举值 | 说明 | +|--------|------| +| `HARM_CATEGORY_UNSPECIFIED` | 未指定 | +| `HARM_CATEGORY_HARASSMENT` | 骚扰 | +| `HARM_CATEGORY_HATE_SPEECH` | 仇恨言论 | +| `HARM_CATEGORY_SEXUALLY_EXPLICIT` | 色情内容 | +| `HARM_CATEGORY_DANGEROUS_CONTENT` | 危险内容 | +| `HARM_CATEGORY_CIVIC_INTEGRITY` | 公民诚信（选举虚假信息等） | + +### 12.3 HarmBlockThreshold 阻断阈值 + +| 枚举值 | 说明 | +|--------|------| +| `HARM_BLOCK_THRESHOLD_UNSPECIFIED` | 未指定，使用默认设置 | +| `BLOCK_LOW_AND_ABOVE` | 阻断低风险及更高级别的内容 | +| `BLOCK_MEDIUM_AND_ABOVE` | 阻断中等风险及更高级别的内容（推荐默认值） | +| `BLOCK_ONLY_HIGH` | 仅阻断高风险内容 | +| `BLOCK_NONE` | 不阻断任何内容（仍会评估风险） | +| `OFF` | 关闭安全过滤（不评估也不阻断） | + +### 12.4 HarmBlockMethod 阻断方法 + +| 枚举值 | 说明 | +|--------|------| +| `HARM_BLOCK_METHOD_UNSPECIFIED` | 未指定 | +| `SEVERITY` | 基于严重程度 | +| `PROBABILITY` | 基于概率 | + +### 12.5 配置示例 + +```json +{ + "safetySettings": [ + { + "category": "HARM_CATEGORY_HARASSMENT", + "threshold": "BLOCK_MEDIUM_AND_ABOVE" + }, + { + "category": "HARM_CATEGORY_HATE_SPEECH", + "threshold": "BLOCK_MEDIUM_AND_ABOVE" + }, + { + "category": "HARM_CATEGORY_SEXUALLY_EXPLICIT", + "threshold": "BLOCK_ONLY_HIGH" + }, + { + "category": "HARM_CATEGORY_DANGEROUS_CONTENT", + "threshold": "BLOCK_MEDIUM_AND_ABOVE" + }, + { + "category": "HARM_CATEGORY_CIVIC_INTEGRITY", + "threshold": "BLOCK_NONE" + } + ] +} +``` + +--- + +## 13. 响应格式 (Response) + +### 13.1 GenerateContentResponse 对象 + +| 字段 | 类型 | 说明 | +|------|------|------| +| `candidates` | Candidate[] | 候选响应列表 | +| `promptFeedback` | PromptFeedback | 输入提示的反馈信息 | +| `usageMetadata` | UsageMetadata | Token 使用量统计 | +| `modelVersion` | string | 模型版本标识，如 `"gemini-2.5-flash-001"` | +| `responseId` | string | 响应唯一标识 | + +### 13.2 Candidate 候选对象 + +| 字段 | 类型 | 说明 | +|------|------|------| +| `content` | Content | 响应内容（含 parts 和 role） | +| `finishReason` | string (FinishReason enum) | 结束原因 | +| `index` | integer | 候选索引（从 0 开始） | +| `citationMetadata` | CitationMetadata | 引用元数据 | +| `groundingMetadata` | GroundingMetadata | 搜索接地元数据 | +| `groundingAttributions` | GroundingAttribution[] | 搜索接地归因 | +| `tokenCount` | integer | Token 计数 | + +### 13.3 CitationMetadata 引用元数据 + +| 字段 | 类型 | 说明 | +|------|------|------| +| `citationSources` | CitationSource[] | 引用来源列表 | + +**CitationSource 对象**： + +| 字段 | 类型 | 说明 | +|------|------|------| +| `startIndex` | integer | 引用开始的字符索引 | +| `endIndex` | integer | 引用结束的字符索引 | +| `uri` | string | 引用来源 URI | +| `license` | string | 许可证信息 | + +### 13.4 GroundingMetadata 搜索接地元数据 + +| 字段 | 类型 | 说明 | +|------|------|------| +| `groundingChunks` | GroundingChunk[] | 接地内容块 | +| `groundingSupports` | GroundingSupport[] | 接地支持信息 | +| `searchEntryPoint` | SearchEntryPoint | 搜索入口 | +| `retrievalMetadata` | RetrievalMetadata | 检索元数据 | +| `webSearchQueries` | string[] | Web 搜索查询列表 | + +### 13.5 PromptFeedback 提示反馈 + +| 字段 | 类型 | 说明 | +|------|------|------| +| `blockReason` | string (BlockReason enum) | 阻断原因 | +| `safetyRatings` | SafetyRating[] | 安全评级列表 | + +**BlockReason 枚举值**： + +| 值 | 说明 | +|----|------| +| `BLOCK_REASON_UNSPECIFIED` | 未指定 | +| `SAFETY` | 因安全过滤被阻断 | +| `OTHER` | 其他原因 | +| `BLOCKLIST` | 因黑名单被阻断 | +| `PROHIBITED_CONTENT` | 禁止内容 | +| `MODEL_REJECT` | 模型拒绝 | + +**SafetyRating 对象**： + +| 字段 | 类型 | 说明 | +|------|------|------| +| `category` | string (HarmCategory) | 安全类别 | +| `probability` | string | 概率级别：`NEGLIGIBLE`、`LOW`、`MEDIUM`、`HIGH` | +| `blocked` | boolean | 是否被阻断 | + +### 13.6 UsageMetadata 使用量统计 + +| 字段 | 类型 | 说明 | +|------|------|------| +| `promptTokenCount` | integer | 输入 Token 数 | +| `candidatesTokenCount` | integer | 输出 Token 数 | +| `totalTokenCount` | integer | 总 Token 数 | +| `cachedContentTokenCount` | integer | 缓存命中 Token 数（使用上下文缓存时） | +| `thoughtsTokenCount` | integer | 思考 Token 数（启用思考模式时） | + +### 13.7 完整响应示例 + +```json +{ + "candidates": [ + { + "content": { + "parts": [ + { + "text": "北京今天的天气是晴天，气温 22°C。" + } + ], + "role": "model" + }, + "finishReason": "STOP", + "index": 0, + "groundingMetadata": { + "webSearchQueries": ["北京今天天气"], + "groundingChunks": [ + { + "web": { + "uri": "https://example.com/weather/beijing", + "title": "北京天气预报" + } + } + ] + } + } + ], + "usageMetadata": { + "promptTokenCount": 15, + "candidatesTokenCount": 20, + "totalTokenCount": 35, + "candidatesTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 20 + } + ] + }, + "modelVersion": "gemini-2.5-flash-001" +} +``` + +--- + +## 14. FinishReason 枚举 + +模型停止生成的原因，位于 Candidate 对象中。 + +| 枚举值 | 说明 | 对应 OpenAI finish_reason | 对应 Claude stop_reason | +|--------|------|--------------------------|------------------------| +| `FINISH_REASON_UNSPECIFIED` | 未指定 | — | — | +| `STOP` | 自然停止（遇到 EOS 或完成回复） | `stop` | `end_turn` | +| `MAX_TOKENS` | 达到最大输出 Token 限制 | `length` | `max_tokens` | +| `SAFETY` | 因安全过滤被阻断 | `content_filter` | — | +| `RECITATION` | 因引用/版权限制被阻断 | — | — | +| `LANGUAGE` | 因语言限制被阻断 | — | — | +| `OTHER` | 其他原因 | — | — | +| `BLOCKLIST` | 因黑名单被阻断 | — | — | +| `PROHIBITED_CONTENT` | 因禁止内容被阻断 | — | — | +| `SPII` | 因敏感个人身份信息被阻断 | — | — | +| `MALFORMED_FUNCTION_CALL` | 函数调用格式错误 | — | — | +| `IMAGE_SAFETY` | 因图片安全被阻断 | — | — | +| `UNEXPECTED_TOOL_CALL` | 非预期的工具调用 | — | — | +| `FAIRNESS_CIVIL_INTEGRITY` | 因公平性/公民诚信被阻断 | — | — | +| `CONTEXT_WINDOW_EXCEEDED` | 超出上下文窗口 | — | — | +| `CANDIDATE_SUMMARY` | 候选摘要（内部使用） | — | — | +| `TOOL_CALL_LIMIT_REACHED` | 达到工具调用次数上限 | — | — | +| `MISSING_THOUGHT_SIGNATURE` | 缺少思考签名 | — | — | +| `MALFORMED_RESPONSE` | 响应格式错误 | — | — | + +--- + +## 15. 流式响应 (SSE Streaming) + +### 15.1 流式端点 + +流式生成使用 `streamGenerateContent` 端点，**必须**添加 `alt=sse` 查询参数： + +``` +POST /v1beta/models/{model}:streamGenerateContent?alt=sse +``` + +### 15.2 SSE 格式特点 + +Gemini 的流式格式与 OpenAI/Claude 有显著差异： + +| 特性 | Gemini | OpenAI | Claude | +|------|--------|--------|--------| +| 数据格式 | `data: {json}\n\n` | `data: {json}\n\n` | `event: xxx\ndata: {json}\n\n` | +| 结束标记 | **无** `[DONE]`，HTTP 连接关闭即结束 | `data: [DONE]` | `event: message_stop` | +| event 字段 | **不使用** | 不使用 | 使用 | +| 完整响应 | 每个 chunk 都是完整的 GenerateContentResponse | chunk 中只有增量 (delta) | 分事件类型 | + +### 15.3 流式响应示例 + +``` +data: {"candidates":[{"content":{"parts":[{"text":"Hello"}],"role":"model"},"index":0}],"usageMetadata":{"promptTokenCount":7,"totalTokenCount":7},"modelVersion":"gemini-2.5-flash-001"} + +data: {"candidates":[{"content":{"parts":[{"text":"! How"}],"role":"model"},"index":0}],"usageMetadata":{"promptTokenCount":7,"candidatesTokenCount":2,"totalTokenCount":9},"modelVersion":"gemini-2.5-flash-001"} + +data: {"candidates":[{"content":{"parts":[{"text":" can I help"}],"role":"model"},"index":0}],"usageMetadata":{"promptTokenCount":7,"candidatesTokenCount":4,"totalTokenCount":11},"modelVersion":"gemini-2.5-flash-001"} + +data: {"candidates":[{"content":{"parts":[{"text":" you today?"}],"role":"model"},"index":0},"finishReason":"STOP"],"usageMetadata":{"promptTokenCount":7,"candidatesTokenCount":7,"totalTokenCount":14},"modelVersion":"gemini-2.5-flash-001"} +``` + +### 15.4 流式响应解析注意事项 + +1. **每个 chunk 都是完整的 GenerateContentResponse**：不需要像 OpenAI 那样拼接 delta，但需要合并多个 chunk 中的 text Part 来重建完整响应 +2. **无 `[DONE]` 标记**：流结束由 HTTP 连接关闭信号决定 +3. **finishReason 出现在最后一个 chunk**：只有最后一个 chunk 的 candidate 会包含 `finishReason` +4. **usageMetadata 逐步累加**：每个 chunk 的 `usageMetadata` 反映截至当前的累计 Token 用量 + +### 15.5 流式 Function Calling 示例 + +``` +data: {"candidates":[{"content":{"parts":[{"functionCall":{"name":"get_weather","args":{"location":"Beijing"}}}],"role":"model"},"index":0}],"usageMetadata":{"promptTokenCount":50,"totalTokenCount":50},"modelVersion":"gemini-2.5-flash-001"} + +data: {"candidates":[{"content":{"parts":[]},"finishReason":"STOP","index":0}],"usageMetadata":{"promptTokenCount":50,"candidatesTokenCount":10,"totalTokenCount":60},"modelVersion":"gemini-2.5-flash-001"} +``` + +### 15.6 流式思考模式示例 + +``` +data: {"candidates":[{"content":{"parts":[{"thought":"让我分析这个问题..."}],"role":"model"},"index":0}],"usageMetadata":{"promptTokenCount":100,"totalTokenCount":100},"modelVersion":"gemini-2.5-flash-001"} + +data: {"candidates":[{"content":{"parts":[{"thought":"首先需要考虑数学公式..."}],"role":"model"},"index":0}],"usageMetadata":{"promptTokenCount":100,"candidatesTokenCount":15,"totalTokenCount":115},"modelVersion":"gemini-2.5-flash-001"} + +data: {"candidates":[{"content":{"parts":[{"text":"答案是 42。"}],"role":"model"},"index":0},"finishReason":"STOP"],"usageMetadata":{"promptTokenCount":100,"candidatesTokenCount":20,"thoughtsTokenCount":15,"totalTokenCount":135},"modelVersion":"gemini-2.5-flash-001"} +``` + +--- + +## 16. 错误格式 + +### 16.1 错误响应结构 + +Gemini 使用 Google RPC 风格的错误格式： + +```json +{ + "error": { + "code": 400, + "message": "Invalid JSON payload received. Unknown name \"models\": Cannot find field.", + "status": "INVALID_ARGUMENT" + } +} +``` + +### 16.2 错误字段说明 + +| 字段 | 类型 | 说明 | +|------|------|------| +| `error.code` | integer | HTTP 状态码 | +| `error.message` | string | 错误描述信息 | +| `error.status` | string | Google RPC 状态码 | +| `error.details` | object[] | 错误详情（部分错误包含） | + +### 16.3 常见 HTTP 状态码与 RPC 状态码映射 + +| HTTP 状态码 | RPC 状态码 | 说明 | +|------------|-----------|------| +| 400 | `INVALID_ARGUMENT` | 请求参数错误（字段缺失、格式不对、枚举值无效等） | +| 401 | `UNAUTHENTICATED` | 认证失败（API Key 无效或过期） | +| 403 | `PERMISSION_DENIED` | 权限不足（API Key 无权访问该资源） | +| 404 | `NOT_FOUND` | 资源不存在（模型名称错误等） | +| 429 | `RESOURCE_EXHAUSTED` | 请求频率超限或配额耗尽 | +| 429 | `RATE_LIMIT_EXCEEDED` | 速率限制 | +| 500 | `INTERNAL` | 服务器内部错误 | +| 503 | `UNAVAILABLE` | 服务暂时不可用 | +| 504 | `DEADLINE_EXCEEDED` | 请求超时 | + +### 16.4 错误示例 + +**认证失败**： + +```json +{ + "error": { + "code": 401, + "message": "API key not valid. Please pass a valid API key.", + "status": "UNAUTHENTICATED" + } +} +``` + +**模型不存在**： + +```json +{ + "error": { + "code": 404, + "message": "Model `gemini-nonexistent` not found.", + "status": "NOT_FOUND" + } +} +``` + +**请求频率超限**： + +```json +{ + "error": { + "code": 429, + "message": "Quota exceeded for quota metric 'Generate Requests Per Minute' and limit 'GenerateRequestsPerMinutePerProjectPerRegion' of service 'generativelanguage.googleapis.com'.", + "status": "RESOURCE_EXHAUSTED" + } +} +``` + +**安全过滤**： + +```json +{ + "candidates": [ + { + "content": { + "parts": [], + "role": "model" + }, + "finishReason": "SAFETY", + "index": 0 + } + ], + "promptFeedback": { + "blockReason": "SAFETY", + "safetyRatings": [ + { + "category": "HARM_CATEGORY_DANGEROUS_CONTENT", + "probability": "HIGH", + "blocked": true + } + ] + } +} +``` + +### 16.5 与其他协议的错误格式对比 + +| 特性 | Gemini | OpenAI | Claude | +|------|--------|--------|--------| +| 顶层字段 | `error` | `error` | 无顶层 error | +| 错误码字段 | `error.code` (HTTP) | `error.type` (字符串) | `error.type` | +| 消息字段 | `error.message` | `error.message` | `error.message` | +| 状态标识 | `error.status` (RPC) | `error.code` (字符串) | — | +| HTTP 状态码 | 是 | 是 | 是 | +| 包装格式 | `{"error": {...}}` | `{"error": {...}}` | `{"type": "error", "error": {...}}` | + +--- + +## 17. 上下文缓存 (Context Caching) + +上下文缓存允许预存储大量上下文内容（如长文档、系统指令），在后续请求中引用，避免重复处理，降低延迟和费用。 + +### 17.1 CachedContent 对象 + +| 字段 | 类型 | 说明 | +|------|------|------| +| `name` | string | 缓存资源名称（系统生成），格式：`cachedContents/{id}` | +| `model` | string | 模型名称，格式：`models/{model}` | +| `displayName` | string | 可选的显示名称 | +| `contents` | Content[] | 缓存的对话内容 | +| `systemInstruction` | Content | 缓存的系统指令 | +| `tools` | Tool[] | 缓存的工具声明 | +| `toolConfig` | ToolConfig | 缓存的工具配置 | +| `ttl` | string (Duration) | 缓存存活时间，格式如 `"3600s"`（1小时） | +| `expireTime` | string (Timestamp) | 过期时间（RFC 3339 格式） | +| `createTime` | string (Timestamp) | 创建时间（系统生成） | +| `updateTime` | string (Timestamp) | 更新时间（系统生成） | +| `usageMetadata` | CachedContentUsageMetadata | 使用量元数据 | + +### 17.2 创建缓存 + +``` +POST /v1beta/cachedContents +``` + +```json +{ + "model": "models/gemini-2.5-flash", + "contents": [ + { + "role": "user", + "parts": [ + {"text": "以下是一份非常长的技术文档内容...（省略数万字）"} + ] + } + ], + "systemInstruction": { + "parts": [ + {"text": "你是一个专业技术文档分析助手。请基于提供的文档内容回答问题。"} + ] + }, + "ttl": "3600s" +} +``` + +**创建响应**： + +```json +{ + "name": "cachedContents/abc123def456", + "model": "models/gemini-2.5-flash", + "createTime": "2025-01-15T10:00:00.000Z", + "updateTime": "2025-01-15T10:00:00.000Z", + "expireTime": "2025-01-15T11:00:00.000Z", + "usageMetadata": { + "totalTokenCount": 50000 + } +} +``` + +### 17.3 使用缓存 + +在 generateContent 请求中通过 `cachedContent` 字段引用缓存： + +```json +{ + "cachedContent": "cachedContents/abc123def456", + "contents": [ + { + "role": "user", + "parts": [ + {"text": "请总结文档的第三章"} + ] + } + ] +} +``` + +**注意**：使用缓存时，请求体中的 `contents` 是追加到缓存内容之后的新消息。缓存的 `systemInstruction`、`tools` 等也会被继承。 + +### 17.4 获取缓存 + +``` +GET /v1beta/cachedContents/{name} +``` + +### 17.5 更新缓存 + +``` +PATCH /v1beta/cachedContents/{name} +``` + +可更新字段：`ttl`、`expireTime` + +```json +{ + "ttl": "7200s" +} +``` + +### 17.6 删除缓存 + +``` +DELETE /v1beta/cachedContents/{name} +``` + +### 17.7 列出缓存 + +``` +GET /v1beta/cachedContents +``` + +### 17.8 CachedContentUsageMetadata + +| 字段 | 类型 | 说明 | +|------|------|------| +| `totalTokenCount` | integer | 缓存内容的总 Token 数 | + +### 17.9 缓存费用优势 + +使用缓存的 Token 价格低于普通输入 Token 价格。响应中的 `usageMetadata.cachedContentTokenCount` 表示本次请求中从缓存命中的 Token 数。 + +--- + +## 18. 向量嵌入 (Embeddings) + +### 18.1 单条嵌入 + +``` +POST /v1beta/models/{model}:embedContent +``` + +**请求体**： + +```json +{ + "model": "models/text-embedding-004", + "content": { + "parts": [ + {"text": "What is the meaning of life?"} + ] + }, + "taskType": "RETRIEVAL_QUERY", + "title": "Life Question", + "outputDimensionality": 256 +} +``` + +**EmbedContentRequest 字段**： + +| 字段 | 类型 | 必填 | 说明 | +|------|------|------|------| +| `model` | string | **是** | 嵌入模型名称 | +| `content` | Content | **是** | 待嵌入的内容 | +| `taskType` | string (TaskType enum) | 否 | 任务类型 | +| `title` | string | 否 | 内容标题（仅 RETRIEVAL_DOCUMENT 类型有效） | +| `outputDimensionality` | integer | 否 | 输出向量维度 | + +**TaskType 枚举**： + +| 值 | 说明 | +|----|------| +| `RETRIEVAL_QUERY` | 检索查询 | +| `RETRIEVAL_DOCUMENT` | 检索文档 | +| `SEMANTIC_SIMILARITY` | 语义相似度 | +| `CLASSIFICATION` | 分类 | +| `CLUSTERING` | 聚类 | +| `QUESTION_ANSWERING` | 问答 | +| `FACT_VERIFICATION` | 事实验证 | +| `CODE_RETRIEVAL_QUERY` | 代码检索查询 | + +**响应**： + +```json +{ + "embedding": { + "values": [0.013168513, -0.008721383, 0.043494854, ...] + } +} +``` + +### 18.2 批量嵌入 + +``` +POST /v1beta/models/{model}:batchEmbedContents +``` + +**请求体**： + +```json +{ + "requests": [ + { + "model": "models/text-embedding-004", + "content": { + "parts": [{"text": "Hello world"}] + } + }, + { + "model": "models/text-embedding-004", + "content": { + "parts": [{"text": "Goodbye world"}] + } + } + ] +} +``` + +**响应**： + +```json +{ + "embeddings": [ + {"values": [0.013168513, -0.008721383, ...]}, + {"values": [0.005394286, -0.011432873, ...]} + ] +} +``` + +### 18.3 可用嵌入模型 + +| 模型 | 最大输入 Token | 输出维度 | 说明 | +|------|---------------|---------|------| +| `text-embedding-004` | 2,048 | 768（可降维到 256） | 推荐通用模型 | +| `embedding-001` | 2,048 | 768 | 旧版模型 | + +--- + +## 19. Token 计数 API + +### 19.1 计数端点 + +``` +POST /v1beta/models/{model}:countTokens +``` + +### 19.2 请求体 + +请求体格式与 GenerateContentRequest 相同，支持 `contents`、`systemInstruction`、`tools` 等字段： + +```json +{ + "contents": [ + { + "role": "user", + "parts": [ + {"text": "Hello, how are you?"} + ] + } + ], + "systemInstruction": { + "parts": [ + {"text": "You are a helpful assistant."} + ] + }, + "tools": [ + { + "functionDeclarations": [ + { + "name": "get_weather", + "description": "Get weather", + "parameters": { + "type": "OBJECT", + "properties": { + "location": {"type": "STRING"} + } + } + } + ] + } + ] +} +``` + +### 19.3 响应 + +```json +{ + "totalTokens": 42 +} +``` + +**CountTokensResponse 字段**： + +| 字段 | 类型 | 说明 | +|------|------|------| +| `totalTokens` | integer | 输入内容的总 Token 数 | +| `cachedContentTokenCount` | integer | 缓存命中的 Token 数（引用缓存时） | + +--- + +## 20. 结构化输出 (Structured Output) + +### 20.1 通过 responseMimeType 指定 + +最简单的方式是设置 `responseMimeType` 为 `application/json`： + +```json +{ + "generationConfig": { + "responseMimeType": "application/json" + } +} +``` + +模型会输出 JSON 格式的响应，但不保证特定的 Schema。 + +### 20.2 通过 responseSchema 指定 + +配合 `responseMimeType: "application/json"` 使用 `responseSchema` 定义输出结构： + +```json +{ + "generationConfig": { + "responseMimeType": "application/json", + "responseSchema": { + "type": "OBJECT", + "properties": { + "name": {"type": "STRING", "description": "人物姓名"}, + "age": {"type": "INTEGER", "description": "年龄"}, + "hobbies": { + "type": "ARRAY", + "items": {"type": "STRING"} + } + }, + "required": ["name", "age"] + } + } +} +``` + +### 20.3 通过 responseJsonSchema 指定 + +增强版 Schema 定义，支持 `$ref` 引用和严格模式： + +```json +{ + "generationConfig": { + "responseMimeType": "application/json", + "responseJsonSchema": { + "schema": { + "type": "OBJECT", + "properties": { + "name": {"type": "STRING"}, + "address": { + "$ref": "#/$defs/Address" + } + }, + "$defs": { + "Address": { + "type": "OBJECT", + "properties": { + "city": {"type": "STRING"}, + "country": {"type": "STRING"} + } + } + } + } + } + } +} +``` + +### 20.4 与 OpenAI Structured Output 对比 + +| 特性 | Gemini | OpenAI | +|------|--------|--------| +| 方式一 | `responseMimeType: "application/json"` | `response_format: {"type": "json_object"}` | +| 方式二 | `responseMimeType` + `responseSchema` | `response_format: {"type": "json_schema", "json_schema": {...}}` | +| Schema 格式 | Gemini 自定义 Schema 格式 | 标准 JSON Schema | +| 枚举类型 | `type: "STRING"` + `enum: [...]` | `"type": "string"` + `"enum": [...]` | + +--- + +## 21. 服务层级 (Service Tier) + +Gemini API 支持不同的服务层级，影响请求处理优先级和速率限制。 + +### 21.1 服务层级类型 + +| 层级 | 说明 | +|------|------| +| `SERVICE_TIER_UNSPECIFIED` | 未指定，使用默认 | +| `STANDARD` | 标准层级，共享资源池 | +| `FLEX` | 弹性层级，延迟可能较高但成本更低 | +| `PRIORITY` | 优先层级，更高的速率限制和更低的延迟 | + +### 21.2 配置方式 + +通过 `generationConfig.routingConfig` 指定： + +```json +{ + "generationConfig": { + "routingConfig": { + "autoMode": "AUTO", + "manualMode": "PRIORITY" + } + } +} +``` + +--- + +## 22. 模型阶段 (Model Stage) + +Google 模型有不同的生命周期阶段。 + +### 22.1 阶段枚举 + +| 阶段 | 说明 | +|------|------| +| `UNSTABLE_EXPERIMENTAL` | 不稳定的实验阶段，API 可能随时变更 | +| `STABLE_EXPERIMENTAL` | 稳定的实验阶段，API 相对稳定 | +| `PREVIEW` | 预览阶段，功能基本确定 | +| `GA` | 正式发布 (General Availability)，推荐生产使用 | +| `RETIRED` | 已退役，不再可用 | + +### 22.2 主流模型列表 + +| 模型 | 说明 | +|------|------| +| `gemini-2.5-flash` | Gemini 2.5 Flash（推荐，支持 Thinking） | +| `gemini-2.5-pro` | Gemini 2.5 Pro（最强推理能力） | +| `gemini-2.0-flash` | Gemini 2.0 Flash | +| `gemini-2.0-flash-lite` | Gemini 2.0 Flash Lite（轻量版） | +| `gemini-1.5-pro` | Gemini 1.5 Pro（支持超长上下文） | +| `gemini-1.5-flash` | Gemini 1.5 Flash | +| `text-embedding-004` | 文本嵌入模型 | + +--- + +## 23. 协议转换注意事项 + +### 23.1 消息格式转换 + +| 转换方向 | 关键差异 | +|---------|---------| +| **OpenAI → Gemini** | `messages` → `contents`；`role: "system"` → `systemInstruction`；`role: "assistant"` → `role: "model"`；`role: "tool"` → `role: "user"` + `functionResponse` | +| **Claude → Gemini** | `messages` → `contents`；`system` → `systemInstruction`；content block 数组 → parts 数组 | +| **Gemini → OpenAI** | `contents` → `messages`；`systemInstruction` → `role: "system"` message；`role: "model"` → `role: "assistant"`；`functionCall` → `tool_calls`；`functionResponse` → `role: "tool"` | +| **Gemini → Claude** | `contents` → `messages`；`systemInstruction` → `system`；parts 数组 → content blocks | + +### 23.2 工具调用转换 + +**OpenAI Tool Calling → Gemini Function Calling**： + +``` +OpenAI: + tools[].type = "function" + tools[].function.name → functionDeclarations[].name + tools[].function.description → functionDeclarations[].description + tools[].function.parameters → functionDeclarations[].parameters（需转换 Schema 格式） + + 响应： + message.tool_calls[].id → 需自行生成映射 + message.tool_calls[].function.name → functionCall.name + message.tool_calls[].function.arguments (JSON string) → functionCall.args (JSON object) + + 工具结果： + role: "tool", tool_call_id → role: "user", functionResponse.name + content (string) → functionResponse.response (JSON object) +``` + +### 23.3 Schema 格式转换 + +Gemini 的 Schema 格式与标准 JSON Schema 有差异： + +| 标准 JSON Schema | Gemini Schema | 说明 | +|-----------------|---------------|------| +| `"type": "string"` | `"type": "STRING"` | 类型名全大写 | +| `"type": "number"` | `"type": "NUMBER"` | — | +| `"type": "integer"` | `"type": "INTEGER"` | — | +| `"type": "boolean"` | `"type": "BOOLEAN"` | — | +| `"type": "array"` | `"type": "ARRAY"` | — | +| `"type": "object"` | `"type": "OBJECT"` | — | +| `"enum": [...]` | `"enum": [...]` | 相同 | +| 不支持 `anyOf` | `"anyOf": [...]` | Gemini 扩展 | +| `"nullable": true` | `"nullable": true` | 相同 | + +### 23.4 流式响应转换 + +**Gemini SSE → OpenAI SSE**： + +1. 解析 Gemini 的每个 `data:` chunk 为 GenerateContentResponse +2. 提取 `candidates[0].content.parts` 中的文本 +3. 转换为 OpenAI 的 delta 格式： + ```json + {"id":"chatcmpl-xxx","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"Hello"},"finish_reason":null}]} + ``` +4. 最后一个 chunk（含 `finishReason`）添加 `finish_reason` 映射 +5. 结束时发送 `data: [DONE]`（Gemini 原生无此标记，需转换层添加） + +**Gemini SSE → Claude SSE**： + +1. 解析 Gemini chunk +2. 首个 chunk 转换为 `event: message_start` +3. 后续文本 chunk 转换为 `event: content_block_delta` + `{"type":"text_delta","text":"..."}` +4. 思考 chunk 转换为 `event: content_block_delta` + `{"type":"thinking_delta","thinking":"..."}` +5. 最后一个 chunk 转换为 `event: message_delta` + `{"delta":{"stop_reason":"end_turn"}}` +6. 发送 `event: message_stop` + +### 23.5 使用量统计转换 + +| Gemini | OpenAI | Claude | +|--------|--------|--------| +| `usageMetadata.promptTokenCount` | `usage.prompt_tokens` | `usage.input_tokens` | +| `usageMetadata.candidatesTokenCount` | `usage.completion_tokens` | `usage.output_tokens` | +| `usageMetadata.totalTokenCount` | `usage.total_tokens` | —（需计算） | +| `usageMetadata.cachedContentTokenCount` | —（OpenAI 无对应） | `usage.cache_read_input_tokens` | +| `usageMetadata.thoughtsTokenCount` | —（OpenAI 无对应） | `usage.cache_creation_input_tokens`（概念不同） | + +### 23.6 错误格式转换 + +**Gemini → OpenAI 错误格式**： + +```json +// Gemini 原始错误 +{"error": {"code": 400, "message": "...", "status": "INVALID_ARGUMENT"}} + +// 转换为 OpenAI 格式 +{"error": {"type": "invalid_request_error", "message": "...", "param": null, "code": null}} +``` + +**Gemini → Claude 错误格式**： + +```json +// 转换为 Claude 格式 +{"type": "error", "error": {"type": "invalid_request_error", "message": "..."}} +``` + +**错误类型映射**： + +| Gemini error.status | OpenAI error.type | Claude error.type | +|--------------------|--------------------|--------------------| +| `UNAUTHENTICATED` | `authentication_error` | `authentication_error` | +| `PERMISSION_DENIED` | `permission_error` | `permission_error` | +| `INVALID_ARGUMENT` | `invalid_request_error` | `invalid_request_error` | +| `RESOURCE_EXHAUSTED` | `rate_limit_error` | `rate_limit_error` | +| `INTERNAL` | `internal_error` | `api_error` | +| `UNAVAILABLE` | `server_error` | `api_error` | +| `NOT_FOUND` | `invalid_request_error` | `not_found_error` | diff --git a/docs/协议文档/Gemini系列/banana介绍.md b/docs/协议文档/Gemini系列/banana介绍.md new file mode 100644 index 0000000..dcb5f44 --- /dev/null +++ b/docs/协议文档/Gemini系列/banana介绍.md @@ -0,0 +1,2260 @@ +# Nano Banana image generation + +Prompt to prototype fully-functional, UI-complete apps, and see Nano Banana 2 integrated with real-world tools, data, and the Gemini ecosystem. All before writing a single line of code. + +- [Try a Nano Banana 2 app](https://aistudio.google.com/apps/bundled/pet_passport) +- Or build your own from prompts: +- ![magazine](https://storage.googleapis.com/generativeai-downloads/images/magazine-2.jpg) ![london](https://storage.googleapis.com/generativeai-downloads/images/Nano%20Banana%20Pro%20outputs%20for%20docs/05-output.jpg) ![restore](https://storage.googleapis.com/generativeai-downloads/images/quetzal.png) ![banana](https://storage.googleapis.com/generativeai-downloads/images/Nano%20Banana%20Pro%20outputs%20for%20docs/06-output.jpg) ![cafe](https://storage.googleapis.com/generativeai-downloads/images/Nano%20Banana%20Pro%20outputs%20for%20docs/02-a-photo-of-an-everyday-scene-at-a-busy-cafe-servin.jpg) ![article](https://storage.googleapis.com/generativeai-downloads/images/Nano%20Banana%20Pro%20outputs%20for%20docs/10-use-search-to-find-how-the-gemini-3-flash-launch-h.jpg) ![dog](https://storage.googleapis.com/generativeai-downloads/images/Nano%20Banana%20Pro%20outputs%20for%20docs/01-an-icon-representing-a-cute-dog-the-background-is-.jpg) ![isometric](https://storage.googleapis.com/generativeai-downloads/images/isometric-pool.jpg) +- ![magazine](https://storage.googleapis.com/generativeai-downloads/images/magazine-2.jpg) Generated by Nano Banana 2 **Prompt:** "A photo of a glossy magazine cover, the minimal blue cover has the large bold words Nano Banana. The text is in a serif font and fills the view. No other text. In front of the text there is a portrait of a person in a sleek and minimal dress. She is playfully holding the number 2, which is the focal point. + Put the issue number and "Feb 2026" date in the corner along with a barcode. The magazine is on a shelf against an orange plastered wall, within a designer store." Create [professional product shots](https://ai.google.dev/gemini-api/docs/image-generation#4_product_mockups_commercial_photography) in [AI Studio](https://aistudio.google.com/apps?features=chat_based_image_editing) +- ![london](https://storage.googleapis.com/generativeai-downloads/images/Nano%20Banana%20Pro%20outputs%20for%20docs/05-output.jpg) Generated by Nano Banana Pro **Prompt:** "Present a clear, 45° top-down isometric miniature 3D cartoon scene of London, featuring its most iconic landmarks and architectural elements. Use soft, refined textures with realistic PBR materials and gentle, lifelike lighting and shadows. Integrate the current weather conditions directly into the city environment to create an immersive atmospheric mood. Use a clean, minimalistic composition with a soft, solid-colored background. At the top-center, place the title "London" in large bold text, a prominent weather icon beneath it, then the date (small text) and temperature (medium text). All text must be centered with consistent spacing, and may subtly overlap the tops of the buildings." Learn more about [search grounding](https://ai.google.dev/gemini-api/docs/image-generation#use-with-grounding) and try it in [AI Studio](https://aistudio.google.com/apps?features=chat_based_image_editing,search_grounding) +- ![quetzal](https://storage.googleapis.com/generativeai-downloads/images/quetzal.png) Generated by Nano Banana 2 **Prompt:** "Use image search to find accurate images of a resplendent quetzal bird. Create a beautiful 3:2 wallpaper of this bird, with a natural top to bottom gradient and minimal composition." Use Google [Image Search](https://ai.google.dev/gemini-api/docs/image-generation#image-search) grounding with Nano Banana 2. Try it in [AI Studio](https://aistudio.google.com/apps?features=chat_based_image_editing,search_grounding) +- ![banana](https://storage.googleapis.com/generativeai-downloads/images/Nano%20Banana%20Pro%20outputs%20for%20docs/06.jpg) Generated by Nano Banana Pro **Prompt:** "Put this logo on a high-end ad for a banana scented perfume. The logo is perfectly integrated into the bottle." Try Nano Banana's [high fidelity detail preservation](https://ai.google.dev/gemini-api/docs/image-generation#5_high-fidelity_detail_preservation) in [AI Studio](https://aistudio.google.com/apps?features=chat_based_image_editing) +- ![cafe](https://storage.googleapis.com/generativeai-downloads/images/Nano%20Banana%20Pro%20outputs%20for%20docs/02-a-photo-of-an-everyday-scene-at-a-busy-cafe-servin.jpg) Generated by Nano Banana Pro **Prompt:** "A photo of an everyday scene at a busy cafe serving breakfast. In the foreground is an anime man with blue hair, one of the people is a pencil sketch, another is a claymation person" Experiment with different [artistic styles](https://ai.google.dev/gemini-api/docs/image-generation#3_style_transfer) with Nano Banana in [AI Studio](https://aistudio.google.com/apps?features=chat_based_image_editing) +- ![article](https://storage.googleapis.com/generativeai-downloads/images/Nano%20Banana%20Pro%20outputs%20for%20docs/10-use-search-to-find-how-the-gemini-3-flash-launch-h.jpg) Generated by Nano Banana Pro **Prompt:** "Use search to find how the Gemini 3 Flash launch has been received. Use this information to write a short article about it (with headings). Return a photo of the article as it appeared in a design focused glossy magazine. It is a photo of a single folded over page, showing the article about Gemini 3 Flash. One hero photo. Headline in serif." Generate [accurate text](https://ai.google.dev/gemini-api/docs/image-generation#3_accurate_text_in_images) from [search](https://ai.google.dev/gemini-api/docs/image-generation#use-with-grounding). Try Nano Banana in [AI Studio](https://aistudio.google.com/apps?features=chat_based_image_editing,search_grounding) +- ![dog](https://storage.googleapis.com/generativeai-downloads/images/Nano%20Banana%20Pro%20outputs%20for%20docs/01-an-icon-representing-a-cute-dog-the-background-is-.jpg) Generated by Nano Banana Pro **Prompt:** "An icon representing a cute dog. The background is white. Make the icons in a colorful and tactile 3D style. No text." Create [icons, stickers, and assets](https://ai.google.dev/gemini-api/docs/image-generation#2_stylized_illustrations_stickers) with Nano Banana in [AI Studio](https://aistudio.google.com/apps?features=chat_based_image_editing,search_grounding) +- ![isometric](https://storage.googleapis.com/generativeai-downloads/images/isometric-pool.jpg) Generated by Nano Banana 2 **Prompt:** "Make a photo that is perfectly isometric. It is not a miniature, it is a captured photo that just happened to be perfectly isometric. It is a photo of a beautiful modern garden. There's a large 2 shaped pool and the words: Nano Banana 2." Try [photorealistic image generation](https://ai.google.dev/gemini-api/docs/image-generation#1_photorealistic_scenes) in [AI Studio](https://aistudio.google.com/apps?features=chat_based_image_editing) + +**Nano Banana** is the name for Gemini's native image generation capabilities. +Gemini can generate and process images conversationally +with text, images, or a combination of both. This lets you create, edit, and +iterate on visuals with unprecedented control. + +Nano Banana refers to three distinct models available in the Gemini API: + +- **Nano Banana 2** : The [Gemini 3.1 Flash Image Preview](https://ai.google.dev/gemini-api/docs/models/gemini-3.1-flash-image-preview) model (`gemini-3.1-flash-image-preview`). This model serves as the high-efficiency counterpart to Gemini 3 Pro Image, optimized for speed and high-volume developer use cases. +- **Nano Banana Pro** : The [Gemini 3 Pro Image Preview](https://ai.google.dev/gemini-api/docs/models/gemini-3-pro-image-preview) model (`gemini-3-pro-image-preview`). This model is designed for professional asset production, utilizing advanced reasoning ("Thinking") to follow complex instructions and render high-fidelity text. +- **Nano Banana** : The [Gemini 2.5 Flash Image](https://ai.google.dev/gemini-api/docs/models/gemini-2.5-flash-image) model (`gemini-2.5-flash-image`). This model is designed for speed and efficiency, optimized for high-volume, low-latency tasks. + +All generated images include a [SynthID watermark](https://ai.google.dev/responsible/docs/safeguards/synthid). + +## Image generation (text-to-image) + +### Python + + from google import genai + from google.genai import types + from PIL import Image + + client = genai.Client() + + prompt = ("Create a picture of a nano banana dish in a fancy restaurant with a Gemini theme") + response = client.models.generate_content( + model="gemini-3.1-flash-image-preview", + contents=[prompt], + ) + + for part in response.parts: + if part.text is not None: + print(part.text) + elif part.inline_data is not None: + image = part.as_image() + image.save("generated_image.png") + +### JavaScript + + import { GoogleGenAI } from "@google/genai"; + import * as fs from "node:fs"; + + async function main() { + + const ai = new GoogleGenAI({}); + + const prompt = + "Create a picture of a nano banana dish in a fancy restaurant with a Gemini theme"; + + const response = await ai.models.generateContent({ + model: "gemini-3.1-flash-image-preview", + contents: prompt, + }); + for (const part of response.candidates[0].content.parts) { + if (part.text) { + console.log(part.text); + } else if (part.inlineData) { + const imageData = part.inlineData.data; + const buffer = Buffer.from(imageData, "base64"); + fs.writeFileSync("gemini-native-image.png", buffer); + console.log("Image saved as gemini-native-image.png"); + } + } + } + + main(); + +### Go + + package main + + import ( + "context" + "fmt" + "log" + "os" + "google.golang.org/genai" + ) + + func main() { + + ctx := context.Background() + client, err := genai.NewClient(ctx, nil) + if err != nil { + log.Fatal(err) + } + + result, _ := client.Models.GenerateContent( + ctx, + "gemini-3.1-flash-image-preview", + genai.Text("Create a picture of a nano banana dish in a " + + " fancy restaurant with a Gemini theme"), + ) + + for _, part := range result.Candidates[0].Content.Parts { + if part.Text != "" { + fmt.Println(part.Text) + } else if part.InlineData != nil { + imageBytes := part.InlineData.Data + outputFilename := "gemini_generated_image.png" + _ = os.WriteFile(outputFilename, imageBytes, 0644) + } + } + } + +### Java + + import com.google.genai.Client; + import com.google.genai.types.GenerateContentConfig; + import com.google.genai.types.GenerateContentResponse; + import com.google.genai.types.Part; + + import java.io.IOException; + import java.nio.file.Files; + import java.nio.file.Paths; + + public class TextToImage { + public static void main(String[] args) throws IOException { + + try (Client client = new Client()) { + GenerateContentConfig config = GenerateContentConfig.builder() + .responseModalities("TEXT", "IMAGE") + .build(); + + GenerateContentResponse response = client.models.generateContent( + "gemini-3.1-flash-image-preview", + "Create a picture of a nano banana dish in a fancy restaurant with a Gemini theme", + config); + + for (Part part : response.parts()) { + if (part.text().isPresent()) { + System.out.println(part.text().get()); + } else if (part.inlineData().isPresent()) { + var blob = part.inlineData().get(); + if (blob.data().isPresent()) { + Files.write(Paths.get("_01_generated_image.png"), blob.data().get()); + } + } + } + } + } + } + +### REST + + curl -s -X POST \ + "https://generativelanguage.googleapis.com/v1beta/models/gemini-3.1-flash-image-preview:generateContent" \ + -H "x-goog-api-key: $GEMINI_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{ + "contents": [{ + "parts": [ + {"text": "Create a picture of a nano banana dish in a fancy restaurant with a Gemini theme"} + ] + }] + }' + +## Image editing (text-and-image-to-image) + +**Reminder** : Make sure you have the necessary rights to any images you upload. +Don't generate content that infringe on others' rights, including videos or +images that deceive, harass, or harm. Your use of this generative AI service is +subject to our [Prohibited Use Policy](https://policies.google.com/terms/generative-ai/use-policy). + +Provide an image and use text prompts to add, remove, or modify elements, +change the style, or adjust the color grading. + +The following example demonstrates uploading `base64` encoded images. +For multiple images, larger payloads, and supported MIME types, check the [Image +understanding](https://ai.google.dev/gemini-api/docs/image-understanding) page. + +### Python + + from google import genai + from google.genai import types + from PIL import Image + + client = genai.Client() + + prompt = ( + "Create a picture of my cat eating a nano-banana in a " + "fancy restaurant under the Gemini constellation", + ) + + image = Image.open("/path/to/cat_image.png") + + response = client.models.generate_content( + model="gemini-3.1-flash-image-preview", + contents=[prompt, image], + ) + + for part in response.parts: + if part.text is not None: + print(part.text) + elif part.inline_data is not None: + image = part.as_image() + image.save("generated_image.png") + +### JavaScript + + import { GoogleGenAI } from "@google/genai"; + import * as fs from "node:fs"; + + async function main() { + + const ai = new GoogleGenAI({}); + + const imagePath = "path/to/cat_image.png"; + const imageData = fs.readFileSync(imagePath); + const base64Image = imageData.toString("base64"); + + const prompt = [ + { text: "Create a picture of my cat eating a nano-banana in a" + + "fancy restaurant under the Gemini constellation" }, + { + inlineData: { + mimeType: "image/png", + data: base64Image, + }, + }, + ]; + + const response = await ai.models.generateContent({ + model: "gemini-3.1-flash-image-preview", + contents: prompt, + }); + for (const part of response.candidates[0].content.parts) { + if (part.text) { + console.log(part.text); + } else if (part.inlineData) { + const imageData = part.inlineData.data; + const buffer = Buffer.from(imageData, "base64"); + fs.writeFileSync("gemini-native-image.png", buffer); + console.log("Image saved as gemini-native-image.png"); + } + } + } + + main(); + +### Go + + package main + + import ( + "context" + "fmt" + "log" + "os" + "google.golang.org/genai" + ) + + func main() { + + ctx := context.Background() + client, err := genai.NewClient(ctx, nil) + if err != nil { + log.Fatal(err) + } + + imagePath := "/path/to/cat_image.png" + imgData, _ := os.ReadFile(imagePath) + + parts := []*genai.Part{ + genai.NewPartFromText("Create a picture of my cat eating a nano-banana in a fancy restaurant under the Gemini constellation"), + &genai.Part{ + InlineData: &genai.Blob{ + MIMEType: "image/png", + Data: imgData, + }, + }, + } + + contents := []*genai.Content{ + genai.NewContentFromParts(parts, genai.RoleUser), + } + + result, _ := client.Models.GenerateContent( + ctx, + "gemini-3.1-flash-image-preview", + contents, + ) + + for _, part := range result.Candidates[0].Content.Parts { + if part.Text != "" { + fmt.Println(part.Text) + } else if part.InlineData != nil { + imageBytes := part.InlineData.Data + outputFilename := "gemini_generated_image.png" + _ = os.WriteFile(outputFilename, imageBytes, 0644) + } + } + } + +### Java + + import com.google.genai.Client; + import com.google.genai.types.Content; + import com.google.genai.types.GenerateContentConfig; + import com.google.genai.types.GenerateContentResponse; + import com.google.genai.types.Part; + + import java.io.IOException; + import java.nio.file.Files; + import java.nio.file.Path; + import java.nio.file.Paths; + + public class TextAndImageToImage { + public static void main(String[] args) throws IOException { + + try (Client client = new Client()) { + GenerateContentConfig config = GenerateContentConfig.builder() + .responseModalities("TEXT", "IMAGE") + .build(); + + GenerateContentResponse response = client.models.generateContent( + "gemini-3.1-flash-image-preview", + Content.fromParts( + Part.fromText(""" + Create a picture of my cat eating a nano-banana in + a fancy restaurant under the Gemini constellation + """), + Part.fromBytes( + Files.readAllBytes( + Path.of("src/main/resources/cat.jpg")), + "image/jpeg")), + config); + + for (Part part : response.parts()) { + if (part.text().isPresent()) { + System.out.println(part.text().get()); + } else if (part.inlineData().isPresent()) { + var blob = part.inlineData().get(); + if (blob.data().isPresent()) { + Files.write(Paths.get("gemini_generated_image.png"), blob.data().get()); + } + } + } + } + } + } + +### REST + + curl -s -X POST \ + "https://generativelanguage.googleapis.com/v1beta/models/gemini-3.1-flash-image-preview:generateContent" \ + -H "x-goog-api-key: $GEMINI_API_KEY" \ + -H 'Content-Type: application/json' \ + -d "{ + \"contents\": [{ + \"parts\":[ + {\"text\": \"'Create a picture of my cat eating a nano-banana in a fancy restaurant under the Gemini constellation\"}, + { + \"inline_data\": { + \"mime_type\":\"image/jpeg\", + \"data\": \"\" + } + } + ] + }] + }" + +### Multi-turn image editing + +Keep generating and editing images conversationally. Chat or multi-turn +conversation is the recommended way to iterate on images. The following +example shows a prompt to generate an infographic about photosynthesis. + +### Python + + from google import genai + from google.genai import types + + client = genai.Client() + + chat = client.chats.create( + model="gemini-3.1-flash-image-preview", + config=types.GenerateContentConfig( + response_modalities=['TEXT', 'IMAGE'], + tools=[{"google_search": {}}] + ) + ) + + message = "Create a vibrant infographic that explains photosynthesis as if it were a recipe for a plant's favorite food. Show the \"ingredients\" (sunlight, water, CO2) and the \"finished dish\" (sugar/energy). The style should be like a page from a colorful kids' cookbook, suitable for a 4th grader." + + response = chat.send_message(message) + + for part in response.parts: + if part.text is not None: + print(part.text) + elif image:= part.as_image(): + image.save("photosynthesis.png") + +### Javascript + + import { GoogleGenAI } from "@google/genai"; + + const ai = new GoogleGenAI({}); + + async function main() { + const chat = ai.chats.create({ + model: "gemini-3.1-flash-image-preview", + config: { + responseModalities: ['TEXT', 'IMAGE'], + tools: [{googleSearch: {}}], + }, + }); + } + + await main(); + + const message = "Create a vibrant infographic that explains photosynthesis as if it were a recipe for a plant's favorite food. Show the \"ingredients\" (sunlight, water, CO2) and the \"finished dish\" (sugar/energy). The style should be like a page from a colorful kids' cookbook, suitable for a 4th grader." + + let response = await chat.sendMessage({message}); + + for (const part of response.candidates[0].content.parts) { + if (part.text) { + console.log(part.text); + } else if (part.inlineData) { + const imageData = part.inlineData.data; + const buffer = Buffer.from(imageData, "base64"); + fs.writeFileSync("photosynthesis.png", buffer); + console.log("Image saved as photosynthesis.png"); + } + } + +### Go + + package main + + import ( + "context" + "fmt" + "log" + "os" + + "google.golang.org/genai" + ) + + func main() { + ctx := context.Background() + client, err := genai.NewClient(ctx, nil) + if err != nil { + log.Fatal(err) + } + defer client.Close() + + model := client.GenerativeModel("gemini-3.1-flash-image-preview") + model.GenerationConfig = &pb.GenerationConfig{ + ResponseModalities: []pb.ResponseModality{genai.Text, genai.Image}, + } + chat := model.StartChat() + + message := "Create a vibrant infographic that explains photosynthesis as if it were a recipe for a plant's favorite food. Show the \"ingredients\" (sunlight, water, CO2) and the \"finished dish\" (sugar/energy). The style should be like a page from a colorful kids' cookbook, suitable for a 4th grader." + + resp, err := chat.SendMessage(ctx, genai.Text(message)) + if err != nil { + log.Fatal(err) + } + + for _, part := range resp.Candidates[0].Content.Parts { + if txt, ok := part.(genai.Text); ok { + fmt.Printf("%s", string(txt)) + } else if img, ok := part.(genai.ImageData); ok { + err := os.WriteFile("photosynthesis.png", img.Data, 0644) + if err != nil { + log.Fatal(err) + } + } + } + } + +### Java + + import com.google.genai.Chat; + import com.google.genai.Client; + import com.google.genai.types.Content; + import com.google.genai.types.GenerateContentConfig; + import com.google.genai.types.GenerateContentResponse; + import com.google.genai.types.GoogleSearch; + import com.google.genai.types.ImageConfig; + import com.google.genai.types.Part; + import com.google.genai.types.RetrievalConfig; + import com.google.genai.types.Tool; + import com.google.genai.types.ToolConfig; + + import java.io.IOException; + import java.nio.file.Files; + import java.nio.file.Path; + import java.nio.file.Paths; + + public class MultiturnImageEditing { + public static void main(String[] args) throws IOException { + + try (Client client = new Client()) { + + GenerateContentConfig config = GenerateContentConfig.builder() + .responseModalities("TEXT", "IMAGE") + .tools(Tool.builder() + .googleSearch(GoogleSearch.builder().build()) + .build()) + .build(); + + Chat chat = client.chats.create("gemini-3.1-flash-image-preview", config); + + GenerateContentResponse response = chat.sendMessage(""" + Create a vibrant infographic that explains photosynthesis + as if it were a recipe for a plant's favorite food. + Show the "ingredients" (sunlight, water, CO2) + and the "finished dish" (sugar/energy). + The style should be like a page from a colorful + kids' cookbook, suitable for a 4th grader. + """); + + for (Part part : response.parts()) { + if (part.text().isPresent()) { + System.out.println(part.text().get()); + } else if (part.inlineData().isPresent()) { + var blob = part.inlineData().get(); + if (blob.data().isPresent()) { + Files.write(Paths.get("photosynthesis.png"), blob.data().get()); + } + } + } + // ... + } + } + } + +### REST + + curl -s -X POST \ + "https://generativelanguage.googleapis.com/v1beta/models/gemini-3.1-flash-image-preview:generateContent" \ + -H "x-goog-api-key: $GEMINI_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{ + "contents": [{ + "role": "user", + "parts": [ + {"text": "Create a vibrant infographic that explains photosynthesis as if it were a recipe for a plants favorite food. Show the \"ingredients\" (sunlight, water, CO2) and the \"finished dish\" (sugar/energy). The style should be like a page from a colorful kids cookbook, suitable for a 4th grader."} + ] + }], + "generationConfig": { + "responseModalities": ["TEXT", "IMAGE"] + } + }' + +![AI-generated infographic about photosynthesis](https://ai.google.dev/static/gemini-api/docs/images/infographic-eng.png) AI-generated infographic about photosynthesis + +You can then use the same chat to change the language on the graphic to Spanish. + +### Python + + message = "Update this infographic to be in Spanish. Do not change any other elements of the image." + aspect_ratio = "16:9" # "1:1","1:4","1:8","2:3","3:2","3:4","4:1","4:3","4:5","5:4","8:1","9:16","16:9","21:9" + resolution = "2K" # "512", "1K", "2K", "4K" + + response = chat.send_message(message, + config=types.GenerateContentConfig( + image_config=types.ImageConfig( + aspect_ratio=aspect_ratio, + image_size=resolution + ), + )) + + for part in response.parts: + if part.text is not None: + print(part.text) + elif image:= part.as_image(): + image.save("photosynthesis_spanish.png") + +### Javascript + + const message = 'Update this infographic to be in Spanish. Do not change any other elements of the image.'; + const aspectRatio = '16:9'; + const resolution = '2K'; + + let response = await chat.sendMessage({ + message, + config: { + responseModalities: ['TEXT', 'IMAGE'], + imageConfig: { + aspectRatio: aspectRatio, + imageSize: resolution, + }, + tools: [{googleSearch: {}}], + }, + }); + + for (const part of response.candidates[0].content.parts) { + if (part.text) { + console.log(part.text); + } else if (part.inlineData) { + const imageData = part.inlineData.data; + const buffer = Buffer.from(imageData, "base64"); + fs.writeFileSync("photosynthesis2.png", buffer); + console.log("Image saved as photosynthesis2.png"); + } + } + +### Go + + message = "Update this infographic to be in Spanish. Do not change any other elements of the image." + aspect_ratio = "16:9" // "1:1","1:4","1:8","2:3","3:2","3:4","4:1","4:3","4:5","5:4","8:1","9:16","16:9","21:9" + resolution = "2K" // "512", "1K", "2K", "4K" + + model.GenerationConfig.ImageConfig = &pb.ImageConfig{ + AspectRatio: aspect_ratio, + ImageSize: resolution, + } + + resp, err = chat.SendMessage(ctx, genai.Text(message)) + if err != nil { + log.Fatal(err) + } + + for _, part := range resp.Candidates[0].Content.Parts { + if txt, ok := part.(genai.Text); ok { + fmt.Printf("%s", string(txt)) + } else if img, ok := part.(genai.ImageData); ok { + err := os.WriteFile("photosynthesis_spanish.png", img.Data, 0644) + if err != nil { + log.Fatal(err) + } + } + } + +### Java + + String aspectRatio = "16:9"; // "1:1","1:4","1:8","2:3","3:2","3:4","4:1","4:3","4:5","5:4","8:1","9:16","16:9","21:9" + String resolution = "2K"; // "512", "1K", "2K", "4K" + + config = GenerateContentConfig.builder() + .responseModalities("TEXT", "IMAGE") + .imageConfig(ImageConfig.builder() + .aspectRatio(aspectRatio) + .imageSize(resolution) + .build()) + .build(); + + response = chat.sendMessage( + "Update this infographic to be in Spanish. " + + "Do not change any other elements of the image.", + config); + + for (Part part : response.parts()) { + if (part.text().isPresent()) { + System.out.println(part.text().get()); + } else if (part.inlineData().isPresent()) { + var blob = part.inlineData().get(); + if (blob.data().isPresent()) { + Files.write(Paths.get("photosynthesis_spanish.png"), blob.data().get()); + } + } + } + +### REST + + curl -s -X POST \ + "https://generativelanguage.googleapis.com/v1beta/models/gemini-3.1-flash-image-preview:generateContent" \ + -H "x-goog-api-key: $GEMINI_API_KEY" \ + -H 'Content-Type: application/json' \ + -d '{ + "contents": [ + { + "role": "user", + "parts": [{"text": "Create a vibrant infographic that explains photosynthesis..."}] + }, + { + "role": "model", + "parts": [{"inline_data": {"mime_type": "image/png", "data": ""}}] + }, + { + "role": "user", + "parts": [{"text": "Update this infographic to be in Spanish. Do not change any other elements of the image."}] + } + ], + "tools": [{"google_search": {}}], + "generationConfig": { + "responseModalities": ["TEXT", "IMAGE"], + "imageConfig": { + "aspectRatio": "16:9", + "imageSize": "2K" + } + } + }' + +![AI-generated infographic of photosynthesis in Spanish](https://ai.google.dev/static/gemini-api/docs/images/infographic-spanish.png) AI-generated infographic of photosynthesis in Spanish + +## New with Gemini 3 Image models + +Gemini 3 offers state-of-the-art image generation and editing models. Gemini 3.1 +Flash Image is optimized for speed and high-volume use-cases, and Gemini 3 +Pro Image is optimized for professional asset production. +Designed to tackle the most challenging workflows through advanced reasoning, +they excel at complex, multi-turn creation and modification tasks. + +- **High-resolution output** : Built-in generation capabilities for 1K, 2K, and 4K visuals. + - **Gemini 3.1 Flash Image** adds the smaller 512 (0.5K) resolution. +- **Advanced text rendering**: Capable of generating legible, stylized text for infographics, menus, diagrams, and marketing assets. +- **Grounding with Google Search** : The model can use Google Search as a tool to verify facts and generate imagery based on real-time data (e.g., current weather maps, stock charts, recent events). + - **Gemini 3.1 Flash Image** adds the integration of Grounding with Google Search for Images alongside Web Search. +- **Thinking mode**: The model utilizes a "thinking" process to reason through complex prompts. It generates interim "thought images" (visible in the backend but not charged) to refine the composition before producing the final high-quality output. +- **Up to 14 reference images**: You can now mix up to 14 reference images to produce the final image. +- **New aspect ratios** : Gemini 3.1 Flash Image Preview adds 1:4, 4:1, 1:8, and 8:1 [aspect ratios](https://ai.google.dev/gemini-api/docs/image-generation#aspect_ratios_and_image_size). + +### Use up to 14 reference images + +Gemini 3 image models let you to mix up to 14 reference images. These 14 images +can include the following: + +| Gemini 3.1 Flash Image Preview | Gemini 3 Pro Image Preview | +|---|---| +| Up to 10 images of objects with high-fidelity to include in the final image | Up to 6 images of objects with high-fidelity to include in the final image | +| Up to 4 images of characters to maintain character consistency | Up to 5 images of characters to maintain character consistency | + +### Python + + from google import genai + from google.genai import types + from PIL import Image + + prompt = "An office group photo of these people, they are making funny faces." + aspect_ratio = "5:4" # "1:1","1:4","1:8","2:3","3:2","3:4","4:1","4:3","4:5","5:4","8:1","9:16","16:9","21:9" + resolution = "2K" # "512", "1K", "2K", "4K" + + client = genai.Client() + + response = client.models.generate_content( + model="gemini-3.1-flash-image-preview", + contents=[ + prompt, + Image.open('person1.png'), + Image.open('person2.png'), + Image.open('person3.png'), + Image.open('person4.png'), + Image.open('person5.png'), + ], + config=types.GenerateContentConfig( + response_modalities=['TEXT', 'IMAGE'], + image_config=types.ImageConfig( + aspect_ratio=aspect_ratio, + image_size=resolution + ), + ) + ) + + for part in response.parts: + if part.text is not None: + print(part.text) + elif image:= part.as_image(): + image.save("office.png") + +### Javascript + + import { GoogleGenAI } from "@google/genai"; + import * as fs from "node:fs"; + + async function main() { + + const ai = new GoogleGenAI({}); + + const prompt = + 'An office group photo of these people, they are making funny faces.'; + const aspectRatio = '5:4'; + const resolution = '2K'; + + const contents = [ + { text: prompt }, + { + inlineData: { + mimeType: "image/jpeg", + data: base64ImageFile1, + }, + }, + { + inlineData: { + mimeType: "image/jpeg", + data: base64ImageFile2, + }, + }, + { + inlineData: { + mimeType: "image/jpeg", + data: base64ImageFile3, + }, + }, + { + inlineData: { + mimeType: "image/jpeg", + data: base64ImageFile4, + }, + }, + { + inlineData: { + mimeType: "image/jpeg", + data: base64ImageFile5, + }, + } + ]; + + const response = await ai.models.generateContent({ + model: 'gemini-3.1-flash-image-preview', + contents: contents, + config: { + responseModalities: ['TEXT', 'IMAGE'], + imageConfig: { + aspectRatio: aspectRatio, + imageSize: resolution, + }, + }, + }); + + for (const part of response.candidates[0].content.parts) { + if (part.text) { + console.log(part.text); + } else if (part.inlineData) { + const imageData = part.inlineData.data; + const buffer = Buffer.from(imageData, "base64"); + fs.writeFileSync("image.png", buffer); + console.log("Image saved as image.png"); + } + } + + } + + main(); + +### Go + + package main + + import ( + "context" + "fmt" + "log" + "os" + + "google.golang.org/genai" + ) + + func main() { + ctx := context.Background() + client, err := genai.NewClient(ctx, nil) + if err != nil { + log.Fatal(err) + } + defer client.Close() + + model := client.GenerativeModel("gemini-3.1-flash-image-preview") + model.GenerationConfig = &pb.GenerationConfig{ + ResponseModalities: []pb.ResponseModality{genai.Text, genai.Image}, + ImageConfig: &pb.ImageConfig{ + AspectRatio: "5:4", + ImageSize: "2K", + }, + } + + img1, err := os.ReadFile("person1.png") + if err != nil { log.Fatal(err) } + img2, err := os.ReadFile("person2.png") + if err != nil { log.Fatal(err) } + img3, err := os.ReadFile("person3.png") + if err != nil { log.Fatal(err) } + img4, err := os.ReadFile("person4.png") + if err != nil { log.Fatal(err) } + img5, err := os.ReadFile("person5.png") + if err != nil { log.Fatal(err) } + + parts := []genai.Part{ + genai.Text("An office group photo of these people, they are making funny faces."), + genai.ImageData{MIMEType: "image/png", Data: img1}, + genai.ImageData{MIMEType: "image/png", Data: img2}, + genai.ImageData{MIMEType: "image/png", Data: img3}, + genai.ImageData{MIMEType: "image/png", Data: img4}, + genai.ImageData{MIMEType: "image/png", Data: img5}, + } + + resp, err := model.GenerateContent(ctx, parts...) + if err != nil { + log.Fatal(err) + } + + for _, part := range resp.Candidates[0].Content.Parts { + if txt, ok := part.(genai.Text); ok { + fmt.Printf("%s", string(txt)) + } else if img, ok := part.(genai.ImageData); ok { + err := os.WriteFile("office.png", img.Data, 0644) + if err != nil { + log.Fatal(err) + } + } + } + } + +### Java + + import com.google.genai.Client; + import com.google.genai.types.Content; + import com.google.genai.types.GenerateContentConfig; + import com.google.genai.types.GenerateContentResponse; + import com.google.genai.types.ImageConfig; + import com.google.genai.types.Part; + + import java.io.IOException; + import java.nio.file.Files; + import java.nio.file.Path; + import java.nio.file.Paths; + + public class GroupPhoto { + public static void main(String[] args) throws IOException { + + try (Client client = new Client()) { + GenerateContentConfig config = GenerateContentConfig.builder() + .responseModalities("TEXT", "IMAGE") + .imageConfig(ImageConfig.builder() + .aspectRatio("5:4") + .imageSize("2K") + .build()) + .build(); + + GenerateContentResponse response = client.models.generateContent( + "gemini-3.1-flash-image-preview", + Content.fromParts( + Part.fromText("An office group photo of these people, they are making funny faces."), + Part.fromBytes(Files.readAllBytes(Path.of("person1.png")), "image/png"), + Part.fromBytes(Files.readAllBytes(Path.of("person2.png")), "image/png"), + Part.fromBytes(Files.readAllBytes(Path.of("person3.png")), "image/png"), + Part.fromBytes(Files.readAllBytes(Path.of("person4.png")), "image/png"), + Part.fromBytes(Files.readAllBytes(Path.of("person5.png")), "image/png") + ), config); + + for (Part part : response.parts()) { + if (part.text().isPresent()) { + System.out.println(part.text().get()); + } else if (part.inlineData().isPresent()) { + var blob = part.inlineData().get(); + if (blob.data().isPresent()) { + Files.write(Paths.get("office.png"), blob.data().get()); + } + } + } + } + } + } + +### REST + + curl -s -X POST \ + "https://generativelanguage.googleapis.com/v1beta/models/gemini-3.1-flash-image-preview:generateContent" \ + -H "x-goog-api-key: $GEMINI_API_KEY" \ + -H 'Content-Type: application/json' \ + -d "{ + \"contents\": [{ + \"parts\":[ + {\"text\": \"An office group photo of these people, they are making funny faces.\"}, + {\"inline_data\": {\"mime_type\":\"image/png\", \"data\": \"\"}}, + {\"inline_data\": {\"mime_type\":\"image/png\", \"data\": \"\"}}, + {\"inline_data\": {\"mime_type\":\"image/png\", \"data\": \"\"}}, + {\"inline_data\": {\"mime_type\":\"image/png\", \"data\": \"\"}}, + {\"inline_data\": {\"mime_type\":\"image/png\", \"data\": \"\"}} + ] + }], + \"generationConfig\": { + \"responseModalities\": [\"TEXT\", \"IMAGE\"], + \"imageConfig\": { + \"aspectRatio\": \"5:4\", + \"imageSize\": \"2K\" + } + } + }" + +![AI-generated office group photo](https://ai.google.dev/static/gemini-api/docs/images/office-group-photo.jpeg) AI-generated office group photo + +### Grounding with Google Search + +Use the [Google Search tool](https://ai.google.dev/gemini-api/docs/google-search) to generate images +based on real-time information, such as weather forecasts, stock charts, or +recent events. + +Note that when using Grounding with Google Search with image generation, +image-based search results are not passed to the generation model and are +excluded from the response (see [Grounding with Google Search for images](https://ai.google.dev/gemini-api/docs/image-generation#image-search)) + +### Python + + from google import genai + prompt = "Visualize the current weather forecast for the next 5 days in San Francisco as a clean, modern weather chart. Add a visual on what I should wear each day" + aspect_ratio = "16:9" # "1:1","1:4","1:8","2:3","3:2","3:4","4:1","4:3","4:5","5:4","8:1","9:16","16:9","21:9" + + client = genai.Client() + + response = client.models.generate_content( + model="gemini-3.1-flash-image-preview", + contents=prompt, + config=types.GenerateContentConfig( + response_modalities=['Text', 'Image'], + image_config=types.ImageConfig( + aspect_ratio=aspect_ratio, + ), + tools=[{"google_search": {}}] + ) + ) + + for part in response.parts: + if part.text is not None: + print(part.text) + elif image:= part.as_image(): + image.save("weather.png") + +### Javascript + + import { GoogleGenAI } from "@google/genai"; + import * as fs from "node:fs"; + + async function main() { + + const ai = new GoogleGenAI({}); + + const prompt = 'Visualize the current weather forecast for the next 5 days in San Francisco as a clean, modern weather chart. Add a visual on what I should wear each day'; + const aspectRatio = '16:9'; + const resolution = '2K'; + + const response = await ai.models.generateContent({ + model: 'gemini-3.1-flash-image-preview', + contents: prompt, + config: { + responseModalities: ['TEXT', 'IMAGE'], + imageConfig: { + aspectRatio: aspectRatio, + imageSize: resolution, + }, + tools: [{ googleSearch: {} }] + }, + }); + + for (const part of response.candidates[0].content.parts) { + if (part.text) { + console.log(part.text); + } else if (part.inlineData) { + const imageData = part.inlineData.data; + const buffer = Buffer.from(imageData, "base64"); + fs.writeFileSync("image.png", buffer); + console.log("Image saved as image.png"); + } + } + + } + + main(); + +### Java + + import com.google.genai.Client; + import com.google.genai.types.GenerateContentConfig; + import com.google.genai.types.GenerateContentResponse; + import com.google.genai.types.GoogleSearch; + import com.google.genai.types.ImageConfig; + import com.google.genai.types.Part; + import com.google.genai.types.Tool; + + import java.io.IOException; + import java.nio.file.Files; + import java.nio.file.Paths; + + public class SearchGrounding { + public static void main(String[] args) throws IOException { + + try (Client client = new Client()) { + GenerateContentConfig config = GenerateContentConfig.builder() + .responseModalities("TEXT", "IMAGE") + .imageConfig(ImageConfig.builder() + .aspectRatio("16:9") + .build()) + .tools(Tool.builder() + .googleSearch(GoogleSearch.builder().build()) + .build()) + .build(); + + GenerateContentResponse response = client.models.generateContent( + "gemini-3.1-flash-image-preview", """ + Visualize the current weather forecast for the next 5 days + in San Francisco as a clean, modern weather chart. + Add a visual on what I should wear each day + """, + config); + + for (Part part : response.parts()) { + if (part.text().isPresent()) { + System.out.println(part.text().get()); + } else if (part.inlineData().isPresent()) { + var blob = part.inlineData().get(); + if (blob.data().isPresent()) { + Files.write(Paths.get("weather.png"), blob.data().get()); + } + } + } + } + } + } + +### REST + + curl -s -X POST \ + "https://generativelanguage.googleapis.com/v1beta/models/gemini-3.1-flash-image-preview:generateContent" \ + -H "x-goog-api-key: $GEMINI_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{ + "contents": [{"parts": [{"text": "Visualize the current weather forecast for the next 5 days in San Francisco as a clean, modern weather chart. Add a visual on what I should wear each day"}]}], + "tools": [{"google_search": {}}], + "generationConfig": { + "responseModalities": ["TEXT", "IMAGE"], + "imageConfig": {"aspectRatio": "16:9"} + } + }' + +![AI-generated five day weather chart for San Francisco](https://ai.google.dev/static/gemini-api/docs/images/weather-forecast.png) AI-generated five day weather chart for San Francisco + +The response includes `groundingMetadata` which contains the following required +fields: + +- **`searchEntryPoint`**: Contains the HTML and CSS to render the required search suggestions. +- **`groundingChunks`**: Returns the top 3 web sources used to ground the generated image + +### Grounding with Google Search for Images (3.1 Flash) + +> [!NOTE] +> **Note:** This feature is only available for the Gemini 3.1 Flash Image model. + +Grounding with Google Search for images allows models to use web images retrieved via +Google Search as visual context for image generation. Image Search is a +new search type within the existing Grounding with Google Search tool, +functioning alongside standard [Web Search](https://ai.google.dev/gemini-api/docs/image-generation#use-with-grounding). + +To enable Image Search, configure the `googleSearch` tool in your API request +and specify `imageSearch` within the `searchTypes` object. Image Search can be +used independently or together with Web Search. + +Note that Grounding with Google Search for images can't be used to search for people. + +### Python + + from google import genai + prompt = "A detailed painting of a Timareta butterfly resting on a flower" + + client = genai.Client() + + response = client.models.generate_content( + model="gemini-3.1-flash-image-preview", + contents=prompt, + config=types.GenerateContentConfig( + response_modalities=["IMAGE"], + tools=[ + types.Tool(google_search=types.GoogleSearch( + search_types=types.SearchTypes( + web_search=types.WebSearch(), + image_search=types.ImageSearch() + ) + )) + ] + ) + ) + + # Display grounding sources if available + if response.candidates and response.candidates[0].grounding_metadata and response.candidates[0].grounding_metadata.search_entry_point: + display(HTML(response.candidates[0].grounding_metadata.search_entry_point.rendered_content)) + +### JavaScript + + import { GoogleGenAI } from "@google/genai"; + + async function main() { + + const ai = new GoogleGenAI({}); + + const prompt = "A detailed painting of a Timareta butterfly resting on a flower"; + + const response = await ai.models.generateContent({ + model: "gemini-3.1-flash-image-preview", + contents: prompt, + config: { + responseModalities: ["IMAGE"], + tools: [ + { + googleSearch: { + searchTypes: { + webSearch: {}, + imageSearch: {} + } + } + } + ] + } + }); + + // Display grounding sources if available + if (response.candidates && response.candidates[0].groundingMetadata && response.candidates[0].groundingMetadata.searchEntryPoint) { + console.log(response.candidates[0].groundingMetadata.searchEntryPoint.renderedContent); + } + } + + main(); + +### Go + + package main + + import ( + "context" + "fmt" + "log" + + "google.golang.org/genai" + pb "google.golang.org/genai/schema" + ) + + func main() { + ctx := context.Background() + client, err := genai.NewClient(ctx, nil) + if err != nil { + log.Fatal(err) + } + defer client.Close() + + model := client.GenerativeModel("gemini-3.1-flash-image-preview") + model.Tools = []*pb.Tool{ + { + GoogleSearch: &pb.GoogleSearch{ + SearchTypes: &pb.SearchTypes{ + WebSearch: &pb.WebSearch{}, + ImageSearch: &pb.ImageSearch{}, + }, + }, + }, + } + model.GenerationConfig = &pb.GenerationConfig{ + ResponseModalities: []pb.ResponseModality{genai.Image}, + } + + prompt := "A detailed painting of a Timareta butterfly resting on a flower" + resp, err := model.GenerateContent(ctx, genai.Text(prompt)) + if err != nil { + log.Fatal(err) + } + + if resp.Candidates[0].GroundingMetadata != nil && resp.Candidates[0].GroundingMetadata.SearchEntryPoint != nil { + fmt.Println(resp.Candidates[0].GroundingMetadata.SearchEntryPoint.RenderedContent) + } + } + +### REST + + curl -s -X POST \ + "https://generativelanguage.googleapis.com/v1beta/models/gemini-3.1-flash-image-preview:generateContent" \ + -H "x-goog-api-key: $GEMINI_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{ + "contents": [{"parts": [{"text": "A detailed painting of a Timareta butterfly resting on a flower"}]}], + "tools": [{"google_search": {"searchTypes": {"webSearch": {}, "imageSearch": {}}}}], + "generationConfig": { + "responseModalities": ["IMAGE"] + } + }' + +**Display requirements** + +When you use Image Search within Grounding with Google Search, you must comply +with the following conditions: + +- **Source attribution**: You must provide a link to the webpage containing the source image (the "containing page," not the image file itself) in a manner that the user will recognize as a link. +- **Direct navigation**: If you also choose to display the source images, you must provide a direct, single-click path from the source images to its containing source webpage. Any other implementation that delays or abstracts the end user's access to the source webpage, including but not limited to any multi-click path or the use of an intermediate image viewer, is not permitted. + +**Response** + +For grounded responses using image search, the API provides clear attribution +and metadata to link its output to verified sources. Key fields in the +`groundingMetadata` object include: + +- **`imageSearchQueries`**: The specific queries used by the model for visual context (image search). +- **`groundingChunks`**: Contains source information for retrieved results. + For image sources, these will be returned as redirect URLs using a new image + chunk type. This chunk includes: + + - **`uri`**: The web page URL for attribution (the landing page). + - **`image_uri`**: The direct image URL. +- **`groundingSupports`**: Provides specific mappings that link the generated + content to its relevant citation source in the chunks. + +- **`searchEntryPoint`**: Includes the "Google Search" chip containing + compliant HTML and CSS to render Search Suggestions. + +### Generate images up to 4K resolution + +Gemini 3 image models generate 1K images by default but can also output 2K, +4K, and 512 (0.5K) (Gemini 3.1 Flash Image only) images. To generate higher +resolution assets, specify the `image_size` in the `generation_config`. + +You must use an uppercase 'K' (e.g. 1K, 2K, 4K). The `512` value does not use a 'K' suffix. Lowercase +parameters (e.g., 1k) will be rejected. + +### Python + + from google import genai + from google.genai import types + + prompt = "Da Vinci style anatomical sketch of a dissected Monarch butterfly. Detailed drawings of the head, wings, and legs on textured parchment with notes in English." + aspect_ratio = "1:1" # "1:1","1:4","1:8","2:3","3:2","3:4","4:1","4:3","4:5","5:4","8:1","9:16","16:9","21:9" + resolution = "1K" # "512", "1K", "2K", "4K" + + client = genai.Client() + + response = client.models.generate_content( + model="gemini-3.1-flash-image-preview", + contents=prompt, + config=types.GenerateContentConfig( + response_modalities=['TEXT', 'IMAGE'], + image_config=types.ImageConfig( + aspect_ratio=aspect_ratio, + image_size=resolution + ), + ) + ) + + for part in response.parts: + if part.text is not None: + print(part.text) + elif image:= part.as_image(): + image.save("butterfly.png") + +### Javascript + + import { GoogleGenAI } from "@google/genai"; + import * as fs from "node:fs"; + + async function main() { + + const ai = new GoogleGenAI({}); + + const prompt = + 'Da Vinci style anatomical sketch of a dissected Monarch butterfly. Detailed drawings of the head, wings, and legs on textured parchment with notes in English.'; + const aspectRatio = '1:1'; + const resolution = '1K'; + + const response = await ai.models.generateContent({ + model: 'gemini-3.1-flash-image-preview', + contents: prompt, + config: { + responseModalities: ['TEXT', 'IMAGE'], + imageConfig: { + aspectRatio: aspectRatio, + imageSize: resolution, + }, + }, + }); + + for (const part of response.candidates[0].content.parts) { + if (part.text) { + console.log(part.text); + } else if (part.inlineData) { + const imageData = part.inlineData.data; + const buffer = Buffer.from(imageData, "base64"); + fs.writeFileSync("image.png", buffer); + console.log("Image saved as image.png"); + } + } + + } + + main(); + +### Go + + package main + + import ( + "context" + "fmt" + "log" + "os" + + "google.golang.org/genai" + ) + + func main() { + ctx := context.Background() + client, err := genai.NewClient(ctx, nil) + if err != nil { + log.Fatal(err) + } + defer client.Close() + + model := client.GenerativeModel("gemini-3.1-flash-image-preview") + model.GenerationConfig = &pb.GenerationConfig{ + ResponseModalities: []pb.ResponseModality{genai.Text, genai.Image}, + ImageConfig: &pb.ImageConfig{ + AspectRatio: "1:1", + ImageSize: "1K", + }, + } + + prompt := "Da Vinci style anatomical sketch of a dissected Monarch butterfly. Detailed drawings of the head, wings, and legs on textured parchment with notes in English." + resp, err := model.GenerateContent(ctx, genai.Text(prompt)) + if err != nil { + log.Fatal(err) + } + + for _, part := range resp.Candidates[0].Content.Parts { + if txt, ok := part.(genai.Text); ok { + fmt.Printf("%s", string(txt)) + } else if img, ok := part.(genai.ImageData); ok { + err := os.WriteFile("butterfly.png", img.Data, 0644) + if err != nil { + log.Fatal(err) + } + } + } + } + +### Java + + import com.google.genai.Client; + import com.google.genai.types.GenerateContentConfig; + import com.google.genai.types.GenerateContentResponse; + import com.google.genai.types.GoogleSearch; + import com.google.genai.types.ImageConfig; + import com.google.genai.types.Part; + import com.google.genai.types.Tool; + + import java.io.IOException; + import java.nio.file.Files; + import java.nio.file.Paths; + + public class HiRes { + public static void main(String[] args) throws IOException { + + try (Client client = new Client()) { + GenerateContentConfig config = GenerateContentConfig.builder() + .responseModalities("TEXT", "IMAGE") + .imageConfig(ImageConfig.builder() + .aspectRatio("16:9") + .imageSize("4K") + .build()) + .build(); + + GenerateContentResponse response = client.models.generateContent( + "gemini-3.1-flash-image-preview", """ + Da Vinci style anatomical sketch of a dissected Monarch butterfly. + Detailed drawings of the head, wings, and legs on textured + parchment with notes in English. + """, + config); + + for (Part part : response.parts()) { + if (part.text().isPresent()) { + System.out.println(part.text().get()); + } else if (part.inlineData().isPresent()) { + var blob = part.inlineData().get(); + if (blob.data().isPresent()) { + Files.write(Paths.get("butterfly.png"), blob.data().get()); + } + } + } + } + } + } + +### REST + + curl -s -X POST \ + "https://generativelanguage.googleapis.com/v1beta/models/gemini-3.1-flash-image-preview:generateContent" \ + -H "x-goog-api-key: $GEMINI_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{ + "contents": [{"parts": [{"text": "Da Vinci style anatomical sketch of a dissected Monarch butterfly. Detailed drawings of the head, wings, and legs on textured parchment with notes in English."}]}], + "tools": [{"google_search": {}}], + "generationConfig": { + "responseModalities": ["TEXT", "IMAGE"], + "imageConfig": {"aspectRatio": "1:1", "imageSize": "1K"} + } + }' + +The following is an example image generated from this prompt: +![AI-generated Da Vinci style anatomical sketch of a dissected Monarch butterfly.](https://ai.google.dev/static/gemini-api/docs/images/gemini3-4k-image.png) AI-generated Da Vinci style anatomical sketch of a dissected Monarch butterfly. + +### Thinking Process + +Gemini 3 image models are thinking models that use a reasoning +process ("Thinking") for complex prompts. This feature is enabled by default and +cannot be disabled in the API. To learn more about the thinking process, see +the [Gemini Thinking](https://ai.google.dev/gemini-api/docs/thinking) guide. + +The model generates up to two interim images to test composition and logic. The +last image within Thinking is also the final rendered image. + +You can check the thoughts that lead to the final image being produced. + +### Python + + for part in response.parts: + if part.thought: + if part.text: + print(part.text) + elif image:= part.as_image(): + image.show() + +### Javascript + + for (const part of response.candidates[0].content.parts) { + if (part.thought) { + if (part.text) { + console.log(part.text); + } else if (part.inlineData) { + const imageData = part.inlineData.data; + const buffer = Buffer.from(imageData, 'base64'); + fs.writeFileSync('image.png', buffer); + console.log('Image saved as image.png'); + } + } + } + +#### Controlling thinking levels + +With Gemini 3.1 Flash Image, you can control the amount of thinking the model +uses to balance quality and latency. The default `thinkingLevel` is `minimal`, +and the supported levels are `minimal` and `high`. Setting the +`thinkingLevel` to `minimal` provides the lowest latency responses. Note that +minimal thinking does not mean the model uses no thinking at all. + +You can add the `includeThoughts` boolean to determine whether the model's +generated thoughts are returned in the response, or remain hidden. + +### Python + + from google import genai + + response = client.models.generate_content( + model="gemini-3.1-flash-image-preview", + contents="A futuristic city built inside a giant glass bottle floating in space", + config=types.GenerateContentConfig( + response_modalities=["IMAGE"], + thinking_config=types.ThinkingConfig( + thinking_level="High", + include_thoughts=True + ), + ) + ) + + for part in response.parts: + if part.thought: # Skip outputting thoughts + continue + if part.text: + display(Markdown(part.text)) + elif image:= part.as_image(): + image.show() + +### JavaScript + + import { GoogleGenAI } from "@google/genai"; + import * as fs from "node:fs"; + + async function main() { + + const ai = new GoogleGenAI({}); + + const response = await ai.models.generateContent({ + model: "gemini-3.1-flash-image-preview", + contents: "A futuristic city built inside a giant glass bottle floating in space", + config: { + responseModalities: ["IMAGE"], + thinkingConfig: { + thinkingLevel: "High", + includeThoughts: true + }, + }, + }); + + for (const part of response.candidates[0].content.parts) { + if (part.thought) { // Skip outputting thoughts + continue; + } + if (part.text) { + console.log(part.text); + } else if (part.inlineData) { + const imageData = part.inlineData.data; + const buffer = Buffer.from(imageData, "base64"); + fs.writeFileSync("image.png", buffer); + console.log("Image saved as image.png"); + } + } + } + main(); + +### Go + + package main + + import ( + "context" + "fmt" + "log" + "os" + + "google.golang.org/genai" + pb "google.golang.org/genai/schema" + ) + + func main() { + ctx := context.Background() + client, err := genai.NewClient(ctx, nil) + if err != nil { + log.Fatal(err) + } + defer client.Close() + + model := client.GenerativeModel("gemini-3.1-flash-image-preview") + model.GenerationConfig = &pb.GenerationConfig{ + ResponseModalities: []pb.ResponseModality{genai.Image}, + ThinkingConfig: &pb.ThinkingConfig{ + ThinkingLevel: "High", + IncludeThoughts: true, + }, + } + + prompt := "A futuristic city built inside a giant glass bottle floating in space" + resp, err := model.GenerateContent(ctx, genai.Text(prompt)) + if err != nil { + log.Fatal(err) + } + + for _, part := range resp.Candidates[0].Content.Parts { + if part.Thought { // Skip outputting thoughts + continue + } + if txt, ok := part.(genai.Text); ok { + fmt.Printf("%s", string(txt)) + } else if img, ok := part.(genai.ImageData); ok { + err := os.WriteFile("image.png", img.Data, 0644) + if err != nil { + log.Fatal(err) + } + } + } + } + +### REST + + curl -s -X POST \ + "https://generativelanguage.googleapis.com/v1beta/models/gemini-3.1-flash-image-preview:generateContent" \ + -H "x-goog-api-key: $GEMINI_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{ + "contents": [{"parts": [{"text": "A futuristic city built inside a giant glass bottle floating in space"}]}], + "generationConfig": { + "responseModalities": ["IMAGE"], + "thinkingConfig": { + "thinkingLevel": "High", + "includeThoughts": true + } + } + }' + +Note that thinking tokens are billed regardless of whether `includeThoughts` is +set to `true` or `false`, as the [thinking process](https://ai.google.dev/gemini-api/docs/image-generation#thinking-process) always +happens by default whether you view the process or not. + +#### Thought Signatures + +Thought signatures are encrypted representations of the +model's internal thought process and are used to preserve reasoning context +across multi-turn interactions. All responses include a `thought_signature` +field. As a general rule, if you receive a thought signature in a model +response, you should pass it back exactly as received when sending the +conversation history in the next turn. Failure to circulate thought signatures +may cause the response to fail. Check the [thought signature](https://ai.google.dev/gemini-api/docs/thought-signatures) +documentation for more explanations of signatures overall. + +> [!NOTE] +> **Note:** If you use the official [Google Gen AI SDKs](https://ai.google.dev/gemini-api/docs/libraries) and use the chat feature (or append the full model response object directly to history), **thought signatures are handled automatically**. You do not need to manually extract or manage them, or change your code. + +Here is how thought signatures work: + +- All `inline_data` parts with image `mimetype` which are part of the response should have signature. +- If there are some text parts at the beginning (before any image) right after the thoughts, the first text part should also have a signature. +- If `inline_data` parts with image `mimetype` are part of thoughts, they won't have signatures. + +The following code shows an example of where thought signatures are included: + + [ + { + "inline_data": { + "data": "", + "mime_type": "image/png" + }, + "thought": true // Thoughts don't have signatures + }, + { + "inline_data": { + "data": "", + "mime_type": "image/png" + }, + "thought": true // Thoughts don't have signatures + }, + { + "inline_data": { + "data": "", + "mime_type": "image/png" + }, + "thought": true // Thoughts don't have signatures + }, + { + "text": "Here is a step-by-step guide to baking macarons, presented in three separate images.\n\n### Step 1: Piping the Batter\n\nThe first step after making your macaron batter is to pipe it onto a baking sheet. This requires a steady hand to create uniform circles.\n\n", + "thought_signature": "" // The first non-thought part always has a signature + }, + { + "inline_data": { + "data": "", + "mime_type": "image/png" + }, + "thought_signature": "" // All image parts have a signatures + }, + { + "text": "\n\n### Step 2: Baking and Developing Feet\n\nOnce piped, the macarons are baked in the oven. A key sign of a successful bake is the development of \"feet\"---the ruffled edge at the base of each macaron shell.\n\n" + // Follow-up text parts don't have signatures + }, + { + "inline_data": { + "data": "", + "mime_type": "image/png" + }, + "thought_signature": "" // All image parts have a signatures + }, + { + "text": "\n\n### Step 3: Assembling the Macaron\n\nThe final step is to pair the cooled macaron shells by size and sandwich them together with your desired filling, creating the classic macaron dessert.\n\n" + }, + { + "inline_data": { + "data": "", + "mime_type": "image/png" + }, + "thought_signature": "" // All image parts have a signatures + } + ] + +## Other image generation modes + +Gemini supports other image interaction modes based on prompt structure and +context, including: + +- **Text to image(s) and text (interleaved):** Outputs images with related text. + - Example prompt: "Generate an illustrated recipe for a paella." +- **Image(s) and text to image(s) and text (interleaved)** : Uses input images and text to create new related images and text. + - Example prompt: (With an image of a furnished room) "What other color sofas would work in my space? can you update the image?" + +## Generate images in batch + +If you need to generate a lot of images, you can use the +[Batch API](https://ai.google.dev/gemini-api/docs/batch-api). You get higher +[rate limits](https://ai.google.dev/gemini-api/docs/rate-limits) in exchange for a turnaround of up +to 24 hours. + +Check the [Batch API image generation documentation](https://ai.google.dev/gemini-api/docs/batch-api#image-generation) and the [cookbook](https://colab.research.google.com/github/google-gemini/cookbook/blob/main/quickstarts/Batch_mode.ipynb) +for Batch API image examples and code. + +## Prompting guide and strategies + +Mastering image generation starts with one fundamental principle: +> **Describe the scene, don't just list keywords.** +> The model's core strength is its deep language understanding. A narrative, +> descriptive paragraph will almost always produce a better, more coherent image +> than a list of disconnected words. + +### Prompts for generating images + +The following strategies will help you create effective prompts to +generate exactly the images you're looking for. + +#### Photography + +For realistic images, use photography terms. Mention camera angles, lens types, +lighting, and fine details to guide the model toward a realistic result. + +| **Prompt** | **Generated output** | +|---|---| +| A photo of a close-up portrait of an elderly Japanese ceramicist with deep, sun-etched wrinkles and a warm, knowing smile. He is carefully inspecting a freshly glazed tea bowl. The setting is his rustic, sun-drenched workshop. The scene is illuminated by soft, golden hour light streaming through a window, highlighting the fine texture of the clay. Captured with an 85mm portrait lens, resulting in a soft, blurred background (bokeh). The overall mood is serene and masterful. Vertical portrait orientation. | ![Elderly Japanese ceramicist](https://ai.google.dev/static/gemini-api/docs/images/photorealistic_example.png) | + +#### Stylized illustrations and stickers + +To create stickers, icons, or assets, be explicit about the style and request a +white background. + +> [!NOTE] +> **Note:** The model does not support generating a transparent background. + +| **Prompt** | **Generated output** | +|---|---| +| A kawaii-style sticker of a happy red panda wearing a tiny bamboo hat. It's munching on a green bamboo leaf. The design features bold, clean outlines, simple cel-shading, and a vibrant color palette. The background must be white. | ![Kawaii red panda sticker](https://ai.google.dev/static/gemini-api/docs/images/red_panda_sticker.png) | + +#### Accurate text in images + +Gemini excels at rendering text. Be clear about the text, the font style +(descriptively), and the overall design. Use Gemini 3 Pro Image Preview for +professional asset production. + +| **Prompt** | **Generated output** | +|---|---| +| Create a modern, minimalist logo for a coffee shop called 'The Daily Grind'. The text should be in a clean, bold, sans-serif font. The color scheme is black and white. Put the logo in a circle. Use a coffee bean in a clever way. | ![Coffee shop logo](https://ai.google.dev/static/gemini-api/docs/images/logo_example.jpg) | + +#### Product mockups and commercial photography + +Perfect for creating clean, professional product shots for ecommerce, +advertising, or branding. + +| **Prompt** | **Generated output** | +|---|---| +| A high-resolution, studio-lit product photograph of a minimalist ceramic coffee mug in matte black, presented on a polished concrete surface. The lighting is a three-point softbox setup designed to create soft, diffused highlights and eliminate harsh shadows. The camera angle is a slightly elevated 45-degree shot to showcase its clean lines. Ultra-realistic, with sharp focus on the steam rising from the coffee. Square image. | ![Ceramic coffee mug product shot](https://ai.google.dev/static/gemini-api/docs/images/product_mockup.png) | + +#### Minimalist and negative space design + +Excellent for creating backgrounds for websites, presentations, or marketing +materials where text will be overlaid. + +| **Prompt** | **Generated output** | +|---|---| +| A minimalist composition featuring a single, delicate red maple leaf positioned in the bottom-right of the frame. The background is a vast, empty off-white canvas, creating significant negative space for text. Soft, diffused lighting from the top left. Square image. | ![Minimalist design with red maple leaf](https://ai.google.dev/static/gemini-api/docs/images/minimalist_design.png) | + +#### Sequential art (Comic panel / Storyboard) + +Builds on character consistency and scene description to create panels for +visual storytelling. For accuracy with text and storytelling ability, these +prompts work best with Gemini 3 Pro and Gemini 3.1 Flash Image Preview. + +| **Prompt** | **Generated output** | +|---|---| +| **Input image:** ![Man in white glasses](https://ai.google.dev/static/gemini-api/docs/images/man_in_white_glasses.jpg) Input image **Prompt:** Make a 3 panel comic in a gritty, noir art style with high-contrast black and white inks. Put the character in a humurous scene. | ![Gritty noir comic panel](https://ai.google.dev/static/gemini-api/docs/images/comic_panel.jpg) | + +#### Grounding with Google Search + +Use Google Search to generate images based on recent or real-time information. +This is useful for news, weather, and other time-sensitive topics. + +| **Prompt** | **Generated output** | +|---|---| +| Make a simple but stylish graphic of last night's Arsenal game in the Champion's League | ![Arsenal football score graphic](https://ai.google.dev/static/gemini-api/docs/images/football-score.jpg) | + +### Prompts for editing images + +These examples show how to provide images alongside your text prompts for +editing, composition, and style transfer. + +#### Adding and removing elements + +Provide an image and describe your change. The model will match the original +image's style, lighting, and perspective. + +| **Prompt** | **Generated output** | +|---|---| +| **Input image:** :cat: Input image **Prompt:** Using the provided image of my cat, please add a small, knitted wizard hat on its head. Make it look like it's sitting comfortably and matches the soft lighting of the photo. | ![Cat with wizard hat](https://ai.google.dev/static/gemini-api/docs/images/cat_with_hat.png) | + +#### Inpainting (Semantic masking) + +Conversationally define a "mask" to edit a specific part of an image while +leaving the rest untouched. + +| **Prompt** | **Generated output** | +|---|---| +| **Input image:** ![A wide shot of a modern, well-lit living room...](https://ai.google.dev/static/gemini-api/docs/images/living_room.png) Input image **Prompt:** Using the provided image of a living room, change only the blue sofa to be a vintage, brown leather chesterfield sofa. Keep the rest of the room, including the pillows on the sofa and the lighting, unchanged. | ![Living room with brown leather sofa](https://ai.google.dev/static/gemini-api/docs/images/living_room_edited.png) | + +#### Style transfer + +Provide an image and ask the model to recreate its content in a different +artistic style. + +| **Prompt** | **Generated output** | +|---|---| +| **Input image:** ![A photorealistic, high-resolution photograph of a busy city street...](https://ai.google.dev/static/gemini-api/docs/images/city.png) Input image **Prompt:** Transform the provided photograph of a modern city street at night into the artistic style of Vincent van Gogh's 'Starry Night'. Preserve the original composition of buildings and cars, but render all elements with swirling, impasto brushstrokes and a dramatic palette of deep blues and bright yellows. | ![City street in Starry Night style](https://ai.google.dev/static/gemini-api/docs/images/city_style_transfer.png) | + +#### Advanced composition: Combining multiple images + +Provide multiple images as context to create a new, composite scene. This is +perfect for product mockups or creative collages. + +| **Prompt** | **Generated output** | +|---|---| +| **Input images:** :dress: Input 1: Dress ![Full-body shot of a woman with her hair in a bun...](https://ai.google.dev/static/gemini-api/docs/images/model.png) Input 2: Model **Prompt:** Create a professional e-commerce fashion photo. Take the blue floral dress from the first image and let the woman from the second image wear it. Generate a realistic, full-body shot of the woman wearing the dress, with the lighting and shadows adjusted to match the outdoor environment. | ![Fashion e-commerce shot](https://ai.google.dev/static/gemini-api/docs/images/fashion_ecommerce_shot.png) | + +#### High-fidelity detail preservation + +To ensure critical details (like a face or logo) are preserved during an edit, +describe them in great detail along with your edit request. + +| **Prompt** | **Generated output** | +|---|---| +| **Input images:** :woman: Input 1: Woman ![A simple, modern logo with the letters 'G' and 'A'...](https://ai.google.dev/static/gemini-api/docs/images/logo.png) Input 2: Logo **Prompt:** Take the first image of the woman with brown hair, blue eyes, and a neutral expression. Add the logo from the second image onto her black t-shirt. Ensure the woman's face and features remain completely unchanged. The logo should look like it's naturally printed on the fabric, following the folds of the shirt. | ![Woman with logo on t-shirt](https://ai.google.dev/static/gemini-api/docs/images/woman_with_logo.png) | + +#### Bring something to life + +Upload a rough sketch or drawing and ask the model to refine it into a finished image. + +| **Prompt** | **Generated output** | +|---|---| +| **Input image:** ![Sketch of a car](https://ai.google.dev/static/gemini-api/docs/images/car-sketch.jpg) Rough sketch of a car **Prompt:** Turn this rough pencil sketch of a futuristic car into a polished photo of the finished concept car in a showroom. Keep the sleek lines and low profile from the sketch but add metallic blue paint and neon rim lighting. | ![Polished photo of concept car](https://ai.google.dev/static/gemini-api/docs/images/car-photo.jpg) | + +#### Character consistency: 360 view + +You can generate 360-degree views of a character by iteratively prompting for +different angles. For best results, include previously generated images in +subsequent prompts to maintain consistency. For complex poses, include a +reference image of the desired pose. + +| **Prompt** | **Generated output** | +|---|---| +| **Input image:** ![Original input of a man in white glasses](https://ai.google.dev/static/gemini-api/docs/images/man_in_white_glasses.jpg) Original image **Prompt:** A studio portrait of this man against white, in profile looking right | ![Output of a man in white glasses looking right](https://ai.google.dev/static/gemini-api/docs/images/man_in_white_glasses_looking_right.jpg) Man in white glasses looking right ![Output of a man in white glasses looking forward](https://ai.google.dev/static/gemini-api/docs/images/man_in_white_glasses_looking_forward.jpg) Man in white glasses looking forward | + +### Best Practices + +To elevate your results from good to great, incorporate these professional +strategies into your workflow. + +- **Be Hyper-Specific:** The more detail you provide, the more control you have. Instead of "fantasy armor," describe it: "ornate elven plate armor, etched with silver leaf patterns, with a high collar and pauldrons shaped like falcon wings." +- **Provide Context and Intent:** Explain the *purpose* of the image. The model's understanding of context will influence the final output. For example, "Create a logo for a high-end, minimalist skincare brand" will yield better results than just "Create a logo." +- **Iterate and Refine:** Don't expect a perfect image on the first try. Use the conversational nature of the model to make small changes. Follow up with prompts like, "That's great, but can you make the lighting a bit warmer?" or "Keep everything the same, but change the character's expression to be more serious." +- **Use Step-by-Step Instructions:** For complex scenes with many elements, break your prompt into steps. "First, create a background of a serene, misty forest at dawn. Then, in the foreground, add a moss-covered ancient stone altar. Finally, place a single, glowing sword on top of the altar." +- **Use "Semantic Negative Prompts":** Instead of saying "no cars," describe the desired scene positively: "an empty, deserted street with no signs of traffic." +- **Control the Camera:** Use photographic and cinematic language to control the composition. Terms like `wide-angle shot`, `macro shot`, `low-angle + perspective`. + +## Limitations + +- For best performance, use the following languages: EN, ar-EG, de-DE, es-MX, fr-FR, hi-IN, id-ID, it-IT, ja-JP, ko-KR, pt-BR, ru-RU, ua-UA, vi-VN, zh-CN. +- Image generation does not support audio or video inputs. +- The model won't always follow the exact number of image outputs that the user explicitly asks for. +- `gemini-2.5-flash-image` works best with up to 3 images as input, while `gemini-3-pro-image-preview` supports 5 images with high fidelity, and up to 14 images in total. `gemini-3.1-flash-image-preview` supports character resemblance of up to 4 characters and the fidelity of up to 10 objects in a single workflow. +- When generating text for an image, Gemini works best if you first generate the text and then ask for an image with the text. +- `gemini-3.1-flash-image-preview` Grounding with Google Search does not support using real-world images of people from web search at this time. +- All generated images include a [SynthID watermark](https://ai.google.dev/responsible/docs/safeguards/synthid). + +## Optional configurations + +You can optionally configure the response modalities and aspect ratio of the +model's output in the `config` field of `generate_content` calls. + +### Output types + +The model defaults to returning text and image responses +(i.e. `response_modalities=['Text', 'Image']`). +You can configure the response to return only images without text using +`response_modalities=['Image']`. + +### Python + + response = client.models.generate_content( + model="gemini-3.1-flash-image-preview", + contents=[prompt], + config=types.GenerateContentConfig( + response_modalities=['Image'] + ) + ) + +### JavaScript + + const response = await ai.models.generateContent({ + model: "gemini-3.1-flash-image-preview", + contents: prompt, + config: { + responseModalities: ['Image'] + } + }); + +### Go + + result, _ := client.Models.GenerateContent( + ctx, + "gemini-3.1-flash-image-preview", + genai.Text("Create a picture of a nano banana dish in a " + + " fancy restaurant with a Gemini theme"), + &genai.GenerateContentConfig{ + ResponseModalities: "Image", + }, + ) + +### Java + + response = client.models.generateContent( + "gemini-3.1-flash-image-preview", + prompt, + GenerateContentConfig.builder() + .responseModalities("IMAGE") + .build()); + +### REST + + curl -s -X POST \ + "https://generativelanguage.googleapis.com/v1beta/models/gemini-3.1-flash-image-preview:generateContent" \ + -H "x-goog-api-key: $GEMINI_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{ + "contents": [{ + "parts": [ + {"text": "Create a picture of a nano banana dish in a fancy restaurant with a Gemini theme"} + ] + }], + "generationConfig": { + "responseModalities": ["Image"] + } + }' + +### Aspect ratios and image size + +The model defaults to matching the output image size to that of your input +image, or otherwise generates 1:1 squares. +You can control the aspect ratio of the output image using the `aspect_ratio` +field under `image_config` in the response request, shown here: + +### Python + + # For gemini-2.5-flash-image + response = client.models.generate_content( + model="gemini-2.5-flash-image", + contents=[prompt], + config=types.GenerateContentConfig( + image_config=types.ImageConfig( + aspect_ratio="16:9", + ) + ) + ) + + # For gemini-3.1-flash-image-preview and gemini-3-pro-image-preview + response = client.models.generate_content( + model="gemini-3.1-flash-image-preview", + contents=[prompt], + config=types.GenerateContentConfig( + image_config=types.ImageConfig( + aspect_ratio="16:9", + image_size="2K", + ) + ) + ) + +### JavaScript + + // For gemini-2.5-flash-image + const response = await ai.models.generateContent({ + model: "gemini-2.5-flash-image", + contents: prompt, + config: { + imageConfig: { + aspectRatio: "16:9", + }, + } + }); + + // For gemini-3.1-flash-image-preview and gemini-3-pro-image-preview + const response_gemini3 = await ai.models.generateContent({ + model: "gemini-3.1-flash-image-preview", + contents: prompt, + config: { + imageConfig: { + aspectRatio: "16:9", + imageSize: "2K", + }, + } + }); + +### Go + + // For gemini-2.5-flash-image + result, _ := client.Models.GenerateContent( + ctx, + "gemini-2.5-flash-image", + genai.Text("Create a picture of a nano banana dish in a " + + " fancy restaurant with a Gemini theme"), + &genai.GenerateContentConfig{ + ImageConfig: &genai.ImageConfig{ + AspectRatio: "16:9", + }, + } + ) + + // For gemini-3.1-flash-image-preview and gemini-3-pro-image-preview + result_gemini3, _ := client.Models.GenerateContent( + ctx, + "gemini-3.1-flash-image-preview", + genai.Text("Create a picture of a nano banana dish in a " + + " fancy restaurant with a Gemini theme"), + &genai.GenerateContentConfig{ + ImageConfig: &genai.ImageConfig{ + AspectRatio: "16:9", + ImageSize: "2K", + }, + } + ) + +### Java + + // For gemini-2.5-flash-image + response = client.models.generateContent( + "gemini-2.5-flash-image", + prompt, + GenerateContentConfig.builder() + .imageConfig(ImageConfig.builder() + .aspectRatio("16:9") + .build()) + .build()); + + // For gemini-3.1-flash-image-preview and gemini-3-pro-image-preview + response_gemini3 = client.models.generateContent( + "gemini-3.1-flash-image-preview", + prompt, + GenerateContentConfig.builder() + .imageConfig(ImageConfig.builder() + .aspectRatio("16:9") + .imageSize("2K") + .build()) + .build()); + +### REST + + # For gemini-2.5-flash-image + curl -s -X POST \ + "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.5-flash-image:generateContent" \ + -H "x-goog-api-key: $GEMINI_API_KEY" \ + -H 'Content-Type: application/json' \ + -d '{ + "contents": [{ + "parts": [ + {"text": "Create a picture of a nano banana dish in a fancy restaurant with a Gemini theme"} + ] + }], + "generationConfig": { + "imageConfig": { + "aspectRatio": "16:9" + } + } + }' + + # For gemini-3-pro-image-preview + curl -s -X POST \ + "https://generativelanguage.googleapis.com/v1beta/models/gemini-3.1-flash-image-preview:generateContent" \ + -H "x-goog-api-key: $GEMINI_API_KEY" \ + -H 'Content-Type: application/json' \ + -d '{ + "contents": [{ + "parts": [ + {"text": "Create a picture of a nano banana dish in a fancy restaurant with a Gemini theme"} + ] + }], + "generationConfig": { + "imageConfig": { + "aspectRatio": "16:9", + "imageSize": "2K" + } + } + }' + +The different ratios available and the size of the image generated are listed in +the following tables: + +### 3.1 Flash Image Preview + +| Aspect ratio | 512 resolution | 0.5K tokens | 1K resolution | 1K tokens | 2K resolution | 2K tokens | 4K resolution | 4K tokens | +|---|---|---|---|---|---|---|---|---| +| **1:1** | 512x512 | 747 | 1024x1024 | 1120 | 2048x2048 | 1680 | 4096x4096 | 2520 | +| **1:4** | 256x1024 | 747 | 512x2048 | 1120 | 1024x4096 | 1680 | 2048x8192 | 2520 | +| **1:8** | 192x1536 | 747 | 384x3072 | 1120 | 768x6144 | 1680 | 1536x12288 | 2520 | +| **2:3** | 424x632 | 747 | 848x1264 | 1120 | 1696x2528 | 1680 | 3392x5056 | 2520 | +| **3:2** | 632x424 | 747 | 1264x848 | 1120 | 2528x1696 | 1680 | 5056x3392 | 2520 | +| **3:4** | 448x600 | 747 | 896x1200 | 1120 | 1792x2400 | 1680 | 3584x4800 | 2520 | +| **4:1** | 1024x256 | 747 | 2048x512 | 1120 | 4096x1024 | 1680 | 8192x2048 | 2520 | +| **4:3** | 600x448 | 747 | 1200x896 | 1120 | 2400x1792 | 1680 | 4800x3584 | 2520 | +| **4:5** | 464x576 | 747 | 928x1152 | 1120 | 1856x2304 | 1680 | 3712x4608 | 2520 | +| **5:4** | 576x464 | 747 | 1152x928 | 1120 | 2304x1856 | 1680 | 4608x3712 | 2520 | +| **8:1** | 1536x192 | 747 | 3072x384 | 1120 | 6144x768 | 1680 | 12288x1536 | 2520 | +| **9:16** | 384x688 | 747 | 768x1376 | 1120 | 1536x2752 | 1680 | 3072x5504 | 2520 | +| **16:9** | 688x384 | 747 | 1376x768 | 1120 | 2752x1536 | 1680 | 5504x3072 | 2520 | +| **21:9** | 792x168 | 747 | 1584x672 | 1120 | 3168x1344 | 1680 | 6336x2688 | 2520 | + +### 3 Pro Image Preview + +| Aspect ratio | 1K resolution | 1K tokens | 2K resolution | 2K tokens | 4K resolution | 4K tokens | +|---|---|---|---|---|---|---| +| **1:1** | 1024x1024 | 1120 | 2048x2048 | 1120 | 4096x4096 | 2000 | +| **2:3** | 848x1264 | 1120 | 1696x2528 | 1120 | 3392x5056 | 2000 | +| **3:2** | 1264x848 | 1120 | 2528x1696 | 1120 | 5056x3392 | 2000 | +| **3:4** | 896x1200 | 1120 | 1792x2400 | 1120 | 3584x4800 | 2000 | +| **4:3** | 1200x896 | 1120 | 2400x1792 | 1120 | 4800x3584 | 2000 | +| **4:5** | 928x1152 | 1120 | 1856x2304 | 1120 | 3712x4608 | 2000 | +| **5:4** | 1152x928 | 1120 | 2304x1856 | 1120 | 4608x3712 | 2000 | +| **9:16** | 768x1376 | 1120 | 1536x2752 | 1120 | 3072x5504 | 2000 | +| **16:9** | 1376x768 | 1120 | 2752x1536 | 1120 | 5504x3072 | 2000 | +| **21:9** | 1584x672 | 1120 | 3168x1344 | 1120 | 6336x2688 | 2000 | + +### Gemini 2.5 Flash Image + +| Aspect ratio | Resolution | Tokens | +|---|---|---| +| 1:1 | 1024x1024 | 1290 | +| 2:3 | 832x1248 | 1290 | +| 3:2 | 1248x832 | 1290 | +| 3:4 | 864x1184 | 1290 | +| 4:3 | 1184x864 | 1290 | +| 4:5 | 896x1152 | 1290 | +| 5:4 | 1152x896 | 1290 | +| 9:16 | 768x1344 | 1290 | +| 16:9 | 1344x768 | 1290 | +| 21:9 | 1536x672 | 1290 | + +## Model selection + +Choose the model best suited for your specific use case. + +- **Gemini 3.1 Flash Image Preview (Nano Banana 2 Preview)** should be your + go-to image generation model, as the best all around performance and + intelligence to cost and latency balance. Check the model [pricing](https://ai.google.dev/gemini-api/docs/pricing#gemini-3.1-flash-image-preview) and [capabilities](https://ai.google.dev/gemini-api/docs/models/gemini-3.1-flash-image-preview) page for more + details. + +- **Gemini 3 Pro Image Preview (Nano Banana Pro Preview)** is designed for + professional asset production and complex instructions. This model features + real-world grounding using Google Search, a default "Thinking" process that + refines composition prior to generation, and can generate images of up to 4K + resolutions. Check the model [pricing](https://ai.google.dev/gemini-api/docs/pricing#gemini-3-pro-image-preview) and [capabilities](https://ai.google.dev/gemini-api/docs/models/gemini-3-pro-image-preview) page for more + details. + +- **Gemini 2.5 Flash Image (Nano Banana)** is designed for speed and + efficiency. This model is optimized for high-volume, low-latency tasks and + generates images at 1024px resolution. Check the model [pricing](https://ai.google.dev/gemini-api/docs/pricing#gemini-2.5-flash-image) and + [capabilities](https://ai.google.dev/gemini-api/docs/models/gemini-2.5-flash-image) page for more + details. + +### When to use Imagen + +In addition to using Gemini's built-in image generation capabilities, you can +also access [Imagen](https://ai.google.dev/gemini-api/docs/imagen), our specialized image generation +model, through the Gemini API. + +Imagen 4 should be your go-to model when starting to generate images +with Imagen. Choose Imagen 4 Ultra for advanced +use-cases or when you need the best image quality (note that can only generate +one image at a time). + +## What's next + +- Find more examples and code samples in the [cookbook guide](https://colab.research.google.com/github/google-gemini/cookbook/blob/main/quickstarts/Get_Started_Nano_Banana.ipynb). +- Check out the [Veo guide](https://ai.google.dev/gemini-api/docs/video) to learn how to generate videos with the Gemini API. +- To learn more about Gemini models, see [Gemini models](https://ai.google.dev/gemini-api/docs/models/gemini). \ No newline at end of file diff --git a/docs/协议文档/Gemini系列/gemini3介绍.md b/docs/协议文档/Gemini系列/gemini3介绍.md new file mode 100644 index 0000000..63bbc5b --- /dev/null +++ b/docs/协议文档/Gemini系列/gemini3介绍.md @@ -0,0 +1,3594 @@ +# Generating content + +The Gemini API supports content generation with images, audio, code, tools, and more. For details on each of these features, read on and check out the task-focused sample code, or read the comprehensive guides. + +- [Text generation](https://ai.google.dev/gemini-api/docs/text-generation) +- [Vision](https://ai.google.dev/gemini-api/docs/vision) +- [Audio](https://ai.google.dev/gemini-api/docs/audio) +- [Embeddings](https://ai.google.dev/gemini-api/docs/embeddings) +- [Long context](https://ai.google.dev/gemini-api/docs/long-context) +- [Code execution](https://ai.google.dev/gemini-api/docs/code-execution) +- [JSON Mode](https://ai.google.dev/gemini-api/docs/json-mode) +- [Function calling](https://ai.google.dev/gemini-api/docs/function-calling) +- [System instructions](https://ai.google.dev/gemini-api/docs/system-instructions) + +## Method: models.generateContent + +- [Endpoint](https://ai.google.dev/api/generate-content#body.HTTP_TEMPLATE) +- [Path parameters](https://ai.google.dev/api/generate-content#body.PATH_PARAMETERS) +- [Request body](https://ai.google.dev/api/generate-content#body.request_body) + - [JSON representation](https://ai.google.dev/api/generate-content#body.request_body.SCHEMA_REPRESENTATION) +- [Response body](https://ai.google.dev/api/generate-content#body.response_body) +- [Authorization scopes](https://ai.google.dev/api/generate-content#body.aspect) +- [Example request](https://ai.google.dev/api/generate-content#body.codeSnippets) + - [Text](https://ai.google.dev/api/generate-content#body.codeSnippets.group) + - [Image](https://ai.google.dev/api/generate-content#body.codeSnippets.group_1) + - [Audio](https://ai.google.dev/api/generate-content#body.codeSnippets.group_2) + - [Video](https://ai.google.dev/api/generate-content#body.codeSnippets.group_3) + - [PDF](https://ai.google.dev/api/generate-content#body.codeSnippets.group_4) + - [Chat](https://ai.google.dev/api/generate-content#body.codeSnippets.group_5) + - [Cache](https://ai.google.dev/api/generate-content#body.codeSnippets.group_6) + - [Tuned Model](https://ai.google.dev/api/generate-content#body.codeSnippets.group_7) + - [JSON Mode](https://ai.google.dev/api/generate-content#body.codeSnippets.group_8) + - [Code execution](https://ai.google.dev/api/generate-content#body.codeSnippets.group_9) + - [Function Calling](https://ai.google.dev/api/generate-content#body.codeSnippets.group_10) + - [Generation config](https://ai.google.dev/api/generate-content#body.codeSnippets.group_11) + - [Safety Settings](https://ai.google.dev/api/generate-content#body.codeSnippets.group_12) + - [System Instruction](https://ai.google.dev/api/generate-content#body.codeSnippets.group_13) + +Generates a model response given an input `GenerateContentRequest`. Refer to the [text generation guide](https://ai.google.dev/gemini-api/docs/text-generation) for detailed usage information. Input capabilities differ between models, including tuned models. Refer to the [model guide](https://ai.google.dev/gemini-api/docs/models/gemini) and [tuning guide](https://ai.google.dev/gemini-api/docs/model-tuning) for details. + +### Endpoint + +post `https://generativelanguage.googleapis.com/v1beta/{model=models/*}:generateContent` + +### Path parameters + +`model` `string` Required. The name of the `Model` to use for generating the completion. + +Format: `models/{model}`. It takes the form `models/{model}`. + +### Request body + +The request body contains data with the following structure: +Fields `contents[]` ``object (`https://ai.google.dev/api/caching#Content`)`` Required. The content of the current conversation with the model. + +For single-turn queries, this is a single instance. For multi-turn queries like [chat](https://ai.google.dev/gemini-api/docs/text-generation#chat), this is a repeated field that contains the conversation history and the latest request. +`tools[]` ``object (`https://ai.google.dev/api/caching#Tool`)`` Optional. A list of `Tools` the `Model` may use to generate the next response. + +A `Tool` is a piece of code that enables the system to interact with external systems to perform an action, or set of actions, outside of knowledge and scope of the `Model`. Supported `Tool`s are `Function` and `codeExecution`. Refer to the [Function calling](https://ai.google.dev/gemini-api/docs/function-calling) and the [Code execution](https://ai.google.dev/gemini-api/docs/code-execution) guides to learn more. +`toolConfig` ``object (`https://ai.google.dev/api/caching#ToolConfig`)`` Optional. Tool configuration for any `Tool` specified in the request. Refer to the [Function calling guide](https://ai.google.dev/gemini-api/docs/function-calling#function_calling_mode) for a usage example. +`safetySettings[]` ``object (`https://ai.google.dev/api/generate-content#v1beta.SafetySetting`)`` Optional. A list of unique `SafetySetting` instances for blocking unsafe content. + +This will be enforced on the `GenerateContentRequest.contents` and `GenerateContentResponse.candidates`. There should not be more than one setting for each `SafetyCategory` type. The API will block any contents and responses that fail to meet the thresholds set by these settings. This list overrides the default settings for each `SafetyCategory` specified in the safetySettings. If there is no `SafetySetting` for a given `SafetyCategory` provided in the list, the API will use the default safety setting for that category. Harm categories HARM_CATEGORY_HATE_SPEECH, HARM_CATEGORY_SEXUALLY_EXPLICIT, HARM_CATEGORY_DANGEROUS_CONTENT, HARM_CATEGORY_HARASSMENT, HARM_CATEGORY_CIVIC_INTEGRITY are supported. Refer to the [guide](https://ai.google.dev/gemini-api/docs/safety-settings) for detailed information on available safety settings. Also refer to the [Safety guidance](https://ai.google.dev/gemini-api/docs/safety-guidance) to learn how to incorporate safety considerations in your AI applications. +`systemInstruction` ``object (`https://ai.google.dev/api/caching#Content`)`` Optional. Developer set [system instruction(s)](https://ai.google.dev/gemini-api/docs/system-instructions). Currently, text only. +`generationConfig` ``object (`https://ai.google.dev/api/generate-content#v1beta.GenerationConfig`)`` Optional. Configuration options for model generation and outputs. +`cachedContent` `string` Optional. The name of the content [cached](https://ai.google.dev/gemini-api/docs/caching) to use as context to serve the prediction. Format: `cachedContents/{cachedContent}` +`serviceTier` ``enum (`https://ai.google.dev/api/generate-content#v1beta.ServiceTier`)`` Optional. The service tier of the request. +`store` `boolean` Optional. Configures the logging behavior for a given request. If set, it takes precedence over the project-level logging config. + +### Example request + +### Text + +### Python + + from google import genai + + client = genai.Client() + response = client.models.generate_content( + model="gemini-2.0-flash", contents="Write a story about a magic backpack." + ) + print(response.text)https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/python/text_generation.py#L26-L32 + +### Node.js + + // Make sure to include the following import: + // import {GoogleGenAI} from '@google/genai'; + const ai = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY }); + + const response = await ai.models.generateContent({ + model: "gemini-2.0-flash", + contents: "Write a story about a magic backpack.", + }); + console.log(response.text);https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/javascript/text_generation.js#L36-L44 + +### Go + + ctx := context.Background() + client, err := genai.NewClient(ctx, &genai.ClientConfig{ + APIKey: os.Getenv("GEMINI_API_KEY"), + Backend: genai.BackendGeminiAPI, + }) + if err != nil { + log.Fatal(err) + } + contents := []*genai.Content{ + genai.NewContentFromText("Write a story about a magic backpack.", genai.RoleUser), + } + response, err := client.Models.GenerateContent(ctx, "gemini-2.0-flash", contents, nil) + if err != nil { + log.Fatal(err) + } + printResponse(response)https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/go/text_generation.go#L16-L31 + +### Shell + + curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash:generateContent?key=$GEMINI_API_KEY" \ + -H 'Content-Type: application/json' \ + -X POST \ + -d '{ + "contents": [{ + "parts":[{"text": "Write a story about a magic backpack."}] + }] + }' 2> /dev/nullhttps://github.com/google-gemini/deprecated-generative-ai-python/blob/7a7cc5474ddaa0255a4410e05361028a24400abd/samples/rest/text_generation.sh#L21-L29 + +### Java + + Client client = new Client(); + + GenerateContentResponse response = + client.models.generateContent( + "gemini-2.0-flash", + "Write a story about a magic backpack.", + null); + + System.out.println(response.text());https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/java/src/main/java/com/example/gemini/TextGeneration.java#L34-L42 + +### Image + +### Python + + from google import genai + import PIL.Image + + client = genai.Client() + organ = PIL.Image.open(media / "organ.jpg") + response = client.models.generate_content( + model="gemini-2.0-flash", contents=["Tell me about this instrument", organ] + ) + print(response.text)https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/python/text_generation.py#L50-L58 + +### Node.js + + // Make sure to include the following import: + // import {GoogleGenAI} from '@google/genai'; + const ai = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY }); + + const organ = await ai.files.upload({ + file: path.join(media, "organ.jpg"), + }); + + const response = await ai.models.generateContent({ + model: "gemini-2.0-flash", + contents: [ + createUserContent([ + "Tell me about this instrument", + createPartFromUri(organ.uri, organ.mimeType) + ]), + ], + }); + console.log(response.text);https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/javascript/text_generation.js#L70-L87 + +### Go + + ctx := context.Background() + client, err := genai.NewClient(ctx, &genai.ClientConfig{ + APIKey: os.Getenv("GEMINI_API_KEY"), + Backend: genai.BackendGeminiAPI, + }) + if err != nil { + log.Fatal(err) + } + + file, err := client.Files.UploadFromPath( + ctx, + filepath.Join(getMedia(), "organ.jpg"), + &genai.UploadFileConfig{ + MIMEType : "image/jpeg", + }, + ) + if err != nil { + log.Fatal(err) + } + parts := []*genai.Part{ + genai.NewPartFromText("Tell me about this instrument"), + genai.NewPartFromURI(file.URI, file.MIMEType), + } + contents := []*genai.Content{ + genai.NewContentFromParts(parts, genai.RoleUser), + } + + response, err := client.Models.GenerateContent(ctx, "gemini-2.0-flash", contents, nil) + if err != nil { + log.Fatal(err) + } + printResponse(response)https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/go/text_generation.go#L66-L97 + +### Shell + + # Use a temporary file to hold the base64 encoded image data + TEMP_B64=$(mktemp) + trap 'rm -f "$TEMP_B64"' EXIT + base64 $B64FLAGS $IMG_PATH > "$TEMP_B64" + + # Use a temporary file to hold the JSON payload + TEMP_JSON=$(mktemp) + trap 'rm -f "$TEMP_JSON"' EXIT + + cat > "$TEMP_JSON" << EOF + { + "contents": [{ + "parts":[ + {"text": "Tell me about this instrument"}, + { + "inline_data": { + "mime_type":"image/jpeg", + "data": "$(cat "$TEMP_B64")" + } + } + ] + }] + } + EOF + + curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash:generateContent?key=$GEMINI_API_KEY" \ + -H 'Content-Type: application/json' \ + -X POST \ + -d "@$TEMP_JSON" 2> /dev/nullhttps://github.com/google-gemini/deprecated-generative-ai-python/blob/7a7cc5474ddaa0255a4410e05361028a24400abd/samples/rest/text_generation.sh#L41-L70 + +### Java + + Client client = new Client(); + + String path = media_path + "organ.jpg"; + byte[] imageData = Files.readAllBytes(Paths.get(path)); + + Content content = + Content.fromParts( + Part.fromText("Tell me about this instrument."), + Part.fromBytes(imageData, "image/jpeg")); + + GenerateContentResponse response = client.models.generateContent("gemini-2.0-flash", content, null); + + System.out.println(response.text());https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/java/src/main/java/com/example/gemini/TextGeneration.java#L70-L82 + +### Audio + +### Python + + from google import genai + + client = genai.Client() + sample_audio = client.files.upload(file=media / "sample.mp3") + response = client.models.generate_content( + model="gemini-2.0-flash", + contents=["Give me a summary of this audio file.", sample_audio], + ) + print(response.text)https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/python/text_generation.py#L118-L126 + +### Node.js + + // Make sure to include the following import: + // import {GoogleGenAI} from '@google/genai'; + const ai = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY }); + + const audio = await ai.files.upload({ + file: path.join(media, "sample.mp3"), + }); + + const response = await ai.models.generateContent({ + model: "gemini-2.0-flash", + contents: [ + createUserContent([ + "Give me a summary of this audio file.", + createPartFromUri(audio.uri, audio.mimeType), + ]), + ], + }); + console.log(response.text);https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/javascript/text_generation.js#L185-L202 + +### Go + + ctx := context.Background() + client, err := genai.NewClient(ctx, &genai.ClientConfig{ + APIKey: os.Getenv("GEMINI_API_KEY"), + Backend: genai.BackendGeminiAPI, + }) + if err != nil { + log.Fatal(err) + } + + file, err := client.Files.UploadFromPath( + ctx, + filepath.Join(getMedia(), "sample.mp3"), + &genai.UploadFileConfig{ + MIMEType : "audio/mpeg", + }, + ) + if err != nil { + log.Fatal(err) + } + + parts := []*genai.Part{ + genai.NewPartFromText("Give me a summary of this audio file."), + genai.NewPartFromURI(file.URI, file.MIMEType), + } + + contents := []*genai.Content{ + genai.NewContentFromParts(parts, genai.RoleUser), + } + + response, err := client.Models.GenerateContent(ctx, "gemini-2.0-flash", contents, nil) + if err != nil { + log.Fatal(err) + } + printResponse(response)https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/go/text_generation.go#L256-L289 + +### Shell + + # Use File API to upload audio data to API request. + MIME_TYPE=$(file -b --mime-type "${AUDIO_PATH}") + NUM_BYTES=$(wc -c < "${AUDIO_PATH}") + DISPLAY_NAME=AUDIO + + tmp_header_file=upload-header.tmp + + # Initial resumable request defining metadata. + # The upload url is in the response headers dump them to a file. + curl "${BASE_URL}/upload/v1beta/files?key=${GEMINI_API_KEY}" \ + -D upload-header.tmp \ + -H "X-Goog-Upload-Protocol: resumable" \ + -H "X-Goog-Upload-Command: start" \ + -H "X-Goog-Upload-Header-Content-Length: ${NUM_BYTES}" \ + -H "X-Goog-Upload-Header-Content-Type: ${MIME_TYPE}" \ + -H "Content-Type: application/json" \ + -d "{'file': {'display_name': '${DISPLAY_NAME}'}}" 2> /dev/null + + upload_url=$(grep -i "x-goog-upload-url: " "${tmp_header_file}" | cut -d" " -f2 | tr -d "\r") + rm "${tmp_header_file}" + + # Upload the actual bytes. + curl "${upload_url}" \ + -H "Content-Length: ${NUM_BYTES}" \ + -H "X-Goog-Upload-Offset: 0" \ + -H "X-Goog-Upload-Command: upload, finalize" \ + --data-binary "@${AUDIO_PATH}" 2> /dev/null > file_info.json + + file_uri=$(jq ".file.uri" file_info.json) + echo file_uri=$file_uri + + curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash:generateContent?key=$GEMINI_API_KEY" \ + -H 'Content-Type: application/json' \ + -X POST \ + -d '{ + "contents": [{ + "parts":[ + {"text": "Please describe this file."}, + {"file_data":{"mime_type": "audio/mpeg", "file_uri": '$file_uri'}}] + }] + }' 2> /dev/null > response.json + + cat response.json + echo + + jq ".candidates[].content.parts[].text" response.jsonhttps://github.com/google-gemini/deprecated-generative-ai-python/blob/7a7cc5474ddaa0255a4410e05361028a24400abd/samples/rest/text_generation.sh#L174-L220 + +### Video + +### Python + + from google import genai + import time + + client = genai.Client() + # Video clip (CC BY 3.0) from https://peach.blender.org/download/ + myfile = client.files.upload(file=media / "Big_Buck_Bunny.mp4") + print(f"{myfile=}") + + # Poll until the video file is completely processed (state becomes ACTIVE). + while not myfile.state or myfile.state.name != "ACTIVE": + print("Processing video...") + print("File state:", myfile.state) + time.sleep(5) + myfile = client.files.get(name=myfile.name) + + response = client.models.generate_content( + model="gemini-2.0-flash", contents=[myfile, "Describe this video clip"] + ) + print(f"{response.text=}")https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/python/text_generation.py#L146-L164 + +### Node.js + + // Make sure to include the following import: + // import {GoogleGenAI} from '@google/genai'; + const ai = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY }); + + let video = await ai.files.upload({ + file: path.join(media, 'Big_Buck_Bunny.mp4'), + }); + + // Poll until the video file is completely processed (state becomes ACTIVE). + while (!video.state || video.state.toString() !== 'ACTIVE') { + console.log('Processing video...'); + console.log('File state: ', video.state); + await sleep(5000); + video = await ai.files.get({name: video.name}); + } + + const response = await ai.models.generateContent({ + model: "gemini-2.0-flash", + contents: [ + createUserContent([ + "Describe this video clip", + createPartFromUri(video.uri, video.mimeType), + ]), + ], + }); + console.log(response.text);https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/javascript/text_generation.js#L237-L262 + +### Go + + ctx := context.Background() + client, err := genai.NewClient(ctx, &genai.ClientConfig{ + APIKey: os.Getenv("GEMINI_API_KEY"), + Backend: genai.BackendGeminiAPI, + }) + if err != nil { + log.Fatal(err) + } + + file, err := client.Files.UploadFromPath( + ctx, + filepath.Join(getMedia(), "Big_Buck_Bunny.mp4"), + &genai.UploadFileConfig{ + MIMEType : "video/mp4", + }, + ) + if err != nil { + log.Fatal(err) + } + + // Poll until the video file is completely processed (state becomes ACTIVE). + for file.State == genai.FileStateUnspecified || file.State != genai.FileStateActive { + fmt.Println("Processing video...") + fmt.Println("File state:", file.State) + time.Sleep(5 * time.Second) + + file, err = client.Files.Get(ctx, file.Name, nil) + if err != nil { + log.Fatal(err) + } + } + + parts := []*genai.Part{ + genai.NewPartFromText("Describe this video clip"), + genai.NewPartFromURI(file.URI, file.MIMEType), + } + + contents := []*genai.Content{ + genai.NewContentFromParts(parts, genai.RoleUser), + } + + response, err := client.Models.GenerateContent(ctx, "gemini-2.0-flash", contents, nil) + if err != nil { + log.Fatal(err) + } + printResponse(response)https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/go/text_generation.go#L342-L387 + +### Shell + + # Use File API to upload audio data to API request. + MIME_TYPE=$(file -b --mime-type "${VIDEO_PATH}") + NUM_BYTES=$(wc -c < "${VIDEO_PATH}") + DISPLAY_NAME=VIDEO + + # Initial resumable request defining metadata. + # The upload url is in the response headers dump them to a file. + curl "${BASE_URL}/upload/v1beta/files?key=${GEMINI_API_KEY}" \ + -D "${tmp_header_file}" \ + -H "X-Goog-Upload-Protocol: resumable" \ + -H "X-Goog-Upload-Command: start" \ + -H "X-Goog-Upload-Header-Content-Length: ${NUM_BYTES}" \ + -H "X-Goog-Upload-Header-Content-Type: ${MIME_TYPE}" \ + -H "Content-Type: application/json" \ + -d "{'file': {'display_name': '${DISPLAY_NAME}'}}" 2> /dev/null + + upload_url=$(grep -i "x-goog-upload-url: " "${tmp_header_file}" | cut -d" " -f2 | tr -d "\r") + rm "${tmp_header_file}" + + # Upload the actual bytes. + curl "${upload_url}" \ + -H "Content-Length: ${NUM_BYTES}" \ + -H "X-Goog-Upload-Offset: 0" \ + -H "X-Goog-Upload-Command: upload, finalize" \ + --data-binary "@${VIDEO_PATH}" 2> /dev/null > file_info.json + + file_uri=$(jq ".file.uri" file_info.json) + echo file_uri=$file_uri + + state=$(jq ".file.state" file_info.json) + echo state=$state + + name=$(jq ".file.name" file_info.json) + echo name=$name + + while [[ "($state)" = *"PROCESSING"* ]]; + do + echo "Processing video..." + sleep 5 + # Get the file of interest to check state + curl https://generativelanguage.googleapis.com/v1beta/files/$name > file_info.json + state=$(jq ".file.state" file_info.json) + done + + curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash:generateContent?key=$GEMINI_API_KEY" \ + -H 'Content-Type: application/json' \ + -X POST \ + -d '{ + "contents": [{ + "parts":[ + {"text": "Transcribe the audio from this video, giving timestamps for salient events in the video. Also provide visual descriptions."}, + {"file_data":{"mime_type": "video/mp4", "file_uri": '$file_uri'}}] + }] + }' 2> /dev/null > response.json + + cat response.json + echo + + jq ".candidates[].content.parts[].text" response.jsonhttps://github.com/google-gemini/deprecated-generative-ai-python/blob/7a7cc5474ddaa0255a4410e05361028a24400abd/samples/rest/text_generation.sh#L272-L331 + +### PDF + +### Python + + from google import genai + + client = genai.Client() + sample_pdf = client.files.upload(file=media / "test.pdf") + response = client.models.generate_content( + model="gemini-2.0-flash", + contents=["Give me a summary of this document:", sample_pdf], + ) + print(f"{response.text=}")https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/python/text_generation.py#L194-L202 + +### Go + + ctx := context.Background() + client, err := genai.NewClient(ctx, &genai.ClientConfig{ + APIKey: os.Getenv("GEMINI_API_KEY"), + Backend: genai.BackendGeminiAPI, + }) + if err != nil { + log.Fatal(err) + } + + file, err := client.Files.UploadFromPath( + ctx, + filepath.Join(getMedia(), "test.pdf"), + &genai.UploadFileConfig{ + MIMEType : "application/pdf", + }, + ) + if err != nil { + log.Fatal(err) + } + + parts := []*genai.Part{ + genai.NewPartFromText("Give me a summary of this document:"), + genai.NewPartFromURI(file.URI, file.MIMEType), + } + + contents := []*genai.Content{ + genai.NewContentFromParts(parts, genai.RoleUser), + } + + response, err := client.Models.GenerateContent(ctx, "gemini-2.0-flash", contents, nil) + if err != nil { + log.Fatal(err) + } + printResponse(response)https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/go/text_generation.go#L452-L485 + +### Shell + + MIME_TYPE=$(file -b --mime-type "${PDF_PATH}") + NUM_BYTES=$(wc -c < "${PDF_PATH}") + DISPLAY_NAME=TEXT + + + echo $MIME_TYPE + tmp_header_file=upload-header.tmp + + # Initial resumable request defining metadata. + # The upload url is in the response headers dump them to a file. + curl "${BASE_URL}/upload/v1beta/files?key=${GEMINI_API_KEY}" \ + -D upload-header.tmp \ + -H "X-Goog-Upload-Protocol: resumable" \ + -H "X-Goog-Upload-Command: start" \ + -H "X-Goog-Upload-Header-Content-Length: ${NUM_BYTES}" \ + -H "X-Goog-Upload-Header-Content-Type: ${MIME_TYPE}" \ + -H "Content-Type: application/json" \ + -d "{'file': {'display_name': '${DISPLAY_NAME}'}}" 2> /dev/null + + upload_url=$(grep -i "x-goog-upload-url: " "${tmp_header_file}" | cut -d" " -f2 | tr -d "\r") + rm "${tmp_header_file}" + + # Upload the actual bytes. + curl "${upload_url}" \ + -H "Content-Length: ${NUM_BYTES}" \ + -H "X-Goog-Upload-Offset: 0" \ + -H "X-Goog-Upload-Command: upload, finalize" \ + --data-binary "@${PDF_PATH}" 2> /dev/null > file_info.json + + file_uri=$(jq ".file.uri" file_info.json) + echo file_uri=$file_uri + + # Now generate content using that file + curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash:generateContent?key=$GEMINI_API_KEY" \ + -H 'Content-Type: application/json' \ + -X POST \ + -d '{ + "contents": [{ + "parts":[ + {"text": "Can you add a few more lines to this poem?"}, + {"file_data":{"mime_type": "application/pdf", "file_uri": '$file_uri'}}] + }] + }' 2> /dev/null > response.json + + cat response.json + echo + + jq ".candidates[].content.parts[].text" response.jsonhttps://github.com/google-gemini/deprecated-generative-ai-python/blob/7a7cc5474ddaa0255a4410e05361028a24400abd/samples/rest/text_generation.sh#L393-L441 + +### Chat + +### Python + + from google import genai + from google.genai import types + + client = genai.Client() + # Pass initial history using the "history" argument + chat = client.chats.create( + model="gemini-2.0-flash", + history=[ + types.Content(role="user", parts=[types.Part(text="Hello")]), + types.Content( + role="model", + parts=[ + types.Part( + text="Great to meet you. What would you like to know?" + ) + ], + ), + ], + ) + response = chat.send_message(message="I have 2 dogs in my house.") + print(response.text) + response = chat.send_message(message="How many paws are in my house?") + print(response.text)https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/python/chat.py#L25-L47 + +### Node.js + + // Make sure to include the following import: + // import {GoogleGenAI} from '@google/genai'; + const ai = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY }); + const chat = ai.chats.create({ + model: "gemini-2.0-flash", + history: [ + { + role: "user", + parts: [{ text: "Hello" }], + }, + { + role: "model", + parts: [{ text: "Great to meet you. What would you like to know?" }], + }, + ], + }); + + const response1 = await chat.sendMessage({ + message: "I have 2 dogs in my house.", + }); + console.log("Chat response 1:", response1.text); + + const response2 = await chat.sendMessage({ + message: "How many paws are in my house?", + }); + console.log("Chat response 2:", response2.text);https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/javascript/chat.js#L33-L58 + +### Go + + ctx := context.Background() + client, err := genai.NewClient(ctx, &genai.ClientConfig{ + APIKey: os.Getenv("GEMINI_API_KEY"), + Backend: genai.BackendGeminiAPI, + }) + if err != nil { + log.Fatal(err) + } + + // Pass initial history using the History field. + history := []*genai.Content{ + genai.NewContentFromText("Hello", genai.RoleUser), + genai.NewContentFromText("Great to meet you. What would you like to know?", genai.RoleModel), + } + + chat, err := client.Chats.Create(ctx, "gemini-2.0-flash", nil, history) + if err != nil { + log.Fatal(err) + } + + firstResp, err := chat.SendMessage(ctx, genai.Part{Text: "I have 2 dogs in my house."}) + if err != nil { + log.Fatal(err) + } + fmt.Println(firstResp.Text()) + + secondResp, err := chat.SendMessage(ctx, genai.Part{Text: "How many paws are in my house?"}) + if err != nil { + log.Fatal(err) + } + fmt.Println(secondResp.Text())https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/go/chat.go#L16-L46 + +### Shell + + curl https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash:generateContent?key=$GEMINI_API_KEY \ + -H 'Content-Type: application/json' \ + -X POST \ + -d '{ + "contents": [ + {"role":"user", + "parts":[{ + "text": "Hello"}]}, + {"role": "model", + "parts":[{ + "text": "Great to meet you. What would you like to know?"}]}, + {"role":"user", + "parts":[{ + "text": "I have two dogs in my house. How many paws are in my house?"}]}, + ] + }' 2> /dev/null | grep "text"https://github.com/google-gemini/deprecated-generative-ai-python/blob/7a7cc5474ddaa0255a4410e05361028a24400abd/samples/rest/chat.sh#L7-L23 + +### Java + + Client client = new Client(); + + Content userContent = Content.fromParts(Part.fromText("Hello")); + Content modelContent = + Content.builder() + .role("model") + .parts( + Collections.singletonList( + Part.fromText("Great to meet you. What would you like to know?") + ) + ).build(); + + Chat chat = client.chats.create( + "gemini-2.0-flash", + GenerateContentConfig.builder() + .systemInstruction(userContent) + .systemInstruction(modelContent) + .build() + ); + + GenerateContentResponse response1 = chat.sendMessage("I have 2 dogs in my house."); + System.out.println(response1.text()); + + GenerateContentResponse response2 = chat.sendMessage("How many paws are in my house?"); + System.out.println(response2.text()); + https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/java/src/main/java/com/example/gemini/ChatSession.java#L32-L57 + +### Cache + +### Python + + from google import genai + from google.genai import types + + client = genai.Client() + document = client.files.upload(file=media / "a11.txt") + model_name = "gemini-1.5-flash-001" + + cache = client.caches.create( + model=model_name, + config=types.CreateCachedContentConfig( + contents=[document], + system_instruction="You are an expert analyzing transcripts.", + ), + ) + print(cache) + + response = client.models.generate_content( + model=model_name, + contents="Please summarize this transcript", + config=types.GenerateContentConfig(cached_content=cache.name), + ) + print(response.text)https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/python/cache.py#L25-L46 + +### Node.js + + // Make sure to include the following import: + // import {GoogleGenAI} from '@google/genai'; + const ai = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY }); + const filePath = path.join(media, "a11.txt"); + const document = await ai.files.upload({ + file: filePath, + config: { mimeType: "text/plain" }, + }); + console.log("Uploaded file name:", document.name); + const modelName = "gemini-1.5-flash-001"; + + const contents = [ + createUserContent(createPartFromUri(document.uri, document.mimeType)), + ]; + + const cache = await ai.caches.create({ + model: modelName, + config: { + contents: contents, + systemInstruction: "You are an expert analyzing transcripts.", + }, + }); + console.log("Cache created:", cache); + + const response = await ai.models.generateContent({ + model: modelName, + contents: "Please summarize this transcript", + config: { cachedContent: cache.name }, + }); + console.log("Response text:", response.text);https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/javascript/cache.js#L33-L62 + +### Go + + ctx := context.Background() + client, err := genai.NewClient(ctx, &genai.ClientConfig{ + APIKey: os.Getenv("GEMINI_API_KEY"), + Backend: genai.BackendGeminiAPI, + }) + if err != nil { + log.Fatal(err) + } + + modelName := "gemini-1.5-flash-001" + document, err := client.Files.UploadFromPath( + ctx, + filepath.Join(getMedia(), "a11.txt"), + &genai.UploadFileConfig{ + MIMEType : "text/plain", + }, + ) + if err != nil { + log.Fatal(err) + } + parts := []*genai.Part{ + genai.NewPartFromURI(document.URI, document.MIMEType), + } + contents := []*genai.Content{ + genai.NewContentFromParts(parts, genai.RoleUser), + } + cache, err := client.Caches.Create(ctx, modelName, &genai.CreateCachedContentConfig{ + Contents: contents, + SystemInstruction: genai.NewContentFromText( + "You are an expert analyzing transcripts.", genai.RoleUser, + ), + }) + if err != nil { + log.Fatal(err) + } + fmt.Println("Cache created:") + fmt.Println(cache) + + // Use the cache for generating content. + response, err := client.Models.GenerateContent( + ctx, + modelName, + genai.Text("Please summarize this transcript"), + &genai.GenerateContentConfig{ + CachedContent: cache.Name, + }, + ) + if err != nil { + log.Fatal(err) + } + printResponse(response)https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/go/cache.go#L16-L66 + +### Tuned Model + +### Python + + # With Gemini 2 we're launching a new SDK. See the following doc for details. + # https://ai.google.dev/gemini-api/docs/migratehttps://github.com/google-gemini/deprecated-generative-ai-python/blob/7a7cc5474ddaa0255a4410e05361028a24400abd/README.md#L23-L24 + +### JSON Mode + +### Python + + from google import genai + from google.genai import types + from typing_extensions import TypedDict + + class Recipe(TypedDict): + recipe_name: str + ingredients: list[str] + + client = genai.Client() + result = client.models.generate_content( + model="gemini-2.0-flash", + contents="List a few popular cookie recipes.", + config=types.GenerateContentConfig( + response_mime_type="application/json", response_schema=list[Recipe] + ), + ) + print(result)https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/python/controlled_generation.py#L25-L41 + +### Node.js + + // Make sure to include the following import: + // import {GoogleGenAI} from '@google/genai'; + const ai = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY }); + const response = await ai.models.generateContent({ + model: "gemini-2.0-flash", + contents: "List a few popular cookie recipes.", + config: { + responseMimeType: "application/json", + responseSchema: { + type: "array", + items: { + type: "object", + properties: { + recipeName: { type: "string" }, + ingredients: { type: "array", items: { type: "string" } }, + }, + required: ["recipeName", "ingredients"], + }, + }, + }, + }); + console.log(response.text);https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/javascript/controlled_generation.js#L33-L54 + +### Go + + ctx := context.Background() + client, err := genai.NewClient(ctx, &genai.ClientConfig{ + APIKey: os.Getenv("GEMINI_API_KEY"), + Backend: genai.BackendGeminiAPI, + }) + if err != nil { + log.Fatal(err) + } + + schema := &genai.Schema{ + Type: genai.TypeArray, + Items: &genai.Schema{ + Type: genai.TypeObject, + Properties: map[string]*genai.Schema{ + "recipe_name": {Type: genai.TypeString}, + "ingredients": { + Type: genai.TypeArray, + Items: &genai.Schema{Type: genai.TypeString}, + }, + }, + Required: []string{"recipe_name"}, + }, + } + + config := &genai.GenerateContentConfig{ + ResponseMIMEType: "application/json", + ResponseSchema: schema, + } + + response, err := client.Models.GenerateContent( + ctx, + "gemini-2.0-flash", + genai.Text("List a few popular cookie recipes."), + config, + ) + if err != nil { + log.Fatal(err) + } + printResponse(response)https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/go/controlled_generation.go#L14-L52 + +### Shell + + curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash:generateContent?key=$GEMINI_API_KEY" \ + -H 'Content-Type: application/json' \ + -d '{ + "contents": [{ + "parts":[ + {"text": "List 5 popular cookie recipes"} + ] + }], + "generationConfig": { + "response_mime_type": "application/json", + "response_schema": { + "type": "ARRAY", + "items": { + "type": "OBJECT", + "properties": { + "recipe_name": {"type":"STRING"}, + } + } + } + } + }' 2> /dev/null | headhttps://github.com/google-gemini/deprecated-generative-ai-python/blob/7a7cc5474ddaa0255a4410e05361028a24400abd/samples/rest/controlled_generation.sh#L5-L25 + +### Java + + Client client = new Client(); + + Schema recipeSchema = Schema.builder() + .type(Array.class.getSimpleName()) + .items(Schema.builder() + .type(Object.class.getSimpleName()) + .properties( + Map.of("recipe_name", Schema.builder() + .type(String.class.getSimpleName()) + .build(), + "ingredients", Schema.builder() + .type(Array.class.getSimpleName()) + .items(Schema.builder() + .type(String.class.getSimpleName()) + .build()) + .build()) + ) + .required(List.of("recipe_name", "ingredients")) + .build()) + .build(); + + GenerateContentConfig config = + GenerateContentConfig.builder() + .responseMimeType("application/json") + .responseSchema(recipeSchema) + .build(); + + GenerateContentResponse response = + client.models.generateContent( + "gemini-2.0-flash", + "List a few popular cookie recipes.", + config); + + System.out.println(response.text());https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/java/src/main/java/com/example/gemini/ControlledGeneration.java#L39-L72 + +### Code execution + +### Python + + from google import genai + from google.genai import types + + client = genai.Client() + response = client.models.generate_content( + model="gemini-2.0-pro-exp-02-05", + contents=( + "Write and execute code that calculates the sum of the first 50 prime numbers. " + "Ensure that only the executable code and its resulting output are generated." + ), + ) + # Each part may contain text, executable code, or an execution result. + for part in response.candidates[0].content.parts: + print(part, "\n") + + print("-" * 80) + # The .text accessor concatenates the parts into a markdown-formatted text. + print("\n", response.text)https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/python/code_execution.py#L22-L39 + +### Go + + ctx := context.Background() + client, err := genai.NewClient(ctx, &genai.ClientConfig{ + APIKey: os.Getenv("GEMINI_API_KEY"), + Backend: genai.BackendGeminiAPI, + }) + if err != nil { + log.Fatal(err) + } + + response, err := client.Models.GenerateContent( + ctx, + "gemini-2.0-pro-exp-02-05", + genai.Text( + `Write and execute code that calculates the sum of the first 50 prime numbers. + Ensure that only the executable code and its resulting output are generated.`, + ), + &genai.GenerateContentConfig{}, + ) + if err != nil { + log.Fatal(err) + } + + // Print the response. + printResponse(response) + + fmt.Println("---") + fmt.Println(response.Text())https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/go/code_execution.go#L14-L40 + +### Java + + Client client = new Client(); + + String prompt = """ + Write and execute code that calculates the sum of the first 50 prime numbers. + Ensure that only the executable code and its resulting output are generated. + """; + + GenerateContentResponse response = + client.models.generateContent( + "gemini-2.0-pro-exp-02-05", + prompt, + null); + + for (Part part : response.candidates().get().getFirst().content().get().parts().get()) { + System.out.println(part + "\n"); + } + + System.out.println("-".repeat(80)); + System.out.println(response.text());https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/java/src/main/java/com/example/gemini/CodeExecution.java#L32-L50 + +### Function Calling + +### Python + + from google import genai + from google.genai import types + + client = genai.Client() + + def add(a: float, b: float) -> float: + """returns a + b.""" + return a + b + + def subtract(a: float, b: float) -> float: + """returns a - b.""" + return a - b + + def multiply(a: float, b: float) -> float: + """returns a * b.""" + return a * b + + def divide(a: float, b: float) -> float: + """returns a / b.""" + return a / b + + # Create a chat session; function calling (via tools) is enabled in the config. + chat = client.chats.create( + model="gemini-2.0-flash", + config=types.GenerateContentConfig(tools=[add, subtract, multiply, divide]), + ) + response = chat.send_message( + message="I have 57 cats, each owns 44 mittens, how many mittens is that in total?" + ) + print(response.text)https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/python/function_calling.py#L22-L51 + +### Go + + ctx := context.Background() + client, err := genai.NewClient(ctx, &genai.ClientConfig{ + APIKey: os.Getenv("GEMINI_API_KEY"), + Backend: genai.BackendGeminiAPI, + }) + if err != nil { + log.Fatal(err) + } + modelName := "gemini-2.0-flash" + + // Create the function declarations for arithmetic operations. + addDeclaration := createArithmeticToolDeclaration("addNumbers", "Return the result of adding two numbers.") + subtractDeclaration := createArithmeticToolDeclaration("subtractNumbers", "Return the result of subtracting the second number from the first.") + multiplyDeclaration := createArithmeticToolDeclaration("multiplyNumbers", "Return the product of two numbers.") + divideDeclaration := createArithmeticToolDeclaration("divideNumbers", "Return the quotient of dividing the first number by the second.") + + // Group the function declarations as a tool. + tools := []*genai.Tool{ + { + FunctionDeclarations: []*genai.FunctionDeclaration{ + addDeclaration, + subtractDeclaration, + multiplyDeclaration, + divideDeclaration, + }, + }, + } + + // Create the content prompt. + contents := []*genai.Content{ + genai.NewContentFromText( + "I have 57 cats, each owns 44 mittens, how many mittens is that in total?", genai.RoleUser, + ), + } + + // Set up the generate content configuration with function calling enabled. + config := &genai.GenerateContentConfig{ + Tools: tools, + ToolConfig: &genai.ToolConfig{ + FunctionCallingConfig: &genai.FunctionCallingConfig{ + // The mode equivalent to FunctionCallingConfigMode.ANY in JS. + Mode: genai.FunctionCallingConfigModeAny, + }, + }, + } + + genContentResp, err := client.Models.GenerateContent(ctx, modelName, contents, config) + if err != nil { + log.Fatal(err) + } + + // Assume the response includes a list of function calls. + if len(genContentResp.FunctionCalls()) == 0 { + log.Println("No function call returned from the AI.") + return nil + } + functionCall := genContentResp.FunctionCalls()[0] + log.Printf("Function call: %+v\n", functionCall) + + // Marshal the Args map into JSON bytes. + argsMap, err := json.Marshal(functionCall.Args) + if err != nil { + log.Fatal(err) + } + + // Unmarshal the JSON bytes into the ArithmeticArgs struct. + var args ArithmeticArgs + if err := json.Unmarshal(argsMap, &args); err != nil { + log.Fatal(err) + } + + // Map the function name to the actual arithmetic function. + var result float64 + switch functionCall.Name { + case "addNumbers": + result = add(args.FirstParam, args.SecondParam) + case "subtractNumbers": + result = subtract(args.FirstParam, args.SecondParam) + case "multiplyNumbers": + result = multiply(args.FirstParam, args.SecondParam) + case "divideNumbers": + result = divide(args.FirstParam, args.SecondParam) + default: + return fmt.Errorf("unimplemented function: %s", functionCall.Name) + } + log.Printf("Function result: %v\n", result) + + // Prepare the final result message as content. + resultContents := []*genai.Content{ + genai.NewContentFromText("The final result is " + fmt.Sprintf("%v", result), genai.RoleUser), + } + + // Use GenerateContent to send the final result. + finalResponse, err := client.Models.GenerateContent(ctx, modelName, resultContents, &genai.GenerateContentConfig{}) + if err != nil { + log.Fatal(err) + } + + printResponse(finalResponse)https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/go/function_calling.go#L63-L161 + +### Node.js + + // Make sure to include the following import: + // import {GoogleGenAI} from '@google/genai'; + const ai = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY }); + + /** + * The add function returns the sum of two numbers. + * @param {number} a + * @param {number} b + * @returns {number} + */ + function add(a, b) { + return a + b; + } + + /** + * The subtract function returns the difference (a - b). + * @param {number} a + * @param {number} b + * @returns {number} + */ + function subtract(a, b) { + return a - b; + } + + /** + * The multiply function returns the product of two numbers. + * @param {number} a + * @param {number} b + * @returns {number} + */ + function multiply(a, b) { + return a * b; + } + + /** + * The divide function returns the quotient of a divided by b. + * @param {number} a + * @param {number} b + * @returns {number} + */ + function divide(a, b) { + return a / b; + } + + const addDeclaration = { + name: "addNumbers", + parameters: { + type: "object", + description: "Return the result of adding two numbers.", + properties: { + firstParam: { + type: "number", + description: + "The first parameter which can be an integer or a floating point number.", + }, + secondParam: { + type: "number", + description: + "The second parameter which can be an integer or a floating point number.", + }, + }, + required: ["firstParam", "secondParam"], + }, + }; + + const subtractDeclaration = { + name: "subtractNumbers", + parameters: { + type: "object", + description: + "Return the result of subtracting the second number from the first.", + properties: { + firstParam: { + type: "number", + description: "The first parameter.", + }, + secondParam: { + type: "number", + description: "The second parameter.", + }, + }, + required: ["firstParam", "secondParam"], + }, + }; + + const multiplyDeclaration = { + name: "multiplyNumbers", + parameters: { + type: "object", + description: "Return the product of two numbers.", + properties: { + firstParam: { + type: "number", + description: "The first parameter.", + }, + secondParam: { + type: "number", + description: "The second parameter.", + }, + }, + required: ["firstParam", "secondParam"], + }, + }; + + const divideDeclaration = { + name: "divideNumbers", + parameters: { + type: "object", + description: + "Return the quotient of dividing the first number by the second.", + properties: { + firstParam: { + type: "number", + description: "The first parameter.", + }, + secondParam: { + type: "number", + description: "The second parameter.", + }, + }, + required: ["firstParam", "secondParam"], + }, + }; + + // Step 1: Call generateContent with function calling enabled. + const generateContentResponse = await ai.models.generateContent({ + model: "gemini-2.0-flash", + contents: + "I have 57 cats, each owns 44 mittens, how many mittens is that in total?", + config: { + toolConfig: { + functionCallingConfig: { + mode: FunctionCallingConfigMode.ANY, + }, + }, + tools: [ + { + functionDeclarations: [ + addDeclaration, + subtractDeclaration, + multiplyDeclaration, + divideDeclaration, + ], + }, + ], + }, + }); + + // Step 2: Extract the function call.( + // Assuming the response contains a 'functionCalls' array. + const functionCall = + generateContentResponse.functionCalls && + generateContentResponse.functionCalls[0]; + console.log(functionCall); + + // Parse the arguments. + const args = functionCall.args; + // Expected args format: { firstParam: number, secondParam: number } + + // Step 3: Invoke the actual function based on the function name. + const functionMapping = { + addNumbers: add, + subtractNumbers: subtract, + multiplyNumbers: multiply, + divideNumbers: divide, + }; + const func = functionMapping[functionCall.name]; + if (!func) { + console.error("Unimplemented error:", functionCall.name); + return generateContentResponse; + } + const resultValue = func(args.firstParam, args.secondParam); + console.log("Function result:", resultValue); + + // Step 4: Use the chat API to send the result as the final answer. + const chat = ai.chats.create({ model: "gemini-2.0-flash" }); + const chatResponse = await chat.sendMessage({ + message: "The final result is " + resultValue, + }); + console.log(chatResponse.text); + return chatResponse; + } + https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/javascript/function_calling.js#L22-L-1 + +### Shell + + + cat > tools.json << EOF + { + "function_declarations": [ + { + "name": "enable_lights", + "description": "Turn on the lighting system." + }, + { + "name": "set_light_color", + "description": "Set the light color. Lights must be enabled for this to work.", + "parameters": { + "type": "object", + "properties": { + "rgb_hex": { + "type": "string", + "description": "The light color as a 6-digit hex string, e.g. ff0000 for red." + } + }, + "required": [ + "rgb_hex" + ] + } + }, + { + "name": "stop_lights", + "description": "Turn off the lighting system." + } + ] + } + EOF + + curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash:generateContent?key=$GEMINI_API_KEY" \ + -H 'Content-Type: application/json' \ + -d @<(echo ' + { + "system_instruction": { + "parts": { + "text": "You are a helpful lighting system bot. You can turn lights on and off, and you can set the color. Do not perform any other tasks." + } + }, + "tools": ['$(cat tools.json)'], + + "tool_config": { + "function_calling_config": {"mode": "auto"} + }, + + "contents": { + "role": "user", + "parts": { + "text": "Turn on the lights please." + } + } + } + ') 2>/dev/null |sed -n '/"content"/,/"finishReason"/p'https://github.com/google-gemini/deprecated-generative-ai-python/blob/7a7cc5474ddaa0255a4410e05361028a24400abd/samples/rest/function_calling.sh#L4-L59 + +### Java + + Client client = new Client(); + + FunctionDeclaration addFunction = + FunctionDeclaration.builder() + .name("addNumbers") + .parameters( + Schema.builder() + .type("object") + .properties(Map.of( + "firstParam", Schema.builder().type("number").description("First number").build(), + "secondParam", Schema.builder().type("number").description("Second number").build())) + .required(Arrays.asList("firstParam", "secondParam")) + .build()) + .build(); + + FunctionDeclaration subtractFunction = + FunctionDeclaration.builder() + .name("subtractNumbers") + .parameters( + Schema.builder() + .type("object") + .properties(Map.of( + "firstParam", Schema.builder().type("number").description("First number").build(), + "secondParam", Schema.builder().type("number").description("Second number").build())) + .required(Arrays.asList("firstParam", "secondParam")) + .build()) + .build(); + + FunctionDeclaration multiplyFunction = + FunctionDeclaration.builder() + .name("multiplyNumbers") + .parameters( + Schema.builder() + .type("object") + .properties(Map.of( + "firstParam", Schema.builder().type("number").description("First number").build(), + "secondParam", Schema.builder().type("number").description("Second number").build())) + .required(Arrays.asList("firstParam", "secondParam")) + .build()) + .build(); + + FunctionDeclaration divideFunction = + FunctionDeclaration.builder() + .name("divideNumbers") + .parameters( + Schema.builder() + .type("object") + .properties(Map.of( + "firstParam", Schema.builder().type("number").description("First number").build(), + "secondParam", Schema.builder().type("number").description("Second number").build())) + .required(Arrays.asList("firstParam", "secondParam")) + .build()) + .build(); + + GenerateContentConfig config = GenerateContentConfig.builder() + .toolConfig(ToolConfig.builder().functionCallingConfig( + FunctionCallingConfig.builder().mode("ANY").build() + ).build()) + .tools( + Collections.singletonList( + Tool.builder().functionDeclarations( + Arrays.asList( + addFunction, + subtractFunction, + divideFunction, + multiplyFunction + ) + ).build() + + ) + ) + .build(); + + GenerateContentResponse response = + client.models.generateContent( + "gemini-2.0-flash", + "I have 57 cats, each owns 44 mittens, how many mittens is that in total?", + config); + + + if (response.functionCalls() == null || response.functionCalls().isEmpty()) { + System.err.println("No function call received"); + return null; + } + + var functionCall = response.functionCalls().getFirst(); + String functionName = functionCall.name().get(); + var arguments = functionCall.args(); + + Map> functionMapping = new HashMap<>(); + functionMapping.put("addNumbers", (a, b) -> a + b); + functionMapping.put("subtractNumbers", (a, b) -> a - b); + functionMapping.put("multiplyNumbers", (a, b) -> a * b); + functionMapping.put("divideNumbers", (a, b) -> b != 0 ? a / b : Double.NaN); + + BiFunction function = functionMapping.get(functionName); + + Number firstParam = (Number) arguments.get().get("firstParam"); + Number secondParam = (Number) arguments.get().get("secondParam"); + Double result = function.apply(firstParam.doubleValue(), secondParam.doubleValue()); + + System.out.println(result);https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/java/src/main/java/com/example/gemini/FunctionCalling.java#L38-L139 + +### Generation config + +### Python + + from google import genai + from google.genai import types + + client = genai.Client() + response = client.models.generate_content( + model="gemini-2.0-flash", + contents="Tell me a story about a magic backpack.", + config=types.GenerateContentConfig( + candidate_count=1, + stop_sequences=["x"], + max_output_tokens=20, + temperature=1.0, + ), + ) + print(response.text)https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/python/configure_model_parameters.py#L22-L36 + +### Node.js + + // Make sure to include the following import: + // import {GoogleGenAI} from '@google/genai'; + const ai = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY }); + + const response = await ai.models.generateContent({ + model: "gemini-2.0-flash", + contents: "Tell me a story about a magic backpack.", + config: { + candidateCount: 1, + stopSequences: ["x"], + maxOutputTokens: 20, + temperature: 1.0, + }, + }); + + console.log(response.text);https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/javascript/configure_model_parameters.js#L22-L37 + +### Go + + ctx := context.Background() + client, err := genai.NewClient(ctx, &genai.ClientConfig{ + APIKey: os.Getenv("GEMINI_API_KEY"), + Backend: genai.BackendGeminiAPI, + }) + if err != nil { + log.Fatal(err) + } + + // Create local variables for parameters. + candidateCount := int32(1) + maxOutputTokens := int32(20) + temperature := float32(1.0) + + response, err := client.Models.GenerateContent( + ctx, + "gemini-2.0-flash", + genai.Text("Tell me a story about a magic backpack."), + &genai.GenerateContentConfig{ + CandidateCount: candidateCount, + StopSequences: []string{"x"}, + MaxOutputTokens: maxOutputTokens, + Temperature: &temperature, + }, + ) + if err != nil { + log.Fatal(err) + } + + printResponse(response)https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/go/configure_model_parameters.go#L13-L42 + +### Shell + + curl https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash:generateContent?key=$GEMINI_API_KEY \ + -H 'Content-Type: application/json' \ + -X POST \ + -d '{ + "contents": [{ + "parts":[ + {"text": "Explain how AI works"} + ] + }], + "generationConfig": { + "stopSequences": [ + "Title" + ], + "temperature": 1.0, + "maxOutputTokens": 800, + "topP": 0.8, + "topK": 10 + } + }' 2> /dev/null | grep "text"https://github.com/google-gemini/deprecated-generative-ai-python/blob/7a7cc5474ddaa0255a4410e05361028a24400abd/samples/rest/configure_model_parameters.sh#L4-L23 + +### Java + + Client client = new Client(); + + GenerateContentConfig config = + GenerateContentConfig.builder() + .candidateCount(1) + .stopSequences(List.of("x")) + .maxOutputTokens(20) + .temperature(1.0F) + .build(); + + GenerateContentResponse response = + client.models.generateContent( + "gemini-2.0-flash", + "Tell me a story about a magic backpack.", + config); + + System.out.println(response.text());https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/java/src/main/java/com/example/gemini/ConfigureModelParameters.java#L29-L45 + +### Safety Settings + +### Python + + from google import genai + from google.genai import types + + client = genai.Client() + unsafe_prompt = ( + "I support Martians Soccer Club and I think Jupiterians Football Club sucks! " + "Write a ironic phrase about them including expletives." + ) + response = client.models.generate_content( + model="gemini-2.0-flash", + contents=unsafe_prompt, + config=types.GenerateContentConfig( + safety_settings=[ + types.SafetySetting( + category="HARM_CATEGORY_HATE_SPEECH", + threshold="BLOCK_MEDIUM_AND_ABOVE", + ), + types.SafetySetting( + category="HARM_CATEGORY_HARASSMENT", threshold="BLOCK_ONLY_HIGH" + ), + ] + ), + ) + try: + print(response.text) + except Exception: + print("No information generated by the model.") + + print(response.candidates[0].safety_ratings)https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/python/safety_settings.py#L48-L76 + +### Node.js + + // Make sure to include the following import: + // import {GoogleGenAI} from '@google/genai'; + const ai = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY }); + const unsafePrompt = + "I support Martians Soccer Club and I think Jupiterians Football Club sucks! Write a ironic phrase about them including expletives."; + + const response = await ai.models.generateContent({ + model: "gemini-2.0-flash", + contents: unsafePrompt, + config: { + safetySettings: [ + { + category: "HARM_CATEGORY_HATE_SPEECH", + threshold: "BLOCK_MEDIUM_AND_ABOVE", + }, + { + category: "HARM_CATEGORY_HARASSMENT", + threshold: "BLOCK_ONLY_HIGH", + }, + ], + }, + }); + + try { + console.log("Generated text:", response.text); + } catch (error) { + console.log("No information generated by the model."); + } + console.log("Safety ratings:", response.candidates[0].safetyRatings); + return response; + } + https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/javascript/safety_settings.js#L49-L-1 + +### Go + + ctx := context.Background() + client, err := genai.NewClient(ctx, &genai.ClientConfig{ + APIKey: os.Getenv("GEMINI_API_KEY"), + Backend: genai.BackendGeminiAPI, + }) + if err != nil { + log.Fatal(err) + } + + unsafePrompt := "I support Martians Soccer Club and I think Jupiterians Football Club sucks! " + + "Write a ironic phrase about them including expletives." + + config := &genai.GenerateContentConfig{ + SafetySettings: []*genai.SafetySetting{ + { + Category: "HARM_CATEGORY_HATE_SPEECH", + Threshold: "BLOCK_MEDIUM_AND_ABOVE", + }, + { + Category: "HARM_CATEGORY_HARASSMENT", + Threshold: "BLOCK_ONLY_HIGH", + }, + }, + } + contents := []*genai.Content{ + genai.NewContentFromText(unsafePrompt, genai.RoleUser), + } + response, err := client.Models.GenerateContent(ctx, "gemini-2.0-flash", contents, config) + if err != nil { + log.Fatal(err) + } + + // Print the generated text. + text := response.Text() + fmt.Println("Generated text:", text) + + // Print the and safety ratings from the first candidate. + if len(response.Candidates) > 0 { + fmt.Println("Finish reason:", response.Candidates[0].FinishReason) + safetyRatings, err := json.MarshalIndent(response.Candidates[0].SafetyRatings, "", " ") + if err != nil { + return err + } + fmt.Println("Safety ratings:", string(safetyRatings)) + } else { + fmt.Println("No candidate returned.") + }https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/go/safety_settings.go#L60-L106 + +### Shell + + echo '{ + "safetySettings": [ + {"category": "HARM_CATEGORY_HARASSMENT", "threshold": "BLOCK_ONLY_HIGH"}, + {"category": "HARM_CATEGORY_HATE_SPEECH", "threshold": "BLOCK_MEDIUM_AND_ABOVE"} + ], + "contents": [{ + "parts":[{ + "text": "'I support Martians Soccer Club and I think Jupiterians Football Club sucks! Write a ironic phrase about them.'"}]}]}' > request.json + + curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash:generateContent?key=$GEMINI_API_KEY" \ + -H 'Content-Type: application/json' \ + -X POST \ + -d @request.json 2> /dev/nullhttps://github.com/google-gemini/deprecated-generative-ai-python/blob/7a7cc5474ddaa0255a4410e05361028a24400abd/samples/rest/safety_settings.sh#L20-L33 + +### Java + + Client client = new Client(); + + String unsafePrompt = """ + I support Martians Soccer Club and I think Jupiterians Football Club sucks! + Write a ironic phrase about them including expletives. + """; + + GenerateContentConfig config = + GenerateContentConfig.builder() + .safetySettings(Arrays.asList( + SafetySetting.builder() + .category("HARM_CATEGORY_HATE_SPEECH") + .threshold("BLOCK_MEDIUM_AND_ABOVE") + .build(), + SafetySetting.builder() + .category("HARM_CATEGORY_HARASSMENT") + .threshold("BLOCK_ONLY_HIGH") + .build() + )).build(); + + GenerateContentResponse response = + client.models.generateContent( + "gemini-2.0-flash", + unsafePrompt, + config); + + try { + System.out.println(response.text()); + } catch (Exception e) { + System.out.println("No information generated by the model"); + } + + System.out.println(response.candidates().get().getFirst().safetyRatings());https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/java/src/main/java/com/example/gemini/SafetySettings.java#L60-L92 + +### System Instruction + +### Python + + from google import genai + from google.genai import types + + client = genai.Client() + response = client.models.generate_content( + model="gemini-2.0-flash", + contents="Good morning! How are you?", + config=types.GenerateContentConfig( + system_instruction="You are a cat. Your name is Neko." + ), + ) + print(response.text)https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/python/system_instruction.py#L22-L33 + +### Node.js + + // Make sure to include the following import: + // import {GoogleGenAI} from '@google/genai'; + const ai = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY }); + const response = await ai.models.generateContent({ + model: "gemini-2.0-flash", + contents: "Good morning! How are you?", + config: { + systemInstruction: "You are a cat. Your name is Neko.", + }, + }); + console.log(response.text);https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/javascript/system_instruction.js#L22-L32 + +### Go + + ctx := context.Background() + client, err := genai.NewClient(ctx, &genai.ClientConfig{ + APIKey: os.Getenv("GEMINI_API_KEY"), + Backend: genai.BackendGeminiAPI, + }) + if err != nil { + log.Fatal(err) + } + + // Construct the user message contents. + contents := []*genai.Content{ + genai.NewContentFromText("Good morning! How are you?", genai.RoleUser), + } + + // Set the system instruction as a *genai.Content. + config := &genai.GenerateContentConfig{ + SystemInstruction: genai.NewContentFromText("You are a cat. Your name is Neko.", genai.RoleUser), + } + + response, err := client.Models.GenerateContent(ctx, "gemini-2.0-flash", contents, config) + if err != nil { + log.Fatal(err) + } + printResponse(response)https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/go/system_instruction.go#L13-L36 + +### Shell + + curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash:generateContent?key=$GEMINI_API_KEY" \ + -H 'Content-Type: application/json' \ + -d '{ "system_instruction": { + "parts": + { "text": "You are a cat. Your name is Neko."}}, + "contents": { + "parts": { + "text": "Hello there"}}}'https://github.com/google-gemini/deprecated-generative-ai-python/blob/7a7cc5474ddaa0255a4410e05361028a24400abd/samples/rest/system_instruction.sh#L4-L12 + +### Java + + Client client = new Client(); + + Part textPart = Part.builder().text("You are a cat. Your name is Neko.").build(); + + Content content = Content.builder().role("system").parts(ImmutableList.of(textPart)).build(); + + GenerateContentConfig config = GenerateContentConfig.builder() + .systemInstruction(content) + .build(); + + GenerateContentResponse response = + client.models.generateContent( + "gemini-2.0-flash", + "Good morning! How are you?", + config); + + System.out.println(response.text());https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/java/src/main/java/com/example/gemini/SystemInstruction.java#L30-L46 + +### Response body + +If successful, the response body contains an instance of `https://ai.google.dev/api/generate-content#v1beta.GenerateContentResponse`. + +## Method: models.streamGenerateContent + +- [Endpoint](https://ai.google.dev/api/generate-content#body.HTTP_TEMPLATE) +- [Path parameters](https://ai.google.dev/api/generate-content#body.PATH_PARAMETERS) +- [Request body](https://ai.google.dev/api/generate-content#body.request_body) + - [JSON representation](https://ai.google.dev/api/generate-content#body.request_body.SCHEMA_REPRESENTATION) +- [Response body](https://ai.google.dev/api/generate-content#body.response_body) +- [Authorization scopes](https://ai.google.dev/api/generate-content#body.aspect) +- [Example request](https://ai.google.dev/api/generate-content#body.codeSnippets) + - [Text](https://ai.google.dev/api/generate-content#body.codeSnippets.group) + - [Image](https://ai.google.dev/api/generate-content#body.codeSnippets.group_1) + - [Audio](https://ai.google.dev/api/generate-content#body.codeSnippets.group_2) + - [Video](https://ai.google.dev/api/generate-content#body.codeSnippets.group_3) + - [PDF](https://ai.google.dev/api/generate-content#body.codeSnippets.group_4) + - [Chat](https://ai.google.dev/api/generate-content#body.codeSnippets.group_5) + +Generates a [streamed response](https://ai.google.dev/gemini-api/docs/text-generation?lang=python#generate-a-text-stream) from the model given an input `GenerateContentRequest`. + +### Endpoint + +post `https://generativelanguage.googleapis.com/v1beta/{model=models/*}:streamGenerateContent` + +### Path parameters + +`model` `string` Required. The name of the `Model` to use for generating the completion. + +Format: `models/{model}`. It takes the form `models/{model}`. + +### Request body + +The request body contains data with the following structure: +Fields `contents[]` ``object (`https://ai.google.dev/api/caching#Content`)`` Required. The content of the current conversation with the model. + +For single-turn queries, this is a single instance. For multi-turn queries like [chat](https://ai.google.dev/gemini-api/docs/text-generation#chat), this is a repeated field that contains the conversation history and the latest request. +`tools[]` ``object (`https://ai.google.dev/api/caching#Tool`)`` Optional. A list of `Tools` the `Model` may use to generate the next response. + +A `Tool` is a piece of code that enables the system to interact with external systems to perform an action, or set of actions, outside of knowledge and scope of the `Model`. Supported `Tool`s are `Function` and `codeExecution`. Refer to the [Function calling](https://ai.google.dev/gemini-api/docs/function-calling) and the [Code execution](https://ai.google.dev/gemini-api/docs/code-execution) guides to learn more. +`toolConfig` ``object (`https://ai.google.dev/api/caching#ToolConfig`)`` Optional. Tool configuration for any `Tool` specified in the request. Refer to the [Function calling guide](https://ai.google.dev/gemini-api/docs/function-calling#function_calling_mode) for a usage example. +`safetySettings[]` ``object (`https://ai.google.dev/api/generate-content#v1beta.SafetySetting`)`` Optional. A list of unique `SafetySetting` instances for blocking unsafe content. + +This will be enforced on the `GenerateContentRequest.contents` and `GenerateContentResponse.candidates`. There should not be more than one setting for each `SafetyCategory` type. The API will block any contents and responses that fail to meet the thresholds set by these settings. This list overrides the default settings for each `SafetyCategory` specified in the safetySettings. If there is no `SafetySetting` for a given `SafetyCategory` provided in the list, the API will use the default safety setting for that category. Harm categories HARM_CATEGORY_HATE_SPEECH, HARM_CATEGORY_SEXUALLY_EXPLICIT, HARM_CATEGORY_DANGEROUS_CONTENT, HARM_CATEGORY_HARASSMENT, HARM_CATEGORY_CIVIC_INTEGRITY are supported. Refer to the [guide](https://ai.google.dev/gemini-api/docs/safety-settings) for detailed information on available safety settings. Also refer to the [Safety guidance](https://ai.google.dev/gemini-api/docs/safety-guidance) to learn how to incorporate safety considerations in your AI applications. +`systemInstruction` ``object (`https://ai.google.dev/api/caching#Content`)`` Optional. Developer set [system instruction(s)](https://ai.google.dev/gemini-api/docs/system-instructions). Currently, text only. +`generationConfig` ``object (`https://ai.google.dev/api/generate-content#v1beta.GenerationConfig`)`` Optional. Configuration options for model generation and outputs. +`cachedContent` `string` Optional. The name of the content [cached](https://ai.google.dev/gemini-api/docs/caching) to use as context to serve the prediction. Format: `cachedContents/{cachedContent}` +`serviceTier` ``enum (`https://ai.google.dev/api/generate-content#v1beta.ServiceTier`)`` Optional. The service tier of the request. +`store` `boolean` Optional. Configures the logging behavior for a given request. If set, it takes precedence over the project-level logging config. + +### Example request + +### Text + +### Python + + from google import genai + + client = genai.Client() + response = client.models.generate_content_stream( + model="gemini-2.0-flash", contents="Write a story about a magic backpack." + ) + for chunk in response: + print(chunk.text) + print("_" * 80)https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/python/text_generation.py#L37-L45 + +### Node.js + + // Make sure to include the following import: + // import {GoogleGenAI} from '@google/genai'; + const ai = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY }); + + const response = await ai.models.generateContentStream({ + model: "gemini-2.0-flash", + contents: "Write a story about a magic backpack.", + }); + let text = ""; + for await (const chunk of response) { + console.log(chunk.text); + text += chunk.text; + }https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/javascript/text_generation.js#L51-L63 + +### Go + + ctx := context.Background() + client, err := genai.NewClient(ctx, &genai.ClientConfig{ + APIKey: os.Getenv("GEMINI_API_KEY"), + Backend: genai.BackendGeminiAPI, + }) + if err != nil { + log.Fatal(err) + } + contents := []*genai.Content{ + genai.NewContentFromText("Write a story about a magic backpack.", genai.RoleUser), + } + for response, err := range client.Models.GenerateContentStream( + ctx, + "gemini-2.0-flash", + contents, + nil, + ) { + if err != nil { + log.Fatal(err) + } + fmt.Print(response.Candidates[0].Content.Parts[0].Text) + }https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/go/text_generation.go#L38-L59 + +### Shell + + curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash:streamGenerateContent?alt=sse&key=${GEMINI_API_KEY}" \ + -H 'Content-Type: application/json' \ + --no-buffer \ + -d '{ "contents":[{"parts":[{"text": "Write a story about a magic backpack."}]}]}'https://github.com/google-gemini/deprecated-generative-ai-python/blob/7a7cc5474ddaa0255a4410e05361028a24400abd/samples/rest/text_generation.sh#L33-L37 + +### Java + + Client client = new Client(); + + ResponseStream responseStream = + client.models.generateContentStream( + "gemini-2.0-flash", + "Write a story about a magic backpack.", + null); + + StringBuilder response = new StringBuilder(); + for (GenerateContentResponse res : responseStream) { + System.out.print(res.text()); + response.append(res.text()); + } + + responseStream.close();https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/java/src/main/java/com/example/gemini/TextGeneration.java#L49-L63 + +### Image + +### Python + + from google import genai + import PIL.Image + + client = genai.Client() + organ = PIL.Image.open(media / "organ.jpg") + response = client.models.generate_content_stream( + model="gemini-2.0-flash", contents=["Tell me about this instrument", organ] + ) + for chunk in response: + print(chunk.text) + print("_" * 80)https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/python/text_generation.py#L63-L73 + +### Node.js + + // Make sure to include the following import: + // import {GoogleGenAI} from '@google/genai'; + const ai = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY }); + + const organ = await ai.files.upload({ + file: path.join(media, "organ.jpg"), + }); + + const response = await ai.models.generateContentStream({ + model: "gemini-2.0-flash", + contents: [ + createUserContent([ + "Tell me about this instrument", + createPartFromUri(organ.uri, organ.mimeType) + ]), + ], + }); + let text = ""; + for await (const chunk of response) { + console.log(chunk.text); + text += chunk.text; + }https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/javascript/text_generation.js#L94-L115 + +### Go + + ctx := context.Background() + client, err := genai.NewClient(ctx, &genai.ClientConfig{ + APIKey: os.Getenv("GEMINI_API_KEY"), + Backend: genai.BackendGeminiAPI, + }) + if err != nil { + log.Fatal(err) + } + file, err := client.Files.UploadFromPath( + ctx, + filepath.Join(getMedia(), "organ.jpg"), + &genai.UploadFileConfig{ + MIMEType : "image/jpeg", + }, + ) + if err != nil { + log.Fatal(err) + } + parts := []*genai.Part{ + genai.NewPartFromText("Tell me about this instrument"), + genai.NewPartFromURI(file.URI, file.MIMEType), + } + contents := []*genai.Content{ + genai.NewContentFromParts(parts, genai.RoleUser), + } + for response, err := range client.Models.GenerateContentStream( + ctx, + "gemini-2.0-flash", + contents, + nil, + ) { + if err != nil { + log.Fatal(err) + } + fmt.Print(response.Candidates[0].Content.Parts[0].Text) + }https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/go/text_generation.go#L104-L139 + +### Shell + + cat > "$TEMP_JSON" << EOF + { + "contents": [{ + "parts":[ + {"text": "Tell me about this instrument"}, + { + "inline_data": { + "mime_type":"image/jpeg", + "data": "$(cat "$TEMP_B64")" + } + } + ] + }] + } + EOF + + curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash:streamGenerateContent?alt=sse&key=$GEMINI_API_KEY" \ + -H 'Content-Type: application/json' \ + -X POST \ + -d "@$TEMP_JSON" 2> /dev/nullhttps://github.com/google-gemini/deprecated-generative-ai-python/blob/7a7cc5474ddaa0255a4410e05361028a24400abd/samples/rest/text_generation.sh#L74-L94 + +### Java + + Client client = new Client(); + + String path = media_path + "organ.jpg"; + byte[] imageData = Files.readAllBytes(Paths.get(path)); + + Content content = + Content.fromParts( + Part.fromText("Tell me about this instrument."), + Part.fromBytes(imageData, "image/jpeg")); + + + ResponseStream responseStream = + client.models.generateContentStream( + "gemini-2.0-flash", + content, + null); + + StringBuilder response = new StringBuilder(); + for (GenerateContentResponse res : responseStream) { + System.out.print(res.text()); + response.append(res.text()); + } + + responseStream.close();https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/java/src/main/java/com/example/gemini/TextGeneration.java#L89-L112 + +### Audio + +### Python + + from google import genai + + client = genai.Client() + sample_audio = client.files.upload(file=media / "sample.mp3") + response = client.models.generate_content_stream( + model="gemini-2.0-flash", + contents=["Give me a summary of this audio file.", sample_audio], + ) + for chunk in response: + print(chunk.text) + print("_" * 80)https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/python/text_generation.py#L131-L141 + +### Go + + ctx := context.Background() + client, err := genai.NewClient(ctx, &genai.ClientConfig{ + APIKey: os.Getenv("GEMINI_API_KEY"), + Backend: genai.BackendGeminiAPI, + }) + if err != nil { + log.Fatal(err) + } + + file, err := client.Files.UploadFromPath( + ctx, + filepath.Join(getMedia(), "sample.mp3"), + &genai.UploadFileConfig{ + MIMEType : "audio/mpeg", + }, + ) + if err != nil { + log.Fatal(err) + } + + parts := []*genai.Part{ + genai.NewPartFromText("Give me a summary of this audio file."), + genai.NewPartFromURI(file.URI, file.MIMEType), + } + + contents := []*genai.Content{ + genai.NewContentFromParts(parts, genai.RoleUser), + } + + for result, err := range client.Models.GenerateContentStream( + ctx, + "gemini-2.0-flash", + contents, + nil, + ) { + if err != nil { + log.Fatal(err) + } + fmt.Print(result.Candidates[0].Content.Parts[0].Text) + }https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/go/text_generation.go#L296-L335 + +### Shell + + # Use File API to upload audio data to API request. + MIME_TYPE=$(file -b --mime-type "${AUDIO_PATH}") + NUM_BYTES=$(wc -c < "${AUDIO_PATH}") + DISPLAY_NAME=AUDIO + + tmp_header_file=upload-header.tmp + + # Initial resumable request defining metadata. + # The upload url is in the response headers dump them to a file. + curl "${BASE_URL}/upload/v1beta/files?key=${GEMINI_API_KEY}" \ + -D upload-header.tmp \ + -H "X-Goog-Upload-Protocol: resumable" \ + -H "X-Goog-Upload-Command: start" \ + -H "X-Goog-Upload-Header-Content-Length: ${NUM_BYTES}" \ + -H "X-Goog-Upload-Header-Content-Type: ${MIME_TYPE}" \ + -H "Content-Type: application/json" \ + -d "{'file': {'display_name': '${DISPLAY_NAME}'}}" 2> /dev/null + + upload_url=$(grep -i "x-goog-upload-url: " "${tmp_header_file}" | cut -d" " -f2 | tr -d "\r") + rm "${tmp_header_file}" + + # Upload the actual bytes. + curl "${upload_url}" \ + -H "Content-Length: ${NUM_BYTES}" \ + -H "X-Goog-Upload-Offset: 0" \ + -H "X-Goog-Upload-Command: upload, finalize" \ + --data-binary "@${AUDIO_PATH}" 2> /dev/null > file_info.json + + file_uri=$(jq ".file.uri" file_info.json) + echo file_uri=$file_uri + + curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash:streamGenerateContent?alt=sse&key=$GEMINI_API_KEY" \ + -H 'Content-Type: application/json' \ + -X POST \ + -d '{ + "contents": [{ + "parts":[ + {"text": "Please describe this file."}, + {"file_data":{"mime_type": "audio/mpeg", "file_uri": '$file_uri'}}] + }] + }' 2> /dev/null > response.json + + cat response.json + echohttps://github.com/google-gemini/deprecated-generative-ai-python/blob/7a7cc5474ddaa0255a4410e05361028a24400abd/samples/rest/text_generation.sh#L224-L268 + +### Video + +### Python + + from google import genai + import time + + client = genai.Client() + # Video clip (CC BY 3.0) from https://peach.blender.org/download/ + myfile = client.files.upload(file=media / "Big_Buck_Bunny.mp4") + print(f"{myfile=}") + + # Poll until the video file is completely processed (state becomes ACTIVE). + while not myfile.state or myfile.state.name != "ACTIVE": + print("Processing video...") + print("File state:", myfile.state) + time.sleep(5) + myfile = client.files.get(name=myfile.name) + + response = client.models.generate_content_stream( + model="gemini-2.0-flash", contents=[myfile, "Describe this video clip"] + ) + for chunk in response: + print(chunk.text) + print("_" * 80)https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/python/text_generation.py#L169-L189 + +### Node.js + + // Make sure to include the following import: + // import {GoogleGenAI} from '@google/genai'; + const ai = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY }); + + let video = await ai.files.upload({ + file: path.join(media, 'Big_Buck_Bunny.mp4'), + }); + + // Poll until the video file is completely processed (state becomes ACTIVE). + while (!video.state || video.state.toString() !== 'ACTIVE') { + console.log('Processing video...'); + console.log('File state: ', video.state); + await sleep(5000); + video = await ai.files.get({name: video.name}); + } + + const response = await ai.models.generateContentStream({ + model: "gemini-2.0-flash", + contents: [ + createUserContent([ + "Describe this video clip", + createPartFromUri(video.uri, video.mimeType), + ]), + ], + }); + let text = ""; + for await (const chunk of response) { + console.log(chunk.text); + text += chunk.text; + }https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/javascript/text_generation.js#L269-L298 + +### Go + + ctx := context.Background() + client, err := genai.NewClient(ctx, &genai.ClientConfig{ + APIKey: os.Getenv("GEMINI_API_KEY"), + Backend: genai.BackendGeminiAPI, + }) + if err != nil { + log.Fatal(err) + } + + file, err := client.Files.UploadFromPath( + ctx, + filepath.Join(getMedia(), "Big_Buck_Bunny.mp4"), + &genai.UploadFileConfig{ + MIMEType : "video/mp4", + }, + ) + if err != nil { + log.Fatal(err) + } + + // Poll until the video file is completely processed (state becomes ACTIVE). + for file.State == genai.FileStateUnspecified || file.State != genai.FileStateActive { + fmt.Println("Processing video...") + fmt.Println("File state:", file.State) + time.Sleep(5 * time.Second) + + file, err = client.Files.Get(ctx, file.Name, nil) + if err != nil { + log.Fatal(err) + } + } + + parts := []*genai.Part{ + genai.NewPartFromText("Describe this video clip"), + genai.NewPartFromURI(file.URI, file.MIMEType), + } + + contents := []*genai.Content{ + genai.NewContentFromParts(parts, genai.RoleUser), + } + + for result, err := range client.Models.GenerateContentStream( + ctx, + "gemini-2.0-flash", + contents, + nil, + ) { + if err != nil { + log.Fatal(err) + } + fmt.Print(result.Candidates[0].Content.Parts[0].Text) + }https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/go/text_generation.go#L394-L445 + +### Shell + + # Use File API to upload audio data to API request. + MIME_TYPE=$(file -b --mime-type "${VIDEO_PATH}") + NUM_BYTES=$(wc -c < "${VIDEO_PATH}") + DISPLAY_NAME=VIDEO_PATH + + # Initial resumable request defining metadata. + # The upload url is in the response headers dump them to a file. + curl "${BASE_URL}/upload/v1beta/files?key=${GEMINI_API_KEY}" \ + -D upload-header.tmp \ + -H "X-Goog-Upload-Protocol: resumable" \ + -H "X-Goog-Upload-Command: start" \ + -H "X-Goog-Upload-Header-Content-Length: ${NUM_BYTES}" \ + -H "X-Goog-Upload-Header-Content-Type: ${MIME_TYPE}" \ + -H "Content-Type: application/json" \ + -d "{'file': {'display_name': '${DISPLAY_NAME}'}}" 2> /dev/null + + upload_url=$(grep -i "x-goog-upload-url: " "${tmp_header_file}" | cut -d" " -f2 | tr -d "\r") + rm "${tmp_header_file}" + + # Upload the actual bytes. + curl "${upload_url}" \ + -H "Content-Length: ${NUM_BYTES}" \ + -H "X-Goog-Upload-Offset: 0" \ + -H "X-Goog-Upload-Command: upload, finalize" \ + --data-binary "@${VIDEO_PATH}" 2> /dev/null > file_info.json + + file_uri=$(jq ".file.uri" file_info.json) + echo file_uri=$file_uri + + state=$(jq ".file.state" file_info.json) + echo state=$state + + while [[ "($state)" = *"PROCESSING"* ]]; + do + echo "Processing video..." + sleep 5 + # Get the file of interest to check state + curl https://generativelanguage.googleapis.com/v1beta/files/$name > file_info.json + state=$(jq ".file.state" file_info.json) + done + + curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash:streamGenerateContent?alt=sse&key=$GEMINI_API_KEY" \ + -H 'Content-Type: application/json' \ + -X POST \ + -d '{ + "contents": [{ + "parts":[ + {"text": "Please describe this file."}, + {"file_data":{"mime_type": "video/mp4", "file_uri": '$file_uri'}}] + }] + }' 2> /dev/null > response.json + + cat response.json + echohttps://github.com/google-gemini/deprecated-generative-ai-python/blob/7a7cc5474ddaa0255a4410e05361028a24400abd/samples/rest/text_generation.sh#L335-L389 + +### PDF + +### Python + + from google import genai + + client = genai.Client() + sample_pdf = client.files.upload(file=media / "test.pdf") + response = client.models.generate_content_stream( + model="gemini-2.0-flash", + contents=["Give me a summary of this document:", sample_pdf], + ) + + for chunk in response: + print(chunk.text) + print("_" * 80)https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/python/text_generation.py#L207-L218 + +### Go + + ctx := context.Background() + client, err := genai.NewClient(ctx, &genai.ClientConfig{ + APIKey: os.Getenv("GEMINI_API_KEY"), + Backend: genai.BackendGeminiAPI, + }) + if err != nil { + log.Fatal(err) + } + + file, err := client.Files.UploadFromPath( + ctx, + filepath.Join(getMedia(), "test.pdf"), + &genai.UploadFileConfig{ + MIMEType : "application/pdf", + }, + ) + if err != nil { + log.Fatal(err) + } + + parts := []*genai.Part{ + genai.NewPartFromText("Give me a summary of this document:"), + genai.NewPartFromURI(file.URI, file.MIMEType), + } + + contents := []*genai.Content{ + genai.NewContentFromParts(parts, genai.RoleUser), + } + + for result, err := range client.Models.GenerateContentStream( + ctx, + "gemini-2.0-flash", + contents, + nil, + ) { + if err != nil { + log.Fatal(err) + } + fmt.Print(result.Candidates[0].Content.Parts[0].Text) + }https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/go/text_generation.go#L492-L531 + +### Shell + + MIME_TYPE=$(file -b --mime-type "${PDF_PATH}") + NUM_BYTES=$(wc -c < "${PDF_PATH}") + DISPLAY_NAME=TEXT + + + echo $MIME_TYPE + tmp_header_file=upload-header.tmp + + # Initial resumable request defining metadata. + # The upload url is in the response headers dump them to a file. + curl "${BASE_URL}/upload/v1beta/files?key=${GEMINI_API_KEY}" \ + -D upload-header.tmp \ + -H "X-Goog-Upload-Protocol: resumable" \ + -H "X-Goog-Upload-Command: start" \ + -H "X-Goog-Upload-Header-Content-Length: ${NUM_BYTES}" \ + -H "X-Goog-Upload-Header-Content-Type: ${MIME_TYPE}" \ + -H "Content-Type: application/json" \ + -d "{'file': {'display_name': '${DISPLAY_NAME}'}}" 2> /dev/null + + upload_url=$(grep -i "x-goog-upload-url: " "${tmp_header_file}" | cut -d" " -f2 | tr -d "\r") + rm "${tmp_header_file}" + + # Upload the actual bytes. + curl "${upload_url}" \ + -H "Content-Length: ${NUM_BYTES}" \ + -H "X-Goog-Upload-Offset: 0" \ + -H "X-Goog-Upload-Command: upload, finalize" \ + --data-binary "@${PDF_PATH}" 2> /dev/null > file_info.json + + file_uri=$(jq ".file.uri" file_info.json) + echo file_uri=$file_uri + + # Now generate content using that file + curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash:streamGenerateContent?alt=sse&key=$GEMINI_API_KEY" \ + -H 'Content-Type: application/json' \ + -X POST \ + -d '{ + "contents": [{ + "parts":[ + {"text": "Can you add a few more lines to this poem?"}, + {"file_data":{"mime_type": "application/pdf", "file_uri": '$file_uri'}}] + }] + }' 2> /dev/null > response.json + + cat response.json + echohttps://github.com/google-gemini/deprecated-generative-ai-python/blob/7a7cc5474ddaa0255a4410e05361028a24400abd/samples/rest/text_generation.sh#L445-L491 + +### Chat + +### Python + + from google import genai + from google.genai import types + + client = genai.Client() + chat = client.chats.create( + model="gemini-2.0-flash", + history=[ + types.Content(role="user", parts=[types.Part(text="Hello")]), + types.Content( + role="model", + parts=[ + types.Part( + text="Great to meet you. What would you like to know?" + ) + ], + ), + ], + ) + response = chat.send_message_stream(message="I have 2 dogs in my house.") + for chunk in response: + print(chunk.text) + print("_" * 80) + response = chat.send_message_stream(message="How many paws are in my house?") + for chunk in response: + print(chunk.text) + print("_" * 80) + + print(chat.get_history())https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/python/chat.py#L52-L79 + +### Node.js + + // Make sure to include the following import: + // import {GoogleGenAI} from '@google/genai'; + const ai = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY }); + const chat = ai.chats.create({ + model: "gemini-2.0-flash", + history: [ + { + role: "user", + parts: [{ text: "Hello" }], + }, + { + role: "model", + parts: [{ text: "Great to meet you. What would you like to know?" }], + }, + ], + }); + + console.log("Streaming response for first message:"); + const stream1 = await chat.sendMessageStream({ + message: "I have 2 dogs in my house.", + }); + for await (const chunk of stream1) { + console.log(chunk.text); + console.log("_".repeat(80)); + } + + console.log("Streaming response for second message:"); + const stream2 = await chat.sendMessageStream({ + message: "How many paws are in my house?", + }); + for await (const chunk of stream2) { + console.log(chunk.text); + console.log("_".repeat(80)); + } + + console.log(chat.getHistory());https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/javascript/chat.js#L66-L101 + +### Go + + ctx := context.Background() + client, err := genai.NewClient(ctx, &genai.ClientConfig{ + APIKey: os.Getenv("GEMINI_API_KEY"), + Backend: genai.BackendGeminiAPI, + }) + if err != nil { + log.Fatal(err) + } + + history := []*genai.Content{ + genai.NewContentFromText("Hello", genai.RoleUser), + genai.NewContentFromText("Great to meet you. What would you like to know?", genai.RoleModel), + } + chat, err := client.Chats.Create(ctx, "gemini-2.0-flash", nil, history) + if err != nil { + log.Fatal(err) + } + + for chunk, err := range chat.SendMessageStream(ctx, genai.Part{Text: "I have 2 dogs in my house."}) { + if err != nil { + log.Fatal(err) + } + fmt.Println(chunk.Text()) + fmt.Println(strings.Repeat("_", 64)) + } + + for chunk, err := range chat.SendMessageStream(ctx, genai.Part{Text: "How many paws are in my house?"}) { + if err != nil { + log.Fatal(err) + } + fmt.Println(chunk.Text()) + fmt.Println(strings.Repeat("_", 64)) + } + + fmt.Println(chat.History(false))https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/go/chat.go#L54-L88 + +### Shell + + curl https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash:streamGenerateContent?alt=sse&key=$GEMINI_API_KEY \ + -H 'Content-Type: application/json' \ + -X POST \ + -d '{ + "contents": [ + {"role":"user", + "parts":[{ + "text": "Hello"}]}, + {"role": "model", + "parts":[{ + "text": "Great to meet you. What would you like to know?"}]}, + {"role":"user", + "parts":[{ + "text": "I have two dogs in my house. How many paws are in my house?"}]}, + ] + }' 2> /dev/null | grep "text"https://github.com/google-gemini/deprecated-generative-ai-python/blob/7a7cc5474ddaa0255a4410e05361028a24400abd/samples/rest/chat.sh#L27-L43 + +### Response body + +If successful, the response body contains a stream of `https://ai.google.dev/api/generate-content#v1beta.GenerateContentResponse` instances. + +## GenerateContentResponse + +- [JSON representation](https://ai.google.dev/api/generate-content#SCHEMA_REPRESENTATION) +- [PromptFeedback](https://ai.google.dev/api/generate-content#PromptFeedback) + - [JSON representation](https://ai.google.dev/api/generate-content#PromptFeedback.SCHEMA_REPRESENTATION) +- [BlockReason](https://ai.google.dev/api/generate-content#BlockReason) +- [UsageMetadata](https://ai.google.dev/api/generate-content#UsageMetadata) + - [JSON representation](https://ai.google.dev/api/generate-content#UsageMetadata.SCHEMA_REPRESENTATION) +- [ModelStatus](https://ai.google.dev/api/generate-content#ModelStatus) + - [JSON representation](https://ai.google.dev/api/generate-content#ModelStatus.SCHEMA_REPRESENTATION) +- [ModelStage](https://ai.google.dev/api/generate-content#ModelStage) + +Response from the model supporting multiple candidate responses. + +Safety ratings and content filtering are reported for both prompt in `GenerateContentResponse.prompt_feedback` and for each candidate in `finishReason` and in `safetyRatings`. The API: - Returns either all requested candidates or none of them - Returns no candidates at all only if there was something wrong with the prompt (check `promptFeedback`) - Reports feedback on each candidate in `finishReason` and `safetyRatings`. +Fields `candidates[]` ``object (`https://ai.google.dev/api/generate-content#v1beta.Candidate`)`` Candidate responses from the model. +`promptFeedback` ``object (`https://ai.google.dev/api/generate-content#PromptFeedback`)`` Returns the prompt's feedback related to the content filters. +`usageMetadata` ``object (`https://ai.google.dev/api/generate-content#UsageMetadata`)`` Output only. Metadata on the generation requests' token usage. +`modelVersion` `string` Output only. The model version used to generate the response. +`responseId` `string` Output only. responseId is used to identify each response. +`modelStatus` ``object (`https://ai.google.dev/api/generate-content#ModelStatus`)`` Output only. The current model status of this model. + +| JSON representation | +|---| +| ``` { "candidates": [ { object (`https://ai.google.dev/api/generate-content#v1beta.Candidate`) } ], "promptFeedback": { object (`https://ai.google.dev/api/generate-content#PromptFeedback`) }, "usageMetadata": { object (`https://ai.google.dev/api/generate-content#UsageMetadata`) }, "modelVersion": string, "responseId": string, "modelStatus": { object (`https://ai.google.dev/api/generate-content#ModelStatus`) } } ``` | + +## PromptFeedback + +A set of the feedback metadata the prompt specified in `GenerateContentRequest.content`. +Fields `blockReason` ``enum (`https://ai.google.dev/api/generate-content#BlockReason`)`` Optional. If set, the prompt was blocked and no candidates are returned. Rephrase the prompt. +`safetyRatings[]` ``object (`https://ai.google.dev/api/generate-content#v1beta.SafetyRating`)`` Ratings for safety of the prompt. There is at most one rating per category. + +| JSON representation | +|---| +| ``` { "blockReason": enum (`https://ai.google.dev/api/generate-content#BlockReason`), "safetyRatings": [ { object (`https://ai.google.dev/api/generate-content#v1beta.SafetyRating`) } ] } ``` | + +## BlockReason + +Specifies the reason why the prompt was blocked. + +| Enums || +|---|---| +| `BLOCK_REASON_UNSPECIFIED` | Default value. This value is unused. | +| `SAFETY` | Prompt was blocked due to safety reasons. Inspect `safetyRatings` to understand which safety category blocked it. | +| `OTHER` | Prompt was blocked due to unknown reasons. | +| `BLOCKLIST` | Prompt was blocked due to the terms which are included from the terminology blocklist. | +| `PROHIBITED_CONTENT` | Prompt was blocked due to prohibited content. | +| `IMAGE_SAFETY` | Candidates blocked due to unsafe image generation content. | + +## UsageMetadata + +Metadata on the generation request's token usage. +Fields `promptTokenCount` `integer` Number of tokens in the prompt. When `cachedContent` is set, this is still the total effective prompt size meaning this includes the number of tokens in the cached content. +`cachedContentTokenCount` `integer` Number of tokens in the cached part of the prompt (the cached content) +`candidatesTokenCount` `integer` Total number of tokens across all the generated response candidates. +`toolUsePromptTokenCount` `integer` Output only. Number of tokens present in tool-use prompt(s). +`thoughtsTokenCount` `integer` Output only. Number of tokens of thoughts for thinking models. +`totalTokenCount` `integer` Total token count for the generation request (prompt + response candidates). +`promptTokensDetails[]` ``object (`https://ai.google.dev/api/generate-content#v1beta.ModalityTokenCount`)`` Output only. List of modalities that were processed in the request input. +`cacheTokensDetails[]` ``object (`https://ai.google.dev/api/generate-content#v1beta.ModalityTokenCount`)`` Output only. List of modalities of the cached content in the request input. +`candidatesTokensDetails[]` ``object (`https://ai.google.dev/api/generate-content#v1beta.ModalityTokenCount`)`` Output only. List of modalities that were returned in the response. +`toolUsePromptTokensDetails[]` ``object (`https://ai.google.dev/api/generate-content#v1beta.ModalityTokenCount`)`` Output only. List of modalities that were processed for tool-use request inputs. + +| JSON representation | +|---| +| ``` { "promptTokenCount": integer, "cachedContentTokenCount": integer, "candidatesTokenCount": integer, "toolUsePromptTokenCount": integer, "thoughtsTokenCount": integer, "totalTokenCount": integer, "promptTokensDetails": [ { object (`https://ai.google.dev/api/generate-content#v1beta.ModalityTokenCount`) } ], "cacheTokensDetails": [ { object (`https://ai.google.dev/api/generate-content#v1beta.ModalityTokenCount`) } ], "candidatesTokensDetails": [ { object (`https://ai.google.dev/api/generate-content#v1beta.ModalityTokenCount`) } ], "toolUsePromptTokensDetails": [ { object (`https://ai.google.dev/api/generate-content#v1beta.ModalityTokenCount`) } ] } ``` | + +## ModelStatus + +The status of the underlying model. This is used to indicate the stage of the underlying model and the retirement time if applicable. +Fields `modelStage` ``enum (`https://ai.google.dev/api/generate-content#ModelStage`)`` The stage of the underlying model. +`retirementTime` ``string (`https://protobuf.dev/reference/protobuf/google.protobuf#timestamp` format)`` The time at which the model will be retired. + +Uses RFC 3339, where generated output will always be Z-normalized and use 0, 3, 6 or 9 fractional digits. Offsets other than "Z" are also accepted. Examples: `"2014-10-02T15:01:23Z"`, `"2014-10-02T15:01:23.045123456Z"` or `"2014-10-02T15:01:23+05:30"`. +`message` `string` A message explaining the model status. + +| JSON representation | +|---| +| ``` { "modelStage": enum (`https://ai.google.dev/api/generate-content#ModelStage`), "retirementTime": string, "message": string } ``` | + +## ModelStage + +Defines the stage of the underlying model. + +| Enums || +|---|---| +| `MODEL_STAGE_UNSPECIFIED` | Unspecified model stage. | +| `UNSTABLE_EXPERIMENTAL` | The underlying model is subject to lots of tunings. > [!WARNING] > This item is deprecated! | +| `EXPERIMENTAL` | Models in this stage are for experimental purposes only. | +| `PREVIEW` | Models in this stage are more mature than experimental models. | +| `STABLE` | Models in this stage are considered stable and ready for production use. | +| `LEGACY` | If the model is on this stage, it means that this model is on the path to deprecation in near future. Only existing customers can use this model. | +| `DEPRECATED` | Models in this stage are deprecated. These models cannot be used. > [!WARNING] > This item is deprecated! | +| `RETIRED` | Models in this stage are retired. These models cannot be used. | + +## Candidate + +- [JSON representation](https://ai.google.dev/api/generate-content#SCHEMA_REPRESENTATION) +- [FinishReason](https://ai.google.dev/api/generate-content#FinishReason) +- [GroundingAttribution](https://ai.google.dev/api/generate-content#GroundingAttribution) + - [JSON representation](https://ai.google.dev/api/generate-content#GroundingAttribution.SCHEMA_REPRESENTATION) +- [AttributionSourceId](https://ai.google.dev/api/generate-content#AttributionSourceId) + - [JSON representation](https://ai.google.dev/api/generate-content#AttributionSourceId.SCHEMA_REPRESENTATION) +- [GroundingPassageId](https://ai.google.dev/api/generate-content#GroundingPassageId) + - [JSON representation](https://ai.google.dev/api/generate-content#GroundingPassageId.SCHEMA_REPRESENTATION) +- [SemanticRetrieverChunk](https://ai.google.dev/api/generate-content#SemanticRetrieverChunk) + - [JSON representation](https://ai.google.dev/api/generate-content#SemanticRetrieverChunk.SCHEMA_REPRESENTATION) +- [GroundingMetadata](https://ai.google.dev/api/generate-content#GroundingMetadata) + - [JSON representation](https://ai.google.dev/api/generate-content#GroundingMetadata.SCHEMA_REPRESENTATION) +- [SearchEntryPoint](https://ai.google.dev/api/generate-content#SearchEntryPoint) + - [JSON representation](https://ai.google.dev/api/generate-content#SearchEntryPoint.SCHEMA_REPRESENTATION) +- [GroundingChunk](https://ai.google.dev/api/generate-content#GroundingChunk) + - [JSON representation](https://ai.google.dev/api/generate-content#GroundingChunk.SCHEMA_REPRESENTATION) +- [Web](https://ai.google.dev/api/generate-content#Web) + - [JSON representation](https://ai.google.dev/api/generate-content#Web.SCHEMA_REPRESENTATION) +- [Image](https://ai.google.dev/api/generate-content#Image) + - [JSON representation](https://ai.google.dev/api/generate-content#Image.SCHEMA_REPRESENTATION) +- [RetrievedContext](https://ai.google.dev/api/generate-content#RetrievedContext) + - [JSON representation](https://ai.google.dev/api/generate-content#RetrievedContext.SCHEMA_REPRESENTATION) +- [CustomMetadata](https://ai.google.dev/api/generate-content#CustomMetadata) + - [JSON representation](https://ai.google.dev/api/generate-content#CustomMetadata.SCHEMA_REPRESENTATION) +- [StringList](https://ai.google.dev/api/generate-content#StringList) + - [JSON representation](https://ai.google.dev/api/generate-content#StringList.SCHEMA_REPRESENTATION) +- [Maps](https://ai.google.dev/api/generate-content#Maps) + - [JSON representation](https://ai.google.dev/api/generate-content#Maps.SCHEMA_REPRESENTATION) +- [PlaceAnswerSources](https://ai.google.dev/api/generate-content#PlaceAnswerSources) + - [JSON representation](https://ai.google.dev/api/generate-content#PlaceAnswerSources.SCHEMA_REPRESENTATION) +- [ReviewSnippet](https://ai.google.dev/api/generate-content#ReviewSnippet) + - [JSON representation](https://ai.google.dev/api/generate-content#ReviewSnippet.SCHEMA_REPRESENTATION) +- [GroundingSupport](https://ai.google.dev/api/generate-content#GroundingSupport) + - [JSON representation](https://ai.google.dev/api/generate-content#GroundingSupport.SCHEMA_REPRESENTATION) +- [Segment](https://ai.google.dev/api/generate-content#Segment) + - [JSON representation](https://ai.google.dev/api/generate-content#Segment.SCHEMA_REPRESENTATION) +- [RetrievalMetadata](https://ai.google.dev/api/generate-content#RetrievalMetadata) + - [JSON representation](https://ai.google.dev/api/generate-content#RetrievalMetadata.SCHEMA_REPRESENTATION) +- [LogprobsResult](https://ai.google.dev/api/generate-content#LogprobsResult) + - [JSON representation](https://ai.google.dev/api/generate-content#LogprobsResult.SCHEMA_REPRESENTATION) +- [TopCandidates](https://ai.google.dev/api/generate-content#TopCandidates) + - [JSON representation](https://ai.google.dev/api/generate-content#TopCandidates.SCHEMA_REPRESENTATION) +- [Candidate](https://ai.google.dev/api/generate-content#Candidate) + - [JSON representation](https://ai.google.dev/api/generate-content#Candidate.SCHEMA_REPRESENTATION) +- [UrlContextMetadata](https://ai.google.dev/api/generate-content#UrlContextMetadata) + - [JSON representation](https://ai.google.dev/api/generate-content#UrlContextMetadata.SCHEMA_REPRESENTATION) +- [UrlMetadata](https://ai.google.dev/api/generate-content#UrlMetadata) + - [JSON representation](https://ai.google.dev/api/generate-content#UrlMetadata.SCHEMA_REPRESENTATION) +- [UrlRetrievalStatus](https://ai.google.dev/api/generate-content#UrlRetrievalStatus) + +A response candidate generated from the model. +Fields `content` ``object (`https://ai.google.dev/api/caching#Content`)`` Output only. Generated content returned from the model. +`finishReason` ``enum (`https://ai.google.dev/api/generate-content#FinishReason`)`` Optional. Output only. The reason why the model stopped generating tokens. + +If empty, the model has not stopped generating tokens. +`safetyRatings[]` ``object (`https://ai.google.dev/api/generate-content#v1beta.SafetyRating`)`` List of ratings for the safety of a response candidate. + +There is at most one rating per category. +`citationMetadata` ``object (`https://ai.google.dev/api/generate-content#v1beta.CitationMetadata`)`` Output only. Citation information for model-generated candidate. + +This field may be populated with recitation information for any text included in the `content`. These are passages that are "recited" from copyrighted material in the foundational LLM's training data. +`tokenCount` `integer` Output only. Token count for this candidate. +`groundingAttributions[]` ``object (`https://ai.google.dev/api/generate-content#GroundingAttribution`)`` Output only. Attribution information for sources that contributed to a grounded answer. + +This field is populated for `GenerateAnswer` calls. +`groundingMetadata` ``object (`https://ai.google.dev/api/generate-content#GroundingMetadata`)`` Output only. Grounding metadata for the candidate. + +This field is populated for `GenerateContent` calls. +`avgLogprobs` `number` Output only. Average log probability score of the candidate. +`logprobsResult` ``object (`https://ai.google.dev/api/generate-content#LogprobsResult`)`` Output only. Log-likelihood scores for the response tokens and top tokens +`urlContextMetadata` ``object (`https://ai.google.dev/api/generate-content#UrlContextMetadata`)`` Output only. Metadata related to url context retrieval tool. +`index` `integer` Output only. Index of the candidate in the list of response candidates. +`finishMessage` `string` Optional. Output only. Details the reason why the model stopped generating tokens. This is populated only when `finishReason` is set. + +| JSON representation | +|---| +| ``` { "content": { object (`https://ai.google.dev/api/caching#Content`) }, "finishReason": enum (`https://ai.google.dev/api/generate-content#FinishReason`), "safetyRatings": [ { object (`https://ai.google.dev/api/generate-content#v1beta.SafetyRating`) } ], "citationMetadata": { object (`https://ai.google.dev/api/generate-content#v1beta.CitationMetadata`) }, "tokenCount": integer, "groundingAttributions": [ { object (`https://ai.google.dev/api/generate-content#GroundingAttribution`) } ], "groundingMetadata": { object (`https://ai.google.dev/api/generate-content#GroundingMetadata`) }, "avgLogprobs": number, "logprobsResult": { object (`https://ai.google.dev/api/generate-content#LogprobsResult`) }, "urlContextMetadata": { object (`https://ai.google.dev/api/generate-content#UrlContextMetadata`) }, "index": integer, "finishMessage": string } ``` | + +## FinishReason + +Defines the reason why the model stopped generating tokens. + +| Enums || +|---|---| +| `FINISH_REASON_UNSPECIFIED` | Default value. This value is unused. | +| `STOP` | Natural stop point of the model or provided stop sequence. | +| `MAX_TOKENS` | The maximum number of tokens as specified in the request was reached. | +| `SAFETY` | The response candidate content was flagged for safety reasons. | +| `RECITATION` | The response candidate content was flagged for recitation reasons. | +| `LANGUAGE` | The response candidate content was flagged for using an unsupported language. | +| `OTHER` | Unknown reason. | +| `BLOCKLIST` | Token generation stopped because the content contains forbidden terms. | +| `PROHIBITED_CONTENT` | Token generation stopped for potentially containing prohibited content. | +| `SPII` | Token generation stopped because the content potentially contains Sensitive Personally Identifiable Information (SPII). | +| `MALFORMED_FUNCTION_CALL` | The function call generated by the model is invalid. | +| `IMAGE_SAFETY` | Token generation stopped because generated images contain safety violations. | +| `IMAGE_PROHIBITED_CONTENT` | Image generation stopped because generated images has other prohibited content. | +| `IMAGE_OTHER` | Image generation stopped because of other miscellaneous issue. | +| `NO_IMAGE` | The model was expected to generate an image, but none was generated. | +| `IMAGE_RECITATION` | Image generation stopped due to recitation. | +| `UNEXPECTED_TOOL_CALL` | Model generated a tool call but no tools were enabled in the request. | +| `TOO_MANY_TOOL_CALLS` | Model called too many tools consecutively, thus the system exited execution. | +| `MISSING_THOUGHT_SIGNATURE` | Request has at least one thought signature missing. | +| `MALFORMED_RESPONSE` | Finished due to malformed response. | + +## GroundingAttribution + +Attribution for a source that contributed to an answer. +Fields `sourceId` ``object (`https://ai.google.dev/api/generate-content#AttributionSourceId`)`` Output only. Identifier for the source contributing to this attribution. +`content` ``object (`https://ai.google.dev/api/caching#Content`)`` Grounding source content that makes up this attribution. + +| JSON representation | +|---| +| ``` { "sourceId": { object (`https://ai.google.dev/api/generate-content#AttributionSourceId`) }, "content": { object (`https://ai.google.dev/api/caching#Content`) } } ``` | + +## AttributionSourceId + +Identifier for the source contributing to this attribution. +Fields `source` `Union type` `source` can be only one of the following: `groundingPassage` ``object (`https://ai.google.dev/api/generate-content#GroundingPassageId`)`` Identifier for an inline passage. +`semanticRetrieverChunk` ``object (`https://ai.google.dev/api/generate-content#SemanticRetrieverChunk`)`` Identifier for a `Chunk` fetched via Semantic Retriever. + +| JSON representation | +|---| +| ``` { // source "groundingPassage": { object (`https://ai.google.dev/api/generate-content#GroundingPassageId`) }, "semanticRetrieverChunk": { object (`https://ai.google.dev/api/generate-content#SemanticRetrieverChunk`) } // Union type } ``` | + +## GroundingPassageId + +Identifier for a part within a `GroundingPassage`. +Fields `passageId` `string` Output only. ID of the passage matching the `GenerateAnswerRequest`'s `GroundingPassage.id`. +`partIndex` `integer` Output only. Index of the part within the `GenerateAnswerRequest`'s `GroundingPassage.content`. + +| JSON representation | +|---| +| ``` { "passageId": string, "partIndex": integer } ``` | + +## SemanticRetrieverChunk + +Identifier for a `Chunk` retrieved via Semantic Retriever specified in the `GenerateAnswerRequest` using `SemanticRetrieverConfig`. +Fields `source` `string` Output only. Name of the source matching the request's `SemanticRetrieverConfig.source`. Example: `corpora/123` or `corpora/123/documents/abc` +`chunk` `string` Output only. Name of the `Chunk` containing the attributed text. Example: `corpora/123/documents/abc/chunks/xyz` + +| JSON representation | +|---| +| ``` { "source": string, "chunk": string } ``` | + +## GroundingMetadata + +Metadata returned to client when grounding is enabled. +Fields `groundingChunks[]` ``object (`https://ai.google.dev/api/generate-content#GroundingChunk`)`` List of supporting references retrieved from specified grounding source. When streaming, this only contains the grounding chunks that have not been included in the grounding metadata of previous responses. +`groundingSupports[]` ``object (`https://ai.google.dev/api/generate-content#GroundingSupport`)`` List of grounding support. +`webSearchQueries[]` `string` Web search queries for the following-up web search. +`imageSearchQueries[]` `string` Image search queries used for grounding. +`searchEntryPoint` ``object (`https://ai.google.dev/api/generate-content#SearchEntryPoint`)`` Optional. Google search entry for the following-up web searches. +`retrievalMetadata` ``object (`https://ai.google.dev/api/generate-content#RetrievalMetadata`)`` Metadata related to retrieval in the grounding flow. +`googleMapsWidgetContextToken` `string` Optional. Resource name of the Google Maps widget context token that can be used with the PlacesContextElement widget in order to render contextual data. Only populated in the case that grounding with Google Maps is enabled. + +| JSON representation | +|---| +| ``` { "groundingChunks": [ { object (`https://ai.google.dev/api/generate-content#GroundingChunk`) } ], "groundingSupports": [ { object (`https://ai.google.dev/api/generate-content#GroundingSupport`) } ], "webSearchQueries": [ string ], "imageSearchQueries": [ string ], "searchEntryPoint": { object (`https://ai.google.dev/api/generate-content#SearchEntryPoint`) }, "retrievalMetadata": { object (`https://ai.google.dev/api/generate-content#RetrievalMetadata`) }, "googleMapsWidgetContextToken": string } ``` | + +## SearchEntryPoint + +Google search entry point. +Fields `renderedContent` `string` Optional. Web content snippet that can be embedded in a web page or an app webview. +`sdkBlob` `string (https://developers.google.com/discovery/v1/type-format format)` Optional. Base64 encoded JSON representing array of \ tuple. + +A base64-encoded string. + +| JSON representation | +|---| +| ``` { "renderedContent": string, "sdkBlob": string } ``` | + +## GroundingChunk + +A `GroundingChunk` represents a segment of supporting evidence that grounds the model's response. It can be a chunk from the web, a retrieved context from a file, or information from Google Maps. +Fields `chunk_type` `Union type` Chunk type. `chunk_type` can be only one of the following: `web` ``object (`https://ai.google.dev/api/generate-content#Web`)`` Grounding chunk from the web. +`image` ``object (`https://ai.google.dev/api/generate-content#Image`)`` Optional. Grounding chunk from image search. +`retrievedContext` ``object (`https://ai.google.dev/api/generate-content#RetrievedContext`)`` Optional. Grounding chunk from context retrieved by the file search tool. +`maps` ``object (`https://ai.google.dev/api/generate-content#Maps`)`` Optional. Grounding chunk from Google Maps. + +| JSON representation | +|---| +| ``` { // chunk_type "web": { object (`https://ai.google.dev/api/generate-content#Web`) }, "image": { object (`https://ai.google.dev/api/generate-content#Image`) }, "retrievedContext": { object (`https://ai.google.dev/api/generate-content#RetrievedContext`) }, "maps": { object (`https://ai.google.dev/api/generate-content#Maps`) } // Union type } ``` | + +## Web + +Chunk from the web. +Fields `uri` `string` Output only. URI reference of the chunk. +`title` `string` Output only. Title of the chunk. + +| JSON representation | +|---| +| ``` { "uri": string, "title": string } ``` | + +## Image + +Chunk from image search. +Fields `sourceUri` `string` The web page URI for attribution. +`imageUri` `string` The image asset URL. +`title` `string` The title of the web page that the image is from. +`domain` `string` The root domain of the web page that the image is from, e.g. "example.com". + +| JSON representation | +|---| +| ``` { "sourceUri": string, "imageUri": string, "title": string, "domain": string } ``` | + +## RetrievedContext + +Chunk from context retrieved by the file search tool. +Fields `customMetadata[]` ``object (`https://ai.google.dev/api/generate-content#CustomMetadata`)`` Optional. User-provided metadata about the retrieved context. +`uri` `string` Optional. URI reference of the semantic retrieval document. +`title` `string` Optional. Title of the document. +`text` `string` Optional. Text of the chunk. +`fileSearchStore` `string` Optional. Name of the `FileSearchStore` containing the document. Example: `fileSearchStores/123` + +| JSON representation | +|---| +| ``` { "customMetadata": [ { object (`https://ai.google.dev/api/generate-content#CustomMetadata`) } ], "uri": string, "title": string, "text": string, "fileSearchStore": string } ``` | + +## CustomMetadata + +User provided metadata about the GroundingFact. +Fields `key` `string` The key of the metadata. +`value` `Union type` The value of the metadata. Can be a string, a list of strings, or a number. `value` can be only one of the following: `stringValue` `string` Optional. The string value of the metadata. +`stringListValue` ``object (`https://ai.google.dev/api/generate-content#StringList`)`` Optional. A list of string values for the metadata. +`numericValue` `number` Optional. The numeric value of the metadata. The expected range for this value depends on the specific `key` used. + +| JSON representation | +|---| +| ``` { "key": string, // value "stringValue": string, "stringListValue": { object (`https://ai.google.dev/api/generate-content#StringList`) }, "numericValue": number // Union type } ``` | + +## StringList + +A list of string values. +Fields `values[]` `string` The string values of the list. + +| JSON representation | +|---| +| ``` { "values": [ string ] } ``` | + +## Maps + +A grounding chunk from Google Maps. A Maps chunk corresponds to a single place. +Fields `uri` `string` URI reference of the place. +`title` `string` Title of the place. +`text` `string` Text description of the place answer. +`placeId` `string` The ID of the place, in `places/{placeId}` format. A user can use this ID to look up that place. +`placeAnswerSources` ``object (`https://ai.google.dev/api/generate-content#PlaceAnswerSources`)`` Sources that provide answers about the features of a given place in Google Maps. + +| JSON representation | +|---| +| ``` { "uri": string, "title": string, "text": string, "placeId": string, "placeAnswerSources": { object (`https://ai.google.dev/api/generate-content#PlaceAnswerSources`) } } ``` | + +## PlaceAnswerSources + +Collection of sources that provide answers about the features of a given place in Google Maps. Each PlaceAnswerSources message corresponds to a specific place in Google Maps. The Google Maps tool used these sources in order to answer questions about features of the place (e.g: "does Bar Foo have Wifi" or "is Foo Bar wheelchair accessible?"). Currently we only support review snippets as sources. +Fields `reviewSnippets[]` ``object (`https://ai.google.dev/api/generate-content#ReviewSnippet`)`` Snippets of reviews that are used to generate answers about the features of a given place in Google Maps. + +| JSON representation | +|---| +| ``` { "reviewSnippets": [ { object (`https://ai.google.dev/api/generate-content#ReviewSnippet`) } ] } ``` | + +## ReviewSnippet + +Encapsulates a snippet of a user review that answers a question about the features of a specific place in Google Maps. +Fields `reviewId` `string` The ID of the review snippet. +`googleMapsUri` `string` A link that corresponds to the user review on Google Maps. +`title` `string` Title of the review. + +| JSON representation | +|---| +| ``` { "reviewId": string, "googleMapsUri": string, "title": string } ``` | + +## GroundingSupport + +Grounding support. +Fields `groundingChunkIndices[]` `integer` Optional. A list of indices (into 'grounding_chunk' in `response.candidate.grounding_metadata`) specifying the citations associated with the claim. For instance \[1,3,4\] means that grounding_chunk\[1\], grounding_chunk\[3\], grounding_chunk\[4\] are the retrieved content attributed to the claim. If the response is streaming, the groundingChunkIndices refer to the indices across all responses. It is the client's responsibility to accumulate the grounding chunks from all responses (while maintaining the same order). +`confidenceScores[]` `number` Optional. Confidence score of the support references. Ranges from 0 to 1. 1 is the most confident. This list must have the same size as the groundingChunkIndices. +`renderedParts[]` `integer` Output only. Indices into the `parts` field of the candidate's content. These indices specify which rendered parts are associated with this support source. +`segment` ``object (`https://ai.google.dev/api/generate-content#Segment`)`` Segment of the content this support belongs to. + +| JSON representation | +|---| +| ``` { "groundingChunkIndices": [ integer ], "confidenceScores": [ number ], "renderedParts": [ integer ], "segment": { object (`https://ai.google.dev/api/generate-content#Segment`) } } ``` | + +## Segment + +Segment of the content. +Fields `partIndex` `integer` The index of a Part object within its parent Content object. +`startIndex` `integer` Start index in the given Part, measured in bytes. Offset from the start of the Part, inclusive, starting at zero. +`endIndex` `integer` End index in the given Part, measured in bytes. Offset from the start of the Part, exclusive, starting at zero. +`text` `string` The text corresponding to the segment from the response. + +| JSON representation | +|---| +| ``` { "partIndex": integer, "startIndex": integer, "endIndex": integer, "text": string } ``` | + +## RetrievalMetadata + +Metadata related to retrieval in the grounding flow. +Fields `googleSearchDynamicRetrievalScore` `number` Optional. Score indicating how likely information from google search could help answer the prompt. The score is in the range \[0, 1\], where 0 is the least likely and 1 is the most likely. This score is only populated when google search grounding and dynamic retrieval is enabled. It will be compared to the threshold to determine whether to trigger google search. + +| JSON representation | +|---| +| ``` { "googleSearchDynamicRetrievalScore": number } ``` | + +## LogprobsResult + +Logprobs Result +Fields `topCandidates[]` ``object (`https://ai.google.dev/api/generate-content#TopCandidates`)`` Length = total number of decoding steps. +`chosenCandidates[]` ``object (`https://ai.google.dev/api/generate-content#Candidate`)`` Length = total number of decoding steps. The chosen candidates may or may not be in topCandidates. +`logProbabilitySum` `number` Sum of log probabilities for all tokens. + +| JSON representation | +|---| +| ``` { "topCandidates": [ { object (`https://ai.google.dev/api/generate-content#TopCandidates`) } ], "chosenCandidates": [ { object (`https://ai.google.dev/api/generate-content#Candidate`) } ], "logProbabilitySum": number } ``` | + +## TopCandidates + +Candidates with top log probabilities at each decoding step. +Fields `candidates[]` ``object (`https://ai.google.dev/api/generate-content#Candidate`)`` Sorted by log probability in descending order. + +| JSON representation | +|---| +| ``` { "candidates": [ { object (`https://ai.google.dev/api/generate-content#Candidate`) } ] } ``` | + +## Candidate + +Candidate for the logprobs token and score. +Fields `token` `string` The candidate's token string value. +`tokenId` `integer` The candidate's token id value. +`logProbability` `number` The candidate's log probability. + +| JSON representation | +|---| +| ``` { "token": string, "tokenId": integer, "logProbability": number } ``` | + +## UrlContextMetadata + +Metadata related to url context retrieval tool. +Fields `urlMetadata[]` ``object (`https://ai.google.dev/api/generate-content#UrlMetadata`)`` List of url context. + +| JSON representation | +|---| +| ``` { "urlMetadata": [ { object (`https://ai.google.dev/api/generate-content#UrlMetadata`) } ] } ``` | + +## UrlMetadata + +Context of the a single url retrieval. +Fields `retrievedUrl` `string` Retrieved url by the tool. +`urlRetrievalStatus` ``enum (`https://ai.google.dev/api/generate-content#UrlRetrievalStatus`)`` Status of the url retrieval. + +| JSON representation | +|---| +| ``` { "retrievedUrl": string, "urlRetrievalStatus": enum (`https://ai.google.dev/api/generate-content#UrlRetrievalStatus`) } ``` | + +## UrlRetrievalStatus + +Status of the url retrieval. + +| Enums || +|---|---| +| `URL_RETRIEVAL_STATUS_UNSPECIFIED` | Default value. This value is unused. | +| `URL_RETRIEVAL_STATUS_SUCCESS` | Url retrieval is successful. | +| `URL_RETRIEVAL_STATUS_ERROR` | Url retrieval is failed due to error. | +| `URL_RETRIEVAL_STATUS_PAYWALL` | Url retrieval is failed because the content is behind paywall. | +| `URL_RETRIEVAL_STATUS_UNSAFE` | Url retrieval is failed because the content is unsafe. | + +## CitationMetadata + +- [JSON representation](https://ai.google.dev/api/generate-content#SCHEMA_REPRESENTATION) +- [CitationSource](https://ai.google.dev/api/generate-content#CitationSource) + - [JSON representation](https://ai.google.dev/api/generate-content#CitationSource.SCHEMA_REPRESENTATION) + +A collection of source attributions for a piece of content. +Fields `citationSources[]` ``object (`https://ai.google.dev/api/generate-content#CitationSource`)`` Citations to sources for a specific response. + +| JSON representation | +|---| +| ``` { "citationSources": [ { object (`https://ai.google.dev/api/generate-content#CitationSource`) } ] } ``` | + +## CitationSource + +A citation to a source for a portion of a specific response. +Fields `startIndex` `integer` Optional. Start of segment of the response that is attributed to this source. + +Index indicates the start of the segment, measured in bytes. +`endIndex` `integer` Optional. End of the attributed segment, exclusive. +`uri` `string` Optional. URI that is attributed as a source for a portion of the text. +`license` `string` Optional. License for the GitHub project that is attributed as a source for segment. + +License info is required for code citations. + +| JSON representation | +|---| +| ``` { "startIndex": integer, "endIndex": integer, "uri": string, "license": string } ``` | + +## GenerationConfig + +- [JSON representation](https://ai.google.dev/api/generate-content#SCHEMA_REPRESENTATION) +- [Modality](https://ai.google.dev/api/generate-content#Modality) +- [SpeechConfig](https://ai.google.dev/api/generate-content#SpeechConfig) + - [JSON representation](https://ai.google.dev/api/generate-content#SpeechConfig.SCHEMA_REPRESENTATION) +- [VoiceConfig](https://ai.google.dev/api/generate-content#VoiceConfig) + - [JSON representation](https://ai.google.dev/api/generate-content#VoiceConfig.SCHEMA_REPRESENTATION) +- [PrebuiltVoiceConfig](https://ai.google.dev/api/generate-content#PrebuiltVoiceConfig) + - [JSON representation](https://ai.google.dev/api/generate-content#PrebuiltVoiceConfig.SCHEMA_REPRESENTATION) +- [MultiSpeakerVoiceConfig](https://ai.google.dev/api/generate-content#MultiSpeakerVoiceConfig) + - [JSON representation](https://ai.google.dev/api/generate-content#MultiSpeakerVoiceConfig.SCHEMA_REPRESENTATION) +- [SpeakerVoiceConfig](https://ai.google.dev/api/generate-content#SpeakerVoiceConfig) + - [JSON representation](https://ai.google.dev/api/generate-content#SpeakerVoiceConfig.SCHEMA_REPRESENTATION) +- [ThinkingConfig](https://ai.google.dev/api/generate-content#ThinkingConfig) + - [JSON representation](https://ai.google.dev/api/generate-content#ThinkingConfig.SCHEMA_REPRESENTATION) +- [ThinkingLevel](https://ai.google.dev/api/generate-content#ThinkingLevel) +- [ImageConfig](https://ai.google.dev/api/generate-content#ImageConfig) + - [JSON representation](https://ai.google.dev/api/generate-content#ImageConfig.SCHEMA_REPRESENTATION) +- [MediaResolution](https://ai.google.dev/api/generate-content#MediaResolution) + +Configuration options for model generation and outputs. Not all parameters are configurable for every model. +Fields `stopSequences[]` `string` Optional. The set of character sequences (up to 5) that will stop output generation. If specified, the API will stop at the first appearance of a `stop_sequence`. The stop sequence will not be included as part of the response. +`responseMimeType` `string` Optional. MIME type of the generated candidate text. Supported MIME types are: `text/plain`: (default) Text output. `application/json`: JSON response in the response candidates. `text/x.enum`: ENUM as a string response in the response candidates. Refer to the [docs](https://ai.google.dev/gemini-api/docs/prompting_with_media#plain_text_formats) for a list of all supported text MIME types. +`responseSchema` ``object (`https://ai.google.dev/api/caching#Schema`)`` Optional. Output schema of the generated candidate text. Schemas must be a subset of the [OpenAPI schema](https://spec.openapis.org/oas/v3.0.3#schema) and can be objects, primitives or arrays. + +If set, a compatible `responseMimeType` must also be set. Compatible MIME types: `application/json`: Schema for JSON response. Refer to the [JSON text generation guide](https://ai.google.dev/gemini-api/docs/json-mode) for more details. +`_responseJsonSchema` ``value (`https://protobuf.dev/reference/protobuf/google.protobuf#value` format)`` Optional. Output schema of the generated response. This is an alternative to `responseSchema` that accepts [JSON Schema](https://json-schema.org/). + +If set, `responseSchema` must be omitted, but `responseMimeType` is required. + +While the full JSON Schema may be sent, not all features are supported. Specifically, only the following properties are supported: + +- `$id` +- `$defs` +- `$ref` +- `$anchor` +- `type` +- `format` +- `title` +- `description` +- `enum` (for strings and numbers) +- `items` +- `prefixItems` +- `minItems` +- `maxItems` +- `minimum` +- `maximum` +- `anyOf` +- `oneOf` (interpreted the same as `anyOf`) +- `properties` +- `additionalProperties` +- `required` + +The non-standard `propertyOrdering` property may also be set. + +Cyclic references are unrolled to a limited degree and, as such, may only be used within non-required properties. (Nullable properties are not sufficient.) If `$ref` is set on a sub-schema, no other properties, except for than those starting as a `$`, may be set. +`responseJsonSchema` ``value (`https://protobuf.dev/reference/protobuf/google.protobuf#value` format)`` Optional. An internal detail. Use `responseJsonSchema` rather than this field. +`responseModalities[]` ``enum (`https://ai.google.dev/api/generate-content#Modality`)`` Optional. The requested modalities of the response. Represents the set of modalities that the model can return, and should be expected in the response. This is an exact match to the modalities of the response. + +A model may have multiple combinations of supported modalities. If the requested modalities do not match any of the supported combinations, an error will be returned. + +An empty list is equivalent to requesting only text. +`candidateCount` `integer` Optional. Number of generated responses to return. If unset, this will default to 1. Please note that this doesn't work for previous generation models (Gemini 1.0 family) +`maxOutputTokens` `integer` Optional. The maximum number of tokens to include in a response candidate. + +Note: The default value varies by model, see the `Model.output_token_limit` attribute of the `Model` returned from the `getModel` function. +`temperature` `number` Optional. Controls the randomness of the output. + +Note: The default value varies by model, see the `Model.temperature` attribute of the `Model` returned from the `getModel` function. + +Values can range from \[0.0, 2.0\]. +`topP` `number` Optional. The maximum cumulative probability of tokens to consider when sampling. + +The model uses combined Top-k and Top-p (nucleus) sampling. + +Tokens are sorted based on their assigned probabilities so that only the most likely tokens are considered. Top-k sampling directly limits the maximum number of tokens to consider, while Nucleus sampling limits the number of tokens based on the cumulative probability. + +Note: The default value varies by `Model` and is specified by the`Model.top_p` attribute returned from the `getModel` function. An empty `topK` attribute indicates that the model doesn't apply top-k sampling and doesn't allow setting `topK` on requests. +`topK` `integer` Optional. The maximum number of tokens to consider when sampling. + +Gemini models use Top-p (nucleus) sampling or a combination of Top-k and nucleus sampling. Top-k sampling considers the set of `topK` most probable tokens. Models running with nucleus sampling don't allow topK setting. + +Note: The default value varies by `Model` and is specified by the`Model.top_p` attribute returned from the `getModel` function. An empty `topK` attribute indicates that the model doesn't apply top-k sampling and doesn't allow setting `topK` on requests. +`seed` `integer` Optional. Seed used in decoding. If not set, the request uses a randomly generated seed. +`presencePenalty` `number` Optional. Presence penalty applied to the next token's logprobs if the token has already been seen in the response. + +This penalty is binary on/off and not dependant on the number of times the token is used (after the first). Use `https://ai.google.dev/api/generate-content#FIELDS.frequency_penalty` for a penalty that increases with each use. + +A positive penalty will discourage the use of tokens that have already been used in the response, increasing the vocabulary. + +A negative penalty will encourage the use of tokens that have already been used in the response, decreasing the vocabulary. +`frequencyPenalty` `number` Optional. Frequency penalty applied to the next token's logprobs, multiplied by the number of times each token has been seen in the respponse so far. + +A positive penalty will discourage the use of tokens that have already been used, proportional to the number of times the token has been used: The more a token is used, the more difficult it is for the model to use that token again increasing the vocabulary of responses. + +Caution: A *negative* penalty will encourage the model to reuse tokens proportional to the number of times the token has been used. Small negative values will reduce the vocabulary of a response. Larger negative values will cause the model to start repeating a common token until it hits the `https://ai.google.dev/api/generate-content#FIELDS.max_output_tokens` limit. +`responseLogprobs` `boolean` Optional. If true, export the logprobs results in response. +`logprobs` `integer` Optional. Only valid if `https://ai.google.dev/api/generate-content#FIELDS.response_logprobs`. This sets the number of top logprobs, including the chosen candidate, to return at each decoding step in the `https://ai.google.dev/api/generate-content#FIELDS.logprobs_result`. The number must be in the range of \[0, 20\]. +`enableEnhancedCivicAnswers` `boolean` Optional. Enables enhanced civic answers. It may not be available for all models. +`speechConfig` ``object (`https://ai.google.dev/api/generate-content#SpeechConfig`)`` Optional. The speech generation config. +`thinkingConfig` ``object (`https://ai.google.dev/api/generate-content#ThinkingConfig`)`` Optional. Config for thinking features. An error will be returned if this field is set for models that don't support thinking. +`imageConfig` ``object (`https://ai.google.dev/api/generate-content#ImageConfig`)`` Optional. Config for image generation. An error will be returned if this field is set for models that don't support these config options. +`mediaResolution` ``enum (`https://ai.google.dev/api/generate-content#MediaResolution`)`` Optional. If specified, the media resolution specified will be used. + +| JSON representation | +|---| +| ``` { "stopSequences": [ string ], "responseMimeType": string, "responseSchema": { object (`https://ai.google.dev/api/caching#Schema`) }, "_responseJsonSchema": value, "responseJsonSchema": value, "responseModalities": [ enum (`https://ai.google.dev/api/generate-content#Modality`) ], "candidateCount": integer, "maxOutputTokens": integer, "temperature": number, "topP": number, "topK": integer, "seed": integer, "presencePenalty": number, "frequencyPenalty": number, "responseLogprobs": boolean, "logprobs": integer, "enableEnhancedCivicAnswers": boolean, "speechConfig": { object (`https://ai.google.dev/api/generate-content#SpeechConfig`) }, "thinkingConfig": { object (`https://ai.google.dev/api/generate-content#ThinkingConfig`) }, "imageConfig": { object (`https://ai.google.dev/api/generate-content#ImageConfig`) }, "mediaResolution": enum (`https://ai.google.dev/api/generate-content#MediaResolution`) } ``` | + +## Modality + +Supported modalities of the response. + +| Enums || +|---|---| +| `MODALITY_UNSPECIFIED` | Default value. | +| `TEXT` | Indicates the model should return text. | +| `IMAGE` | Indicates the model should return images. | +| `AUDIO` | Indicates the model should return audio. | + +## SpeechConfig + +Config for speech generation and transcription. +Fields `voiceConfig` ``object (`https://ai.google.dev/api/generate-content#VoiceConfig`)`` The configuration in case of single-voice output. +`multiSpeakerVoiceConfig` ``object (`https://ai.google.dev/api/generate-content#MultiSpeakerVoiceConfig`)`` Optional. The configuration for the multi-speaker setup. It is mutually exclusive with the voiceConfig field. +`languageCode` `string` Optional. The IETF [BCP-47](https://www.rfc-editor.org/rfc/bcp/bcp47.txt) language code that the user configured the app to use. Used for speech recognition and synthesis. + +Valid values are: `de-DE`, `en-AU`, `en-GB`, `en-IN`, `en-US`, `es-US`, `fr-FR`, `hi-IN`, `pt-BR`, `ar-XA`, `es-ES`, `fr-CA`, `id-ID`, `it-IT`, `ja-JP`, `tr-TR`, `vi-VN`, `bn-IN`, `gu-IN`, `kn-IN`, `ml-IN`, `mr-IN`, `ta-IN`, `te-IN`, `nl-NL`, `ko-KR`, `cmn-CN`, `pl-PL`, `ru-RU`, and `th-TH`. + +| JSON representation | +|---| +| ``` { "voiceConfig": { object (`https://ai.google.dev/api/generate-content#VoiceConfig`) }, "multiSpeakerVoiceConfig": { object (`https://ai.google.dev/api/generate-content#MultiSpeakerVoiceConfig`) }, "languageCode": string } ``` | + +## VoiceConfig + +The configuration for the voice to use. +Fields `voice_config` `Union type` The configuration for the speaker to use. `voice_config` can be only one of the following: `prebuiltVoiceConfig` ``object (`https://ai.google.dev/api/generate-content#PrebuiltVoiceConfig`)`` The configuration for the prebuilt voice to use. + +| JSON representation | +|---| +| ``` { // voice_config "prebuiltVoiceConfig": { object (`https://ai.google.dev/api/generate-content#PrebuiltVoiceConfig`) } // Union type } ``` | + +## PrebuiltVoiceConfig + +The configuration for the prebuilt speaker to use. +Fields `voiceName` `string` The name of the preset voice to use. + +| JSON representation | +|---| +| ``` { "voiceName": string } ``` | + +## MultiSpeakerVoiceConfig + +The configuration for the multi-speaker setup. +Fields `speakerVoiceConfigs[]` ``object (`https://ai.google.dev/api/generate-content#SpeakerVoiceConfig`)`` Required. All the enabled speaker voices. + +| JSON representation | +|---| +| ``` { "speakerVoiceConfigs": [ { object (`https://ai.google.dev/api/generate-content#SpeakerVoiceConfig`) } ] } ``` | + +## SpeakerVoiceConfig + +The configuration for a single speaker in a multi speaker setup. +Fields `speaker` `string` Required. The name of the speaker to use. Should be the same as in the prompt. +`voiceConfig` ``object (`https://ai.google.dev/api/generate-content#VoiceConfig`)`` Required. The configuration for the voice to use. + +| JSON representation | +|---| +| ``` { "speaker": string, "voiceConfig": { object (`https://ai.google.dev/api/generate-content#VoiceConfig`) } } ``` | + +## ThinkingConfig + +Config for thinking features. +Fields `includeThoughts` `boolean` Indicates whether to include thoughts in the response. If true, thoughts are returned only when available. +`thinkingBudget` `integer` The number of thoughts tokens that the model should generate. +`thinkingLevel` ``enum (`https://ai.google.dev/api/generate-content#ThinkingLevel`)`` Optional. Controls the maximum depth of the model's internal reasoning process before it produces a response. If not specified, the default is HIGH. Recommended for Gemini 3 or later models. Use with earlier models results in an error. + +| JSON representation | +|---| +| ``` { "includeThoughts": boolean, "thinkingBudget": integer, "thinkingLevel": enum (`https://ai.google.dev/api/generate-content#ThinkingLevel`) } ``` | + +## ThinkingLevel + +Allow user to specify how much to think using enum instead of integer budget. + +| Enums || +|---|---| +| `THINKING_LEVEL_UNSPECIFIED` | Default value. | +| `MINIMAL` | Little to no thinking. | +| `LOW` | Low thinking level. | +| `MEDIUM` | Medium thinking level. | +| `HIGH` | High thinking level. | + +## ImageConfig + +Config for image generation features. +Fields `aspectRatio` `string` Optional. The aspect ratio of the image to generate. Supported aspect ratios: `1:1`, `1:4`, `4:1`, `1:8`, `8:1`, `2:3`, `3:2`, `3:4`, `4:3`, `4:5`, `5:4`, `9:16`, `16:9`, or `21:9`. + +If not specified, the model will choose a default aspect ratio based on any reference images provided. +`imageSize` `string` Optional. Specifies the size of generated images. Supported values are `512`, `1K`, `2K`, `4K`. If not specified, the model will use default value `1K`. + +| JSON representation | +|---| +| ``` { "aspectRatio": string, "imageSize": string } ``` | + +## MediaResolution + +Media resolution for the input media. + +| Enums || +|---|---| +| `MEDIA_RESOLUTION_UNSPECIFIED` | Media resolution has not been set. | +| `MEDIA_RESOLUTION_LOW` | Media resolution set to low (64 tokens). | +| `MEDIA_RESOLUTION_MEDIUM` | Media resolution set to medium (256 tokens). | +| `MEDIA_RESOLUTION_HIGH` | Media resolution set to high (zoomed reframing with 256 tokens). | + +## HarmCategory + +The category of a rating. + +These categories cover various kinds of harms that developers may wish to adjust. + +| Enums || +|---|---| +| `HARM_CATEGORY_UNSPECIFIED` | Category is unspecified. | +| `HARM_CATEGORY_DEROGATORY` | **PaLM** - Negative or harmful comments targeting identity and/or protected attribute. | +| `HARM_CATEGORY_TOXICITY` | **PaLM** - Content that is rude, disrespectful, or profane. | +| `HARM_CATEGORY_VIOLENCE` | **PaLM** - Describes scenarios depicting violence against an individual or group, or general descriptions of gore. | +| `HARM_CATEGORY_SEXUAL` | **PaLM** - Contains references to sexual acts or other lewd content. | +| `HARM_CATEGORY_MEDICAL` | **PaLM** - Promotes unchecked medical advice. | +| `HARM_CATEGORY_DANGEROUS` | **PaLM** - Dangerous content that promotes, facilitates, or encourages harmful acts. | +| `HARM_CATEGORY_HARASSMENT` | **Gemini** - Harassment content. | +| `HARM_CATEGORY_HATE_SPEECH` | **Gemini** - Hate speech and content. | +| `HARM_CATEGORY_SEXUALLY_EXPLICIT` | **Gemini** - Sexually explicit content. | +| `HARM_CATEGORY_DANGEROUS_CONTENT` | **Gemini** - Dangerous content. | +| `HARM_CATEGORY_CIVIC_INTEGRITY` | **Gemini** - Content that may be used to harm civic integrity. DEPRECATED: use enableEnhancedCivicAnswers instead. > [!WARNING] > This item is deprecated! | + +## ModalityTokenCount + +- [JSON representation](https://ai.google.dev/api/generate-content#SCHEMA_REPRESENTATION) +- [Modality](https://ai.google.dev/api/generate-content#Modality) + +Represents token counting info for a single modality. +Fields `modality` ``enum (`https://ai.google.dev/api/generate-content#Modality`)`` The modality associated with this token count. +`tokenCount` `integer` Number of tokens. + +| JSON representation | +|---| +| ``` { "modality": enum (`https://ai.google.dev/api/generate-content#Modality`), "tokenCount": integer } ``` | + +## Modality + +Content Part modality + +| Enums || +|---|---| +| `MODALITY_UNSPECIFIED` | Unspecified modality. | +| `TEXT` | Plain text. | +| `IMAGE` | Image. | +| `VIDEO` | Video. | +| `AUDIO` | Audio. | +| `DOCUMENT` | Document, e.g. PDF. | + +## SafetyRating + +- [JSON representation](https://ai.google.dev/api/generate-content#SCHEMA_REPRESENTATION) +- [HarmProbability](https://ai.google.dev/api/generate-content#HarmProbability) + +Safety rating for a piece of content. + +The safety rating contains the category of harm and the harm probability level in that category for a piece of content. Content is classified for safety across a number of harm categories and the probability of the harm classification is included here. +Fields `category` ``enum (`https://ai.google.dev/api/generate-content#v1beta.HarmCategory`)`` Required. The category for this rating. +`probability` ``enum (`https://ai.google.dev/api/generate-content#HarmProbability`)`` Required. The probability of harm for this content. +`blocked` `boolean` Was this content blocked because of this rating? + +| JSON representation | +|---| +| ``` { "category": enum (`https://ai.google.dev/api/generate-content#v1beta.HarmCategory`), "probability": enum (`https://ai.google.dev/api/generate-content#HarmProbability`), "blocked": boolean } ``` | + +## HarmProbability + +The probability that a piece of content is harmful. + +The classification system gives the probability of the content being unsafe. This does not indicate the severity of harm for a piece of content. + +| Enums || +|---|---| +| `HARM_PROBABILITY_UNSPECIFIED` | Probability is unspecified. | +| `NEGLIGIBLE` | Content has a negligible chance of being unsafe. | +| `LOW` | Content has a low chance of being unsafe. | +| `MEDIUM` | Content has a medium chance of being unsafe. | +| `HIGH` | Content has a high chance of being unsafe. | + +## SafetySetting + +- [JSON representation](https://ai.google.dev/api/generate-content#SCHEMA_REPRESENTATION) +- [HarmBlockThreshold](https://ai.google.dev/api/generate-content#HarmBlockThreshold) + +Safety setting, affecting the safety-blocking behavior. + +Passing a safety setting for a category changes the allowed probability that content is blocked. +Fields `category` ``enum (`https://ai.google.dev/api/generate-content#v1beta.HarmCategory`)`` Required. The category for this setting. +`threshold` ``enum (`https://ai.google.dev/api/generate-content#HarmBlockThreshold`)`` Required. Controls the probability threshold at which harm is blocked. + +| JSON representation | +|---| +| ``` { "category": enum (`https://ai.google.dev/api/generate-content#v1beta.HarmCategory`), "threshold": enum (`https://ai.google.dev/api/generate-content#HarmBlockThreshold`) } ``` | + +## HarmBlockThreshold + +Block at and beyond a specified harm probability. + +| Enums || +|---|---| +| `HARM_BLOCK_THRESHOLD_UNSPECIFIED` | Threshold is unspecified. | +| `BLOCK_LOW_AND_ABOVE` | Content with NEGLIGIBLE will be allowed. | +| `BLOCK_MEDIUM_AND_ABOVE` | Content with NEGLIGIBLE and LOW will be allowed. | +| `BLOCK_ONLY_HIGH` | Content with NEGLIGIBLE, LOW, and MEDIUM will be allowed. | +| `BLOCK_NONE` | All content will be allowed. | +| `OFF` | Turn off the safety filter. | + +## ServiceTier + +Service tier of the request. + +| Enums || +|---|---| +| `unspecified` | Default service tier, which is standard. | +| `standard` | Standard service tier. | +| `flex` | Flex service tier. | +| `priority` | Priority service tier. | \ No newline at end of file diff --git a/docs/协议文档/Gemini系列/模型.md b/docs/协议文档/Gemini系列/模型.md new file mode 100644 index 0000000..e00d98e --- /dev/null +++ b/docs/协议文档/Gemini系列/模型.md @@ -0,0 +1,292 @@ +# Models + +The models endpoint provides a way for you to programmatically list the available models, and retrieve extended metadata such as supported functionality and context window sizing. Read more in [the Models guide](https://ai.google.dev/gemini-api/docs/models/gemini). + +## Method: models.get + +- [Endpoint](https://ai.google.dev/api/models#body.HTTP_TEMPLATE) +- [Path parameters](https://ai.google.dev/api/models#body.PATH_PARAMETERS) +- [Request body](https://ai.google.dev/api/models#body.request_body) +- [Response body](https://ai.google.dev/api/models#body.response_body) +- [Authorization scopes](https://ai.google.dev/api/models#body.aspect) +- [Example request](https://ai.google.dev/api/models#body.codeSnippets) + - [Get](https://ai.google.dev/api/models#body.codeSnippets.group) + +Gets information about a specific `Model` such as its version number, token limits, [parameters](https://ai.google.dev/gemini-api/docs/models/generative-models#model-parameters) and other metadata. Refer to the [Gemini models guide](https://ai.google.dev/gemini-api/docs/models/gemini) for detailed model information. + +### Endpoint + +get `https://generativelanguage.googleapis.com/v1beta/{name=models/*}` + +### Path parameters + +`name` `string` Required. The resource name of the model. + +This name should match a model name returned by the `models.list` method. + +Format: `models/{model}` It takes the form `models/{model}`. + +### Request body + +The request body must be empty. + +### Example request + +### Python + + from google import genai + + client = genai.Client() + model_info = client.models.get(model="gemini-2.0-flash") + print(model_info)https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/python/models.py#L41-L45 + +### Go + + ctx := context.Background() + client, err := genai.NewClient(ctx, &genai.ClientConfig{ + APIKey: os.Getenv("GEMINI_API_KEY"), + Backend: genai.BackendGeminiAPI, + }) + if err != nil { + log.Fatal(err) + } + + modelInfo, err := client.Models.Get(ctx, "gemini-2.0-flash", nil) + if err != nil { + log.Fatal(err) + } + + fmt.Println(modelInfo)https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/go/models.go#L55-L69 + +### Shell + + curl https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash?key=$GEMINI_API_KEYhttps://github.com/google-gemini/deprecated-generative-ai-python/blob/7a7cc5474ddaa0255a4410e05361028a24400abd/samples/rest/models.sh#L9-L10 + +### Response body + +If successful, the response body contains an instance of `https://ai.google.dev/api/models#Model`. + +## Method: models.list + +- [Endpoint](https://ai.google.dev/api/models#body.HTTP_TEMPLATE) +- [Query parameters](https://ai.google.dev/api/models#body.QUERY_PARAMETERS) +- [Request body](https://ai.google.dev/api/models#body.request_body) +- [Response body](https://ai.google.dev/api/models#body.response_body) + - [JSON representation](https://ai.google.dev/api/models#body.ListModelsResponse.SCHEMA_REPRESENTATION) +- [Authorization scopes](https://ai.google.dev/api/models#body.aspect) +- [Example request](https://ai.google.dev/api/models#body.codeSnippets) + - [List](https://ai.google.dev/api/models#body.codeSnippets.group) + +Lists the [`Model`s](https://ai.google.dev/gemini-api/docs/models/gemini) available through the Gemini API. + +### Endpoint + +get `https://generativelanguage.googleapis.com/v1beta/models` + +### Query parameters + +`pageSize` `integer` The maximum number of `Models` to return (per page). + +If unspecified, 50 models will be returned per page. This method returns at most 1000 models per page, even if you pass a larger pageSize. +`pageToken` `string` A page token, received from a previous `models.list` call. + +Provide the `pageToken` returned by one request as an argument to the next request to retrieve the next page. + +When paginating, all other parameters provided to `models.list` must match the call that provided the page token. + +### Request body + +The request body must be empty. + +### Example request + +### Python + + from google import genai + + client = genai.Client() + + print("List of models that support generateContent:\n") + for m in client.models.list(): + for action in m.supported_actions: + if action == "generateContent": + print(m.name) + + print("List of models that support embedContent:\n") + for m in client.models.list(): + for action in m.supported_actions: + if action == "embedContent": + print(m.name)https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/python/models.py#L22-L36 + +### Go + + ctx := context.Background() + client, err := genai.NewClient(ctx, &genai.ClientConfig{ + APIKey: os.Getenv("GEMINI_API_KEY"), + Backend: genai.BackendGeminiAPI, + }) + if err != nil { + log.Fatal(err) + } + + + // Retrieve the list of models. + models, err := client.Models.List(ctx, &genai.ListModelsConfig{}) + if err != nil { + log.Fatal(err) + } + + fmt.Println("List of models that support generateContent:") + for _, m := range models.Items { + for _, action := range m.SupportedActions { + if action == "generateContent" { + fmt.Println(m.Name) + break + } + } + } + + fmt.Println("\nList of models that support embedContent:") + for _, m := range models.Items { + for _, action := range m.SupportedActions { + if action == "embedContent" { + fmt.Println(m.Name) + break + } + } + }https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/go/models.go#L14-L48 + +### Shell + + curl https://generativelanguage.googleapis.com/v1beta/models?key=$GEMINI_API_KEYhttps://github.com/google-gemini/deprecated-generative-ai-python/blob/7a7cc5474ddaa0255a4410e05361028a24400abd/samples/rest/models.sh#L4-L5 + +### Response body + +Response from `ListModel` containing a paginated list of Models. + +If successful, the response body contains data with the following structure: +Fields `models[]` ``object (`https://ai.google.dev/api/models#Model`)`` The returned Models. +`nextPageToken` `string` A token, which can be sent as `pageToken` to retrieve the next page. + +If this field is omitted, there are no more pages. + +| JSON representation | +|---| +| ``` { "models": [ { object (`https://ai.google.dev/api/models#Model`) } ], "nextPageToken": string } ``` | + +## REST Resource: models + +- [Resource: Model](https://ai.google.dev/api/models#Model) + - [JSON representation](https://ai.google.dev/api/models#Model.SCHEMA_REPRESENTATION) +- [Methods](https://ai.google.dev/api/models#METHODS_SUMMARY) + +## Resource: Model + +Information about a Generative Language Model. +Fields `name` `string` Required. The resource name of the `Model`. Refer to [Model variants](https://ai.google.dev/gemini-api/docs/models/gemini#model-variations) for all allowed values. + +Format: `models/{model}` with a `{model}` naming convention of: + +- "{baseModelId}-{version}" + +Examples: + +- `models/gemini-1.5-flash-001` + `baseModelId` `string` Required. The name of the base model, pass this to the generation request. + +Examples: + +- `gemini-1.5-flash` + `version` `string` Required. The version number of the model. + +This represents the major version (`1.0` or `1.5`) +`displayName` `string` The human-readable name of the model. E.g. "Gemini 1.5 Flash". + +The name can be up to 128 characters long and can consist of any UTF-8 characters. +`description` `string` A short description of the model. +`inputTokenLimit` `integer` Maximum number of input tokens allowed for this model. +`outputTokenLimit` `integer` Maximum number of output tokens available for this model. +`supportedGenerationMethods[]` `string` The model's supported generation methods. + +The corresponding API method names are defined as Pascal case strings, such as `generateMessage` and `generateContent`. +`thinking` `boolean` Whether the model supports thinking. +`temperature` `number` Controls the randomness of the output. + +Values can range over `[0.0,maxTemperature]`, inclusive. A higher value will produce responses that are more varied, while a value closer to `0.0` will typically result in less surprising responses from the model. This value specifies default to be used by the backend while making the call to the model. +`maxTemperature` `number` The maximum temperature this model can use. +`topP` `number` For [Nucleus sampling](https://ai.google.dev/gemini-api/docs/prompting-strategies#top-p). + +Nucleus sampling considers the smallest set of tokens whose probability sum is at least `topP`. This value specifies default to be used by the backend while making the call to the model. +`topK` `integer` For Top-k sampling. + +Top-k sampling considers the set of `topK` most probable tokens. This value specifies default to be used by the backend while making the call to the model. If empty, indicates the model doesn't use top-k sampling, and `topK` isn't allowed as a generation parameter. + +| JSON representation | +|---| +| ``` { "name": string, "baseModelId": string, "version": string, "displayName": string, "description": string, "inputTokenLimit": integer, "outputTokenLimit": integer, "supportedGenerationMethods": [ string ], "thinking": boolean, "temperature": number, "maxTemperature": number, "topP": number, "topK": integer } ``` | + +## Method: models.predict + +- [Endpoint](https://ai.google.dev/api/models#body.HTTP_TEMPLATE) +- [Path parameters](https://ai.google.dev/api/models#body.PATH_PARAMETERS) +- [Request body](https://ai.google.dev/api/models#body.request_body) + - [JSON representation](https://ai.google.dev/api/models#body.request_body.SCHEMA_REPRESENTATION) +- [Response body](https://ai.google.dev/api/models#body.response_body) + - [JSON representation](https://ai.google.dev/api/models#body.PredictResponse.SCHEMA_REPRESENTATION) +- [Authorization scopes](https://ai.google.dev/api/models#body.aspect) + +Performs a prediction request. + +### Endpoint + +post `https://generativelanguage.googleapis.com/v1beta/{model=models/*}:predict` + +### Path parameters + +`model` `string` Required. The name of the model for prediction. Format: `name=models/{model}`. It takes the form `models/{model}`. + +### Request body + +The request body contains data with the following structure: +Fields `instances[]` ``value (`https://protobuf.dev/reference/protobuf/google.protobuf#value` format)`` Required. The instances that are the input to the prediction call. +`parameters` ``value (`https://protobuf.dev/reference/protobuf/google.protobuf#value` format)`` Optional. The parameters that govern the prediction call. + +### Response body + +Response message for \[PredictionService.Predict\]. + +If successful, the response body contains data with the following structure: +Fields `predictions[]` ``value (`https://protobuf.dev/reference/protobuf/google.protobuf#value` format)`` The outputs of the prediction call. + +| JSON representation | +|---| +| ``` { "predictions": [ value ] } ``` | + +## Method: models.predictLongRunning + +- [Endpoint](https://ai.google.dev/api/models#body.HTTP_TEMPLATE) +- [Path parameters](https://ai.google.dev/api/models#body.PATH_PARAMETERS) +- [Request body](https://ai.google.dev/api/models#body.request_body) + - [JSON representation](https://ai.google.dev/api/models#body.request_body.SCHEMA_REPRESENTATION) +- [Response body](https://ai.google.dev/api/models#body.response_body) +- [Authorization scopes](https://ai.google.dev/api/models#body.aspect) + +Same as models.predict but returns an LRO. + +### Endpoint + +post `https://generativelanguage.googleapis.com/v1beta/{model=models/*}:predictLongRunning` + +### Path parameters + +`model` `string` Required. The name of the model for prediction. Format: `name=models/{model}`. + +### Request body + +The request body contains data with the following structure: +Fields `instances[]` ``value (`https://protobuf.dev/reference/protobuf/google.protobuf#value` format)`` Required. The instances that are the input to the prediction call. +`parameters` ``value (`https://protobuf.dev/reference/protobuf/google.protobuf#value` format)`` Optional. The parameters that govern the prediction call. + +### Response body + +If successful, the response body contains an instance of `https://ai.google.dev/api/batch-api#Operation`. \ No newline at end of file diff --git a/docs/协议文档/Gemini系列/生成内容.md b/docs/协议文档/Gemini系列/生成内容.md new file mode 100644 index 0000000..63bbc5b --- /dev/null +++ b/docs/协议文档/Gemini系列/生成内容.md @@ -0,0 +1,3594 @@ +# Generating content + +The Gemini API supports content generation with images, audio, code, tools, and more. For details on each of these features, read on and check out the task-focused sample code, or read the comprehensive guides. + +- [Text generation](https://ai.google.dev/gemini-api/docs/text-generation) +- [Vision](https://ai.google.dev/gemini-api/docs/vision) +- [Audio](https://ai.google.dev/gemini-api/docs/audio) +- [Embeddings](https://ai.google.dev/gemini-api/docs/embeddings) +- [Long context](https://ai.google.dev/gemini-api/docs/long-context) +- [Code execution](https://ai.google.dev/gemini-api/docs/code-execution) +- [JSON Mode](https://ai.google.dev/gemini-api/docs/json-mode) +- [Function calling](https://ai.google.dev/gemini-api/docs/function-calling) +- [System instructions](https://ai.google.dev/gemini-api/docs/system-instructions) + +## Method: models.generateContent + +- [Endpoint](https://ai.google.dev/api/generate-content#body.HTTP_TEMPLATE) +- [Path parameters](https://ai.google.dev/api/generate-content#body.PATH_PARAMETERS) +- [Request body](https://ai.google.dev/api/generate-content#body.request_body) + - [JSON representation](https://ai.google.dev/api/generate-content#body.request_body.SCHEMA_REPRESENTATION) +- [Response body](https://ai.google.dev/api/generate-content#body.response_body) +- [Authorization scopes](https://ai.google.dev/api/generate-content#body.aspect) +- [Example request](https://ai.google.dev/api/generate-content#body.codeSnippets) + - [Text](https://ai.google.dev/api/generate-content#body.codeSnippets.group) + - [Image](https://ai.google.dev/api/generate-content#body.codeSnippets.group_1) + - [Audio](https://ai.google.dev/api/generate-content#body.codeSnippets.group_2) + - [Video](https://ai.google.dev/api/generate-content#body.codeSnippets.group_3) + - [PDF](https://ai.google.dev/api/generate-content#body.codeSnippets.group_4) + - [Chat](https://ai.google.dev/api/generate-content#body.codeSnippets.group_5) + - [Cache](https://ai.google.dev/api/generate-content#body.codeSnippets.group_6) + - [Tuned Model](https://ai.google.dev/api/generate-content#body.codeSnippets.group_7) + - [JSON Mode](https://ai.google.dev/api/generate-content#body.codeSnippets.group_8) + - [Code execution](https://ai.google.dev/api/generate-content#body.codeSnippets.group_9) + - [Function Calling](https://ai.google.dev/api/generate-content#body.codeSnippets.group_10) + - [Generation config](https://ai.google.dev/api/generate-content#body.codeSnippets.group_11) + - [Safety Settings](https://ai.google.dev/api/generate-content#body.codeSnippets.group_12) + - [System Instruction](https://ai.google.dev/api/generate-content#body.codeSnippets.group_13) + +Generates a model response given an input `GenerateContentRequest`. Refer to the [text generation guide](https://ai.google.dev/gemini-api/docs/text-generation) for detailed usage information. Input capabilities differ between models, including tuned models. Refer to the [model guide](https://ai.google.dev/gemini-api/docs/models/gemini) and [tuning guide](https://ai.google.dev/gemini-api/docs/model-tuning) for details. + +### Endpoint + +post `https://generativelanguage.googleapis.com/v1beta/{model=models/*}:generateContent` + +### Path parameters + +`model` `string` Required. The name of the `Model` to use for generating the completion. + +Format: `models/{model}`. It takes the form `models/{model}`. + +### Request body + +The request body contains data with the following structure: +Fields `contents[]` ``object (`https://ai.google.dev/api/caching#Content`)`` Required. The content of the current conversation with the model. + +For single-turn queries, this is a single instance. For multi-turn queries like [chat](https://ai.google.dev/gemini-api/docs/text-generation#chat), this is a repeated field that contains the conversation history and the latest request. +`tools[]` ``object (`https://ai.google.dev/api/caching#Tool`)`` Optional. A list of `Tools` the `Model` may use to generate the next response. + +A `Tool` is a piece of code that enables the system to interact with external systems to perform an action, or set of actions, outside of knowledge and scope of the `Model`. Supported `Tool`s are `Function` and `codeExecution`. Refer to the [Function calling](https://ai.google.dev/gemini-api/docs/function-calling) and the [Code execution](https://ai.google.dev/gemini-api/docs/code-execution) guides to learn more. +`toolConfig` ``object (`https://ai.google.dev/api/caching#ToolConfig`)`` Optional. Tool configuration for any `Tool` specified in the request. Refer to the [Function calling guide](https://ai.google.dev/gemini-api/docs/function-calling#function_calling_mode) for a usage example. +`safetySettings[]` ``object (`https://ai.google.dev/api/generate-content#v1beta.SafetySetting`)`` Optional. A list of unique `SafetySetting` instances for blocking unsafe content. + +This will be enforced on the `GenerateContentRequest.contents` and `GenerateContentResponse.candidates`. There should not be more than one setting for each `SafetyCategory` type. The API will block any contents and responses that fail to meet the thresholds set by these settings. This list overrides the default settings for each `SafetyCategory` specified in the safetySettings. If there is no `SafetySetting` for a given `SafetyCategory` provided in the list, the API will use the default safety setting for that category. Harm categories HARM_CATEGORY_HATE_SPEECH, HARM_CATEGORY_SEXUALLY_EXPLICIT, HARM_CATEGORY_DANGEROUS_CONTENT, HARM_CATEGORY_HARASSMENT, HARM_CATEGORY_CIVIC_INTEGRITY are supported. Refer to the [guide](https://ai.google.dev/gemini-api/docs/safety-settings) for detailed information on available safety settings. Also refer to the [Safety guidance](https://ai.google.dev/gemini-api/docs/safety-guidance) to learn how to incorporate safety considerations in your AI applications. +`systemInstruction` ``object (`https://ai.google.dev/api/caching#Content`)`` Optional. Developer set [system instruction(s)](https://ai.google.dev/gemini-api/docs/system-instructions). Currently, text only. +`generationConfig` ``object (`https://ai.google.dev/api/generate-content#v1beta.GenerationConfig`)`` Optional. Configuration options for model generation and outputs. +`cachedContent` `string` Optional. The name of the content [cached](https://ai.google.dev/gemini-api/docs/caching) to use as context to serve the prediction. Format: `cachedContents/{cachedContent}` +`serviceTier` ``enum (`https://ai.google.dev/api/generate-content#v1beta.ServiceTier`)`` Optional. The service tier of the request. +`store` `boolean` Optional. Configures the logging behavior for a given request. If set, it takes precedence over the project-level logging config. + +### Example request + +### Text + +### Python + + from google import genai + + client = genai.Client() + response = client.models.generate_content( + model="gemini-2.0-flash", contents="Write a story about a magic backpack." + ) + print(response.text)https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/python/text_generation.py#L26-L32 + +### Node.js + + // Make sure to include the following import: + // import {GoogleGenAI} from '@google/genai'; + const ai = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY }); + + const response = await ai.models.generateContent({ + model: "gemini-2.0-flash", + contents: "Write a story about a magic backpack.", + }); + console.log(response.text);https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/javascript/text_generation.js#L36-L44 + +### Go + + ctx := context.Background() + client, err := genai.NewClient(ctx, &genai.ClientConfig{ + APIKey: os.Getenv("GEMINI_API_KEY"), + Backend: genai.BackendGeminiAPI, + }) + if err != nil { + log.Fatal(err) + } + contents := []*genai.Content{ + genai.NewContentFromText("Write a story about a magic backpack.", genai.RoleUser), + } + response, err := client.Models.GenerateContent(ctx, "gemini-2.0-flash", contents, nil) + if err != nil { + log.Fatal(err) + } + printResponse(response)https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/go/text_generation.go#L16-L31 + +### Shell + + curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash:generateContent?key=$GEMINI_API_KEY" \ + -H 'Content-Type: application/json' \ + -X POST \ + -d '{ + "contents": [{ + "parts":[{"text": "Write a story about a magic backpack."}] + }] + }' 2> /dev/nullhttps://github.com/google-gemini/deprecated-generative-ai-python/blob/7a7cc5474ddaa0255a4410e05361028a24400abd/samples/rest/text_generation.sh#L21-L29 + +### Java + + Client client = new Client(); + + GenerateContentResponse response = + client.models.generateContent( + "gemini-2.0-flash", + "Write a story about a magic backpack.", + null); + + System.out.println(response.text());https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/java/src/main/java/com/example/gemini/TextGeneration.java#L34-L42 + +### Image + +### Python + + from google import genai + import PIL.Image + + client = genai.Client() + organ = PIL.Image.open(media / "organ.jpg") + response = client.models.generate_content( + model="gemini-2.0-flash", contents=["Tell me about this instrument", organ] + ) + print(response.text)https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/python/text_generation.py#L50-L58 + +### Node.js + + // Make sure to include the following import: + // import {GoogleGenAI} from '@google/genai'; + const ai = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY }); + + const organ = await ai.files.upload({ + file: path.join(media, "organ.jpg"), + }); + + const response = await ai.models.generateContent({ + model: "gemini-2.0-flash", + contents: [ + createUserContent([ + "Tell me about this instrument", + createPartFromUri(organ.uri, organ.mimeType) + ]), + ], + }); + console.log(response.text);https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/javascript/text_generation.js#L70-L87 + +### Go + + ctx := context.Background() + client, err := genai.NewClient(ctx, &genai.ClientConfig{ + APIKey: os.Getenv("GEMINI_API_KEY"), + Backend: genai.BackendGeminiAPI, + }) + if err != nil { + log.Fatal(err) + } + + file, err := client.Files.UploadFromPath( + ctx, + filepath.Join(getMedia(), "organ.jpg"), + &genai.UploadFileConfig{ + MIMEType : "image/jpeg", + }, + ) + if err != nil { + log.Fatal(err) + } + parts := []*genai.Part{ + genai.NewPartFromText("Tell me about this instrument"), + genai.NewPartFromURI(file.URI, file.MIMEType), + } + contents := []*genai.Content{ + genai.NewContentFromParts(parts, genai.RoleUser), + } + + response, err := client.Models.GenerateContent(ctx, "gemini-2.0-flash", contents, nil) + if err != nil { + log.Fatal(err) + } + printResponse(response)https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/go/text_generation.go#L66-L97 + +### Shell + + # Use a temporary file to hold the base64 encoded image data + TEMP_B64=$(mktemp) + trap 'rm -f "$TEMP_B64"' EXIT + base64 $B64FLAGS $IMG_PATH > "$TEMP_B64" + + # Use a temporary file to hold the JSON payload + TEMP_JSON=$(mktemp) + trap 'rm -f "$TEMP_JSON"' EXIT + + cat > "$TEMP_JSON" << EOF + { + "contents": [{ + "parts":[ + {"text": "Tell me about this instrument"}, + { + "inline_data": { + "mime_type":"image/jpeg", + "data": "$(cat "$TEMP_B64")" + } + } + ] + }] + } + EOF + + curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash:generateContent?key=$GEMINI_API_KEY" \ + -H 'Content-Type: application/json' \ + -X POST \ + -d "@$TEMP_JSON" 2> /dev/nullhttps://github.com/google-gemini/deprecated-generative-ai-python/blob/7a7cc5474ddaa0255a4410e05361028a24400abd/samples/rest/text_generation.sh#L41-L70 + +### Java + + Client client = new Client(); + + String path = media_path + "organ.jpg"; + byte[] imageData = Files.readAllBytes(Paths.get(path)); + + Content content = + Content.fromParts( + Part.fromText("Tell me about this instrument."), + Part.fromBytes(imageData, "image/jpeg")); + + GenerateContentResponse response = client.models.generateContent("gemini-2.0-flash", content, null); + + System.out.println(response.text());https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/java/src/main/java/com/example/gemini/TextGeneration.java#L70-L82 + +### Audio + +### Python + + from google import genai + + client = genai.Client() + sample_audio = client.files.upload(file=media / "sample.mp3") + response = client.models.generate_content( + model="gemini-2.0-flash", + contents=["Give me a summary of this audio file.", sample_audio], + ) + print(response.text)https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/python/text_generation.py#L118-L126 + +### Node.js + + // Make sure to include the following import: + // import {GoogleGenAI} from '@google/genai'; + const ai = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY }); + + const audio = await ai.files.upload({ + file: path.join(media, "sample.mp3"), + }); + + const response = await ai.models.generateContent({ + model: "gemini-2.0-flash", + contents: [ + createUserContent([ + "Give me a summary of this audio file.", + createPartFromUri(audio.uri, audio.mimeType), + ]), + ], + }); + console.log(response.text);https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/javascript/text_generation.js#L185-L202 + +### Go + + ctx := context.Background() + client, err := genai.NewClient(ctx, &genai.ClientConfig{ + APIKey: os.Getenv("GEMINI_API_KEY"), + Backend: genai.BackendGeminiAPI, + }) + if err != nil { + log.Fatal(err) + } + + file, err := client.Files.UploadFromPath( + ctx, + filepath.Join(getMedia(), "sample.mp3"), + &genai.UploadFileConfig{ + MIMEType : "audio/mpeg", + }, + ) + if err != nil { + log.Fatal(err) + } + + parts := []*genai.Part{ + genai.NewPartFromText("Give me a summary of this audio file."), + genai.NewPartFromURI(file.URI, file.MIMEType), + } + + contents := []*genai.Content{ + genai.NewContentFromParts(parts, genai.RoleUser), + } + + response, err := client.Models.GenerateContent(ctx, "gemini-2.0-flash", contents, nil) + if err != nil { + log.Fatal(err) + } + printResponse(response)https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/go/text_generation.go#L256-L289 + +### Shell + + # Use File API to upload audio data to API request. + MIME_TYPE=$(file -b --mime-type "${AUDIO_PATH}") + NUM_BYTES=$(wc -c < "${AUDIO_PATH}") + DISPLAY_NAME=AUDIO + + tmp_header_file=upload-header.tmp + + # Initial resumable request defining metadata. + # The upload url is in the response headers dump them to a file. + curl "${BASE_URL}/upload/v1beta/files?key=${GEMINI_API_KEY}" \ + -D upload-header.tmp \ + -H "X-Goog-Upload-Protocol: resumable" \ + -H "X-Goog-Upload-Command: start" \ + -H "X-Goog-Upload-Header-Content-Length: ${NUM_BYTES}" \ + -H "X-Goog-Upload-Header-Content-Type: ${MIME_TYPE}" \ + -H "Content-Type: application/json" \ + -d "{'file': {'display_name': '${DISPLAY_NAME}'}}" 2> /dev/null + + upload_url=$(grep -i "x-goog-upload-url: " "${tmp_header_file}" | cut -d" " -f2 | tr -d "\r") + rm "${tmp_header_file}" + + # Upload the actual bytes. + curl "${upload_url}" \ + -H "Content-Length: ${NUM_BYTES}" \ + -H "X-Goog-Upload-Offset: 0" \ + -H "X-Goog-Upload-Command: upload, finalize" \ + --data-binary "@${AUDIO_PATH}" 2> /dev/null > file_info.json + + file_uri=$(jq ".file.uri" file_info.json) + echo file_uri=$file_uri + + curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash:generateContent?key=$GEMINI_API_KEY" \ + -H 'Content-Type: application/json' \ + -X POST \ + -d '{ + "contents": [{ + "parts":[ + {"text": "Please describe this file."}, + {"file_data":{"mime_type": "audio/mpeg", "file_uri": '$file_uri'}}] + }] + }' 2> /dev/null > response.json + + cat response.json + echo + + jq ".candidates[].content.parts[].text" response.jsonhttps://github.com/google-gemini/deprecated-generative-ai-python/blob/7a7cc5474ddaa0255a4410e05361028a24400abd/samples/rest/text_generation.sh#L174-L220 + +### Video + +### Python + + from google import genai + import time + + client = genai.Client() + # Video clip (CC BY 3.0) from https://peach.blender.org/download/ + myfile = client.files.upload(file=media / "Big_Buck_Bunny.mp4") + print(f"{myfile=}") + + # Poll until the video file is completely processed (state becomes ACTIVE). + while not myfile.state or myfile.state.name != "ACTIVE": + print("Processing video...") + print("File state:", myfile.state) + time.sleep(5) + myfile = client.files.get(name=myfile.name) + + response = client.models.generate_content( + model="gemini-2.0-flash", contents=[myfile, "Describe this video clip"] + ) + print(f"{response.text=}")https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/python/text_generation.py#L146-L164 + +### Node.js + + // Make sure to include the following import: + // import {GoogleGenAI} from '@google/genai'; + const ai = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY }); + + let video = await ai.files.upload({ + file: path.join(media, 'Big_Buck_Bunny.mp4'), + }); + + // Poll until the video file is completely processed (state becomes ACTIVE). + while (!video.state || video.state.toString() !== 'ACTIVE') { + console.log('Processing video...'); + console.log('File state: ', video.state); + await sleep(5000); + video = await ai.files.get({name: video.name}); + } + + const response = await ai.models.generateContent({ + model: "gemini-2.0-flash", + contents: [ + createUserContent([ + "Describe this video clip", + createPartFromUri(video.uri, video.mimeType), + ]), + ], + }); + console.log(response.text);https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/javascript/text_generation.js#L237-L262 + +### Go + + ctx := context.Background() + client, err := genai.NewClient(ctx, &genai.ClientConfig{ + APIKey: os.Getenv("GEMINI_API_KEY"), + Backend: genai.BackendGeminiAPI, + }) + if err != nil { + log.Fatal(err) + } + + file, err := client.Files.UploadFromPath( + ctx, + filepath.Join(getMedia(), "Big_Buck_Bunny.mp4"), + &genai.UploadFileConfig{ + MIMEType : "video/mp4", + }, + ) + if err != nil { + log.Fatal(err) + } + + // Poll until the video file is completely processed (state becomes ACTIVE). + for file.State == genai.FileStateUnspecified || file.State != genai.FileStateActive { + fmt.Println("Processing video...") + fmt.Println("File state:", file.State) + time.Sleep(5 * time.Second) + + file, err = client.Files.Get(ctx, file.Name, nil) + if err != nil { + log.Fatal(err) + } + } + + parts := []*genai.Part{ + genai.NewPartFromText("Describe this video clip"), + genai.NewPartFromURI(file.URI, file.MIMEType), + } + + contents := []*genai.Content{ + genai.NewContentFromParts(parts, genai.RoleUser), + } + + response, err := client.Models.GenerateContent(ctx, "gemini-2.0-flash", contents, nil) + if err != nil { + log.Fatal(err) + } + printResponse(response)https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/go/text_generation.go#L342-L387 + +### Shell + + # Use File API to upload audio data to API request. + MIME_TYPE=$(file -b --mime-type "${VIDEO_PATH}") + NUM_BYTES=$(wc -c < "${VIDEO_PATH}") + DISPLAY_NAME=VIDEO + + # Initial resumable request defining metadata. + # The upload url is in the response headers dump them to a file. + curl "${BASE_URL}/upload/v1beta/files?key=${GEMINI_API_KEY}" \ + -D "${tmp_header_file}" \ + -H "X-Goog-Upload-Protocol: resumable" \ + -H "X-Goog-Upload-Command: start" \ + -H "X-Goog-Upload-Header-Content-Length: ${NUM_BYTES}" \ + -H "X-Goog-Upload-Header-Content-Type: ${MIME_TYPE}" \ + -H "Content-Type: application/json" \ + -d "{'file': {'display_name': '${DISPLAY_NAME}'}}" 2> /dev/null + + upload_url=$(grep -i "x-goog-upload-url: " "${tmp_header_file}" | cut -d" " -f2 | tr -d "\r") + rm "${tmp_header_file}" + + # Upload the actual bytes. + curl "${upload_url}" \ + -H "Content-Length: ${NUM_BYTES}" \ + -H "X-Goog-Upload-Offset: 0" \ + -H "X-Goog-Upload-Command: upload, finalize" \ + --data-binary "@${VIDEO_PATH}" 2> /dev/null > file_info.json + + file_uri=$(jq ".file.uri" file_info.json) + echo file_uri=$file_uri + + state=$(jq ".file.state" file_info.json) + echo state=$state + + name=$(jq ".file.name" file_info.json) + echo name=$name + + while [[ "($state)" = *"PROCESSING"* ]]; + do + echo "Processing video..." + sleep 5 + # Get the file of interest to check state + curl https://generativelanguage.googleapis.com/v1beta/files/$name > file_info.json + state=$(jq ".file.state" file_info.json) + done + + curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash:generateContent?key=$GEMINI_API_KEY" \ + -H 'Content-Type: application/json' \ + -X POST \ + -d '{ + "contents": [{ + "parts":[ + {"text": "Transcribe the audio from this video, giving timestamps for salient events in the video. Also provide visual descriptions."}, + {"file_data":{"mime_type": "video/mp4", "file_uri": '$file_uri'}}] + }] + }' 2> /dev/null > response.json + + cat response.json + echo + + jq ".candidates[].content.parts[].text" response.jsonhttps://github.com/google-gemini/deprecated-generative-ai-python/blob/7a7cc5474ddaa0255a4410e05361028a24400abd/samples/rest/text_generation.sh#L272-L331 + +### PDF + +### Python + + from google import genai + + client = genai.Client() + sample_pdf = client.files.upload(file=media / "test.pdf") + response = client.models.generate_content( + model="gemini-2.0-flash", + contents=["Give me a summary of this document:", sample_pdf], + ) + print(f"{response.text=}")https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/python/text_generation.py#L194-L202 + +### Go + + ctx := context.Background() + client, err := genai.NewClient(ctx, &genai.ClientConfig{ + APIKey: os.Getenv("GEMINI_API_KEY"), + Backend: genai.BackendGeminiAPI, + }) + if err != nil { + log.Fatal(err) + } + + file, err := client.Files.UploadFromPath( + ctx, + filepath.Join(getMedia(), "test.pdf"), + &genai.UploadFileConfig{ + MIMEType : "application/pdf", + }, + ) + if err != nil { + log.Fatal(err) + } + + parts := []*genai.Part{ + genai.NewPartFromText("Give me a summary of this document:"), + genai.NewPartFromURI(file.URI, file.MIMEType), + } + + contents := []*genai.Content{ + genai.NewContentFromParts(parts, genai.RoleUser), + } + + response, err := client.Models.GenerateContent(ctx, "gemini-2.0-flash", contents, nil) + if err != nil { + log.Fatal(err) + } + printResponse(response)https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/go/text_generation.go#L452-L485 + +### Shell + + MIME_TYPE=$(file -b --mime-type "${PDF_PATH}") + NUM_BYTES=$(wc -c < "${PDF_PATH}") + DISPLAY_NAME=TEXT + + + echo $MIME_TYPE + tmp_header_file=upload-header.tmp + + # Initial resumable request defining metadata. + # The upload url is in the response headers dump them to a file. + curl "${BASE_URL}/upload/v1beta/files?key=${GEMINI_API_KEY}" \ + -D upload-header.tmp \ + -H "X-Goog-Upload-Protocol: resumable" \ + -H "X-Goog-Upload-Command: start" \ + -H "X-Goog-Upload-Header-Content-Length: ${NUM_BYTES}" \ + -H "X-Goog-Upload-Header-Content-Type: ${MIME_TYPE}" \ + -H "Content-Type: application/json" \ + -d "{'file': {'display_name': '${DISPLAY_NAME}'}}" 2> /dev/null + + upload_url=$(grep -i "x-goog-upload-url: " "${tmp_header_file}" | cut -d" " -f2 | tr -d "\r") + rm "${tmp_header_file}" + + # Upload the actual bytes. + curl "${upload_url}" \ + -H "Content-Length: ${NUM_BYTES}" \ + -H "X-Goog-Upload-Offset: 0" \ + -H "X-Goog-Upload-Command: upload, finalize" \ + --data-binary "@${PDF_PATH}" 2> /dev/null > file_info.json + + file_uri=$(jq ".file.uri" file_info.json) + echo file_uri=$file_uri + + # Now generate content using that file + curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash:generateContent?key=$GEMINI_API_KEY" \ + -H 'Content-Type: application/json' \ + -X POST \ + -d '{ + "contents": [{ + "parts":[ + {"text": "Can you add a few more lines to this poem?"}, + {"file_data":{"mime_type": "application/pdf", "file_uri": '$file_uri'}}] + }] + }' 2> /dev/null > response.json + + cat response.json + echo + + jq ".candidates[].content.parts[].text" response.jsonhttps://github.com/google-gemini/deprecated-generative-ai-python/blob/7a7cc5474ddaa0255a4410e05361028a24400abd/samples/rest/text_generation.sh#L393-L441 + +### Chat + +### Python + + from google import genai + from google.genai import types + + client = genai.Client() + # Pass initial history using the "history" argument + chat = client.chats.create( + model="gemini-2.0-flash", + history=[ + types.Content(role="user", parts=[types.Part(text="Hello")]), + types.Content( + role="model", + parts=[ + types.Part( + text="Great to meet you. What would you like to know?" + ) + ], + ), + ], + ) + response = chat.send_message(message="I have 2 dogs in my house.") + print(response.text) + response = chat.send_message(message="How many paws are in my house?") + print(response.text)https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/python/chat.py#L25-L47 + +### Node.js + + // Make sure to include the following import: + // import {GoogleGenAI} from '@google/genai'; + const ai = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY }); + const chat = ai.chats.create({ + model: "gemini-2.0-flash", + history: [ + { + role: "user", + parts: [{ text: "Hello" }], + }, + { + role: "model", + parts: [{ text: "Great to meet you. What would you like to know?" }], + }, + ], + }); + + const response1 = await chat.sendMessage({ + message: "I have 2 dogs in my house.", + }); + console.log("Chat response 1:", response1.text); + + const response2 = await chat.sendMessage({ + message: "How many paws are in my house?", + }); + console.log("Chat response 2:", response2.text);https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/javascript/chat.js#L33-L58 + +### Go + + ctx := context.Background() + client, err := genai.NewClient(ctx, &genai.ClientConfig{ + APIKey: os.Getenv("GEMINI_API_KEY"), + Backend: genai.BackendGeminiAPI, + }) + if err != nil { + log.Fatal(err) + } + + // Pass initial history using the History field. + history := []*genai.Content{ + genai.NewContentFromText("Hello", genai.RoleUser), + genai.NewContentFromText("Great to meet you. What would you like to know?", genai.RoleModel), + } + + chat, err := client.Chats.Create(ctx, "gemini-2.0-flash", nil, history) + if err != nil { + log.Fatal(err) + } + + firstResp, err := chat.SendMessage(ctx, genai.Part{Text: "I have 2 dogs in my house."}) + if err != nil { + log.Fatal(err) + } + fmt.Println(firstResp.Text()) + + secondResp, err := chat.SendMessage(ctx, genai.Part{Text: "How many paws are in my house?"}) + if err != nil { + log.Fatal(err) + } + fmt.Println(secondResp.Text())https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/go/chat.go#L16-L46 + +### Shell + + curl https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash:generateContent?key=$GEMINI_API_KEY \ + -H 'Content-Type: application/json' \ + -X POST \ + -d '{ + "contents": [ + {"role":"user", + "parts":[{ + "text": "Hello"}]}, + {"role": "model", + "parts":[{ + "text": "Great to meet you. What would you like to know?"}]}, + {"role":"user", + "parts":[{ + "text": "I have two dogs in my house. How many paws are in my house?"}]}, + ] + }' 2> /dev/null | grep "text"https://github.com/google-gemini/deprecated-generative-ai-python/blob/7a7cc5474ddaa0255a4410e05361028a24400abd/samples/rest/chat.sh#L7-L23 + +### Java + + Client client = new Client(); + + Content userContent = Content.fromParts(Part.fromText("Hello")); + Content modelContent = + Content.builder() + .role("model") + .parts( + Collections.singletonList( + Part.fromText("Great to meet you. What would you like to know?") + ) + ).build(); + + Chat chat = client.chats.create( + "gemini-2.0-flash", + GenerateContentConfig.builder() + .systemInstruction(userContent) + .systemInstruction(modelContent) + .build() + ); + + GenerateContentResponse response1 = chat.sendMessage("I have 2 dogs in my house."); + System.out.println(response1.text()); + + GenerateContentResponse response2 = chat.sendMessage("How many paws are in my house?"); + System.out.println(response2.text()); + https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/java/src/main/java/com/example/gemini/ChatSession.java#L32-L57 + +### Cache + +### Python + + from google import genai + from google.genai import types + + client = genai.Client() + document = client.files.upload(file=media / "a11.txt") + model_name = "gemini-1.5-flash-001" + + cache = client.caches.create( + model=model_name, + config=types.CreateCachedContentConfig( + contents=[document], + system_instruction="You are an expert analyzing transcripts.", + ), + ) + print(cache) + + response = client.models.generate_content( + model=model_name, + contents="Please summarize this transcript", + config=types.GenerateContentConfig(cached_content=cache.name), + ) + print(response.text)https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/python/cache.py#L25-L46 + +### Node.js + + // Make sure to include the following import: + // import {GoogleGenAI} from '@google/genai'; + const ai = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY }); + const filePath = path.join(media, "a11.txt"); + const document = await ai.files.upload({ + file: filePath, + config: { mimeType: "text/plain" }, + }); + console.log("Uploaded file name:", document.name); + const modelName = "gemini-1.5-flash-001"; + + const contents = [ + createUserContent(createPartFromUri(document.uri, document.mimeType)), + ]; + + const cache = await ai.caches.create({ + model: modelName, + config: { + contents: contents, + systemInstruction: "You are an expert analyzing transcripts.", + }, + }); + console.log("Cache created:", cache); + + const response = await ai.models.generateContent({ + model: modelName, + contents: "Please summarize this transcript", + config: { cachedContent: cache.name }, + }); + console.log("Response text:", response.text);https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/javascript/cache.js#L33-L62 + +### Go + + ctx := context.Background() + client, err := genai.NewClient(ctx, &genai.ClientConfig{ + APIKey: os.Getenv("GEMINI_API_KEY"), + Backend: genai.BackendGeminiAPI, + }) + if err != nil { + log.Fatal(err) + } + + modelName := "gemini-1.5-flash-001" + document, err := client.Files.UploadFromPath( + ctx, + filepath.Join(getMedia(), "a11.txt"), + &genai.UploadFileConfig{ + MIMEType : "text/plain", + }, + ) + if err != nil { + log.Fatal(err) + } + parts := []*genai.Part{ + genai.NewPartFromURI(document.URI, document.MIMEType), + } + contents := []*genai.Content{ + genai.NewContentFromParts(parts, genai.RoleUser), + } + cache, err := client.Caches.Create(ctx, modelName, &genai.CreateCachedContentConfig{ + Contents: contents, + SystemInstruction: genai.NewContentFromText( + "You are an expert analyzing transcripts.", genai.RoleUser, + ), + }) + if err != nil { + log.Fatal(err) + } + fmt.Println("Cache created:") + fmt.Println(cache) + + // Use the cache for generating content. + response, err := client.Models.GenerateContent( + ctx, + modelName, + genai.Text("Please summarize this transcript"), + &genai.GenerateContentConfig{ + CachedContent: cache.Name, + }, + ) + if err != nil { + log.Fatal(err) + } + printResponse(response)https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/go/cache.go#L16-L66 + +### Tuned Model + +### Python + + # With Gemini 2 we're launching a new SDK. See the following doc for details. + # https://ai.google.dev/gemini-api/docs/migratehttps://github.com/google-gemini/deprecated-generative-ai-python/blob/7a7cc5474ddaa0255a4410e05361028a24400abd/README.md#L23-L24 + +### JSON Mode + +### Python + + from google import genai + from google.genai import types + from typing_extensions import TypedDict + + class Recipe(TypedDict): + recipe_name: str + ingredients: list[str] + + client = genai.Client() + result = client.models.generate_content( + model="gemini-2.0-flash", + contents="List a few popular cookie recipes.", + config=types.GenerateContentConfig( + response_mime_type="application/json", response_schema=list[Recipe] + ), + ) + print(result)https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/python/controlled_generation.py#L25-L41 + +### Node.js + + // Make sure to include the following import: + // import {GoogleGenAI} from '@google/genai'; + const ai = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY }); + const response = await ai.models.generateContent({ + model: "gemini-2.0-flash", + contents: "List a few popular cookie recipes.", + config: { + responseMimeType: "application/json", + responseSchema: { + type: "array", + items: { + type: "object", + properties: { + recipeName: { type: "string" }, + ingredients: { type: "array", items: { type: "string" } }, + }, + required: ["recipeName", "ingredients"], + }, + }, + }, + }); + console.log(response.text);https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/javascript/controlled_generation.js#L33-L54 + +### Go + + ctx := context.Background() + client, err := genai.NewClient(ctx, &genai.ClientConfig{ + APIKey: os.Getenv("GEMINI_API_KEY"), + Backend: genai.BackendGeminiAPI, + }) + if err != nil { + log.Fatal(err) + } + + schema := &genai.Schema{ + Type: genai.TypeArray, + Items: &genai.Schema{ + Type: genai.TypeObject, + Properties: map[string]*genai.Schema{ + "recipe_name": {Type: genai.TypeString}, + "ingredients": { + Type: genai.TypeArray, + Items: &genai.Schema{Type: genai.TypeString}, + }, + }, + Required: []string{"recipe_name"}, + }, + } + + config := &genai.GenerateContentConfig{ + ResponseMIMEType: "application/json", + ResponseSchema: schema, + } + + response, err := client.Models.GenerateContent( + ctx, + "gemini-2.0-flash", + genai.Text("List a few popular cookie recipes."), + config, + ) + if err != nil { + log.Fatal(err) + } + printResponse(response)https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/go/controlled_generation.go#L14-L52 + +### Shell + + curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash:generateContent?key=$GEMINI_API_KEY" \ + -H 'Content-Type: application/json' \ + -d '{ + "contents": [{ + "parts":[ + {"text": "List 5 popular cookie recipes"} + ] + }], + "generationConfig": { + "response_mime_type": "application/json", + "response_schema": { + "type": "ARRAY", + "items": { + "type": "OBJECT", + "properties": { + "recipe_name": {"type":"STRING"}, + } + } + } + } + }' 2> /dev/null | headhttps://github.com/google-gemini/deprecated-generative-ai-python/blob/7a7cc5474ddaa0255a4410e05361028a24400abd/samples/rest/controlled_generation.sh#L5-L25 + +### Java + + Client client = new Client(); + + Schema recipeSchema = Schema.builder() + .type(Array.class.getSimpleName()) + .items(Schema.builder() + .type(Object.class.getSimpleName()) + .properties( + Map.of("recipe_name", Schema.builder() + .type(String.class.getSimpleName()) + .build(), + "ingredients", Schema.builder() + .type(Array.class.getSimpleName()) + .items(Schema.builder() + .type(String.class.getSimpleName()) + .build()) + .build()) + ) + .required(List.of("recipe_name", "ingredients")) + .build()) + .build(); + + GenerateContentConfig config = + GenerateContentConfig.builder() + .responseMimeType("application/json") + .responseSchema(recipeSchema) + .build(); + + GenerateContentResponse response = + client.models.generateContent( + "gemini-2.0-flash", + "List a few popular cookie recipes.", + config); + + System.out.println(response.text());https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/java/src/main/java/com/example/gemini/ControlledGeneration.java#L39-L72 + +### Code execution + +### Python + + from google import genai + from google.genai import types + + client = genai.Client() + response = client.models.generate_content( + model="gemini-2.0-pro-exp-02-05", + contents=( + "Write and execute code that calculates the sum of the first 50 prime numbers. " + "Ensure that only the executable code and its resulting output are generated." + ), + ) + # Each part may contain text, executable code, or an execution result. + for part in response.candidates[0].content.parts: + print(part, "\n") + + print("-" * 80) + # The .text accessor concatenates the parts into a markdown-formatted text. + print("\n", response.text)https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/python/code_execution.py#L22-L39 + +### Go + + ctx := context.Background() + client, err := genai.NewClient(ctx, &genai.ClientConfig{ + APIKey: os.Getenv("GEMINI_API_KEY"), + Backend: genai.BackendGeminiAPI, + }) + if err != nil { + log.Fatal(err) + } + + response, err := client.Models.GenerateContent( + ctx, + "gemini-2.0-pro-exp-02-05", + genai.Text( + `Write and execute code that calculates the sum of the first 50 prime numbers. + Ensure that only the executable code and its resulting output are generated.`, + ), + &genai.GenerateContentConfig{}, + ) + if err != nil { + log.Fatal(err) + } + + // Print the response. + printResponse(response) + + fmt.Println("---") + fmt.Println(response.Text())https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/go/code_execution.go#L14-L40 + +### Java + + Client client = new Client(); + + String prompt = """ + Write and execute code that calculates the sum of the first 50 prime numbers. + Ensure that only the executable code and its resulting output are generated. + """; + + GenerateContentResponse response = + client.models.generateContent( + "gemini-2.0-pro-exp-02-05", + prompt, + null); + + for (Part part : response.candidates().get().getFirst().content().get().parts().get()) { + System.out.println(part + "\n"); + } + + System.out.println("-".repeat(80)); + System.out.println(response.text());https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/java/src/main/java/com/example/gemini/CodeExecution.java#L32-L50 + +### Function Calling + +### Python + + from google import genai + from google.genai import types + + client = genai.Client() + + def add(a: float, b: float) -> float: + """returns a + b.""" + return a + b + + def subtract(a: float, b: float) -> float: + """returns a - b.""" + return a - b + + def multiply(a: float, b: float) -> float: + """returns a * b.""" + return a * b + + def divide(a: float, b: float) -> float: + """returns a / b.""" + return a / b + + # Create a chat session; function calling (via tools) is enabled in the config. + chat = client.chats.create( + model="gemini-2.0-flash", + config=types.GenerateContentConfig(tools=[add, subtract, multiply, divide]), + ) + response = chat.send_message( + message="I have 57 cats, each owns 44 mittens, how many mittens is that in total?" + ) + print(response.text)https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/python/function_calling.py#L22-L51 + +### Go + + ctx := context.Background() + client, err := genai.NewClient(ctx, &genai.ClientConfig{ + APIKey: os.Getenv("GEMINI_API_KEY"), + Backend: genai.BackendGeminiAPI, + }) + if err != nil { + log.Fatal(err) + } + modelName := "gemini-2.0-flash" + + // Create the function declarations for arithmetic operations. + addDeclaration := createArithmeticToolDeclaration("addNumbers", "Return the result of adding two numbers.") + subtractDeclaration := createArithmeticToolDeclaration("subtractNumbers", "Return the result of subtracting the second number from the first.") + multiplyDeclaration := createArithmeticToolDeclaration("multiplyNumbers", "Return the product of two numbers.") + divideDeclaration := createArithmeticToolDeclaration("divideNumbers", "Return the quotient of dividing the first number by the second.") + + // Group the function declarations as a tool. + tools := []*genai.Tool{ + { + FunctionDeclarations: []*genai.FunctionDeclaration{ + addDeclaration, + subtractDeclaration, + multiplyDeclaration, + divideDeclaration, + }, + }, + } + + // Create the content prompt. + contents := []*genai.Content{ + genai.NewContentFromText( + "I have 57 cats, each owns 44 mittens, how many mittens is that in total?", genai.RoleUser, + ), + } + + // Set up the generate content configuration with function calling enabled. + config := &genai.GenerateContentConfig{ + Tools: tools, + ToolConfig: &genai.ToolConfig{ + FunctionCallingConfig: &genai.FunctionCallingConfig{ + // The mode equivalent to FunctionCallingConfigMode.ANY in JS. + Mode: genai.FunctionCallingConfigModeAny, + }, + }, + } + + genContentResp, err := client.Models.GenerateContent(ctx, modelName, contents, config) + if err != nil { + log.Fatal(err) + } + + // Assume the response includes a list of function calls. + if len(genContentResp.FunctionCalls()) == 0 { + log.Println("No function call returned from the AI.") + return nil + } + functionCall := genContentResp.FunctionCalls()[0] + log.Printf("Function call: %+v\n", functionCall) + + // Marshal the Args map into JSON bytes. + argsMap, err := json.Marshal(functionCall.Args) + if err != nil { + log.Fatal(err) + } + + // Unmarshal the JSON bytes into the ArithmeticArgs struct. + var args ArithmeticArgs + if err := json.Unmarshal(argsMap, &args); err != nil { + log.Fatal(err) + } + + // Map the function name to the actual arithmetic function. + var result float64 + switch functionCall.Name { + case "addNumbers": + result = add(args.FirstParam, args.SecondParam) + case "subtractNumbers": + result = subtract(args.FirstParam, args.SecondParam) + case "multiplyNumbers": + result = multiply(args.FirstParam, args.SecondParam) + case "divideNumbers": + result = divide(args.FirstParam, args.SecondParam) + default: + return fmt.Errorf("unimplemented function: %s", functionCall.Name) + } + log.Printf("Function result: %v\n", result) + + // Prepare the final result message as content. + resultContents := []*genai.Content{ + genai.NewContentFromText("The final result is " + fmt.Sprintf("%v", result), genai.RoleUser), + } + + // Use GenerateContent to send the final result. + finalResponse, err := client.Models.GenerateContent(ctx, modelName, resultContents, &genai.GenerateContentConfig{}) + if err != nil { + log.Fatal(err) + } + + printResponse(finalResponse)https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/go/function_calling.go#L63-L161 + +### Node.js + + // Make sure to include the following import: + // import {GoogleGenAI} from '@google/genai'; + const ai = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY }); + + /** + * The add function returns the sum of two numbers. + * @param {number} a + * @param {number} b + * @returns {number} + */ + function add(a, b) { + return a + b; + } + + /** + * The subtract function returns the difference (a - b). + * @param {number} a + * @param {number} b + * @returns {number} + */ + function subtract(a, b) { + return a - b; + } + + /** + * The multiply function returns the product of two numbers. + * @param {number} a + * @param {number} b + * @returns {number} + */ + function multiply(a, b) { + return a * b; + } + + /** + * The divide function returns the quotient of a divided by b. + * @param {number} a + * @param {number} b + * @returns {number} + */ + function divide(a, b) { + return a / b; + } + + const addDeclaration = { + name: "addNumbers", + parameters: { + type: "object", + description: "Return the result of adding two numbers.", + properties: { + firstParam: { + type: "number", + description: + "The first parameter which can be an integer or a floating point number.", + }, + secondParam: { + type: "number", + description: + "The second parameter which can be an integer or a floating point number.", + }, + }, + required: ["firstParam", "secondParam"], + }, + }; + + const subtractDeclaration = { + name: "subtractNumbers", + parameters: { + type: "object", + description: + "Return the result of subtracting the second number from the first.", + properties: { + firstParam: { + type: "number", + description: "The first parameter.", + }, + secondParam: { + type: "number", + description: "The second parameter.", + }, + }, + required: ["firstParam", "secondParam"], + }, + }; + + const multiplyDeclaration = { + name: "multiplyNumbers", + parameters: { + type: "object", + description: "Return the product of two numbers.", + properties: { + firstParam: { + type: "number", + description: "The first parameter.", + }, + secondParam: { + type: "number", + description: "The second parameter.", + }, + }, + required: ["firstParam", "secondParam"], + }, + }; + + const divideDeclaration = { + name: "divideNumbers", + parameters: { + type: "object", + description: + "Return the quotient of dividing the first number by the second.", + properties: { + firstParam: { + type: "number", + description: "The first parameter.", + }, + secondParam: { + type: "number", + description: "The second parameter.", + }, + }, + required: ["firstParam", "secondParam"], + }, + }; + + // Step 1: Call generateContent with function calling enabled. + const generateContentResponse = await ai.models.generateContent({ + model: "gemini-2.0-flash", + contents: + "I have 57 cats, each owns 44 mittens, how many mittens is that in total?", + config: { + toolConfig: { + functionCallingConfig: { + mode: FunctionCallingConfigMode.ANY, + }, + }, + tools: [ + { + functionDeclarations: [ + addDeclaration, + subtractDeclaration, + multiplyDeclaration, + divideDeclaration, + ], + }, + ], + }, + }); + + // Step 2: Extract the function call.( + // Assuming the response contains a 'functionCalls' array. + const functionCall = + generateContentResponse.functionCalls && + generateContentResponse.functionCalls[0]; + console.log(functionCall); + + // Parse the arguments. + const args = functionCall.args; + // Expected args format: { firstParam: number, secondParam: number } + + // Step 3: Invoke the actual function based on the function name. + const functionMapping = { + addNumbers: add, + subtractNumbers: subtract, + multiplyNumbers: multiply, + divideNumbers: divide, + }; + const func = functionMapping[functionCall.name]; + if (!func) { + console.error("Unimplemented error:", functionCall.name); + return generateContentResponse; + } + const resultValue = func(args.firstParam, args.secondParam); + console.log("Function result:", resultValue); + + // Step 4: Use the chat API to send the result as the final answer. + const chat = ai.chats.create({ model: "gemini-2.0-flash" }); + const chatResponse = await chat.sendMessage({ + message: "The final result is " + resultValue, + }); + console.log(chatResponse.text); + return chatResponse; + } + https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/javascript/function_calling.js#L22-L-1 + +### Shell + + + cat > tools.json << EOF + { + "function_declarations": [ + { + "name": "enable_lights", + "description": "Turn on the lighting system." + }, + { + "name": "set_light_color", + "description": "Set the light color. Lights must be enabled for this to work.", + "parameters": { + "type": "object", + "properties": { + "rgb_hex": { + "type": "string", + "description": "The light color as a 6-digit hex string, e.g. ff0000 for red." + } + }, + "required": [ + "rgb_hex" + ] + } + }, + { + "name": "stop_lights", + "description": "Turn off the lighting system." + } + ] + } + EOF + + curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash:generateContent?key=$GEMINI_API_KEY" \ + -H 'Content-Type: application/json' \ + -d @<(echo ' + { + "system_instruction": { + "parts": { + "text": "You are a helpful lighting system bot. You can turn lights on and off, and you can set the color. Do not perform any other tasks." + } + }, + "tools": ['$(cat tools.json)'], + + "tool_config": { + "function_calling_config": {"mode": "auto"} + }, + + "contents": { + "role": "user", + "parts": { + "text": "Turn on the lights please." + } + } + } + ') 2>/dev/null |sed -n '/"content"/,/"finishReason"/p'https://github.com/google-gemini/deprecated-generative-ai-python/blob/7a7cc5474ddaa0255a4410e05361028a24400abd/samples/rest/function_calling.sh#L4-L59 + +### Java + + Client client = new Client(); + + FunctionDeclaration addFunction = + FunctionDeclaration.builder() + .name("addNumbers") + .parameters( + Schema.builder() + .type("object") + .properties(Map.of( + "firstParam", Schema.builder().type("number").description("First number").build(), + "secondParam", Schema.builder().type("number").description("Second number").build())) + .required(Arrays.asList("firstParam", "secondParam")) + .build()) + .build(); + + FunctionDeclaration subtractFunction = + FunctionDeclaration.builder() + .name("subtractNumbers") + .parameters( + Schema.builder() + .type("object") + .properties(Map.of( + "firstParam", Schema.builder().type("number").description("First number").build(), + "secondParam", Schema.builder().type("number").description("Second number").build())) + .required(Arrays.asList("firstParam", "secondParam")) + .build()) + .build(); + + FunctionDeclaration multiplyFunction = + FunctionDeclaration.builder() + .name("multiplyNumbers") + .parameters( + Schema.builder() + .type("object") + .properties(Map.of( + "firstParam", Schema.builder().type("number").description("First number").build(), + "secondParam", Schema.builder().type("number").description("Second number").build())) + .required(Arrays.asList("firstParam", "secondParam")) + .build()) + .build(); + + FunctionDeclaration divideFunction = + FunctionDeclaration.builder() + .name("divideNumbers") + .parameters( + Schema.builder() + .type("object") + .properties(Map.of( + "firstParam", Schema.builder().type("number").description("First number").build(), + "secondParam", Schema.builder().type("number").description("Second number").build())) + .required(Arrays.asList("firstParam", "secondParam")) + .build()) + .build(); + + GenerateContentConfig config = GenerateContentConfig.builder() + .toolConfig(ToolConfig.builder().functionCallingConfig( + FunctionCallingConfig.builder().mode("ANY").build() + ).build()) + .tools( + Collections.singletonList( + Tool.builder().functionDeclarations( + Arrays.asList( + addFunction, + subtractFunction, + divideFunction, + multiplyFunction + ) + ).build() + + ) + ) + .build(); + + GenerateContentResponse response = + client.models.generateContent( + "gemini-2.0-flash", + "I have 57 cats, each owns 44 mittens, how many mittens is that in total?", + config); + + + if (response.functionCalls() == null || response.functionCalls().isEmpty()) { + System.err.println("No function call received"); + return null; + } + + var functionCall = response.functionCalls().getFirst(); + String functionName = functionCall.name().get(); + var arguments = functionCall.args(); + + Map> functionMapping = new HashMap<>(); + functionMapping.put("addNumbers", (a, b) -> a + b); + functionMapping.put("subtractNumbers", (a, b) -> a - b); + functionMapping.put("multiplyNumbers", (a, b) -> a * b); + functionMapping.put("divideNumbers", (a, b) -> b != 0 ? a / b : Double.NaN); + + BiFunction function = functionMapping.get(functionName); + + Number firstParam = (Number) arguments.get().get("firstParam"); + Number secondParam = (Number) arguments.get().get("secondParam"); + Double result = function.apply(firstParam.doubleValue(), secondParam.doubleValue()); + + System.out.println(result);https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/java/src/main/java/com/example/gemini/FunctionCalling.java#L38-L139 + +### Generation config + +### Python + + from google import genai + from google.genai import types + + client = genai.Client() + response = client.models.generate_content( + model="gemini-2.0-flash", + contents="Tell me a story about a magic backpack.", + config=types.GenerateContentConfig( + candidate_count=1, + stop_sequences=["x"], + max_output_tokens=20, + temperature=1.0, + ), + ) + print(response.text)https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/python/configure_model_parameters.py#L22-L36 + +### Node.js + + // Make sure to include the following import: + // import {GoogleGenAI} from '@google/genai'; + const ai = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY }); + + const response = await ai.models.generateContent({ + model: "gemini-2.0-flash", + contents: "Tell me a story about a magic backpack.", + config: { + candidateCount: 1, + stopSequences: ["x"], + maxOutputTokens: 20, + temperature: 1.0, + }, + }); + + console.log(response.text);https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/javascript/configure_model_parameters.js#L22-L37 + +### Go + + ctx := context.Background() + client, err := genai.NewClient(ctx, &genai.ClientConfig{ + APIKey: os.Getenv("GEMINI_API_KEY"), + Backend: genai.BackendGeminiAPI, + }) + if err != nil { + log.Fatal(err) + } + + // Create local variables for parameters. + candidateCount := int32(1) + maxOutputTokens := int32(20) + temperature := float32(1.0) + + response, err := client.Models.GenerateContent( + ctx, + "gemini-2.0-flash", + genai.Text("Tell me a story about a magic backpack."), + &genai.GenerateContentConfig{ + CandidateCount: candidateCount, + StopSequences: []string{"x"}, + MaxOutputTokens: maxOutputTokens, + Temperature: &temperature, + }, + ) + if err != nil { + log.Fatal(err) + } + + printResponse(response)https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/go/configure_model_parameters.go#L13-L42 + +### Shell + + curl https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash:generateContent?key=$GEMINI_API_KEY \ + -H 'Content-Type: application/json' \ + -X POST \ + -d '{ + "contents": [{ + "parts":[ + {"text": "Explain how AI works"} + ] + }], + "generationConfig": { + "stopSequences": [ + "Title" + ], + "temperature": 1.0, + "maxOutputTokens": 800, + "topP": 0.8, + "topK": 10 + } + }' 2> /dev/null | grep "text"https://github.com/google-gemini/deprecated-generative-ai-python/blob/7a7cc5474ddaa0255a4410e05361028a24400abd/samples/rest/configure_model_parameters.sh#L4-L23 + +### Java + + Client client = new Client(); + + GenerateContentConfig config = + GenerateContentConfig.builder() + .candidateCount(1) + .stopSequences(List.of("x")) + .maxOutputTokens(20) + .temperature(1.0F) + .build(); + + GenerateContentResponse response = + client.models.generateContent( + "gemini-2.0-flash", + "Tell me a story about a magic backpack.", + config); + + System.out.println(response.text());https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/java/src/main/java/com/example/gemini/ConfigureModelParameters.java#L29-L45 + +### Safety Settings + +### Python + + from google import genai + from google.genai import types + + client = genai.Client() + unsafe_prompt = ( + "I support Martians Soccer Club and I think Jupiterians Football Club sucks! " + "Write a ironic phrase about them including expletives." + ) + response = client.models.generate_content( + model="gemini-2.0-flash", + contents=unsafe_prompt, + config=types.GenerateContentConfig( + safety_settings=[ + types.SafetySetting( + category="HARM_CATEGORY_HATE_SPEECH", + threshold="BLOCK_MEDIUM_AND_ABOVE", + ), + types.SafetySetting( + category="HARM_CATEGORY_HARASSMENT", threshold="BLOCK_ONLY_HIGH" + ), + ] + ), + ) + try: + print(response.text) + except Exception: + print("No information generated by the model.") + + print(response.candidates[0].safety_ratings)https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/python/safety_settings.py#L48-L76 + +### Node.js + + // Make sure to include the following import: + // import {GoogleGenAI} from '@google/genai'; + const ai = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY }); + const unsafePrompt = + "I support Martians Soccer Club and I think Jupiterians Football Club sucks! Write a ironic phrase about them including expletives."; + + const response = await ai.models.generateContent({ + model: "gemini-2.0-flash", + contents: unsafePrompt, + config: { + safetySettings: [ + { + category: "HARM_CATEGORY_HATE_SPEECH", + threshold: "BLOCK_MEDIUM_AND_ABOVE", + }, + { + category: "HARM_CATEGORY_HARASSMENT", + threshold: "BLOCK_ONLY_HIGH", + }, + ], + }, + }); + + try { + console.log("Generated text:", response.text); + } catch (error) { + console.log("No information generated by the model."); + } + console.log("Safety ratings:", response.candidates[0].safetyRatings); + return response; + } + https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/javascript/safety_settings.js#L49-L-1 + +### Go + + ctx := context.Background() + client, err := genai.NewClient(ctx, &genai.ClientConfig{ + APIKey: os.Getenv("GEMINI_API_KEY"), + Backend: genai.BackendGeminiAPI, + }) + if err != nil { + log.Fatal(err) + } + + unsafePrompt := "I support Martians Soccer Club and I think Jupiterians Football Club sucks! " + + "Write a ironic phrase about them including expletives." + + config := &genai.GenerateContentConfig{ + SafetySettings: []*genai.SafetySetting{ + { + Category: "HARM_CATEGORY_HATE_SPEECH", + Threshold: "BLOCK_MEDIUM_AND_ABOVE", + }, + { + Category: "HARM_CATEGORY_HARASSMENT", + Threshold: "BLOCK_ONLY_HIGH", + }, + }, + } + contents := []*genai.Content{ + genai.NewContentFromText(unsafePrompt, genai.RoleUser), + } + response, err := client.Models.GenerateContent(ctx, "gemini-2.0-flash", contents, config) + if err != nil { + log.Fatal(err) + } + + // Print the generated text. + text := response.Text() + fmt.Println("Generated text:", text) + + // Print the and safety ratings from the first candidate. + if len(response.Candidates) > 0 { + fmt.Println("Finish reason:", response.Candidates[0].FinishReason) + safetyRatings, err := json.MarshalIndent(response.Candidates[0].SafetyRatings, "", " ") + if err != nil { + return err + } + fmt.Println("Safety ratings:", string(safetyRatings)) + } else { + fmt.Println("No candidate returned.") + }https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/go/safety_settings.go#L60-L106 + +### Shell + + echo '{ + "safetySettings": [ + {"category": "HARM_CATEGORY_HARASSMENT", "threshold": "BLOCK_ONLY_HIGH"}, + {"category": "HARM_CATEGORY_HATE_SPEECH", "threshold": "BLOCK_MEDIUM_AND_ABOVE"} + ], + "contents": [{ + "parts":[{ + "text": "'I support Martians Soccer Club and I think Jupiterians Football Club sucks! Write a ironic phrase about them.'"}]}]}' > request.json + + curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash:generateContent?key=$GEMINI_API_KEY" \ + -H 'Content-Type: application/json' \ + -X POST \ + -d @request.json 2> /dev/nullhttps://github.com/google-gemini/deprecated-generative-ai-python/blob/7a7cc5474ddaa0255a4410e05361028a24400abd/samples/rest/safety_settings.sh#L20-L33 + +### Java + + Client client = new Client(); + + String unsafePrompt = """ + I support Martians Soccer Club and I think Jupiterians Football Club sucks! + Write a ironic phrase about them including expletives. + """; + + GenerateContentConfig config = + GenerateContentConfig.builder() + .safetySettings(Arrays.asList( + SafetySetting.builder() + .category("HARM_CATEGORY_HATE_SPEECH") + .threshold("BLOCK_MEDIUM_AND_ABOVE") + .build(), + SafetySetting.builder() + .category("HARM_CATEGORY_HARASSMENT") + .threshold("BLOCK_ONLY_HIGH") + .build() + )).build(); + + GenerateContentResponse response = + client.models.generateContent( + "gemini-2.0-flash", + unsafePrompt, + config); + + try { + System.out.println(response.text()); + } catch (Exception e) { + System.out.println("No information generated by the model"); + } + + System.out.println(response.candidates().get().getFirst().safetyRatings());https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/java/src/main/java/com/example/gemini/SafetySettings.java#L60-L92 + +### System Instruction + +### Python + + from google import genai + from google.genai import types + + client = genai.Client() + response = client.models.generate_content( + model="gemini-2.0-flash", + contents="Good morning! How are you?", + config=types.GenerateContentConfig( + system_instruction="You are a cat. Your name is Neko." + ), + ) + print(response.text)https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/python/system_instruction.py#L22-L33 + +### Node.js + + // Make sure to include the following import: + // import {GoogleGenAI} from '@google/genai'; + const ai = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY }); + const response = await ai.models.generateContent({ + model: "gemini-2.0-flash", + contents: "Good morning! How are you?", + config: { + systemInstruction: "You are a cat. Your name is Neko.", + }, + }); + console.log(response.text);https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/javascript/system_instruction.js#L22-L32 + +### Go + + ctx := context.Background() + client, err := genai.NewClient(ctx, &genai.ClientConfig{ + APIKey: os.Getenv("GEMINI_API_KEY"), + Backend: genai.BackendGeminiAPI, + }) + if err != nil { + log.Fatal(err) + } + + // Construct the user message contents. + contents := []*genai.Content{ + genai.NewContentFromText("Good morning! How are you?", genai.RoleUser), + } + + // Set the system instruction as a *genai.Content. + config := &genai.GenerateContentConfig{ + SystemInstruction: genai.NewContentFromText("You are a cat. Your name is Neko.", genai.RoleUser), + } + + response, err := client.Models.GenerateContent(ctx, "gemini-2.0-flash", contents, config) + if err != nil { + log.Fatal(err) + } + printResponse(response)https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/go/system_instruction.go#L13-L36 + +### Shell + + curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash:generateContent?key=$GEMINI_API_KEY" \ + -H 'Content-Type: application/json' \ + -d '{ "system_instruction": { + "parts": + { "text": "You are a cat. Your name is Neko."}}, + "contents": { + "parts": { + "text": "Hello there"}}}'https://github.com/google-gemini/deprecated-generative-ai-python/blob/7a7cc5474ddaa0255a4410e05361028a24400abd/samples/rest/system_instruction.sh#L4-L12 + +### Java + + Client client = new Client(); + + Part textPart = Part.builder().text("You are a cat. Your name is Neko.").build(); + + Content content = Content.builder().role("system").parts(ImmutableList.of(textPart)).build(); + + GenerateContentConfig config = GenerateContentConfig.builder() + .systemInstruction(content) + .build(); + + GenerateContentResponse response = + client.models.generateContent( + "gemini-2.0-flash", + "Good morning! How are you?", + config); + + System.out.println(response.text());https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/java/src/main/java/com/example/gemini/SystemInstruction.java#L30-L46 + +### Response body + +If successful, the response body contains an instance of `https://ai.google.dev/api/generate-content#v1beta.GenerateContentResponse`. + +## Method: models.streamGenerateContent + +- [Endpoint](https://ai.google.dev/api/generate-content#body.HTTP_TEMPLATE) +- [Path parameters](https://ai.google.dev/api/generate-content#body.PATH_PARAMETERS) +- [Request body](https://ai.google.dev/api/generate-content#body.request_body) + - [JSON representation](https://ai.google.dev/api/generate-content#body.request_body.SCHEMA_REPRESENTATION) +- [Response body](https://ai.google.dev/api/generate-content#body.response_body) +- [Authorization scopes](https://ai.google.dev/api/generate-content#body.aspect) +- [Example request](https://ai.google.dev/api/generate-content#body.codeSnippets) + - [Text](https://ai.google.dev/api/generate-content#body.codeSnippets.group) + - [Image](https://ai.google.dev/api/generate-content#body.codeSnippets.group_1) + - [Audio](https://ai.google.dev/api/generate-content#body.codeSnippets.group_2) + - [Video](https://ai.google.dev/api/generate-content#body.codeSnippets.group_3) + - [PDF](https://ai.google.dev/api/generate-content#body.codeSnippets.group_4) + - [Chat](https://ai.google.dev/api/generate-content#body.codeSnippets.group_5) + +Generates a [streamed response](https://ai.google.dev/gemini-api/docs/text-generation?lang=python#generate-a-text-stream) from the model given an input `GenerateContentRequest`. + +### Endpoint + +post `https://generativelanguage.googleapis.com/v1beta/{model=models/*}:streamGenerateContent` + +### Path parameters + +`model` `string` Required. The name of the `Model` to use for generating the completion. + +Format: `models/{model}`. It takes the form `models/{model}`. + +### Request body + +The request body contains data with the following structure: +Fields `contents[]` ``object (`https://ai.google.dev/api/caching#Content`)`` Required. The content of the current conversation with the model. + +For single-turn queries, this is a single instance. For multi-turn queries like [chat](https://ai.google.dev/gemini-api/docs/text-generation#chat), this is a repeated field that contains the conversation history and the latest request. +`tools[]` ``object (`https://ai.google.dev/api/caching#Tool`)`` Optional. A list of `Tools` the `Model` may use to generate the next response. + +A `Tool` is a piece of code that enables the system to interact with external systems to perform an action, or set of actions, outside of knowledge and scope of the `Model`. Supported `Tool`s are `Function` and `codeExecution`. Refer to the [Function calling](https://ai.google.dev/gemini-api/docs/function-calling) and the [Code execution](https://ai.google.dev/gemini-api/docs/code-execution) guides to learn more. +`toolConfig` ``object (`https://ai.google.dev/api/caching#ToolConfig`)`` Optional. Tool configuration for any `Tool` specified in the request. Refer to the [Function calling guide](https://ai.google.dev/gemini-api/docs/function-calling#function_calling_mode) for a usage example. +`safetySettings[]` ``object (`https://ai.google.dev/api/generate-content#v1beta.SafetySetting`)`` Optional. A list of unique `SafetySetting` instances for blocking unsafe content. + +This will be enforced on the `GenerateContentRequest.contents` and `GenerateContentResponse.candidates`. There should not be more than one setting for each `SafetyCategory` type. The API will block any contents and responses that fail to meet the thresholds set by these settings. This list overrides the default settings for each `SafetyCategory` specified in the safetySettings. If there is no `SafetySetting` for a given `SafetyCategory` provided in the list, the API will use the default safety setting for that category. Harm categories HARM_CATEGORY_HATE_SPEECH, HARM_CATEGORY_SEXUALLY_EXPLICIT, HARM_CATEGORY_DANGEROUS_CONTENT, HARM_CATEGORY_HARASSMENT, HARM_CATEGORY_CIVIC_INTEGRITY are supported. Refer to the [guide](https://ai.google.dev/gemini-api/docs/safety-settings) for detailed information on available safety settings. Also refer to the [Safety guidance](https://ai.google.dev/gemini-api/docs/safety-guidance) to learn how to incorporate safety considerations in your AI applications. +`systemInstruction` ``object (`https://ai.google.dev/api/caching#Content`)`` Optional. Developer set [system instruction(s)](https://ai.google.dev/gemini-api/docs/system-instructions). Currently, text only. +`generationConfig` ``object (`https://ai.google.dev/api/generate-content#v1beta.GenerationConfig`)`` Optional. Configuration options for model generation and outputs. +`cachedContent` `string` Optional. The name of the content [cached](https://ai.google.dev/gemini-api/docs/caching) to use as context to serve the prediction. Format: `cachedContents/{cachedContent}` +`serviceTier` ``enum (`https://ai.google.dev/api/generate-content#v1beta.ServiceTier`)`` Optional. The service tier of the request. +`store` `boolean` Optional. Configures the logging behavior for a given request. If set, it takes precedence over the project-level logging config. + +### Example request + +### Text + +### Python + + from google import genai + + client = genai.Client() + response = client.models.generate_content_stream( + model="gemini-2.0-flash", contents="Write a story about a magic backpack." + ) + for chunk in response: + print(chunk.text) + print("_" * 80)https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/python/text_generation.py#L37-L45 + +### Node.js + + // Make sure to include the following import: + // import {GoogleGenAI} from '@google/genai'; + const ai = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY }); + + const response = await ai.models.generateContentStream({ + model: "gemini-2.0-flash", + contents: "Write a story about a magic backpack.", + }); + let text = ""; + for await (const chunk of response) { + console.log(chunk.text); + text += chunk.text; + }https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/javascript/text_generation.js#L51-L63 + +### Go + + ctx := context.Background() + client, err := genai.NewClient(ctx, &genai.ClientConfig{ + APIKey: os.Getenv("GEMINI_API_KEY"), + Backend: genai.BackendGeminiAPI, + }) + if err != nil { + log.Fatal(err) + } + contents := []*genai.Content{ + genai.NewContentFromText("Write a story about a magic backpack.", genai.RoleUser), + } + for response, err := range client.Models.GenerateContentStream( + ctx, + "gemini-2.0-flash", + contents, + nil, + ) { + if err != nil { + log.Fatal(err) + } + fmt.Print(response.Candidates[0].Content.Parts[0].Text) + }https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/go/text_generation.go#L38-L59 + +### Shell + + curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash:streamGenerateContent?alt=sse&key=${GEMINI_API_KEY}" \ + -H 'Content-Type: application/json' \ + --no-buffer \ + -d '{ "contents":[{"parts":[{"text": "Write a story about a magic backpack."}]}]}'https://github.com/google-gemini/deprecated-generative-ai-python/blob/7a7cc5474ddaa0255a4410e05361028a24400abd/samples/rest/text_generation.sh#L33-L37 + +### Java + + Client client = new Client(); + + ResponseStream responseStream = + client.models.generateContentStream( + "gemini-2.0-flash", + "Write a story about a magic backpack.", + null); + + StringBuilder response = new StringBuilder(); + for (GenerateContentResponse res : responseStream) { + System.out.print(res.text()); + response.append(res.text()); + } + + responseStream.close();https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/java/src/main/java/com/example/gemini/TextGeneration.java#L49-L63 + +### Image + +### Python + + from google import genai + import PIL.Image + + client = genai.Client() + organ = PIL.Image.open(media / "organ.jpg") + response = client.models.generate_content_stream( + model="gemini-2.0-flash", contents=["Tell me about this instrument", organ] + ) + for chunk in response: + print(chunk.text) + print("_" * 80)https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/python/text_generation.py#L63-L73 + +### Node.js + + // Make sure to include the following import: + // import {GoogleGenAI} from '@google/genai'; + const ai = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY }); + + const organ = await ai.files.upload({ + file: path.join(media, "organ.jpg"), + }); + + const response = await ai.models.generateContentStream({ + model: "gemini-2.0-flash", + contents: [ + createUserContent([ + "Tell me about this instrument", + createPartFromUri(organ.uri, organ.mimeType) + ]), + ], + }); + let text = ""; + for await (const chunk of response) { + console.log(chunk.text); + text += chunk.text; + }https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/javascript/text_generation.js#L94-L115 + +### Go + + ctx := context.Background() + client, err := genai.NewClient(ctx, &genai.ClientConfig{ + APIKey: os.Getenv("GEMINI_API_KEY"), + Backend: genai.BackendGeminiAPI, + }) + if err != nil { + log.Fatal(err) + } + file, err := client.Files.UploadFromPath( + ctx, + filepath.Join(getMedia(), "organ.jpg"), + &genai.UploadFileConfig{ + MIMEType : "image/jpeg", + }, + ) + if err != nil { + log.Fatal(err) + } + parts := []*genai.Part{ + genai.NewPartFromText("Tell me about this instrument"), + genai.NewPartFromURI(file.URI, file.MIMEType), + } + contents := []*genai.Content{ + genai.NewContentFromParts(parts, genai.RoleUser), + } + for response, err := range client.Models.GenerateContentStream( + ctx, + "gemini-2.0-flash", + contents, + nil, + ) { + if err != nil { + log.Fatal(err) + } + fmt.Print(response.Candidates[0].Content.Parts[0].Text) + }https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/go/text_generation.go#L104-L139 + +### Shell + + cat > "$TEMP_JSON" << EOF + { + "contents": [{ + "parts":[ + {"text": "Tell me about this instrument"}, + { + "inline_data": { + "mime_type":"image/jpeg", + "data": "$(cat "$TEMP_B64")" + } + } + ] + }] + } + EOF + + curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash:streamGenerateContent?alt=sse&key=$GEMINI_API_KEY" \ + -H 'Content-Type: application/json' \ + -X POST \ + -d "@$TEMP_JSON" 2> /dev/nullhttps://github.com/google-gemini/deprecated-generative-ai-python/blob/7a7cc5474ddaa0255a4410e05361028a24400abd/samples/rest/text_generation.sh#L74-L94 + +### Java + + Client client = new Client(); + + String path = media_path + "organ.jpg"; + byte[] imageData = Files.readAllBytes(Paths.get(path)); + + Content content = + Content.fromParts( + Part.fromText("Tell me about this instrument."), + Part.fromBytes(imageData, "image/jpeg")); + + + ResponseStream responseStream = + client.models.generateContentStream( + "gemini-2.0-flash", + content, + null); + + StringBuilder response = new StringBuilder(); + for (GenerateContentResponse res : responseStream) { + System.out.print(res.text()); + response.append(res.text()); + } + + responseStream.close();https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/java/src/main/java/com/example/gemini/TextGeneration.java#L89-L112 + +### Audio + +### Python + + from google import genai + + client = genai.Client() + sample_audio = client.files.upload(file=media / "sample.mp3") + response = client.models.generate_content_stream( + model="gemini-2.0-flash", + contents=["Give me a summary of this audio file.", sample_audio], + ) + for chunk in response: + print(chunk.text) + print("_" * 80)https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/python/text_generation.py#L131-L141 + +### Go + + ctx := context.Background() + client, err := genai.NewClient(ctx, &genai.ClientConfig{ + APIKey: os.Getenv("GEMINI_API_KEY"), + Backend: genai.BackendGeminiAPI, + }) + if err != nil { + log.Fatal(err) + } + + file, err := client.Files.UploadFromPath( + ctx, + filepath.Join(getMedia(), "sample.mp3"), + &genai.UploadFileConfig{ + MIMEType : "audio/mpeg", + }, + ) + if err != nil { + log.Fatal(err) + } + + parts := []*genai.Part{ + genai.NewPartFromText("Give me a summary of this audio file."), + genai.NewPartFromURI(file.URI, file.MIMEType), + } + + contents := []*genai.Content{ + genai.NewContentFromParts(parts, genai.RoleUser), + } + + for result, err := range client.Models.GenerateContentStream( + ctx, + "gemini-2.0-flash", + contents, + nil, + ) { + if err != nil { + log.Fatal(err) + } + fmt.Print(result.Candidates[0].Content.Parts[0].Text) + }https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/go/text_generation.go#L296-L335 + +### Shell + + # Use File API to upload audio data to API request. + MIME_TYPE=$(file -b --mime-type "${AUDIO_PATH}") + NUM_BYTES=$(wc -c < "${AUDIO_PATH}") + DISPLAY_NAME=AUDIO + + tmp_header_file=upload-header.tmp + + # Initial resumable request defining metadata. + # The upload url is in the response headers dump them to a file. + curl "${BASE_URL}/upload/v1beta/files?key=${GEMINI_API_KEY}" \ + -D upload-header.tmp \ + -H "X-Goog-Upload-Protocol: resumable" \ + -H "X-Goog-Upload-Command: start" \ + -H "X-Goog-Upload-Header-Content-Length: ${NUM_BYTES}" \ + -H "X-Goog-Upload-Header-Content-Type: ${MIME_TYPE}" \ + -H "Content-Type: application/json" \ + -d "{'file': {'display_name': '${DISPLAY_NAME}'}}" 2> /dev/null + + upload_url=$(grep -i "x-goog-upload-url: " "${tmp_header_file}" | cut -d" " -f2 | tr -d "\r") + rm "${tmp_header_file}" + + # Upload the actual bytes. + curl "${upload_url}" \ + -H "Content-Length: ${NUM_BYTES}" \ + -H "X-Goog-Upload-Offset: 0" \ + -H "X-Goog-Upload-Command: upload, finalize" \ + --data-binary "@${AUDIO_PATH}" 2> /dev/null > file_info.json + + file_uri=$(jq ".file.uri" file_info.json) + echo file_uri=$file_uri + + curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash:streamGenerateContent?alt=sse&key=$GEMINI_API_KEY" \ + -H 'Content-Type: application/json' \ + -X POST \ + -d '{ + "contents": [{ + "parts":[ + {"text": "Please describe this file."}, + {"file_data":{"mime_type": "audio/mpeg", "file_uri": '$file_uri'}}] + }] + }' 2> /dev/null > response.json + + cat response.json + echohttps://github.com/google-gemini/deprecated-generative-ai-python/blob/7a7cc5474ddaa0255a4410e05361028a24400abd/samples/rest/text_generation.sh#L224-L268 + +### Video + +### Python + + from google import genai + import time + + client = genai.Client() + # Video clip (CC BY 3.0) from https://peach.blender.org/download/ + myfile = client.files.upload(file=media / "Big_Buck_Bunny.mp4") + print(f"{myfile=}") + + # Poll until the video file is completely processed (state becomes ACTIVE). + while not myfile.state or myfile.state.name != "ACTIVE": + print("Processing video...") + print("File state:", myfile.state) + time.sleep(5) + myfile = client.files.get(name=myfile.name) + + response = client.models.generate_content_stream( + model="gemini-2.0-flash", contents=[myfile, "Describe this video clip"] + ) + for chunk in response: + print(chunk.text) + print("_" * 80)https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/python/text_generation.py#L169-L189 + +### Node.js + + // Make sure to include the following import: + // import {GoogleGenAI} from '@google/genai'; + const ai = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY }); + + let video = await ai.files.upload({ + file: path.join(media, 'Big_Buck_Bunny.mp4'), + }); + + // Poll until the video file is completely processed (state becomes ACTIVE). + while (!video.state || video.state.toString() !== 'ACTIVE') { + console.log('Processing video...'); + console.log('File state: ', video.state); + await sleep(5000); + video = await ai.files.get({name: video.name}); + } + + const response = await ai.models.generateContentStream({ + model: "gemini-2.0-flash", + contents: [ + createUserContent([ + "Describe this video clip", + createPartFromUri(video.uri, video.mimeType), + ]), + ], + }); + let text = ""; + for await (const chunk of response) { + console.log(chunk.text); + text += chunk.text; + }https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/javascript/text_generation.js#L269-L298 + +### Go + + ctx := context.Background() + client, err := genai.NewClient(ctx, &genai.ClientConfig{ + APIKey: os.Getenv("GEMINI_API_KEY"), + Backend: genai.BackendGeminiAPI, + }) + if err != nil { + log.Fatal(err) + } + + file, err := client.Files.UploadFromPath( + ctx, + filepath.Join(getMedia(), "Big_Buck_Bunny.mp4"), + &genai.UploadFileConfig{ + MIMEType : "video/mp4", + }, + ) + if err != nil { + log.Fatal(err) + } + + // Poll until the video file is completely processed (state becomes ACTIVE). + for file.State == genai.FileStateUnspecified || file.State != genai.FileStateActive { + fmt.Println("Processing video...") + fmt.Println("File state:", file.State) + time.Sleep(5 * time.Second) + + file, err = client.Files.Get(ctx, file.Name, nil) + if err != nil { + log.Fatal(err) + } + } + + parts := []*genai.Part{ + genai.NewPartFromText("Describe this video clip"), + genai.NewPartFromURI(file.URI, file.MIMEType), + } + + contents := []*genai.Content{ + genai.NewContentFromParts(parts, genai.RoleUser), + } + + for result, err := range client.Models.GenerateContentStream( + ctx, + "gemini-2.0-flash", + contents, + nil, + ) { + if err != nil { + log.Fatal(err) + } + fmt.Print(result.Candidates[0].Content.Parts[0].Text) + }https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/go/text_generation.go#L394-L445 + +### Shell + + # Use File API to upload audio data to API request. + MIME_TYPE=$(file -b --mime-type "${VIDEO_PATH}") + NUM_BYTES=$(wc -c < "${VIDEO_PATH}") + DISPLAY_NAME=VIDEO_PATH + + # Initial resumable request defining metadata. + # The upload url is in the response headers dump them to a file. + curl "${BASE_URL}/upload/v1beta/files?key=${GEMINI_API_KEY}" \ + -D upload-header.tmp \ + -H "X-Goog-Upload-Protocol: resumable" \ + -H "X-Goog-Upload-Command: start" \ + -H "X-Goog-Upload-Header-Content-Length: ${NUM_BYTES}" \ + -H "X-Goog-Upload-Header-Content-Type: ${MIME_TYPE}" \ + -H "Content-Type: application/json" \ + -d "{'file': {'display_name': '${DISPLAY_NAME}'}}" 2> /dev/null + + upload_url=$(grep -i "x-goog-upload-url: " "${tmp_header_file}" | cut -d" " -f2 | tr -d "\r") + rm "${tmp_header_file}" + + # Upload the actual bytes. + curl "${upload_url}" \ + -H "Content-Length: ${NUM_BYTES}" \ + -H "X-Goog-Upload-Offset: 0" \ + -H "X-Goog-Upload-Command: upload, finalize" \ + --data-binary "@${VIDEO_PATH}" 2> /dev/null > file_info.json + + file_uri=$(jq ".file.uri" file_info.json) + echo file_uri=$file_uri + + state=$(jq ".file.state" file_info.json) + echo state=$state + + while [[ "($state)" = *"PROCESSING"* ]]; + do + echo "Processing video..." + sleep 5 + # Get the file of interest to check state + curl https://generativelanguage.googleapis.com/v1beta/files/$name > file_info.json + state=$(jq ".file.state" file_info.json) + done + + curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash:streamGenerateContent?alt=sse&key=$GEMINI_API_KEY" \ + -H 'Content-Type: application/json' \ + -X POST \ + -d '{ + "contents": [{ + "parts":[ + {"text": "Please describe this file."}, + {"file_data":{"mime_type": "video/mp4", "file_uri": '$file_uri'}}] + }] + }' 2> /dev/null > response.json + + cat response.json + echohttps://github.com/google-gemini/deprecated-generative-ai-python/blob/7a7cc5474ddaa0255a4410e05361028a24400abd/samples/rest/text_generation.sh#L335-L389 + +### PDF + +### Python + + from google import genai + + client = genai.Client() + sample_pdf = client.files.upload(file=media / "test.pdf") + response = client.models.generate_content_stream( + model="gemini-2.0-flash", + contents=["Give me a summary of this document:", sample_pdf], + ) + + for chunk in response: + print(chunk.text) + print("_" * 80)https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/python/text_generation.py#L207-L218 + +### Go + + ctx := context.Background() + client, err := genai.NewClient(ctx, &genai.ClientConfig{ + APIKey: os.Getenv("GEMINI_API_KEY"), + Backend: genai.BackendGeminiAPI, + }) + if err != nil { + log.Fatal(err) + } + + file, err := client.Files.UploadFromPath( + ctx, + filepath.Join(getMedia(), "test.pdf"), + &genai.UploadFileConfig{ + MIMEType : "application/pdf", + }, + ) + if err != nil { + log.Fatal(err) + } + + parts := []*genai.Part{ + genai.NewPartFromText("Give me a summary of this document:"), + genai.NewPartFromURI(file.URI, file.MIMEType), + } + + contents := []*genai.Content{ + genai.NewContentFromParts(parts, genai.RoleUser), + } + + for result, err := range client.Models.GenerateContentStream( + ctx, + "gemini-2.0-flash", + contents, + nil, + ) { + if err != nil { + log.Fatal(err) + } + fmt.Print(result.Candidates[0].Content.Parts[0].Text) + }https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/go/text_generation.go#L492-L531 + +### Shell + + MIME_TYPE=$(file -b --mime-type "${PDF_PATH}") + NUM_BYTES=$(wc -c < "${PDF_PATH}") + DISPLAY_NAME=TEXT + + + echo $MIME_TYPE + tmp_header_file=upload-header.tmp + + # Initial resumable request defining metadata. + # The upload url is in the response headers dump them to a file. + curl "${BASE_URL}/upload/v1beta/files?key=${GEMINI_API_KEY}" \ + -D upload-header.tmp \ + -H "X-Goog-Upload-Protocol: resumable" \ + -H "X-Goog-Upload-Command: start" \ + -H "X-Goog-Upload-Header-Content-Length: ${NUM_BYTES}" \ + -H "X-Goog-Upload-Header-Content-Type: ${MIME_TYPE}" \ + -H "Content-Type: application/json" \ + -d "{'file': {'display_name': '${DISPLAY_NAME}'}}" 2> /dev/null + + upload_url=$(grep -i "x-goog-upload-url: " "${tmp_header_file}" | cut -d" " -f2 | tr -d "\r") + rm "${tmp_header_file}" + + # Upload the actual bytes. + curl "${upload_url}" \ + -H "Content-Length: ${NUM_BYTES}" \ + -H "X-Goog-Upload-Offset: 0" \ + -H "X-Goog-Upload-Command: upload, finalize" \ + --data-binary "@${PDF_PATH}" 2> /dev/null > file_info.json + + file_uri=$(jq ".file.uri" file_info.json) + echo file_uri=$file_uri + + # Now generate content using that file + curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash:streamGenerateContent?alt=sse&key=$GEMINI_API_KEY" \ + -H 'Content-Type: application/json' \ + -X POST \ + -d '{ + "contents": [{ + "parts":[ + {"text": "Can you add a few more lines to this poem?"}, + {"file_data":{"mime_type": "application/pdf", "file_uri": '$file_uri'}}] + }] + }' 2> /dev/null > response.json + + cat response.json + echohttps://github.com/google-gemini/deprecated-generative-ai-python/blob/7a7cc5474ddaa0255a4410e05361028a24400abd/samples/rest/text_generation.sh#L445-L491 + +### Chat + +### Python + + from google import genai + from google.genai import types + + client = genai.Client() + chat = client.chats.create( + model="gemini-2.0-flash", + history=[ + types.Content(role="user", parts=[types.Part(text="Hello")]), + types.Content( + role="model", + parts=[ + types.Part( + text="Great to meet you. What would you like to know?" + ) + ], + ), + ], + ) + response = chat.send_message_stream(message="I have 2 dogs in my house.") + for chunk in response: + print(chunk.text) + print("_" * 80) + response = chat.send_message_stream(message="How many paws are in my house?") + for chunk in response: + print(chunk.text) + print("_" * 80) + + print(chat.get_history())https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/python/chat.py#L52-L79 + +### Node.js + + // Make sure to include the following import: + // import {GoogleGenAI} from '@google/genai'; + const ai = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY }); + const chat = ai.chats.create({ + model: "gemini-2.0-flash", + history: [ + { + role: "user", + parts: [{ text: "Hello" }], + }, + { + role: "model", + parts: [{ text: "Great to meet you. What would you like to know?" }], + }, + ], + }); + + console.log("Streaming response for first message:"); + const stream1 = await chat.sendMessageStream({ + message: "I have 2 dogs in my house.", + }); + for await (const chunk of stream1) { + console.log(chunk.text); + console.log("_".repeat(80)); + } + + console.log("Streaming response for second message:"); + const stream2 = await chat.sendMessageStream({ + message: "How many paws are in my house?", + }); + for await (const chunk of stream2) { + console.log(chunk.text); + console.log("_".repeat(80)); + } + + console.log(chat.getHistory());https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/javascript/chat.js#L66-L101 + +### Go + + ctx := context.Background() + client, err := genai.NewClient(ctx, &genai.ClientConfig{ + APIKey: os.Getenv("GEMINI_API_KEY"), + Backend: genai.BackendGeminiAPI, + }) + if err != nil { + log.Fatal(err) + } + + history := []*genai.Content{ + genai.NewContentFromText("Hello", genai.RoleUser), + genai.NewContentFromText("Great to meet you. What would you like to know?", genai.RoleModel), + } + chat, err := client.Chats.Create(ctx, "gemini-2.0-flash", nil, history) + if err != nil { + log.Fatal(err) + } + + for chunk, err := range chat.SendMessageStream(ctx, genai.Part{Text: "I have 2 dogs in my house."}) { + if err != nil { + log.Fatal(err) + } + fmt.Println(chunk.Text()) + fmt.Println(strings.Repeat("_", 64)) + } + + for chunk, err := range chat.SendMessageStream(ctx, genai.Part{Text: "How many paws are in my house?"}) { + if err != nil { + log.Fatal(err) + } + fmt.Println(chunk.Text()) + fmt.Println(strings.Repeat("_", 64)) + } + + fmt.Println(chat.History(false))https://github.com/google-gemini/api-examples/blob/4ce9033e1d2f857db3f728d78399e3d7ded8ef05/go/chat.go#L54-L88 + +### Shell + + curl https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash:streamGenerateContent?alt=sse&key=$GEMINI_API_KEY \ + -H 'Content-Type: application/json' \ + -X POST \ + -d '{ + "contents": [ + {"role":"user", + "parts":[{ + "text": "Hello"}]}, + {"role": "model", + "parts":[{ + "text": "Great to meet you. What would you like to know?"}]}, + {"role":"user", + "parts":[{ + "text": "I have two dogs in my house. How many paws are in my house?"}]}, + ] + }' 2> /dev/null | grep "text"https://github.com/google-gemini/deprecated-generative-ai-python/blob/7a7cc5474ddaa0255a4410e05361028a24400abd/samples/rest/chat.sh#L27-L43 + +### Response body + +If successful, the response body contains a stream of `https://ai.google.dev/api/generate-content#v1beta.GenerateContentResponse` instances. + +## GenerateContentResponse + +- [JSON representation](https://ai.google.dev/api/generate-content#SCHEMA_REPRESENTATION) +- [PromptFeedback](https://ai.google.dev/api/generate-content#PromptFeedback) + - [JSON representation](https://ai.google.dev/api/generate-content#PromptFeedback.SCHEMA_REPRESENTATION) +- [BlockReason](https://ai.google.dev/api/generate-content#BlockReason) +- [UsageMetadata](https://ai.google.dev/api/generate-content#UsageMetadata) + - [JSON representation](https://ai.google.dev/api/generate-content#UsageMetadata.SCHEMA_REPRESENTATION) +- [ModelStatus](https://ai.google.dev/api/generate-content#ModelStatus) + - [JSON representation](https://ai.google.dev/api/generate-content#ModelStatus.SCHEMA_REPRESENTATION) +- [ModelStage](https://ai.google.dev/api/generate-content#ModelStage) + +Response from the model supporting multiple candidate responses. + +Safety ratings and content filtering are reported for both prompt in `GenerateContentResponse.prompt_feedback` and for each candidate in `finishReason` and in `safetyRatings`. The API: - Returns either all requested candidates or none of them - Returns no candidates at all only if there was something wrong with the prompt (check `promptFeedback`) - Reports feedback on each candidate in `finishReason` and `safetyRatings`. +Fields `candidates[]` ``object (`https://ai.google.dev/api/generate-content#v1beta.Candidate`)`` Candidate responses from the model. +`promptFeedback` ``object (`https://ai.google.dev/api/generate-content#PromptFeedback`)`` Returns the prompt's feedback related to the content filters. +`usageMetadata` ``object (`https://ai.google.dev/api/generate-content#UsageMetadata`)`` Output only. Metadata on the generation requests' token usage. +`modelVersion` `string` Output only. The model version used to generate the response. +`responseId` `string` Output only. responseId is used to identify each response. +`modelStatus` ``object (`https://ai.google.dev/api/generate-content#ModelStatus`)`` Output only. The current model status of this model. + +| JSON representation | +|---| +| ``` { "candidates": [ { object (`https://ai.google.dev/api/generate-content#v1beta.Candidate`) } ], "promptFeedback": { object (`https://ai.google.dev/api/generate-content#PromptFeedback`) }, "usageMetadata": { object (`https://ai.google.dev/api/generate-content#UsageMetadata`) }, "modelVersion": string, "responseId": string, "modelStatus": { object (`https://ai.google.dev/api/generate-content#ModelStatus`) } } ``` | + +## PromptFeedback + +A set of the feedback metadata the prompt specified in `GenerateContentRequest.content`. +Fields `blockReason` ``enum (`https://ai.google.dev/api/generate-content#BlockReason`)`` Optional. If set, the prompt was blocked and no candidates are returned. Rephrase the prompt. +`safetyRatings[]` ``object (`https://ai.google.dev/api/generate-content#v1beta.SafetyRating`)`` Ratings for safety of the prompt. There is at most one rating per category. + +| JSON representation | +|---| +| ``` { "blockReason": enum (`https://ai.google.dev/api/generate-content#BlockReason`), "safetyRatings": [ { object (`https://ai.google.dev/api/generate-content#v1beta.SafetyRating`) } ] } ``` | + +## BlockReason + +Specifies the reason why the prompt was blocked. + +| Enums || +|---|---| +| `BLOCK_REASON_UNSPECIFIED` | Default value. This value is unused. | +| `SAFETY` | Prompt was blocked due to safety reasons. Inspect `safetyRatings` to understand which safety category blocked it. | +| `OTHER` | Prompt was blocked due to unknown reasons. | +| `BLOCKLIST` | Prompt was blocked due to the terms which are included from the terminology blocklist. | +| `PROHIBITED_CONTENT` | Prompt was blocked due to prohibited content. | +| `IMAGE_SAFETY` | Candidates blocked due to unsafe image generation content. | + +## UsageMetadata + +Metadata on the generation request's token usage. +Fields `promptTokenCount` `integer` Number of tokens in the prompt. When `cachedContent` is set, this is still the total effective prompt size meaning this includes the number of tokens in the cached content. +`cachedContentTokenCount` `integer` Number of tokens in the cached part of the prompt (the cached content) +`candidatesTokenCount` `integer` Total number of tokens across all the generated response candidates. +`toolUsePromptTokenCount` `integer` Output only. Number of tokens present in tool-use prompt(s). +`thoughtsTokenCount` `integer` Output only. Number of tokens of thoughts for thinking models. +`totalTokenCount` `integer` Total token count for the generation request (prompt + response candidates). +`promptTokensDetails[]` ``object (`https://ai.google.dev/api/generate-content#v1beta.ModalityTokenCount`)`` Output only. List of modalities that were processed in the request input. +`cacheTokensDetails[]` ``object (`https://ai.google.dev/api/generate-content#v1beta.ModalityTokenCount`)`` Output only. List of modalities of the cached content in the request input. +`candidatesTokensDetails[]` ``object (`https://ai.google.dev/api/generate-content#v1beta.ModalityTokenCount`)`` Output only. List of modalities that were returned in the response. +`toolUsePromptTokensDetails[]` ``object (`https://ai.google.dev/api/generate-content#v1beta.ModalityTokenCount`)`` Output only. List of modalities that were processed for tool-use request inputs. + +| JSON representation | +|---| +| ``` { "promptTokenCount": integer, "cachedContentTokenCount": integer, "candidatesTokenCount": integer, "toolUsePromptTokenCount": integer, "thoughtsTokenCount": integer, "totalTokenCount": integer, "promptTokensDetails": [ { object (`https://ai.google.dev/api/generate-content#v1beta.ModalityTokenCount`) } ], "cacheTokensDetails": [ { object (`https://ai.google.dev/api/generate-content#v1beta.ModalityTokenCount`) } ], "candidatesTokensDetails": [ { object (`https://ai.google.dev/api/generate-content#v1beta.ModalityTokenCount`) } ], "toolUsePromptTokensDetails": [ { object (`https://ai.google.dev/api/generate-content#v1beta.ModalityTokenCount`) } ] } ``` | + +## ModelStatus + +The status of the underlying model. This is used to indicate the stage of the underlying model and the retirement time if applicable. +Fields `modelStage` ``enum (`https://ai.google.dev/api/generate-content#ModelStage`)`` The stage of the underlying model. +`retirementTime` ``string (`https://protobuf.dev/reference/protobuf/google.protobuf#timestamp` format)`` The time at which the model will be retired. + +Uses RFC 3339, where generated output will always be Z-normalized and use 0, 3, 6 or 9 fractional digits. Offsets other than "Z" are also accepted. Examples: `"2014-10-02T15:01:23Z"`, `"2014-10-02T15:01:23.045123456Z"` or `"2014-10-02T15:01:23+05:30"`. +`message` `string` A message explaining the model status. + +| JSON representation | +|---| +| ``` { "modelStage": enum (`https://ai.google.dev/api/generate-content#ModelStage`), "retirementTime": string, "message": string } ``` | + +## ModelStage + +Defines the stage of the underlying model. + +| Enums || +|---|---| +| `MODEL_STAGE_UNSPECIFIED` | Unspecified model stage. | +| `UNSTABLE_EXPERIMENTAL` | The underlying model is subject to lots of tunings. > [!WARNING] > This item is deprecated! | +| `EXPERIMENTAL` | Models in this stage are for experimental purposes only. | +| `PREVIEW` | Models in this stage are more mature than experimental models. | +| `STABLE` | Models in this stage are considered stable and ready for production use. | +| `LEGACY` | If the model is on this stage, it means that this model is on the path to deprecation in near future. Only existing customers can use this model. | +| `DEPRECATED` | Models in this stage are deprecated. These models cannot be used. > [!WARNING] > This item is deprecated! | +| `RETIRED` | Models in this stage are retired. These models cannot be used. | + +## Candidate + +- [JSON representation](https://ai.google.dev/api/generate-content#SCHEMA_REPRESENTATION) +- [FinishReason](https://ai.google.dev/api/generate-content#FinishReason) +- [GroundingAttribution](https://ai.google.dev/api/generate-content#GroundingAttribution) + - [JSON representation](https://ai.google.dev/api/generate-content#GroundingAttribution.SCHEMA_REPRESENTATION) +- [AttributionSourceId](https://ai.google.dev/api/generate-content#AttributionSourceId) + - [JSON representation](https://ai.google.dev/api/generate-content#AttributionSourceId.SCHEMA_REPRESENTATION) +- [GroundingPassageId](https://ai.google.dev/api/generate-content#GroundingPassageId) + - [JSON representation](https://ai.google.dev/api/generate-content#GroundingPassageId.SCHEMA_REPRESENTATION) +- [SemanticRetrieverChunk](https://ai.google.dev/api/generate-content#SemanticRetrieverChunk) + - [JSON representation](https://ai.google.dev/api/generate-content#SemanticRetrieverChunk.SCHEMA_REPRESENTATION) +- [GroundingMetadata](https://ai.google.dev/api/generate-content#GroundingMetadata) + - [JSON representation](https://ai.google.dev/api/generate-content#GroundingMetadata.SCHEMA_REPRESENTATION) +- [SearchEntryPoint](https://ai.google.dev/api/generate-content#SearchEntryPoint) + - [JSON representation](https://ai.google.dev/api/generate-content#SearchEntryPoint.SCHEMA_REPRESENTATION) +- [GroundingChunk](https://ai.google.dev/api/generate-content#GroundingChunk) + - [JSON representation](https://ai.google.dev/api/generate-content#GroundingChunk.SCHEMA_REPRESENTATION) +- [Web](https://ai.google.dev/api/generate-content#Web) + - [JSON representation](https://ai.google.dev/api/generate-content#Web.SCHEMA_REPRESENTATION) +- [Image](https://ai.google.dev/api/generate-content#Image) + - [JSON representation](https://ai.google.dev/api/generate-content#Image.SCHEMA_REPRESENTATION) +- [RetrievedContext](https://ai.google.dev/api/generate-content#RetrievedContext) + - [JSON representation](https://ai.google.dev/api/generate-content#RetrievedContext.SCHEMA_REPRESENTATION) +- [CustomMetadata](https://ai.google.dev/api/generate-content#CustomMetadata) + - [JSON representation](https://ai.google.dev/api/generate-content#CustomMetadata.SCHEMA_REPRESENTATION) +- [StringList](https://ai.google.dev/api/generate-content#StringList) + - [JSON representation](https://ai.google.dev/api/generate-content#StringList.SCHEMA_REPRESENTATION) +- [Maps](https://ai.google.dev/api/generate-content#Maps) + - [JSON representation](https://ai.google.dev/api/generate-content#Maps.SCHEMA_REPRESENTATION) +- [PlaceAnswerSources](https://ai.google.dev/api/generate-content#PlaceAnswerSources) + - [JSON representation](https://ai.google.dev/api/generate-content#PlaceAnswerSources.SCHEMA_REPRESENTATION) +- [ReviewSnippet](https://ai.google.dev/api/generate-content#ReviewSnippet) + - [JSON representation](https://ai.google.dev/api/generate-content#ReviewSnippet.SCHEMA_REPRESENTATION) +- [GroundingSupport](https://ai.google.dev/api/generate-content#GroundingSupport) + - [JSON representation](https://ai.google.dev/api/generate-content#GroundingSupport.SCHEMA_REPRESENTATION) +- [Segment](https://ai.google.dev/api/generate-content#Segment) + - [JSON representation](https://ai.google.dev/api/generate-content#Segment.SCHEMA_REPRESENTATION) +- [RetrievalMetadata](https://ai.google.dev/api/generate-content#RetrievalMetadata) + - [JSON representation](https://ai.google.dev/api/generate-content#RetrievalMetadata.SCHEMA_REPRESENTATION) +- [LogprobsResult](https://ai.google.dev/api/generate-content#LogprobsResult) + - [JSON representation](https://ai.google.dev/api/generate-content#LogprobsResult.SCHEMA_REPRESENTATION) +- [TopCandidates](https://ai.google.dev/api/generate-content#TopCandidates) + - [JSON representation](https://ai.google.dev/api/generate-content#TopCandidates.SCHEMA_REPRESENTATION) +- [Candidate](https://ai.google.dev/api/generate-content#Candidate) + - [JSON representation](https://ai.google.dev/api/generate-content#Candidate.SCHEMA_REPRESENTATION) +- [UrlContextMetadata](https://ai.google.dev/api/generate-content#UrlContextMetadata) + - [JSON representation](https://ai.google.dev/api/generate-content#UrlContextMetadata.SCHEMA_REPRESENTATION) +- [UrlMetadata](https://ai.google.dev/api/generate-content#UrlMetadata) + - [JSON representation](https://ai.google.dev/api/generate-content#UrlMetadata.SCHEMA_REPRESENTATION) +- [UrlRetrievalStatus](https://ai.google.dev/api/generate-content#UrlRetrievalStatus) + +A response candidate generated from the model. +Fields `content` ``object (`https://ai.google.dev/api/caching#Content`)`` Output only. Generated content returned from the model. +`finishReason` ``enum (`https://ai.google.dev/api/generate-content#FinishReason`)`` Optional. Output only. The reason why the model stopped generating tokens. + +If empty, the model has not stopped generating tokens. +`safetyRatings[]` ``object (`https://ai.google.dev/api/generate-content#v1beta.SafetyRating`)`` List of ratings for the safety of a response candidate. + +There is at most one rating per category. +`citationMetadata` ``object (`https://ai.google.dev/api/generate-content#v1beta.CitationMetadata`)`` Output only. Citation information for model-generated candidate. + +This field may be populated with recitation information for any text included in the `content`. These are passages that are "recited" from copyrighted material in the foundational LLM's training data. +`tokenCount` `integer` Output only. Token count for this candidate. +`groundingAttributions[]` ``object (`https://ai.google.dev/api/generate-content#GroundingAttribution`)`` Output only. Attribution information for sources that contributed to a grounded answer. + +This field is populated for `GenerateAnswer` calls. +`groundingMetadata` ``object (`https://ai.google.dev/api/generate-content#GroundingMetadata`)`` Output only. Grounding metadata for the candidate. + +This field is populated for `GenerateContent` calls. +`avgLogprobs` `number` Output only. Average log probability score of the candidate. +`logprobsResult` ``object (`https://ai.google.dev/api/generate-content#LogprobsResult`)`` Output only. Log-likelihood scores for the response tokens and top tokens +`urlContextMetadata` ``object (`https://ai.google.dev/api/generate-content#UrlContextMetadata`)`` Output only. Metadata related to url context retrieval tool. +`index` `integer` Output only. Index of the candidate in the list of response candidates. +`finishMessage` `string` Optional. Output only. Details the reason why the model stopped generating tokens. This is populated only when `finishReason` is set. + +| JSON representation | +|---| +| ``` { "content": { object (`https://ai.google.dev/api/caching#Content`) }, "finishReason": enum (`https://ai.google.dev/api/generate-content#FinishReason`), "safetyRatings": [ { object (`https://ai.google.dev/api/generate-content#v1beta.SafetyRating`) } ], "citationMetadata": { object (`https://ai.google.dev/api/generate-content#v1beta.CitationMetadata`) }, "tokenCount": integer, "groundingAttributions": [ { object (`https://ai.google.dev/api/generate-content#GroundingAttribution`) } ], "groundingMetadata": { object (`https://ai.google.dev/api/generate-content#GroundingMetadata`) }, "avgLogprobs": number, "logprobsResult": { object (`https://ai.google.dev/api/generate-content#LogprobsResult`) }, "urlContextMetadata": { object (`https://ai.google.dev/api/generate-content#UrlContextMetadata`) }, "index": integer, "finishMessage": string } ``` | + +## FinishReason + +Defines the reason why the model stopped generating tokens. + +| Enums || +|---|---| +| `FINISH_REASON_UNSPECIFIED` | Default value. This value is unused. | +| `STOP` | Natural stop point of the model or provided stop sequence. | +| `MAX_TOKENS` | The maximum number of tokens as specified in the request was reached. | +| `SAFETY` | The response candidate content was flagged for safety reasons. | +| `RECITATION` | The response candidate content was flagged for recitation reasons. | +| `LANGUAGE` | The response candidate content was flagged for using an unsupported language. | +| `OTHER` | Unknown reason. | +| `BLOCKLIST` | Token generation stopped because the content contains forbidden terms. | +| `PROHIBITED_CONTENT` | Token generation stopped for potentially containing prohibited content. | +| `SPII` | Token generation stopped because the content potentially contains Sensitive Personally Identifiable Information (SPII). | +| `MALFORMED_FUNCTION_CALL` | The function call generated by the model is invalid. | +| `IMAGE_SAFETY` | Token generation stopped because generated images contain safety violations. | +| `IMAGE_PROHIBITED_CONTENT` | Image generation stopped because generated images has other prohibited content. | +| `IMAGE_OTHER` | Image generation stopped because of other miscellaneous issue. | +| `NO_IMAGE` | The model was expected to generate an image, but none was generated. | +| `IMAGE_RECITATION` | Image generation stopped due to recitation. | +| `UNEXPECTED_TOOL_CALL` | Model generated a tool call but no tools were enabled in the request. | +| `TOO_MANY_TOOL_CALLS` | Model called too many tools consecutively, thus the system exited execution. | +| `MISSING_THOUGHT_SIGNATURE` | Request has at least one thought signature missing. | +| `MALFORMED_RESPONSE` | Finished due to malformed response. | + +## GroundingAttribution + +Attribution for a source that contributed to an answer. +Fields `sourceId` ``object (`https://ai.google.dev/api/generate-content#AttributionSourceId`)`` Output only. Identifier for the source contributing to this attribution. +`content` ``object (`https://ai.google.dev/api/caching#Content`)`` Grounding source content that makes up this attribution. + +| JSON representation | +|---| +| ``` { "sourceId": { object (`https://ai.google.dev/api/generate-content#AttributionSourceId`) }, "content": { object (`https://ai.google.dev/api/caching#Content`) } } ``` | + +## AttributionSourceId + +Identifier for the source contributing to this attribution. +Fields `source` `Union type` `source` can be only one of the following: `groundingPassage` ``object (`https://ai.google.dev/api/generate-content#GroundingPassageId`)`` Identifier for an inline passage. +`semanticRetrieverChunk` ``object (`https://ai.google.dev/api/generate-content#SemanticRetrieverChunk`)`` Identifier for a `Chunk` fetched via Semantic Retriever. + +| JSON representation | +|---| +| ``` { // source "groundingPassage": { object (`https://ai.google.dev/api/generate-content#GroundingPassageId`) }, "semanticRetrieverChunk": { object (`https://ai.google.dev/api/generate-content#SemanticRetrieverChunk`) } // Union type } ``` | + +## GroundingPassageId + +Identifier for a part within a `GroundingPassage`. +Fields `passageId` `string` Output only. ID of the passage matching the `GenerateAnswerRequest`'s `GroundingPassage.id`. +`partIndex` `integer` Output only. Index of the part within the `GenerateAnswerRequest`'s `GroundingPassage.content`. + +| JSON representation | +|---| +| ``` { "passageId": string, "partIndex": integer } ``` | + +## SemanticRetrieverChunk + +Identifier for a `Chunk` retrieved via Semantic Retriever specified in the `GenerateAnswerRequest` using `SemanticRetrieverConfig`. +Fields `source` `string` Output only. Name of the source matching the request's `SemanticRetrieverConfig.source`. Example: `corpora/123` or `corpora/123/documents/abc` +`chunk` `string` Output only. Name of the `Chunk` containing the attributed text. Example: `corpora/123/documents/abc/chunks/xyz` + +| JSON representation | +|---| +| ``` { "source": string, "chunk": string } ``` | + +## GroundingMetadata + +Metadata returned to client when grounding is enabled. +Fields `groundingChunks[]` ``object (`https://ai.google.dev/api/generate-content#GroundingChunk`)`` List of supporting references retrieved from specified grounding source. When streaming, this only contains the grounding chunks that have not been included in the grounding metadata of previous responses. +`groundingSupports[]` ``object (`https://ai.google.dev/api/generate-content#GroundingSupport`)`` List of grounding support. +`webSearchQueries[]` `string` Web search queries for the following-up web search. +`imageSearchQueries[]` `string` Image search queries used for grounding. +`searchEntryPoint` ``object (`https://ai.google.dev/api/generate-content#SearchEntryPoint`)`` Optional. Google search entry for the following-up web searches. +`retrievalMetadata` ``object (`https://ai.google.dev/api/generate-content#RetrievalMetadata`)`` Metadata related to retrieval in the grounding flow. +`googleMapsWidgetContextToken` `string` Optional. Resource name of the Google Maps widget context token that can be used with the PlacesContextElement widget in order to render contextual data. Only populated in the case that grounding with Google Maps is enabled. + +| JSON representation | +|---| +| ``` { "groundingChunks": [ { object (`https://ai.google.dev/api/generate-content#GroundingChunk`) } ], "groundingSupports": [ { object (`https://ai.google.dev/api/generate-content#GroundingSupport`) } ], "webSearchQueries": [ string ], "imageSearchQueries": [ string ], "searchEntryPoint": { object (`https://ai.google.dev/api/generate-content#SearchEntryPoint`) }, "retrievalMetadata": { object (`https://ai.google.dev/api/generate-content#RetrievalMetadata`) }, "googleMapsWidgetContextToken": string } ``` | + +## SearchEntryPoint + +Google search entry point. +Fields `renderedContent` `string` Optional. Web content snippet that can be embedded in a web page or an app webview. +`sdkBlob` `string (https://developers.google.com/discovery/v1/type-format format)` Optional. Base64 encoded JSON representing array of \ tuple. + +A base64-encoded string. + +| JSON representation | +|---| +| ``` { "renderedContent": string, "sdkBlob": string } ``` | + +## GroundingChunk + +A `GroundingChunk` represents a segment of supporting evidence that grounds the model's response. It can be a chunk from the web, a retrieved context from a file, or information from Google Maps. +Fields `chunk_type` `Union type` Chunk type. `chunk_type` can be only one of the following: `web` ``object (`https://ai.google.dev/api/generate-content#Web`)`` Grounding chunk from the web. +`image` ``object (`https://ai.google.dev/api/generate-content#Image`)`` Optional. Grounding chunk from image search. +`retrievedContext` ``object (`https://ai.google.dev/api/generate-content#RetrievedContext`)`` Optional. Grounding chunk from context retrieved by the file search tool. +`maps` ``object (`https://ai.google.dev/api/generate-content#Maps`)`` Optional. Grounding chunk from Google Maps. + +| JSON representation | +|---| +| ``` { // chunk_type "web": { object (`https://ai.google.dev/api/generate-content#Web`) }, "image": { object (`https://ai.google.dev/api/generate-content#Image`) }, "retrievedContext": { object (`https://ai.google.dev/api/generate-content#RetrievedContext`) }, "maps": { object (`https://ai.google.dev/api/generate-content#Maps`) } // Union type } ``` | + +## Web + +Chunk from the web. +Fields `uri` `string` Output only. URI reference of the chunk. +`title` `string` Output only. Title of the chunk. + +| JSON representation | +|---| +| ``` { "uri": string, "title": string } ``` | + +## Image + +Chunk from image search. +Fields `sourceUri` `string` The web page URI for attribution. +`imageUri` `string` The image asset URL. +`title` `string` The title of the web page that the image is from. +`domain` `string` The root domain of the web page that the image is from, e.g. "example.com". + +| JSON representation | +|---| +| ``` { "sourceUri": string, "imageUri": string, "title": string, "domain": string } ``` | + +## RetrievedContext + +Chunk from context retrieved by the file search tool. +Fields `customMetadata[]` ``object (`https://ai.google.dev/api/generate-content#CustomMetadata`)`` Optional. User-provided metadata about the retrieved context. +`uri` `string` Optional. URI reference of the semantic retrieval document. +`title` `string` Optional. Title of the document. +`text` `string` Optional. Text of the chunk. +`fileSearchStore` `string` Optional. Name of the `FileSearchStore` containing the document. Example: `fileSearchStores/123` + +| JSON representation | +|---| +| ``` { "customMetadata": [ { object (`https://ai.google.dev/api/generate-content#CustomMetadata`) } ], "uri": string, "title": string, "text": string, "fileSearchStore": string } ``` | + +## CustomMetadata + +User provided metadata about the GroundingFact. +Fields `key` `string` The key of the metadata. +`value` `Union type` The value of the metadata. Can be a string, a list of strings, or a number. `value` can be only one of the following: `stringValue` `string` Optional. The string value of the metadata. +`stringListValue` ``object (`https://ai.google.dev/api/generate-content#StringList`)`` Optional. A list of string values for the metadata. +`numericValue` `number` Optional. The numeric value of the metadata. The expected range for this value depends on the specific `key` used. + +| JSON representation | +|---| +| ``` { "key": string, // value "stringValue": string, "stringListValue": { object (`https://ai.google.dev/api/generate-content#StringList`) }, "numericValue": number // Union type } ``` | + +## StringList + +A list of string values. +Fields `values[]` `string` The string values of the list. + +| JSON representation | +|---| +| ``` { "values": [ string ] } ``` | + +## Maps + +A grounding chunk from Google Maps. A Maps chunk corresponds to a single place. +Fields `uri` `string` URI reference of the place. +`title` `string` Title of the place. +`text` `string` Text description of the place answer. +`placeId` `string` The ID of the place, in `places/{placeId}` format. A user can use this ID to look up that place. +`placeAnswerSources` ``object (`https://ai.google.dev/api/generate-content#PlaceAnswerSources`)`` Sources that provide answers about the features of a given place in Google Maps. + +| JSON representation | +|---| +| ``` { "uri": string, "title": string, "text": string, "placeId": string, "placeAnswerSources": { object (`https://ai.google.dev/api/generate-content#PlaceAnswerSources`) } } ``` | + +## PlaceAnswerSources + +Collection of sources that provide answers about the features of a given place in Google Maps. Each PlaceAnswerSources message corresponds to a specific place in Google Maps. The Google Maps tool used these sources in order to answer questions about features of the place (e.g: "does Bar Foo have Wifi" or "is Foo Bar wheelchair accessible?"). Currently we only support review snippets as sources. +Fields `reviewSnippets[]` ``object (`https://ai.google.dev/api/generate-content#ReviewSnippet`)`` Snippets of reviews that are used to generate answers about the features of a given place in Google Maps. + +| JSON representation | +|---| +| ``` { "reviewSnippets": [ { object (`https://ai.google.dev/api/generate-content#ReviewSnippet`) } ] } ``` | + +## ReviewSnippet + +Encapsulates a snippet of a user review that answers a question about the features of a specific place in Google Maps. +Fields `reviewId` `string` The ID of the review snippet. +`googleMapsUri` `string` A link that corresponds to the user review on Google Maps. +`title` `string` Title of the review. + +| JSON representation | +|---| +| ``` { "reviewId": string, "googleMapsUri": string, "title": string } ``` | + +## GroundingSupport + +Grounding support. +Fields `groundingChunkIndices[]` `integer` Optional. A list of indices (into 'grounding_chunk' in `response.candidate.grounding_metadata`) specifying the citations associated with the claim. For instance \[1,3,4\] means that grounding_chunk\[1\], grounding_chunk\[3\], grounding_chunk\[4\] are the retrieved content attributed to the claim. If the response is streaming, the groundingChunkIndices refer to the indices across all responses. It is the client's responsibility to accumulate the grounding chunks from all responses (while maintaining the same order). +`confidenceScores[]` `number` Optional. Confidence score of the support references. Ranges from 0 to 1. 1 is the most confident. This list must have the same size as the groundingChunkIndices. +`renderedParts[]` `integer` Output only. Indices into the `parts` field of the candidate's content. These indices specify which rendered parts are associated with this support source. +`segment` ``object (`https://ai.google.dev/api/generate-content#Segment`)`` Segment of the content this support belongs to. + +| JSON representation | +|---| +| ``` { "groundingChunkIndices": [ integer ], "confidenceScores": [ number ], "renderedParts": [ integer ], "segment": { object (`https://ai.google.dev/api/generate-content#Segment`) } } ``` | + +## Segment + +Segment of the content. +Fields `partIndex` `integer` The index of a Part object within its parent Content object. +`startIndex` `integer` Start index in the given Part, measured in bytes. Offset from the start of the Part, inclusive, starting at zero. +`endIndex` `integer` End index in the given Part, measured in bytes. Offset from the start of the Part, exclusive, starting at zero. +`text` `string` The text corresponding to the segment from the response. + +| JSON representation | +|---| +| ``` { "partIndex": integer, "startIndex": integer, "endIndex": integer, "text": string } ``` | + +## RetrievalMetadata + +Metadata related to retrieval in the grounding flow. +Fields `googleSearchDynamicRetrievalScore` `number` Optional. Score indicating how likely information from google search could help answer the prompt. The score is in the range \[0, 1\], where 0 is the least likely and 1 is the most likely. This score is only populated when google search grounding and dynamic retrieval is enabled. It will be compared to the threshold to determine whether to trigger google search. + +| JSON representation | +|---| +| ``` { "googleSearchDynamicRetrievalScore": number } ``` | + +## LogprobsResult + +Logprobs Result +Fields `topCandidates[]` ``object (`https://ai.google.dev/api/generate-content#TopCandidates`)`` Length = total number of decoding steps. +`chosenCandidates[]` ``object (`https://ai.google.dev/api/generate-content#Candidate`)`` Length = total number of decoding steps. The chosen candidates may or may not be in topCandidates. +`logProbabilitySum` `number` Sum of log probabilities for all tokens. + +| JSON representation | +|---| +| ``` { "topCandidates": [ { object (`https://ai.google.dev/api/generate-content#TopCandidates`) } ], "chosenCandidates": [ { object (`https://ai.google.dev/api/generate-content#Candidate`) } ], "logProbabilitySum": number } ``` | + +## TopCandidates + +Candidates with top log probabilities at each decoding step. +Fields `candidates[]` ``object (`https://ai.google.dev/api/generate-content#Candidate`)`` Sorted by log probability in descending order. + +| JSON representation | +|---| +| ``` { "candidates": [ { object (`https://ai.google.dev/api/generate-content#Candidate`) } ] } ``` | + +## Candidate + +Candidate for the logprobs token and score. +Fields `token` `string` The candidate's token string value. +`tokenId` `integer` The candidate's token id value. +`logProbability` `number` The candidate's log probability. + +| JSON representation | +|---| +| ``` { "token": string, "tokenId": integer, "logProbability": number } ``` | + +## UrlContextMetadata + +Metadata related to url context retrieval tool. +Fields `urlMetadata[]` ``object (`https://ai.google.dev/api/generate-content#UrlMetadata`)`` List of url context. + +| JSON representation | +|---| +| ``` { "urlMetadata": [ { object (`https://ai.google.dev/api/generate-content#UrlMetadata`) } ] } ``` | + +## UrlMetadata + +Context of the a single url retrieval. +Fields `retrievedUrl` `string` Retrieved url by the tool. +`urlRetrievalStatus` ``enum (`https://ai.google.dev/api/generate-content#UrlRetrievalStatus`)`` Status of the url retrieval. + +| JSON representation | +|---| +| ``` { "retrievedUrl": string, "urlRetrievalStatus": enum (`https://ai.google.dev/api/generate-content#UrlRetrievalStatus`) } ``` | + +## UrlRetrievalStatus + +Status of the url retrieval. + +| Enums || +|---|---| +| `URL_RETRIEVAL_STATUS_UNSPECIFIED` | Default value. This value is unused. | +| `URL_RETRIEVAL_STATUS_SUCCESS` | Url retrieval is successful. | +| `URL_RETRIEVAL_STATUS_ERROR` | Url retrieval is failed due to error. | +| `URL_RETRIEVAL_STATUS_PAYWALL` | Url retrieval is failed because the content is behind paywall. | +| `URL_RETRIEVAL_STATUS_UNSAFE` | Url retrieval is failed because the content is unsafe. | + +## CitationMetadata + +- [JSON representation](https://ai.google.dev/api/generate-content#SCHEMA_REPRESENTATION) +- [CitationSource](https://ai.google.dev/api/generate-content#CitationSource) + - [JSON representation](https://ai.google.dev/api/generate-content#CitationSource.SCHEMA_REPRESENTATION) + +A collection of source attributions for a piece of content. +Fields `citationSources[]` ``object (`https://ai.google.dev/api/generate-content#CitationSource`)`` Citations to sources for a specific response. + +| JSON representation | +|---| +| ``` { "citationSources": [ { object (`https://ai.google.dev/api/generate-content#CitationSource`) } ] } ``` | + +## CitationSource + +A citation to a source for a portion of a specific response. +Fields `startIndex` `integer` Optional. Start of segment of the response that is attributed to this source. + +Index indicates the start of the segment, measured in bytes. +`endIndex` `integer` Optional. End of the attributed segment, exclusive. +`uri` `string` Optional. URI that is attributed as a source for a portion of the text. +`license` `string` Optional. License for the GitHub project that is attributed as a source for segment. + +License info is required for code citations. + +| JSON representation | +|---| +| ``` { "startIndex": integer, "endIndex": integer, "uri": string, "license": string } ``` | + +## GenerationConfig + +- [JSON representation](https://ai.google.dev/api/generate-content#SCHEMA_REPRESENTATION) +- [Modality](https://ai.google.dev/api/generate-content#Modality) +- [SpeechConfig](https://ai.google.dev/api/generate-content#SpeechConfig) + - [JSON representation](https://ai.google.dev/api/generate-content#SpeechConfig.SCHEMA_REPRESENTATION) +- [VoiceConfig](https://ai.google.dev/api/generate-content#VoiceConfig) + - [JSON representation](https://ai.google.dev/api/generate-content#VoiceConfig.SCHEMA_REPRESENTATION) +- [PrebuiltVoiceConfig](https://ai.google.dev/api/generate-content#PrebuiltVoiceConfig) + - [JSON representation](https://ai.google.dev/api/generate-content#PrebuiltVoiceConfig.SCHEMA_REPRESENTATION) +- [MultiSpeakerVoiceConfig](https://ai.google.dev/api/generate-content#MultiSpeakerVoiceConfig) + - [JSON representation](https://ai.google.dev/api/generate-content#MultiSpeakerVoiceConfig.SCHEMA_REPRESENTATION) +- [SpeakerVoiceConfig](https://ai.google.dev/api/generate-content#SpeakerVoiceConfig) + - [JSON representation](https://ai.google.dev/api/generate-content#SpeakerVoiceConfig.SCHEMA_REPRESENTATION) +- [ThinkingConfig](https://ai.google.dev/api/generate-content#ThinkingConfig) + - [JSON representation](https://ai.google.dev/api/generate-content#ThinkingConfig.SCHEMA_REPRESENTATION) +- [ThinkingLevel](https://ai.google.dev/api/generate-content#ThinkingLevel) +- [ImageConfig](https://ai.google.dev/api/generate-content#ImageConfig) + - [JSON representation](https://ai.google.dev/api/generate-content#ImageConfig.SCHEMA_REPRESENTATION) +- [MediaResolution](https://ai.google.dev/api/generate-content#MediaResolution) + +Configuration options for model generation and outputs. Not all parameters are configurable for every model. +Fields `stopSequences[]` `string` Optional. The set of character sequences (up to 5) that will stop output generation. If specified, the API will stop at the first appearance of a `stop_sequence`. The stop sequence will not be included as part of the response. +`responseMimeType` `string` Optional. MIME type of the generated candidate text. Supported MIME types are: `text/plain`: (default) Text output. `application/json`: JSON response in the response candidates. `text/x.enum`: ENUM as a string response in the response candidates. Refer to the [docs](https://ai.google.dev/gemini-api/docs/prompting_with_media#plain_text_formats) for a list of all supported text MIME types. +`responseSchema` ``object (`https://ai.google.dev/api/caching#Schema`)`` Optional. Output schema of the generated candidate text. Schemas must be a subset of the [OpenAPI schema](https://spec.openapis.org/oas/v3.0.3#schema) and can be objects, primitives or arrays. + +If set, a compatible `responseMimeType` must also be set. Compatible MIME types: `application/json`: Schema for JSON response. Refer to the [JSON text generation guide](https://ai.google.dev/gemini-api/docs/json-mode) for more details. +`_responseJsonSchema` ``value (`https://protobuf.dev/reference/protobuf/google.protobuf#value` format)`` Optional. Output schema of the generated response. This is an alternative to `responseSchema` that accepts [JSON Schema](https://json-schema.org/). + +If set, `responseSchema` must be omitted, but `responseMimeType` is required. + +While the full JSON Schema may be sent, not all features are supported. Specifically, only the following properties are supported: + +- `$id` +- `$defs` +- `$ref` +- `$anchor` +- `type` +- `format` +- `title` +- `description` +- `enum` (for strings and numbers) +- `items` +- `prefixItems` +- `minItems` +- `maxItems` +- `minimum` +- `maximum` +- `anyOf` +- `oneOf` (interpreted the same as `anyOf`) +- `properties` +- `additionalProperties` +- `required` + +The non-standard `propertyOrdering` property may also be set. + +Cyclic references are unrolled to a limited degree and, as such, may only be used within non-required properties. (Nullable properties are not sufficient.) If `$ref` is set on a sub-schema, no other properties, except for than those starting as a `$`, may be set. +`responseJsonSchema` ``value (`https://protobuf.dev/reference/protobuf/google.protobuf#value` format)`` Optional. An internal detail. Use `responseJsonSchema` rather than this field. +`responseModalities[]` ``enum (`https://ai.google.dev/api/generate-content#Modality`)`` Optional. The requested modalities of the response. Represents the set of modalities that the model can return, and should be expected in the response. This is an exact match to the modalities of the response. + +A model may have multiple combinations of supported modalities. If the requested modalities do not match any of the supported combinations, an error will be returned. + +An empty list is equivalent to requesting only text. +`candidateCount` `integer` Optional. Number of generated responses to return. If unset, this will default to 1. Please note that this doesn't work for previous generation models (Gemini 1.0 family) +`maxOutputTokens` `integer` Optional. The maximum number of tokens to include in a response candidate. + +Note: The default value varies by model, see the `Model.output_token_limit` attribute of the `Model` returned from the `getModel` function. +`temperature` `number` Optional. Controls the randomness of the output. + +Note: The default value varies by model, see the `Model.temperature` attribute of the `Model` returned from the `getModel` function. + +Values can range from \[0.0, 2.0\]. +`topP` `number` Optional. The maximum cumulative probability of tokens to consider when sampling. + +The model uses combined Top-k and Top-p (nucleus) sampling. + +Tokens are sorted based on their assigned probabilities so that only the most likely tokens are considered. Top-k sampling directly limits the maximum number of tokens to consider, while Nucleus sampling limits the number of tokens based on the cumulative probability. + +Note: The default value varies by `Model` and is specified by the`Model.top_p` attribute returned from the `getModel` function. An empty `topK` attribute indicates that the model doesn't apply top-k sampling and doesn't allow setting `topK` on requests. +`topK` `integer` Optional. The maximum number of tokens to consider when sampling. + +Gemini models use Top-p (nucleus) sampling or a combination of Top-k and nucleus sampling. Top-k sampling considers the set of `topK` most probable tokens. Models running with nucleus sampling don't allow topK setting. + +Note: The default value varies by `Model` and is specified by the`Model.top_p` attribute returned from the `getModel` function. An empty `topK` attribute indicates that the model doesn't apply top-k sampling and doesn't allow setting `topK` on requests. +`seed` `integer` Optional. Seed used in decoding. If not set, the request uses a randomly generated seed. +`presencePenalty` `number` Optional. Presence penalty applied to the next token's logprobs if the token has already been seen in the response. + +This penalty is binary on/off and not dependant on the number of times the token is used (after the first). Use `https://ai.google.dev/api/generate-content#FIELDS.frequency_penalty` for a penalty that increases with each use. + +A positive penalty will discourage the use of tokens that have already been used in the response, increasing the vocabulary. + +A negative penalty will encourage the use of tokens that have already been used in the response, decreasing the vocabulary. +`frequencyPenalty` `number` Optional. Frequency penalty applied to the next token's logprobs, multiplied by the number of times each token has been seen in the respponse so far. + +A positive penalty will discourage the use of tokens that have already been used, proportional to the number of times the token has been used: The more a token is used, the more difficult it is for the model to use that token again increasing the vocabulary of responses. + +Caution: A *negative* penalty will encourage the model to reuse tokens proportional to the number of times the token has been used. Small negative values will reduce the vocabulary of a response. Larger negative values will cause the model to start repeating a common token until it hits the `https://ai.google.dev/api/generate-content#FIELDS.max_output_tokens` limit. +`responseLogprobs` `boolean` Optional. If true, export the logprobs results in response. +`logprobs` `integer` Optional. Only valid if `https://ai.google.dev/api/generate-content#FIELDS.response_logprobs`. This sets the number of top logprobs, including the chosen candidate, to return at each decoding step in the `https://ai.google.dev/api/generate-content#FIELDS.logprobs_result`. The number must be in the range of \[0, 20\]. +`enableEnhancedCivicAnswers` `boolean` Optional. Enables enhanced civic answers. It may not be available for all models. +`speechConfig` ``object (`https://ai.google.dev/api/generate-content#SpeechConfig`)`` Optional. The speech generation config. +`thinkingConfig` ``object (`https://ai.google.dev/api/generate-content#ThinkingConfig`)`` Optional. Config for thinking features. An error will be returned if this field is set for models that don't support thinking. +`imageConfig` ``object (`https://ai.google.dev/api/generate-content#ImageConfig`)`` Optional. Config for image generation. An error will be returned if this field is set for models that don't support these config options. +`mediaResolution` ``enum (`https://ai.google.dev/api/generate-content#MediaResolution`)`` Optional. If specified, the media resolution specified will be used. + +| JSON representation | +|---| +| ``` { "stopSequences": [ string ], "responseMimeType": string, "responseSchema": { object (`https://ai.google.dev/api/caching#Schema`) }, "_responseJsonSchema": value, "responseJsonSchema": value, "responseModalities": [ enum (`https://ai.google.dev/api/generate-content#Modality`) ], "candidateCount": integer, "maxOutputTokens": integer, "temperature": number, "topP": number, "topK": integer, "seed": integer, "presencePenalty": number, "frequencyPenalty": number, "responseLogprobs": boolean, "logprobs": integer, "enableEnhancedCivicAnswers": boolean, "speechConfig": { object (`https://ai.google.dev/api/generate-content#SpeechConfig`) }, "thinkingConfig": { object (`https://ai.google.dev/api/generate-content#ThinkingConfig`) }, "imageConfig": { object (`https://ai.google.dev/api/generate-content#ImageConfig`) }, "mediaResolution": enum (`https://ai.google.dev/api/generate-content#MediaResolution`) } ``` | + +## Modality + +Supported modalities of the response. + +| Enums || +|---|---| +| `MODALITY_UNSPECIFIED` | Default value. | +| `TEXT` | Indicates the model should return text. | +| `IMAGE` | Indicates the model should return images. | +| `AUDIO` | Indicates the model should return audio. | + +## SpeechConfig + +Config for speech generation and transcription. +Fields `voiceConfig` ``object (`https://ai.google.dev/api/generate-content#VoiceConfig`)`` The configuration in case of single-voice output. +`multiSpeakerVoiceConfig` ``object (`https://ai.google.dev/api/generate-content#MultiSpeakerVoiceConfig`)`` Optional. The configuration for the multi-speaker setup. It is mutually exclusive with the voiceConfig field. +`languageCode` `string` Optional. The IETF [BCP-47](https://www.rfc-editor.org/rfc/bcp/bcp47.txt) language code that the user configured the app to use. Used for speech recognition and synthesis. + +Valid values are: `de-DE`, `en-AU`, `en-GB`, `en-IN`, `en-US`, `es-US`, `fr-FR`, `hi-IN`, `pt-BR`, `ar-XA`, `es-ES`, `fr-CA`, `id-ID`, `it-IT`, `ja-JP`, `tr-TR`, `vi-VN`, `bn-IN`, `gu-IN`, `kn-IN`, `ml-IN`, `mr-IN`, `ta-IN`, `te-IN`, `nl-NL`, `ko-KR`, `cmn-CN`, `pl-PL`, `ru-RU`, and `th-TH`. + +| JSON representation | +|---| +| ``` { "voiceConfig": { object (`https://ai.google.dev/api/generate-content#VoiceConfig`) }, "multiSpeakerVoiceConfig": { object (`https://ai.google.dev/api/generate-content#MultiSpeakerVoiceConfig`) }, "languageCode": string } ``` | + +## VoiceConfig + +The configuration for the voice to use. +Fields `voice_config` `Union type` The configuration for the speaker to use. `voice_config` can be only one of the following: `prebuiltVoiceConfig` ``object (`https://ai.google.dev/api/generate-content#PrebuiltVoiceConfig`)`` The configuration for the prebuilt voice to use. + +| JSON representation | +|---| +| ``` { // voice_config "prebuiltVoiceConfig": { object (`https://ai.google.dev/api/generate-content#PrebuiltVoiceConfig`) } // Union type } ``` | + +## PrebuiltVoiceConfig + +The configuration for the prebuilt speaker to use. +Fields `voiceName` `string` The name of the preset voice to use. + +| JSON representation | +|---| +| ``` { "voiceName": string } ``` | + +## MultiSpeakerVoiceConfig + +The configuration for the multi-speaker setup. +Fields `speakerVoiceConfigs[]` ``object (`https://ai.google.dev/api/generate-content#SpeakerVoiceConfig`)`` Required. All the enabled speaker voices. + +| JSON representation | +|---| +| ``` { "speakerVoiceConfigs": [ { object (`https://ai.google.dev/api/generate-content#SpeakerVoiceConfig`) } ] } ``` | + +## SpeakerVoiceConfig + +The configuration for a single speaker in a multi speaker setup. +Fields `speaker` `string` Required. The name of the speaker to use. Should be the same as in the prompt. +`voiceConfig` ``object (`https://ai.google.dev/api/generate-content#VoiceConfig`)`` Required. The configuration for the voice to use. + +| JSON representation | +|---| +| ``` { "speaker": string, "voiceConfig": { object (`https://ai.google.dev/api/generate-content#VoiceConfig`) } } ``` | + +## ThinkingConfig + +Config for thinking features. +Fields `includeThoughts` `boolean` Indicates whether to include thoughts in the response. If true, thoughts are returned only when available. +`thinkingBudget` `integer` The number of thoughts tokens that the model should generate. +`thinkingLevel` ``enum (`https://ai.google.dev/api/generate-content#ThinkingLevel`)`` Optional. Controls the maximum depth of the model's internal reasoning process before it produces a response. If not specified, the default is HIGH. Recommended for Gemini 3 or later models. Use with earlier models results in an error. + +| JSON representation | +|---| +| ``` { "includeThoughts": boolean, "thinkingBudget": integer, "thinkingLevel": enum (`https://ai.google.dev/api/generate-content#ThinkingLevel`) } ``` | + +## ThinkingLevel + +Allow user to specify how much to think using enum instead of integer budget. + +| Enums || +|---|---| +| `THINKING_LEVEL_UNSPECIFIED` | Default value. | +| `MINIMAL` | Little to no thinking. | +| `LOW` | Low thinking level. | +| `MEDIUM` | Medium thinking level. | +| `HIGH` | High thinking level. | + +## ImageConfig + +Config for image generation features. +Fields `aspectRatio` `string` Optional. The aspect ratio of the image to generate. Supported aspect ratios: `1:1`, `1:4`, `4:1`, `1:8`, `8:1`, `2:3`, `3:2`, `3:4`, `4:3`, `4:5`, `5:4`, `9:16`, `16:9`, or `21:9`. + +If not specified, the model will choose a default aspect ratio based on any reference images provided. +`imageSize` `string` Optional. Specifies the size of generated images. Supported values are `512`, `1K`, `2K`, `4K`. If not specified, the model will use default value `1K`. + +| JSON representation | +|---| +| ``` { "aspectRatio": string, "imageSize": string } ``` | + +## MediaResolution + +Media resolution for the input media. + +| Enums || +|---|---| +| `MEDIA_RESOLUTION_UNSPECIFIED` | Media resolution has not been set. | +| `MEDIA_RESOLUTION_LOW` | Media resolution set to low (64 tokens). | +| `MEDIA_RESOLUTION_MEDIUM` | Media resolution set to medium (256 tokens). | +| `MEDIA_RESOLUTION_HIGH` | Media resolution set to high (zoomed reframing with 256 tokens). | + +## HarmCategory + +The category of a rating. + +These categories cover various kinds of harms that developers may wish to adjust. + +| Enums || +|---|---| +| `HARM_CATEGORY_UNSPECIFIED` | Category is unspecified. | +| `HARM_CATEGORY_DEROGATORY` | **PaLM** - Negative or harmful comments targeting identity and/or protected attribute. | +| `HARM_CATEGORY_TOXICITY` | **PaLM** - Content that is rude, disrespectful, or profane. | +| `HARM_CATEGORY_VIOLENCE` | **PaLM** - Describes scenarios depicting violence against an individual or group, or general descriptions of gore. | +| `HARM_CATEGORY_SEXUAL` | **PaLM** - Contains references to sexual acts or other lewd content. | +| `HARM_CATEGORY_MEDICAL` | **PaLM** - Promotes unchecked medical advice. | +| `HARM_CATEGORY_DANGEROUS` | **PaLM** - Dangerous content that promotes, facilitates, or encourages harmful acts. | +| `HARM_CATEGORY_HARASSMENT` | **Gemini** - Harassment content. | +| `HARM_CATEGORY_HATE_SPEECH` | **Gemini** - Hate speech and content. | +| `HARM_CATEGORY_SEXUALLY_EXPLICIT` | **Gemini** - Sexually explicit content. | +| `HARM_CATEGORY_DANGEROUS_CONTENT` | **Gemini** - Dangerous content. | +| `HARM_CATEGORY_CIVIC_INTEGRITY` | **Gemini** - Content that may be used to harm civic integrity. DEPRECATED: use enableEnhancedCivicAnswers instead. > [!WARNING] > This item is deprecated! | + +## ModalityTokenCount + +- [JSON representation](https://ai.google.dev/api/generate-content#SCHEMA_REPRESENTATION) +- [Modality](https://ai.google.dev/api/generate-content#Modality) + +Represents token counting info for a single modality. +Fields `modality` ``enum (`https://ai.google.dev/api/generate-content#Modality`)`` The modality associated with this token count. +`tokenCount` `integer` Number of tokens. + +| JSON representation | +|---| +| ``` { "modality": enum (`https://ai.google.dev/api/generate-content#Modality`), "tokenCount": integer } ``` | + +## Modality + +Content Part modality + +| Enums || +|---|---| +| `MODALITY_UNSPECIFIED` | Unspecified modality. | +| `TEXT` | Plain text. | +| `IMAGE` | Image. | +| `VIDEO` | Video. | +| `AUDIO` | Audio. | +| `DOCUMENT` | Document, e.g. PDF. | + +## SafetyRating + +- [JSON representation](https://ai.google.dev/api/generate-content#SCHEMA_REPRESENTATION) +- [HarmProbability](https://ai.google.dev/api/generate-content#HarmProbability) + +Safety rating for a piece of content. + +The safety rating contains the category of harm and the harm probability level in that category for a piece of content. Content is classified for safety across a number of harm categories and the probability of the harm classification is included here. +Fields `category` ``enum (`https://ai.google.dev/api/generate-content#v1beta.HarmCategory`)`` Required. The category for this rating. +`probability` ``enum (`https://ai.google.dev/api/generate-content#HarmProbability`)`` Required. The probability of harm for this content. +`blocked` `boolean` Was this content blocked because of this rating? + +| JSON representation | +|---| +| ``` { "category": enum (`https://ai.google.dev/api/generate-content#v1beta.HarmCategory`), "probability": enum (`https://ai.google.dev/api/generate-content#HarmProbability`), "blocked": boolean } ``` | + +## HarmProbability + +The probability that a piece of content is harmful. + +The classification system gives the probability of the content being unsafe. This does not indicate the severity of harm for a piece of content. + +| Enums || +|---|---| +| `HARM_PROBABILITY_UNSPECIFIED` | Probability is unspecified. | +| `NEGLIGIBLE` | Content has a negligible chance of being unsafe. | +| `LOW` | Content has a low chance of being unsafe. | +| `MEDIUM` | Content has a medium chance of being unsafe. | +| `HIGH` | Content has a high chance of being unsafe. | + +## SafetySetting + +- [JSON representation](https://ai.google.dev/api/generate-content#SCHEMA_REPRESENTATION) +- [HarmBlockThreshold](https://ai.google.dev/api/generate-content#HarmBlockThreshold) + +Safety setting, affecting the safety-blocking behavior. + +Passing a safety setting for a category changes the allowed probability that content is blocked. +Fields `category` ``enum (`https://ai.google.dev/api/generate-content#v1beta.HarmCategory`)`` Required. The category for this setting. +`threshold` ``enum (`https://ai.google.dev/api/generate-content#HarmBlockThreshold`)`` Required. Controls the probability threshold at which harm is blocked. + +| JSON representation | +|---| +| ``` { "category": enum (`https://ai.google.dev/api/generate-content#v1beta.HarmCategory`), "threshold": enum (`https://ai.google.dev/api/generate-content#HarmBlockThreshold`) } ``` | + +## HarmBlockThreshold + +Block at and beyond a specified harm probability. + +| Enums || +|---|---| +| `HARM_BLOCK_THRESHOLD_UNSPECIFIED` | Threshold is unspecified. | +| `BLOCK_LOW_AND_ABOVE` | Content with NEGLIGIBLE will be allowed. | +| `BLOCK_MEDIUM_AND_ABOVE` | Content with NEGLIGIBLE and LOW will be allowed. | +| `BLOCK_ONLY_HIGH` | Content with NEGLIGIBLE, LOW, and MEDIUM will be allowed. | +| `BLOCK_NONE` | All content will be allowed. | +| `OFF` | Turn off the safety filter. | + +## ServiceTier + +Service tier of the request. + +| Enums || +|---|---| +| `unspecified` | Default service tier, which is standard. | +| `standard` | Standard service tier. | +| `flex` | Flex service tier. | +| `priority` | Priority service tier. | \ No newline at end of file diff --git a/docs/协议文档/OpenAI/OpenAI-Responses-API-完整文档.md b/docs/协议文档/OpenAI/OpenAI-Responses-API-完整文档.md new file mode 100644 index 0000000..def71d3 --- /dev/null +++ b/docs/协议文档/OpenAI/OpenAI-Responses-API-完整文档.md @@ -0,0 +1,1752 @@ +# OpenAI Responses API 完整技术文档 + +> 文档来源：OpenAI 官方 API Reference (https://platform.openai.com/docs/api-reference/responses) +> 整理日期：2026-04-19 + +--- + +## 目录 + +1. [端点概览](#1-端点概览) +2. [请求参数详解](#2-请求参数详解) +3. [响应结构详解](#3-响应结构详解) +4. [输出项类型 (Output Item Types)](#4-输出项类型-output-item-types) +5. [SSE 流式事件类型](#5-sse-流式事件类型) +6. [输入格式 (Input Formats)](#6-输入格式-input-formats) +7. [工具类型 (Tool Types)](#7-工具类型-tool-types) +8. [推理参数 (Reasoning)](#8-推理参数-reasoning) +9. [错误格式](#9-错误格式) +10. [与 Chat Completions API 的差异](#10-与-chat-completions-api-的差异) + +--- + +## 1. 端点概览 + +### 主端点 + +``` +POST https://api.openai.com/v1/responses +``` + +创建一个模型 Response，支持文本/图像/文件输入，支持函数调用、网络搜索、文件搜索、代码解释器等工具。模型选择与 API key 关联的项目绑定。 + +**认证方式**：Bearer Token（API Key） + +``` +Authorization: Bearer $OPENAI_API_KEY +``` + +### 子端点 + +| 方法 | 路径 | 功能 | +|------|------|------| +| POST | `/v1/responses` | 创建 Response | +| GET | `/v1/responses/{response_id}` | 检索已有 Response | +| DELETE | `/v1/responses/{response_id}` | 删除 Response | +| POST | `/v1/responses/{response_id}/cancel` | 取消进行中的 Response | +| POST | `/v1/responses/compact` | 压缩对话上下文 | +| GET | `/v1/responses/{response_id}/input_items` | 获取输入项列表 | +| POST | `/v1/responses/input_tokens` | 预估输入 token 数 | + +--- + +## 2. 请求参数详解 + +### 顶层参数 + +| 字段 | 类型 | 必填 | 默认值 | 说明 | +|------|------|------|--------|------| +| `model` | string | **是** | — | 模型 ID，如 `gpt-4o`、`o3`、`gpt-5.1` | +| `input` | string \| array | **是** | — | 用户输入。可以是纯文本字符串，也可以是内容项数组。详见第6节 | +| `instructions` | string | 否 | `""` | 系统级指令，相当于 Chat Completions 的 system message | +| `temperature` | float64 | 否 | `1.0` | 采样温度，0~2。越高越随机 | +| `top_p` | float64 | 否 | `1.0` | 核采样概率阈值 | +| `max_output_tokens` | uint | 否 | 模型默认 | 最大输出 token 数 | +| `tools` | array | 否 | `[]` | 可用工具列表。详见第7节 | +| `tool_choice` | string \| object | 否 | `"auto"` | 工具选择策略。详见下方 | +| `text` | object | 否 | `{format: {type: "text"}}` | 文本输出格式配置。详见下方 | +| `stream` | bool | 否 | `false` | 是否启用流式响应 | +| `stream_options` | object | 否 | — | 流式选项。详见下方 | +| `store` | bool | 否 | `true` | 是否存储响应以支持后续引用 | +| `metadata` | object | 否 | `{}` | 自定义元数据，键值对，最多 16 个键 | +| `reasoning` | object | 否 | — | 推理配置。详见第8节 | +| `truncation` | string | 否 | `"disabled"` | 上下文截断策略：`"auto"` 或 `"disabled"` | +| `previous_response_id` | string | 否 | — | 前一个 Response 的 ID，用于对话延续 | +| `conversation` | object | 否 | — | 对话状态管理对象，含 `id` 字段 | +| `include` | array | 否 | `[]` | 额外包含的数据，如 `["message.input_image.image_url"]` | +| `parallel_tool_calls` | bool | 否 | `true` | 是否允许并行工具调用 | +| `max_tool_calls` | uint | 否 | — | 最大工具调用次数 | +| `prompt` | object | 否 | — | 提示词配置（用于 prompt cache） | +| `prompt_cache_key` | string | 否 | — | 提示词缓存键 | +| `prompt_cache_retention` | string | 否 | — | 缓存保留时间 | +| `safety_identifier` | string | 否 | — | 安全标识符 | +| `service_tier` | string | 否 | `"auto"` | 服务层级：`"auto"`、`"default"`、`"flex"`、`"priority"` | +| `verbosity` | string | 否 | `"medium"` | 回复详细程度：`"low"`、`"medium"`、`"high"` | +| `logprobs` | bool | 否 | `false` | 是否返回 log probabilities | +| `top_logprobs` | int | 否 | `0` | 返回的 top log probabilities 数量（0~20） | +| `context_management` | object | 否 | — | 上下文管理配置。详见下方 | +| `user` | string | 否 | — | 终端用户标识符，用于监控和滥用检测 | +| `background` | bool | 否 | `false` | 是否以后台模式运行（异步） | + +### tool_choice 可选值 + +| 值 | 类型 | 说明 | +|----|------|------| +| `"none"` | string | 不调用任何工具 | +| `"auto"` | string | 模型自动决定是否调用工具 | +| `"required"` | string | 必须调用至少一个工具 | +| `{"type": "function", "name": "func_name"}` | object | 指定调用某个函数工具 | +| `{"type": "mcp", "server_label": "...", "name": "..."}` | object | 指定调用某个 MCP 工具 | +| `{"type": "custom", "name": "..."}` | object | 指定调用某个自定义工具 | +| `{"type": "allowed_tools", "tools": [...]}` | object | 限制允许使用的工具列表 | +| `"apply_patch"` | string | 强制使用 apply_patch 工具 | +| `"shell"` | string | 强制使用 shell 工具 | + +### text.format 对象 + +| 值类型 | 格式 | 说明 | +|--------|------|------| +| 纯文本 | `{"type": "text"}` | 默认，无格式约束 | +| JSON Schema | `{"type": "json_schema", "name": "...", "schema": {...}, "strict": true}` | 结构化输出，JSON Schema 约束 | +| JSON Object | `{"type": "json_object"}` | 旧版 JSON 输出（已过时，建议用 json_schema） | + +**json_schema 格式详细字段**： + +| 字段 | 类型 | 必填 | 说明 | +|------|------|------|------| +| `type` | string | 是 | 固定值 `"json_schema"` | +| `name` | string | 是 | Schema 名称 | +| `schema` | object | 是 | JSON Schema 定义 | +| `strict` | bool | 否 | 是否严格模式（默认 false） | +| `description` | string | 否 | Schema 描述 | + +### stream_options 对象 + +| 字段 | 类型 | 说明 | +|------|------|------| +| `include_obfuscation` | bool | 是否在流中包含混淆信息 | + +### context_management 对象 + +| 字段 | 类型 | 说明 | +|------|------|------| +| `compaction` | object | 上下文压缩配置 | +| `compaction.compact_threshold` | int | 触发压缩的 token 阈值 | + +### service_tier 说明 + +| 值 | 说明 | +|----|------| +| `"auto"` | 自动选择最佳层级（默认） | +| `"default"` | 标准层级 | +| `"flex"` | 弹性层级，延迟可能更高但成本更低 | +| `"priority"` | 优先层级，最低延迟 | + +### 请求示例 + +**基础文本请求**： + +```json +{ + "model": "gpt-4o", + "input": "Tell me a joke about programming." +} +``` + +**带工具和系统指令的请求**： + +```json +{ + "model": "gpt-4o", + "instructions": "You are a helpful assistant that provides concise answers.", + "input": "What is the weather in San Francisco?", + "tools": [ + { + "type": "function", + "name": "get_weather", + "description": "Get the current weather in a location", + "parameters": { + "type": "object", + "properties": { + "location": {"type": "string", "description": "City name"} + }, + "required": ["location"] + } + } + ], + "tool_choice": "auto", + "temperature": 0.7, + "max_output_tokens": 1024, + "store": true +} +``` + +**带结构化输出的请求**： + +```json +{ + "model": "gpt-4o", + "input": "List the top 3 programming languages.", + "text": { + "format": { + "type": "json_schema", + "name": "language_list", + "strict": true, + "schema": { + "type": "object", + "properties": { + "languages": { + "type": "array", + "items": { + "type": "object", + "properties": { + "name": {"type": "string"}, + "rank": {"type": "integer"} + }, + "required": ["name", "rank"] + } + } + }, + "required": ["languages"] + } + } + } +} +``` + +**流式请求**： + +```json +{ + "model": "gpt-4o", + "input": "Explain quantum computing in simple terms.", + "stream": true +} +``` + +**带对话历史的请求（使用 previous_response_id）**： + +```json +{ + "model": "gpt-4o", + "input": "What about Tokyo?", + "previous_response_id": "resp_abc123" +} +``` + +**带推理参数的请求**： + +```json +{ + "model": "o3", + "input": "Solve this math problem step by step: ...", + "reasoning": { + "effort": "high", + "summary": "detailed" + } +} +``` + +--- + +## 3. 响应结构详解 + +### 非流式响应对象 + +| 字段 | 类型 | 说明 | +|------|------|------| +| `id` | string | Response 唯一标识符，格式 `resp_xxxxx` | +| `object` | string | 固定值 `"response"` | +| `created_at` | int | 创建时间（Unix 时间戳，秒） | +| `completed_at` | int \| null | 完成时间（Unix 时间戳，秒） | +| `status` | string | 状态值。见下方状态枚举 | +| `error` | object \| null | 错误信息（如果有） | +| `incomplete_details` | object \| null | 未完成详情 | +| `instructions` | string \| null | 使用的系统指令 | +| `max_output_tokens` | int \| null | 最大输出 token 限制 | +| `model` | string | 实际使用的模型 ID | +| `output` | array | 输出项数组。详见第4节 | +| `parallel_tool_calls` | bool | 是否允许并行工具调用 | +| `previous_response_id` | string \| null | 前一个 Response 的 ID | +| `reasoning` | object \| null | 推理配置。详见第8节 | +| `store` | bool | 是否存储 | +| `temperature` | float64 \| null | 使用的温度值 | +| `text` | object \| null | 文本格式配置 | +| `tool_choice` | any | 工具选择策略 | +| `tools` | array | 可用工具列表 | +| `top_p` | float64 \| null | 使用的 top_p 值 | +| `truncation` | string \| null | 截断策略 | +| `usage` | object \| null | Token 使用量。见下方 | +| `user` | string \| null | 终端用户标识 | +| `metadata` | object \| null | 自定义元数据 | + +### status 枚举 + +| 值 | 说明 | +|----|------| +| `"completed"` | 响应已完成 | +| `"failed"` | 响应失败 | +| `"in_progress"` | 响应正在进行 | +| `"cancelled"` | 响应已取消 | +| `"queued"` | 响应排队中（后台模式） | +| `"incomplete"` | 响应未完成（token 耗尽等） | + +### incomplete_details 对象 + +| 字段 | 类型 | 说明 | +|------|------|------| +| `reason` | string | 未完成原因，如 `"max_output_tokens"` | + +### usage 对象 + +| 字段 | 类型 | 说明 | +|------|------|------| +| `input_tokens` | int | 输入 token 数 | +| `output_tokens` | int | 输出 token 数 | +| `total_tokens` | int | 总 token 数 | +| `input_tokens_details` | object | 输入 token 细分 | +| `output_tokens_details` | object | 输出 token 细分 | + +### input_tokens_details 对象 + +| 字段 | 类型 | 说明 | +|------|------|------| +| `cached_tokens` | int | 缓存命中的 token 数 | +| `text_tokens` | int | 文本 token 数 | +| `audio_tokens` | int | 音频 token 数 | +| `image_tokens` | int | 图像 token 数 | + +### output_tokens_details 对象 + +| 字段 | 类型 | 说明 | +|------|------|------| +| `text_tokens` | int | 文本 token 数 | +| `audio_tokens` | int | 音频 token 数 | +| `reasoning_tokens` | int | 推理 token 数 | + +### 响应示例 + +```json +{ + "id": "resp_abc123", + "object": "response", + "created_at": 1745000000, + "completed_at": 1745000001, + "status": "completed", + "error": null, + "incomplete_details": null, + "instructions": "You are a helpful assistant.", + "max_output_tokens": null, + "model": "gpt-4o-2024-08-06", + "output": [ + { + "type": "message", + "id": "msg_abc123", + "status": "completed", + "role": "assistant", + "content": [ + { + "type": "output_text", + "text": "Why do programmers prefer dark mode? Because light attracts bugs!", + "annotations": [] + } + ] + } + ], + "parallel_tool_calls": true, + "previous_response_id": null, + "reasoning": null, + "store": true, + "temperature": 1.0, + "text": {"format": {"type": "text"}}, + "tool_choice": "auto", + "tools": [], + "top_p": 1.0, + "truncation": "disabled", + "usage": { + "input_tokens": 25, + "output_tokens": 15, + "total_tokens": 40, + "input_tokens_details": { + "cached_tokens": 0, + "text_tokens": 25 + }, + "output_tokens_details": { + "text_tokens": 15, + "reasoning_tokens": 0 + } + }, + "user": null, + "metadata": {} +} +``` + +--- + +## 4. 输出项类型 (Output Item Types) + +`output` 数组中的每个项都有一个 `type` 字段标识其类型。 + +### 4.1 message（文本消息） + +模型生成的文本消息。 + +| 字段 | 类型 | 说明 | +|------|------|------| +| `type` | string | `"message"` | +| `id` | string | 消息 ID，格式 `msg_xxxxx` | +| `status` | string | `"completed"` 或 `"in_progress"` | +| `role` | string | `"assistant"` | +| `content` | array | 内容块数组，每项为 `output_text` 或 `refusal` | + +**content 中的 output_text 项**： + +| 字段 | 类型 | 说明 | +|------|------|------| +| `type` | string | `"output_text"` | +| `text` | string | 文本内容 | +| `annotations` | array | 注解列表（如 URL 引用、文件引用） | + +**annotations 类型**： + +| type | 字段 | 说明 | +|------|------|------| +| `url_citation` | `url`, `title`, `start_index`, `end_index` | URL 引用（来自 web_search） | +| `file_citation` | `file_id`, `filename`, `start_index`, `end_index` | 文件引用（来自 file_search） | +| `container_file_citation` | `file_id`, `start_index`, `end_index` | 容器文件引用（来自 code_interpreter） | + +### 4.2 function_call（函数调用） + +模型决定调用一个函数。 + +| 字段 | 类型 | 说明 | +|------|------|------| +| `type` | string | `"function_call"` | +| `id` | string | 调用项 ID | +| `call_id` | string | 函数调用 ID（用于匹配 function_call_output） | +| `name` | string | 函数名称 | +| `arguments` | string | 函数参数（JSON 字符串） | +| `status` | string | `"completed"` 或 `"in_progress"` | + +### 4.3 function_call_output（函数调用结果） + +函数调用的返回结果（通常出现在后续请求的 input 中）。 + +| 字段 | 类型 | 说明 | +|------|------|------| +| `type` | string | `"function_call_output"` | +| `call_id` | string | 对应的 function_call 的 call_id | +| `output` | string | 函数返回的结果 | + +### 4.4 web_search_call（网络搜索调用） + +| 字段 | 类型 | 说明 | +|------|------|------| +| `type` | string | `"web_search_call"` | +| `id` | string | 搜索调用 ID | +| `status` | string | `"completed"` / `"in_progress"` / `"failed"` | +| `action` | string | 搜索动作类型：`"search"` / `"open_page"` / `"find_in_page"` | +| `query` | string | 搜索查询 | + +**web_search action 类型**： + +| action | 说明 | 关联字段 | +|--------|------|----------| +| `search` | 执行搜索 | `query`（搜索词） | +| `open_page` | 打开搜索结果页面 | — | +| `find_in_page` | 在页面中查找内容 | — | + +### 4.5 file_search_call（文件搜索调用） + +| 字段 | 类型 | 说明 | +|------|------|------| +| `type` | string | `"file_search_call"` | +| `id` | string | 搜索调用 ID | +| `status` | string | `"completed"` / `"in_progress"` / `"failed"` | +| `queries` | array | 搜索查询列表 | +| `results` | array | 搜索结果列表（仅当 include 中请求时返回） | + +### 4.6 code_interpreter_call（代码解释器调用） + +| 字段 | 类型 | 说明 | +|------|------|------| +| `type` | string | `"code_interpreter_call"` | +| `id` | string | 调用 ID | +| `status` | string | `"completed"` / `"in_progress"` / `"failed"` | +| `code` | string | 执行的代码 | +| `container_id` | string | 容器 ID | +| `outputs` | array | 输出列表（logs 和 image 类型） | + +**code_interpreter output 类型**： + +| type | 字段 | 说明 | +|------|------|------| +| `logs` | `logs`（string） | 代码执行的标准输出日志 | +| `image` | `file_id`（string） | 生成的图像文件 ID | + +### 4.7 reasoning（推理过程） + +推理模型的思考过程输出。 + +| 子类型 | 说明 | +|--------|------| +| `reasoning.summary_text` | 推理摘要文本 | +| `reasoning.reasoning_text` | 完整推理文本 | + +### 4.8 compaction（上下文压缩） + +当对话过长触发自动压缩时出现。 + +| 字段 | 类型 | 说明 | +|------|------|------| +| `type` | string | `"compaction"` | +| `id` | string | 压缩操作 ID | +| `encrypted_content` | string | 加密后的压缩内容 | + +### 4.9 computer_call（计算机操作调用） + +用于 computer_use_preview 工具。 + +| 字段 | 类型 | 说明 | +|------|------|------| +| `type` | string | `"computer_call"` | +| `id` | string | 调用 ID | +| `action` | object | 要执行的操作 | +| `pending_safety_checks` | array | 待确认的安全检查列表 | + +**computer_call action 类型**： + +| action | 字段 | 说明 | +|--------|------|------| +| `click` | `x`, `y`, `button` | 鼠标点击 | +| `double_click` | `x`, `y` | 双击 | +| `drag` | `path`（数组 of `{x,y}`） | 拖拽路径 | +| `keypress` | `keys`（数组） | 按键 | +| `move` | `x`, `y` | 移动鼠标 | +| `screenshot` | — | 截屏 | +| `scroll` | `x`, `y`, `scroll_x`, `scroll_y` | 滚动 | +| `type` | `text` | 输入文字 | +| `wait` | — | 等待 | + +### 4.10 image_generation_call（图像生成调用） + +| 字段 | 类型 | 说明 | +|------|------|------| +| `type` | string | `"image_generation_call"` | +| `id` | string | 调用 ID | +| `status` | string | `"completed"` / `"in_progress"` / `"failed"` | +| `result` | string | 生成的图像（base64 或 URL） | +| `revised_prompt` | string | 修正后的提示词 | + +### 4.11 mcp_call（MCP 工具调用） + +| 字段 | 类型 | 说明 | +|------|------|------| +| `type` | string | `"mcp_call"` | +| `id` | string | 调用 ID | +| `server_label` | string | MCP 服务器标签 | +| `name` | string | 工具名称 | +| `arguments` | object | 工具参数 | +| `error` | string \| null | 错误信息 | +| `output` | string \| null | 工具输出 | + +### 4.12 mcp_list_tools（MCP 工具列表） + +| 字段 | 类型 | 说明 | +|------|------|------| +| `type` | string | `"mcp_list_tools"` | +| `id` | string | 列表 ID | +| `server_label` | string | MCP 服务器标签 | +| `tools` | array | 可用工具列表 | +| `error` | string \| null | 错误信息 | + +### 4.13 mcp_approval_request / mcp_approval_response + +MCP 工具执行前需要用户批准的请求和响应。 + +### 4.14 local_shell_call / local_shell_call_output + +本地 Shell 命令执行（Codex CLI 等场景）。 + +| 字段 | 类型 | 说明 | +|------|------|------| +| `type` | string | `"local_shell_call"` | +| `id` | string | 调用 ID | +| `action` | array | 命令动作列表 | + +**local_shell_call_output**： + +| 字段 | 类型 | 说明 | +|------|------|------| +| `type` | string | `"local_shell_call_output"` | +| `call_id` | string | 对应的 call ID | +| `outcome` | string | `"exit"` / `"timeout"` | +| `exit_code` | int | 退出码（outcome=exit 时） | +| `output` | string | 命令输出 | + +### 4.15 shell_call / shell_call_output + +远程 Shell 命令执行。 + +| 字段 | 类型 | 说明 | +|------|------|------| +| `type` | string | `"shell_call"` | +| `id` | string | 调用 ID | +| `action` | object | `{type: "exec", commands: [...], timeout: int, max_output_length: int}` | + +### 4.16 apply_patch_call / apply_patch_call_output + +文件补丁操作。 + +| 字段 | 类型 | 说明 | +|------|------|------| +| `type` | string | `"apply_patch_call"` | +| `id` | string | 调用 ID | +| `action` | array | 操作列表 | + +**apply_patch_call 的 action 类型**： + +| type | 字段 | 说明 | +|------|------|------| +| `create_file` | `path`, `content` | 创建文件 | +| `delete_file` | `path` | 删除文件 | +| `update_file` | `path`, `diff` | 更新文件 | + +### 4.17 custom_tool_call / custom_tool_call_output + +自定义工具调用和输出。 + +### 4.18 item_reference + +对已有项的引用。 + +--- + +## 5. SSE 流式事件类型 + +启用 `stream: true` 后，响应以 SSE（Server-Sent Events）格式返回。每个事件包含 `event:` 和 `data:` 两个字段。 + +### 事件格式 + +``` +event: {event_type} +data: {json_payload} +``` + +### 事件生命周期总览 + +``` +response.created + → response.in_progress + → response.output_item.added (每个输出项开始) + → response.content_part.added (每个内容块开始) + → response.output_text.delta (文本增量，多次) + → response.output_text.done (文本完成) + → response.content_part.done (内容块完成) + → response.output_item.done (输出项完成) + → response.function_call_arguments.delta (函数参数增量，多次) + → response.function_call_arguments.done (函数参数完成) + → response.reasoning_summary_text.delta (推理摘要增量，多次) + → response.completed (整体完成) + → response.error (错误) + → response.failed (失败) +``` + +### 5.1 response.created + +Response 对象创建时触发。包含完整的 Response 对象（status 为 `in_progress`）。 + +```json +{ + "type": "response.created", + "response": { + "id": "resp_abc123", + "object": "response", + "created_at": 1745000000, + "status": "in_progress", + "output": [], + ... + } +} +``` + +### 5.2 response.in_progress + +Response 开始处理时触发。与 `response.created` 类似但稍后触发。 + +```json +{ + "type": "response.in_progress", + "response": { ... } +} +``` + +### 5.3 response.output_item.added + +新的输出项（message 或 function_call）开始生成时触发。 + +```json +{ + "type": "response.output_item.added", + "output_index": 0, + "item": { + "type": "message", + "id": "msg_abc123", + "status": "in_progress", + "role": "assistant", + "content": [] + } +} +``` + +### 5.4 response.content_part.added + +新的内容块（如 output_text）开始生成时触发。 + +```json +{ + "type": "response.content_part.added", + "item_id": "msg_abc123", + "output_index": 0, + "content_index": 0, + "part": { + "type": "output_text", + "text": "", + "annotations": [] + } +} +``` + +### 5.5 response.output_text.delta + +文本增量输出，是流式响应中最频繁的事件。 + +```json +{ + "type": "response.output_text.delta", + "item_id": "msg_abc123", + "output_index": 0, + "content_index": 0, + "delta": "Hello" +} +``` + +### 5.6 response.output_text.done + +文本输出完成时触发，包含完整文本。 + +```json +{ + "type": "response.output_text.done", + "item_id": "msg_abc123", + "output_index": 0, + "content_index": 0, + "text": "Hello, world!" +} +``` + +### 5.7 response.content_part.done + +内容块完成时触发，包含最终内容。 + +```json +{ + "type": "response.content_part.done", + "item_id": "msg_abc123", + "output_index": 0, + "content_index": 0, + "part": { + "type": "output_text", + "text": "Hello, world!", + "annotations": [] + } +} +``` + +### 5.8 response.output_item.done + +输出项完成时触发，包含完整的输出项数据。 + +```json +{ + "type": "response.output_item.done", + "output_index": 0, + "item": { + "type": "message", + "id": "msg_abc123", + "status": "completed", + "role": "assistant", + "content": [ + { + "type": "output_text", + "text": "Hello, world!", + "annotations": [] + } + ] + } +} +``` + +### 5.9 response.function_call_arguments.delta + +函数调用参数的增量输出。 + +```json +{ + "type": "response.function_call_arguments.delta", + "item_id": "fc_abc123", + "output_index": 1, + "delta": "{\"location\":" +} +``` + +### 5.10 response.function_call_arguments.done + +函数调用参数完整输出完成。 + +```json +{ + "type": "response.function_call_arguments.done", + "item_id": "fc_abc123", + "output_index": 1, + "arguments": "{\"location\":\"San Francisco\"}" +} +``` + +### 5.11 response.reasoning_summary_text.delta + +推理模型的摘要文本增量输出。 + +```json +{ + "type": "response.reasoning_summary_text.delta", + "item_id": "msg_abc123", + "output_index": 0, + "summary_index": 0, + "delta": "The problem requires..." +} +``` + +### 5.12 response.completed + +整个 Response 完成时触发，包含完整的 Response 对象（含最终 usage 和 output）。 + +```json +{ + "type": "response.completed", + "response": { + "id": "resp_abc123", + "object": "response", + "created_at": 1745000000, + "completed_at": 1745000001, + "status": "completed", + "output": [ ... ], + "usage": { + "input_tokens": 25, + "output_tokens": 50, + "total_tokens": 75 + }, + ... + } +} +``` + +### 5.13 response.error + +响应过程中发生错误。 + +```json +{ + "type": "response.error", + "response": { ... } +} +``` + +### 5.14 response.failed + +响应失败时触发。 + +```json +{ + "type": "response.failed", + "response": { ... } +} +``` + +### 完整流式交互示例 + +``` +event: response.created +data: {"type":"response.created","response":{"id":"resp_abc","status":"in_progress","output":[]}} + +event: response.output_item.added +data: {"type":"response.output_item.added","output_index":0,"item":{"type":"message","id":"msg_abc","status":"in_progress","role":"assistant","content":[]}} + +event: response.content_part.added +data: {"type":"response.content_part.added","item_id":"msg_abc","output_index":0,"content_index":0,"part":{"type":"output_text","text":"","annotations":[]}} + +event: response.output_text.delta +data: {"type":"response.output_text.delta","item_id":"msg_abc","output_index":0,"content_index":0,"delta":"Hello"} + +event: response.output_text.delta +data: {"type":"response.output_text.delta","item_id":"msg_abc","output_index":0,"content_index":0,"delta":" world"} + +event: response.output_text.done +data: {"type":"response.output_text.done","item_id":"msg_abc","output_index":0,"content_index":0,"text":"Hello world"} + +event: response.content_part.done +data: {"type":"response.content_part.done","item_id":"msg_abc","output_index":0,"content_index":0,"part":{"type":"output_text","text":"Hello world","annotations":[]}} + +event: response.output_item.done +data: {"type":"response.output_item.done","output_index":0,"item":{"type":"message","id":"msg_abc","status":"completed","role":"assistant","content":[{"type":"output_text","text":"Hello world","annotations":[]}]}} + +event: response.completed +data: {"type":"response.completed","response":{"id":"resp_abc","status":"completed","output":[...],"usage":{"input_tokens":10,"output_tokens":5,"total_tokens":15}}} +``` + +--- + +## 6. 输入格式 (Input Formats) + +`input` 字段支持三种格式：纯文本字符串、内容项数组、消息对象数组。 + +### 6.1 纯文本字符串 + +最简单的输入方式，等同于单条 user 消息。 + +```json +{ + "model": "gpt-4o", + "input": "What is the capital of France?" +} +``` + +### 6.2 内容项数组 + +适合多模态输入，每项指定类型。 + +#### input_text（文本输入） + +```json +{ + "type": "input_text", + "text": "Describe this image." +} +``` + +#### input_image（图像输入） + +| 字段 | 类型 | 说明 | +|------|------|------| +| `type` | string | `"input_image"` | +| `image_url` | string | 图像 URL（与 `file_id` 二选一） | +| `file_id` | string | 已上传文件的 ID | +| `detail` | string | 图像细节级别：`"auto"` / `"low"` / `"high"` | + +```json +{ + "type": "input_image", + "image_url": "https://example.com/photo.jpg", + "detail": "high" +} +``` + +#### input_file（文件输入） + +| 字段 | 类型 | 说明 | +|------|------|------| +| `type` | string | `"input_file"` | +| `file_data` | string | Base64 编码的文件内容 | +| `file_id` | string | 已上传文件的 ID | +| `filename` | string | 文件名（带扩展名，用于类型推断） | +| `url` | string | 文件 URL | + +```json +{ + "type": "input_file", + "file_data": "data:text/csv;base64,...", + "filename": "data.csv" +} +``` + +### 6.3 消息对象数组 + +类似 Chat Completions 的 messages 格式，用于传递多轮对话。 + +每条消息是一个带有 `type: "message"` 的对象： + +```json +{ + "type": "message", + "role": "user", + "content": "Hello!" +} +``` + +#### 支持的 role + +| role | 说明 | +|------|------| +| `"user"` | 用户消息 | +| `"assistant"` | 助手消息 | +| `"system"` | 系统消息（建议使用顶层 `instructions` 代替） | +| `"developer"` | 开发者消息 | + +#### content 格式 + +content 可以是字符串或内容项数组： + +**字符串**： +```json +{ + "type": "message", + "role": "user", + "content": "Hello!" +} +``` + +**内容项数组**： +```json +{ + "type": "message", + "role": "user", + "content": [ + {"type": "input_text", "text": "What is in this image?"}, + {"type": "input_image", "image_url": "https://example.com/photo.jpg"} + ] +} +``` + +**助手消息使用 output_text**： +```json +{ + "type": "message", + "role": "assistant", + "content": [ + {"type": "output_text", "text": "The image shows a sunset."} + ] +} +``` + +### 6.4 function_call_output（函数结果输入） + +用于将函数调用结果传回模型： + +```json +{ + "type": "function_call_output", + "call_id": "call_abc123", + "output": "{\"temperature\": 72, \"condition\": \"sunny\"}" +} +``` + +### 6.5 完整多轮对话示例 + +```json +{ + "model": "gpt-4o", + "input": [ + {"type": "message", "role": "user", "content": "What's the weather in SF?"}, + { + "type": "message", + "role": "assistant", + "content": [], + }, + { + "type": "function_call", + "call_id": "call_001", + "name": "get_weather", + "arguments": "{\"location\":\"San Francisco\"}" + }, + { + "type": "function_call_output", + "call_id": "call_001", + "output": "{\"temperature\": 72, \"condition\": \"sunny\"}" + } + ], + "tools": [ + { + "type": "function", + "name": "get_weather", + "description": "Get weather for a location", + "parameters": { + "type": "object", + "properties": { + "location": {"type": "string"} + }, + "required": ["location"] + } + } + ] +} +``` + +--- + +## 7. 工具类型 (Tool Types) + +`tools` 数组中的每个工具都有一个 `type` 字段。 + +### 7.1 function（函数工具） + +最常用的工具类型，定义可供模型调用的函数。 + +```json +{ + "type": "function", + "name": "get_weather", + "description": "Get current weather for a location", + "parameters": { + "type": "object", + "properties": { + "location": {"type": "string", "description": "City name"}, + "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]} + }, + "required": ["location"] + }, + "strict": false +} +``` + +| 字段 | 类型 | 必填 | 说明 | +|------|------|------|------| +| `type` | string | 是 | `"function"` | +| `name` | string | 是 | 函数名称 | +| `description` | string | 否 | 函数描述 | +| `parameters` | object | 否 | JSON Schema 格式的参数定义 | +| `strict` | bool | 否 | 是否严格遵循 schema（默认 false） | + +### 7.2 web_search（网络搜索） + +允许模型搜索互联网。 + +```json +{ + "type": "web_search", + "user_location": { + "type": "approximate", + "city": "San Francisco", + "country": "US" + }, + "search_context_size": "medium", + "allowed_domains": ["wikipedia.org", "github.com"] +} +``` + +| 字段 | 类型 | 说明 | +|------|------|------| +| `type` | string | `"web_search"` 或 `"web_search_2025_08_26"` | +| `user_location` | object | 用户位置（影响搜索结果本地化） | +| `search_context_size` | string | 搜索上下文大小：`"low"` / `"medium"` / `"high"` | +| `allowed_domains` | array | 限制搜索的域名列表 | + +**user_location 对象**： + +| 字段 | 类型 | 说明 | +|------|------|------| +| `type` | string | `"approximate"` | +| `city` | string | 城市 | +| `country` | string | 国家代码 | +| `region` | string | 地区 | +| `timezone` | string | 时区 | + +### 7.3 file_search（文件搜索） + +在向量存储中搜索相关文件。 + +```json +{ + "type": "file_search", + "vector_store_ids": ["vs_abc123"], + "filters": { + "type": "eq", + "key": "category", + "value": "report" + }, + "ranking": { + "ranker": "auto", + "score_threshold": 0.5 + }, + "max_num_results": 10 +} +``` + +| 字段 | 类型 | 说明 | +|------|------|------| +| `type` | string | `"file_search"` | +| `vector_store_ids` | array | 向量存储 ID 列表 | +| `filters` | object | 搜索过滤器 | +| `ranking` | object | 排序配置 | +| `max_num_results` | int | 最大返回结果数 | + +**过滤器类型**： + +**ComparisonFilter（比较过滤器）**： + +| 字段 | 类型 | 说明 | +|------|------|------| +| `type` | string | 比较操作符：`"eq"` / `"ne"` / `"gt"` / `"gte"` / `"lt"` / `"lte"` | +| `key` | string | 过滤字段名 | +| `value` | any | 过滤值 | + +**CompoundFilter（复合过滤器）**： + +| 字段 | 类型 | 说明 | +|------|------|------| +| `type` | string | `"and"` / `"or"` | +| `filters` | array | 子过滤器数组 | + +**ranking 对象**： + +| 字段 | 类型 | 说明 | +|------|------|------| +| `ranker` | string | 排序算法：`"auto"` / `"rrf"` | +| `score_threshold` | float | 最低相关性分数阈值（0~1） | +| `rrf` | object | Reciprocal Rank Fusion 权重 | + +**rrf 权重对象**： + +| 字段 | 类型 | 说明 | +|------|------|------| +| `embedding_weight` | float | 向量嵌入权重 | +| `text_weight` | float | 文本权重 | + +### 7.4 code_interpreter（代码解释器） + +允许模型执行代码。 + +```json +{ + "type": "code_interpreter", + "container": { + "type": "auto", + "file_ids": ["file_abc123"] + } +} +``` + +| 字段 | 类型 | 说明 | +|------|------|------| +| `type` | string | `"code_interpreter"` | +| `container` | object | 容器配置 | + +**container 对象**： + +| 字段 | 类型 | 说明 | +|------|------|------| +| `type` | string | `"auto"` | +| `file_ids` | array | 预加载到容器中的文件 ID | + +### 7.5 image_generation（图像生成） + +允许模型生成图像。 + +```json +{ + "type": "image_generation", + "quality": "high", + "size": "1024x1024", + "output_format": "png", + "partial_images": 2, + "input_image_mask": { + "image_url": "data:image/png;base64,...", + "mask": {"type": "auto"} + }, + "moderation": "auto" +} +``` + +| 字段 | 类型 | 说明 | +|------|------|------| +| `type` | string | `"image_generation"` | +| `quality` | string | 图像质量：`"low"` / `"medium"` / `"high"` | +| `size` | string | 尺寸：`"1024x1024"` / `"1536x1024"` / `"1024x1536"` | +| `output_format` | string | 输出格式：`"png"` / `"jpg"` / `"webp"` | +| `partial_images` | int | 流式返回中间图像数量（0~3） | +| `input_image_mask` | object | 图像编辑遮罩 | +| `moderation` | string | 内容审核级别：`"auto"` / `"low"` | + +### 7.6 computer_use_preview（计算机使用） + +允许模型操控虚拟计算机环境。 + +```json +{ + "type": "computer_use_preview", + "display_width": 1024, + "display_height": 768, + "environment": "browser" +} +``` + +| 字段 | 类型 | 说明 | +|------|------|------| +| `type` | string | `"computer_use_preview"` | +| `display_width` | int | 显示宽度（像素） | +| `display_height` | int | 显示高度（像素） | +| `environment` | string | 环境类型：`"browser"` / `"mac"` / `"windows"` / `"ubuntu"` | + +### 7.7 mcp（Model Context Protocol） + +连接外部 MCP 服务器获取工具。 + +```json +{ + "type": "mcp", + "server_label": "github", + "server_url": "https://mcp.github.com/sse", + "allowed_tools": ["create_issue", "search_code"], + "require_approval": "never", + "oauth_token": "...", + "headers": { + "Authorization": "Bearer ..." + } +} +``` + +| 字段 | 类型 | 说明 | +|------|------|------| +| `type` | string | `"mcp"` | +| `server_label` | string | 服务器标签（唯一标识） | +| `server_url` | string | MCP 服务器 URL（与 `connector_id` 二选一） | +| `connector_id` | string | 预构建的连接器 ID | +| `allowed_tools` | array | 允许使用的工具列表 | +| `require_approval` | string | 批准要求：`"never"` / `"always"` | +| `oauth_token` | string | OAuth 令牌 | +| `headers` | object | 自定义请求头 | + +**预构建 MCP 连接器 ID**： + +| connector_id | 服务 | +|--------------|------| +| `connector_dropbox` | Dropbox | +| `connector_gmail` | Gmail | +| `connector_googlecalendar` | Google Calendar | +| `connector_googledrive` | Google Drive | +| `connector_microsoftteams` | Microsoft Teams | +| `connector_outlookcalendar` | Outlook Calendar | +| `connector_outlookemail` | Outlook Email | +| `connector_sharepoint` | SharePoint | + +### 7.8 local_shell（本地 Shell） + +允许模型执行本地 shell 命令（Codex CLI 等）。 + +```json +{ + "type": "local_shell" +} +``` + +### 7.9 shell（远程 Shell） + +允许模型执行远程 shell 命令。 + +```json +{ + "type": "shell", + "commands": ["ls -la", "cat README.md"], + "timeout": 30, + "max_output_length": 10000 +} +``` + +### 7.10 custom（自定义工具） + +自定义工具，支持文本或结构化输入。 + +```json +{ + "type": "custom", + "name": "my_tool", + "input_format": { + "type": "text" + } +} +``` + +**input_format 类型**： + +| 格式 | 说明 | +|------|------| +| `{"type": "text"}` | 文本输入 | +| `{"type": "grammar", "lark": "..."}` | Lark 语法定义的结构化输入 | +| `{"type": "grammar", "regex": "..."}` | 正则表达式约束的结构化输入 | + +### 7.11 apply_patch + +文件补丁操作工具。 + +```json +{ + "type": "apply_patch" +} +``` + +--- + +## 8. 推理参数 (Reasoning) + +### reasoning 对象 + +| 字段 | 类型 | 必填 | 默认值 | 说明 | +|------|------|------|--------|------| +| `effort` | string | 否 | 模型相关 | 推理努力程度。见下方枚举 | +| `summary` | string | 否 | 模型相关 | 推理摘要模式。见下方枚举 | + +### effort 枚举 + +| 值 | 说明 | 备注 | +|----|------|------| +| `"none"` | 不进行推理 | gpt-5.1 默认值 | +| `"minimal"` | 最小推理 | — | +| `"low"` | 低推理 | budget_tokens <= 2048 时映射 | +| `"medium"` | 中等推理 | budget_tokens <= 16384 时映射，大多数推理模型默认 | +| `"high"` | 高推理 | gpt-5-pro 仅支持此级别 | +| `"xhigh"` | 超高推理 | gpt-5.1-codex-max 及之后模型支持 | + +**各模型的默认 effort**： + +| 模型 | 默认 effort | 支持的 effort | +|------|------------|---------------| +| gpt-5.1 | `none` | none, minimal, low, medium, high | +| gpt-5-pro | `high` | 仅 high | +| o3 | `medium` | low, medium, high | +| o4-mini | `medium` | low, medium, high | +| gpt-4o | 不支持推理 | — | + +### summary 枚举 + +| 值 | 说明 | +|----|------| +| `"auto"` | 自动决定是否生成摘要 | +| `"concise"` | 生成简洁摘要 | +| `"detailed"` | 生成详细摘要 | + +> **注意**：`generate_summary` 已废弃，使用 `summary` 代替。 + +### 请求示例 + +```json +{ + "model": "o3", + "input": "Solve: What is 15! / (12! * 3!)?", + "reasoning": { + "effort": "high", + "summary": "detailed" + } +} +``` + +### 响应中的 reasoning 字段 + +```json +{ + "reasoning": { + "effort": "high", + "summary": "detailed" + } +} +``` + +### 与 Claude thinking 的映射关系 + +在协议转换层（team-api converter.go），Claude 的 `thinking.budget_tokens` 映射为 Responses 的 `reasoning.effort`： + +| budget_tokens | reasoning.effort | +|---------------|-----------------| +| <= 2048 | `"low"` | +| <= 16384 | `"medium"` | +| > 16384 | `"high"` | + +--- + +## 9. 错误格式 + +### 响应级错误 + +错误包含在 Response 对象的 `error` 字段中： + +```json +{ + "id": "resp_abc123", + "object": "response", + "status": "failed", + "error": { + "code": "rate_limit_exceeded", + "message": "You have exceeded your rate limit." + } +} +``` + +### HTTP 错误响应 + +非成功 HTTP 状态码返回的 JSON 错误体： + +```json +{ + "error": { + "message": "Incorrect API key provided: sk_****1234.", + "type": "invalid_request_error", + "param": null, + "code": "invalid_api_key" + } +} +``` + +### 常见错误类型 + +| HTTP 状态码 | error.type | 说明 | +|------------|-----------|------| +| 400 | `invalid_request_error` | 请求参数错误 | +| 401 | `authentication_error` | API Key 无效 | +| 402 | `insufficient_quota` | 额度不足 | +| 403 | `permission_error` | 权限不足 | +| 404 | `not_found_error` | 资源不存在 | +| 429 | `rate_limit_error` | 请求频率超限 | +| 500 | `internal_error` | 服务器内部错误 | +| 503 | `server_error` | 服务不可用 | + +### 流式错误 + +流式模式下通过 SSE 事件传递错误： + +``` +event: response.error +data: {"type":"response.error","response":{"status":"failed","error":{"code":"...","message":"..."}}} + +event: response.failed +data: {"type":"response.failed","response":{"status":"failed","error":{"code":"...","message":"..."}}} +``` + +--- + +## 10. 与 Chat Completions API 的差异 + +### 10.1 端点差异 + +| 项目 | Chat Completions | Responses API | +|------|-----------------|---------------| +| 端点 | `POST /v1/chat/completions` | `POST /v1/responses` | +| 设计理念 | 单轮补全 | 对话状态管理 | +| 对话历史 | 客户端维护 messages 数组 | 服务端存储，通过 `previous_response_id` 引用 | + +### 10.2 请求参数差异 + +| Chat Completions | Responses API | 说明 | +|-----------------|---------------|------| +| `messages` | `input` + `instructions` | messages 拆分为 input（用户输入）和 instructions（系统指令） | +| `system` message | `instructions` | 系统提示移到顶层字段 | +| `developer` message | `instructions` | 开发者提示也移到顶层字段 | +| `max_tokens` | `max_output_tokens` | 字段名变化 | +| `max_completion_tokens` | `max_output_tokens` | 统一为一个字段 | +| `response_format` | `text.format` | 嵌套层级变化 | +| `reasoning_effort` | `reasoning.effort` | 从平级字段变为嵌套对象 | +| — | `reasoning.summary` | 新增推理摘要控制 | +| — | `previous_response_id` | 新增对话状态引用 | +| — | `conversation` | 新增对话 ID 管理 | +| — | `store` | 新增存储控制 | +| — | `background` | 新增后台执行模式 | +| — | `include` | 新增额外数据包含控制 | +| — | `service_tier` | 新增服务层级选择 | +| — | `verbosity` | 新增回复详细程度控制 | +| — | `context_management` | 新增上下文压缩管理 | +| — | `max_tool_calls` | 新增工具调用次数限制 | +| — | `prompt` / `prompt_cache_key` | 新增提示词缓存机制 | +| `logprobs` | `logprobs` | 相同 | +| `top_logprobs` | `top_logprobs` | 相同 | +| `tools[].function` | `tools[]` | Responses 的 function 工具扁平化（无嵌套 function 对象） | +| `tool_choice` | `tool_choice` | Responses 支持更多类型（mcp、custom、apply_patch 等） | + +### 10.3 响应结构差异 + +| Chat Completions | Responses API | 说明 | +|-----------------|---------------|------| +| `id: "chatcmpl-..."` | `id: "resp_..."` | ID 前缀不同 | +| `object: "chat.completion"` | `object: "response"` | 对象类型不同 | +| `choices[].message.content` | `output[].content[].text` | 从 choices 数组变为 output 数组，content 拆分为内容块 | +| `choices[].message.tool_calls` | `output[]`（独立 function_call 项） | 工具调用从消息内嵌变为独立输出项 | +| `choices[].finish_reason` | `status` | finish_reason 变为顶层 status | +| `usage.prompt_tokens` | `usage.input_tokens` | 字段名变化 | +| `usage.completion_tokens` | `usage.output_tokens` | 字段名变化 | +| — | `output[].type` | 新增输出项类型标识 | +| — | `output[].id` | 新增输出项独立 ID | +| — | `completed_at` | 新增完成时间戳 | + +### 10.4 工具调用差异 + +**Chat Completions 工具调用流程**： + +``` +1. 模型返回 tool_calls 数组（在 message 内） +2. 客户端执行函数 +3. 客户端将结果作为 role="tool" 的 message 发回 +``` + +**Responses API 工具调用流程**： + +``` +1. 模型返回独立的 function_call 输出项（在 output 数组中） +2. 客户端执行函数 +3. 客户端将结果作为 function_call_output 输入项发回 +4. 通过 call_id 匹配调用和结果 +``` + +### 10.5 流式响应差异 + +| Chat Completions | Responses API | 说明 | +|-----------------|---------------|------| +| `data: {"choices":[{"delta":...}]}` | `event: response.output_text.delta\ndata: {"delta":...}` | 格式完全不同 | +| 无事件类型 | 有明确事件类型 | Responses 每个事件有 `event:` 行 | +| `data: [DONE]` | `event: response.completed` | 结束信号不同 | +| 单一流 | 结构化多级事件流 | Responses 有 created/added/delta/done/completed 多级 | +| delta 中混合所有类型 | 按类型分开发送 | 文本 delta 和工具调用 delta 分开 | + +### 10.6 协议转换映射（team-api 实现细节） + +在 team-api 的 `converter.go` 中实现了双向转换： + +**Responses → Chat Completions（请求）**： + +| Responses 字段 | Chat Completions 字段 | +|---------------|----------------------| +| `instructions` | `messages[0]` (role=system) | +| `input` (string) | `messages[1]` (role=user, content=string) | +| `input` (array, type=message) | `messages[n]` (role=message.role) | +| `input` (array, type=function_call_output) | `messages[n]` (role=tool, tool_call_id=call_id) | +| `tools[].type=function` | `tools[].type=function, function={name,description,parameters}` | +| `reasoning.effort` | `reasoning_effort` | +| `text.format` (json_schema) | `response_format` | +| `stream` + `true` | `stream=true, stream_options={include_usage:true}` | + +**Chat Completions → Responses（请求）**： + +| Chat Completions 字段 | Responses 字段 | +|----------------------|---------------| +| `messages[role=system/developer]` | `instructions`（合并为单个字符串） | +| `messages[role=user]` | `input[]` (type=message, role=user) | +| `messages[role=assistant]` | `input[]` (type=message, role=assistant) | +| `messages[role=assistant].tool_calls` | `input[]` (type=function_call) | +| `messages[role=tool]` | `input[]` (type=function_call_output) | +| `max_tokens` / `max_completion_tokens` | `max_output_tokens`（取较大值） | +| `response_format` | `text.format` | +| `reasoning_effort` | `reasoning.effort` + `reasoning.summary="detailed"` | +| `tools[].function` | `tools[]` (扁平化) | + +**Responses → Chat Completions（响应）**： + +| Responses 字段 | Chat Completions 字段 | +|---------------|----------------------| +| `output[type=message].content[type=output_text].text` | `choices[0].message.content` | +| `output[type=function_call]` | `choices[0].message.tool_calls[]` | +| `status="completed"` | `choices[0].finish_reason="stop"` | +| `status="completed"` + 有 tool_calls | `choices[0].finish_reason="tool_calls"` | +| `usage.input_tokens` | `usage.prompt_tokens` | +| `usage.output_tokens` | `usage.completion_tokens` | +| `usage.total_tokens` | `usage.total_tokens` | + +**Responses SSE → Chat SSE（流式）**： + +| Responses 事件 | Chat Completions 输出 | +|---------------|----------------------| +| `response.output_text.delta` | `choices[0].delta.content` | +| `response.reasoning_summary_text.delta` | `choices[0].delta.reasoning_content` | +| `response.output_item.added` (function_call) | `choices[0].delta.tool_calls[]` | +| `response.function_call_arguments.delta` | `choices[0].delta.tool_calls[].function.arguments` | +| `response.completed` | `choices[0].finish_reason` + usage chunk + `[DONE]` | +| `response.error` / `response.failed` | 错误返回 | + +--- + +## 附录 A：team-api DTO 结构体参考 + +以下为 `relay/dto/openai_responses.go` 中定义的核心结构体，供开发参考。 + +### OpenAIResponsesRequest + +```go +type OpenAIResponsesRequest struct { + Model string `json:"model"` + Input json.RawMessage `json:"input,omitempty"` + Include json.RawMessage `json:"include,omitempty"` + Instructions json.RawMessage `json:"instructions,omitempty"` + MaxOutputTokens *uint `json:"max_output_tokens,omitempty"` + Metadata json.RawMessage `json:"metadata,omitempty"` + ParallelToolCalls json.RawMessage `json:"parallel_tool_calls,omitempty"` + PreviousResponseID string `json:"previous_response_id,omitempty"` + Reasoning *Reasoning `json:"reasoning,omitempty"` + Store json.RawMessage `json:"store,omitempty"` + Stream *bool `json:"stream,omitempty"` + StreamOptions *StreamOptions `json:"stream_options,omitempty"` + Temperature *float64 `json:"temperature,omitempty"` + Text json.RawMessage `json:"text,omitempty"` + ToolChoice json.RawMessage `json:"tool_choice,omitempty"` + Tools json.RawMessage `json:"tools,omitempty"` + TopP *float64 `json:"top_p,omitempty"` + TopLogProbs *int `json:"top_logprobs,omitempty"` + Truncation json.RawMessage `json:"truncation,omitempty"` + User json.RawMessage `json:"user,omitempty"` + MaxToolCalls *uint `json:"max_tool_calls,omitempty"` + Prompt json.RawMessage `json:"prompt,omitempty"` + ServiceTier string `json:"service_tier,omitempty"` + Conversation json.RawMessage `json:"conversation,omitempty"` + ContextManagement json.RawMessage `json:"context_management,omitempty"` +} +``` + +### OpenAIResponsesResponse + +```go +type OpenAIResponsesResponse struct { + ID string `json:"id"` + Object string `json:"object"` + CreatedAt int `json:"created_at"` + CompletedAt int `json:"completed_at,omitempty"` + Status json.RawMessage `json:"status"` + Error any `json:"error"` + IncompleteDetails any `json:"incomplete_details"` + Instructions any `json:"instructions"` + MaxOutputTokens *int `json:"max_output_tokens"` + Model string `json:"model"` + Output []ResponsesOutput `json:"output"` + ParallelToolCalls bool `json:"parallel_tool_calls"` + PreviousResponseID any `json:"previous_response_id"` + Reasoning *ResponsesReasoning `json:"reasoning"` + Store bool `json:"store"` + Temperature *float64 `json:"temperature"` + Text *ResponsesText `json:"text,omitempty"` + ToolChoice any `json:"tool_choice"` + Tools []any `json:"tools"` + TopP *float64 `json:"top_p"` + Truncation any `json:"truncation"` + Usage *ResponsesUsage `json:"usage,omitempty"` + User any `json:"user"` + Metadata any `json:"metadata"` +} +``` + +### ResponsesStreamResponse + +```go +type ResponsesStreamResponse struct { + Type string `json:"type"` + Response *OpenAIResponsesResponse `json:"response,omitempty"` + Delta string `json:"delta,omitempty"` + Item *ResponsesOutput `json:"item,omitempty"` + OutputIndex *int `json:"output_index,omitempty"` + ContentIndex *int `json:"content_index,omitempty"` + SummaryIndex *int `json:"summary_index,omitempty"` + ItemID string `json:"item_id,omitempty"` + Part *ResponsesSummaryPart `json:"part,omitempty"` +} +``` + +--- + +## 附录 B：关键代码文件索引 + +| 文件 | 路径 | 说明 | +|------|------|------| +| DTO 定义 | `relay/dto/openai_responses.go` | 所有 Responses API 请求/响应结构体 | +| 协议转换 | `relay/channel/openai/converter.go` | Responses ↔ Chat Completions 双向转换 | +| 响应处理 | `relay/channel/openai/responses.go` | Chat 响应 → Responses 格式的 SSE 转换 | diff --git a/docs/协议文档/OpenAI/OpenAI-Responses协议文档.md b/docs/协议文档/OpenAI/OpenAI-Responses协议文档.md new file mode 100644 index 0000000..e92f5bf --- /dev/null +++ b/docs/协议文档/OpenAI/OpenAI-Responses协议文档.md @@ -0,0 +1,1777 @@ +# OpenAI Responses API 协议文档 + +> 基于 OpenAI 官方 API Reference 整理，用于协议转换器实现参考。 +> 端点基础路径：`https://api.openai.com/v1/responses` +> 官方文档：https://platform.openai.com/docs/api-reference/responses + +--- + +## 目录 + +- [1. API 概述](#1-api-概述) +- [2. 创建响应 — POST /v1/responses](#2-创建响应--post-v1responses) + - [2.1 请求体参数](#21-请求体参数) + - [2.2 输入项类型（Input Items）](#22-输入项类型input-items) + - [2.3 输出项类型（Output Items）](#23-输出项类型output-items) + - [2.4 工具定义（Tools）](#24-工具定义tools) + - [2.5 工具选择（Tool Choice）](#25-工具选择tool-choice) + - [2.6 推理配置（Reasoning）](#26-推理配置reasoning) + - [2.7 文本输出格式（Text Format）](#27-文本输出格式text-format) + - [2.8 上下文管理（Context Management）](#28-上下文管理context-management) + - [2.9 对话管理（Conversation）](#29-对话管理conversation) + - [2.10 流式选项（Stream Options）](#210-流式选项stream-options) + - [2.11 其他请求参数](#211-其他请求参数) +- [3. 响应对象（Response Object）](#3-响应对象response-object) + - [3.1 完整响应结构](#31-完整响应结构) + - [3.2 响应示例](#32-响应示例) + - [3.3 状态值（Status）](#33-状态值status) + - [3.4 Usage 统计](#34-usage-统计) + - [3.5 错误对象](#35-错误对象) + - [3.6 不完整详情](#36-不完整详情) +- [4. 其他端点](#4-其他端点) + - [4.1 获取响应 — GET /v1/responses/{response_id}](#41-获取响应--get-v1responsesresponse_id) + - [4.2 删除响应 — DELETE /v1/responses/{response_id}](#42-删除响应--delete-v1responsesresponse_id) + - [4.3 取消响应 — POST /v1/responses/{response_id}/cancel](#43-取消响应--post-v1responsesresponse_idcancel) + - [4.4 压缩对话 — POST /v1/responses/compact](#44-压缩对话--post-v1responsescompact) + - [4.5 列出输入项 — GET /v1/responses/{response_id}/input_items](#45-列出输入项--get-v1responsesresponse_idinput_items) + - [4.6 计算输入 Token — POST /v1/responses/input_tokens](#46-计算输入-token--post-v1responsesinput_tokens) +- [5. 与 Chat Completions API 的对比](#5-与-chat-completions-api-的对比) + +--- + +## 1. API 概述 + +OpenAI Responses API 是 OpenAI 推出的新一代模型响应 API，旨在替代 Chat Completions API。它提供了更丰富的功能： + +- **统一的输入/输出模型**：使用 `input` 和 `output` 数组代替传统的 `messages` 结构 +- **内置工具支持**：原生支持 Web 搜索、文件搜索、代码解释器、图像生成、计算机控制等内置工具 +- **MCP 协议集成**：支持通过 Model Context Protocol 连接第三方工具服务器 +- **推理控制**：支持调节推理力度（effort）和生成推理摘要（summary） +- **上下文管理**：支持对话压缩（compaction）以管理长对话 +- **后台模式**：支持异步处理响应 +- **结构化输出**：支持 JSON Schema 约束的结构化输出 +- **多轮对话**：通过 `previous_response_id` 或 `conversation` 对象实现 + +端点总览： + +| 方法 | 路径 | 功能 | +|------|------|------| +| POST | `/v1/responses` | 创建模型响应 | +| GET | `/v1/responses/{response_id}` | 获取响应 | +| DELETE | `/v1/responses/{response_id}` | 删除响应 | +| POST | `/v1/responses/{response_id}/cancel` | 取消响应 | +| POST | `/v1/responses/compact` | 压缩对话 | +| GET | `/v1/responses/{response_id}/input_items` | 列出输入项 | +| POST | `/v1/responses/input_tokens` | 计算输入 Token 数 | + +--- + +## 2. 创建响应 — POST /v1/responses + +创建模型响应。提供文本、图片或文件输入以生成文本或 JSON 输出。可以让模型调用自定义代码或使用内置工具（如 Web 搜索、文件搜索）。 + +### 2.1 请求体参数 + +| 参数名 | 类型 | 必填 | 默认值 | 说明 | +|--------|------|------|--------|------| +| `model` | string | **是** | — | 模型 ID，如 `gpt-4o`、`o3`、`gpt-4.1` | +| `input` | string / array | **是** | — | 模型输入，可以是字符串（等同于 user 角色的文本输入）或输入项数组 | +| `instructions` | string | 否 | null | 系统（或开发者）消息，插入到模型上下文中。使用 `previous_response_id` 时不会继承之前的 instructions | +| `tools` | array | 否 | `[]` | 模型可以调用的工具数组 | +| `tool_choice` | string / object | 否 | `"auto"` | 模型如何选择工具 | +| `temperature` | number | 否 | 1 | 采样温度，0-2 之间。建议与 `top_p` 只调整一个 | +| `top_p` | number | 否 | 1 | 核采样概率质量。建议与 `temperature` 只调整一个 | +| `max_output_tokens` | integer | 否 | null | 输出 Token 上限（含可见输出和推理 Token） | +| `max_tool_calls` | integer | 否 | — | 内置工具最大总调用次数（跨所有内置工具合计） | +| `stream` | boolean | 否 | false | 是否启用流式响应 | +| `stream_options` | object | 否 | null | 流式选项，仅在 `stream: true` 时设置 | +| `store` | boolean | 否 | true | 是否存储响应以供后续 API 检索 | +| `reasoning` | object | 否 | — | 推理配置 | +| `text` | object | 否 | — | 文本输出格式配置 | +| `previous_response_id` | string | 否 | null | 上一个响应的唯一 ID，用于创建多轮对话。不能与 `conversation` 同时使用 | +| `conversation` | object | 否 | null | 对话对象，指定此响应所属的对话 | +| `background` | boolean | 否 | false | 是否在后台运行模型响应 | +| `include` | array | 否 | — | 指定要包含的附加输出数据 | +| `parallel_tool_calls` | boolean | 否 | true | 是否允许模型并行调用工具 | +| `truncation` | string | 否 | `"disabled"` | 截断策略：`"auto"` 或 `"disabled"` | +| `metadata` | object | 否 | `{}` | 最多 16 个键值对的元数据。键最长 64 字符，值最长 512 字符 | +| `user` | string | 否 | null | 终端用户标识（**已弃用**，使用 `prompt_cache_key` 和 `safety_identifier` 替代） | +| `prompt_cache_key` | string | 否 | — | 用于缓存优化的稳定标识符，替代 `user` 字段 | +| `prompt_cache_retention` | string | 否 | — | 提示缓存保留策略，设为 `24h` 启用扩展缓存 | +| `safety_identifier` | string | 否 | — | 用于检测违规用户的安全标识符 | +| `service_tier` | string | 否 | `"auto"` | 服务等级：`"auto"` / `"default"` / `"flex"` / `"priority"` | +| `prompt` | object | 否 | — | 提示模板引用及其变量 | +| `logprobs` | integer | 否 | — | 返回每个位置最可能的 Token 数量，0-20 | + +#### `include` 参数支持的值 + +| 值 | 说明 | +|----|------| +| `web_search_call.action.sources` | 包含 Web 搜索工具调用的来源 | +| `code_interpreter_call.outputs` | 包含代码解释器中 Python 代码执行的输出 | +| `computer_call_output.output.image_url` | 包含计算机调用的截图图片 URL | +| `file_search_call.results` | 包含文件搜索工具调用的搜索结果 | +| `message.input_image.image_url` | 包含输入消息中的图片 URL | +| `message.output_text.logprobs` | 包含助手消息的 logprobs | +| `reasoning.encrypted_content` | 包含推理项的加密内容，用于无状态多轮对话 | + +### 2.2 输入项类型（Input Items） + +`input` 参数支持多种形式： + +1. **纯字符串**：等同于 user 角色的文本输入 +2. **输入项数组**：包含不同类型内容项的数组 + +#### 2.2.1 input_text — 文本输入 + +```json +{ + "type": "input_text", + "text": "Hello, how are you?" +} +``` + +| 字段 | 类型 | 必填 | 说明 | +|------|------|------|------| +| `type` | string | 是 | 固定值 `"input_text"` | +| `text` | string | 是 | 文本内容 | + +#### 2.2.2 input_image — 图片输入 + +```json +{ + "type": "input_image", + "image_url": "https://example.com/image.png", + "detail": "auto" +} +``` + +| 字段 | 类型 | 必填 | 说明 | +|------|------|------|------| +| `type` | string | 是 | 固定值 `"input_image"` | +| `image_url` | string | 否* | 图片 URL（完整 URL 或 base64 data URL）。与 `file_id` 二选一 | +| `file_id` | string | 否* | 上传文件的 ID。与 `image_url` 二选一 | +| `detail` | string | 否 | 图片细节级别：`"high"` / `"low"` / `"auto"`，默认 `"auto"` | + +#### 2.2.3 input_file — 文件输入 + +```json +{ + "type": "input_file", + "file_url": "https://example.com/document.pdf", + "filename": "document.pdf" +} +``` + +| 字段 | 类型 | 必填 | 说明 | +|------|------|------|------| +| `type` | string | 是 | 固定值 `"input_file"` | +| `file_url` | string | 否* | 文件 URL。与 `file_id` 和 `content` 三选一 | +| `file_id` | string | 否* | 上传文件的 ID。与 `file_url` 和 `content` 三选一 | +| `content` | string | 否* | 文件内容。与 `file_url` 和 `file_id` 三选一 | +| `filename` | string | 否 | 文件名 | + +#### 2.2.4 message — 消息输入 + +用于带角色指示的消息输入。`developer` 或 `system` 角色的指令优先级高于 `user` 角色。`assistant` 角色的消息被视为模型在之前交互中生成的。 + +**作为输入项使用（简写格式）：** + +```json +{ + "role": "user", + "content": "Tell me a story." +} +``` + +**完整格式（从 API 返回时）：** + +```json +{ + "type": "message", + "id": "msg_abc123", + "role": "user", + "content": [ + { + "type": "input_text", + "text": "Tell me a story." + } + ], + "status": "completed" +} +``` + +| 字段 | 类型 | 必填 | 说明 | +|------|------|------|------| +| `type` | string | 条件 | 固定值 `"message"`，API 返回时总有，输入时可省略 | +| `id` | string | 否 | 消息唯一 ID，API 返回时填充 | +| `role` | string | 是 | 角色：`"user"` / `"assistant"` / `"system"` / `"developer"` | +| `content` | string / array | 是 | 消息内容，可以是字符串或内容项数组 | +| `status` | string | 否 | 项目状态：`"in_progress"` / `"completed"` / `"incomplete"` | + +#### 2.2.5 item_reference — 项引用 + +```json +{ + "type": "item_reference", + "id": "msg_abc123" +} +``` + +| 字段 | 类型 | 必填 | 说明 | +|------|------|------|------| +| `type` | string | 是 | 固定值 `"item_reference"` | +| `id` | string | 是 | 要引用的项 ID | + +#### 2.2.6 function_call_output — 函数调用输出 + +```json +{ + "type": "function_call_output", + "call_id": "call_abc123", + "output": "{\"temperature\": 72}" +} +``` + +| 字段 | 类型 | 必填 | 说明 | +|------|------|------|------| +| `type` | string | 是 | 固定值 `"function_call_output"` | +| `call_id` | string | 是 | 对应函数调用的唯一 ID | +| `output` | string / array | 是 | 函数调用的输出，可以是 JSON 字符串或内容输出数组 | +| `id` | string | 否 | API 返回时填充的唯一 ID | +| `status` | string | 否 | 项目状态 | + +`output` 的内容输出数组可包含以下类型： +- `input_text`：文本输出 +- `input_image`：图片输出 +- `input_file`：文件输出 + +#### 2.2.7 computer_call_output — 计算机调用输出 + +```json +{ + "type": "computer_call_output", + "call_id": "call_abc123", + "output": { + "type": "computer_screenshot", + "image_url": "data:image/png;base64,..." + }, + "acknowledged_safety_checks": [ + { + "id": "safety_check_001", + "code": "malicious_content", + "message": "Potential malicious content detected" + } + ] +} +``` + +| 字段 | 类型 | 必填 | 说明 | +|------|------|------|------| +| `type` | string | 是 | 固定值 `"computer_call_output"` | +| `call_id` | string | 是 | 对应计算机调用的 ID | +| `output` | object | 是 | 计算机截图对象 | +| `acknowledged_safety_checks` | array | 否 | 已确认的安全检查列表 | +| `id` | string | 否 | API 返回时填充 | +| `status` | string | 否 | 项目状态 | + +**computer_screenshot 对象：** + +| 字段 | 类型 | 必填 | 说明 | +|------|------|------|------| +| `type` | string | 是 | 固定值 `"computer_screenshot"` | +| `image_url` | string | 否 | 截图图片 URL | +| `file_id` | string | 否 | 截图文件 ID | + +#### 2.2.8 mcp_approval_response — MCP 审批响应 + +```json +{ + "type": "mcp_approval_response", + "approval_request_id": "apr_abc123", + "approve": true, + "reason": "This operation is safe" +} +``` + +| 字段 | 类型 | 必填 | 说明 | +|------|------|------|------| +| `type` | string | 是 | 固定值 `"mcp_approval_response"` | +| `approval_request_id` | string | 是 | 要回应的审批请求 ID | +| `approve` | boolean | 是 | 是否批准 | +| `id` | string | 否 | 唯一 ID | +| `reason` | string | 否 | 决定原因 | + +### 2.3 输出项类型（Output Items） + +响应的 `output` 数组可以包含以下类型的项。 + +#### 2.3.1 message — 输出消息 + +模型生成的消息输出。 + +```json +{ + "type": "message", + "id": "msg_abc123", + "status": "completed", + "role": "assistant", + "content": [ + { + "type": "output_text", + "text": "Hello! How can I help you?", + "annotations": [] + } + ] +} +``` + +| 字段 | 类型 | 说明 | +|------|------|------| +| `type` | string | 固定值 `"message"` | +| `id` | string | 输出消息唯一 ID | +| `status` | string | 状态：`"in_progress"` / `"completed"` / `"incomplete"` | +| `role` | string | 固定值 `"assistant"` | +| `content` | array | 内容数组，包含 `output_text` 和/或 `refusal` | + +**output_text 内容类型：** + +```json +{ + "type": "output_text", + "text": "Hello!", + "annotations": [ + { + "type": "url_citation", + "url": "https://example.com", + "title": "Example", + "start_index": 0, + "end_index": 10 + } + ] +} +``` + +| 字段 | 类型 | 说明 | +|------|------|------| +| `type` | string | 固定值 `"output_text"` | +| `text` | string | 模型输出的文本 | +| `annotations` | array | 注解数组，包含以下类型 | + +**注解类型：** + +| 类型 | 字段 | 说明 | +|------|------|------| +| `file_citation` | `type`, `file_id`, `filename`, `index` | 文件引用 | +| `url_citation` | `type`, `url`, `title`, `start_index`, `end_index` | URL 引用 | +| `container_file_citation` | `type`, `file_id`, `filename`, `start_index`, `end_index`, `index` | 容器文件引用 | +| `file_path` | `type`, `file_id`, `index` | 文件路径引用 | + +**refusal 内容类型：** + +```json +{ + "type": "refusal", + "refusal": "I'm sorry, but I cannot assist with that request." +} +``` + +#### 2.3.2 function_call — 函数调用 + +```json +{ + "type": "function_call", + "id": "fc_abc123", + "call_id": "call_abc123", + "name": "get_weather", + "arguments": "{\"location\": \"San Francisco\"}", + "status": "completed" +} +``` + +| 字段 | 类型 | 说明 | +|------|------|------| +| `type` | string | 固定值 `"function_call"` | +| `id` | string | 函数调用的唯一 ID | +| `call_id` | string | 用于映射函数调用输出的标识符 | +| `name` | string | 要运行的函数名 | +| `arguments` | string | 传递给函数的 JSON 字符串参数 | +| `status` | string | 状态：`"in_progress"` / `"completed"` / `"incomplete"` | + +#### 2.3.3 web_search_call — Web 搜索调用 + +```json +{ + "type": "web_search_call", + "id": "ws_abc123", + "status": "completed", + "action": { + "type": "search", + "query": "latest news about AI", + "sources": [ + { + "type": "url", + "url": "https://example.com/article" + } + ] + } +} +``` + +| 字段 | 类型 | 说明 | +|------|------|------| +| `type` | string | 固定值 `"web_search_call"` | +| `id` | string | Web 搜索调用的唯一 ID | +| `status` | string | 状态 | +| `action` | object | 搜索动作描述 | + +**action 类型：** + +| 类型 | 说明 | 关键字段 | +|------|------|----------| +| `search` | 执行 Web 搜索查询 | `query`（已弃用）, `sources` | +| `open_page` | 打开搜索结果中的 URL | `url`, `sources` | +| `find_in_page` | 在已加载页面中搜索 | `pattern`, `url` | + +#### 2.3.4 file_search_call — 文件搜索调用 + +```json +{ + "type": "file_search_call", + "id": "fs_abc123", + "status": "completed", + "queries": ["quarterly revenue"], + "results": [ + { + "file_id": "file-abc123", + "text": "Q4 revenue was $5.2M...", + "score": 0.95 + } + ] +} +``` + +| 字段 | 类型 | 说明 | +|------|------|------| +| `type` | string | 固定值 `"file_search_call"` | +| `id` | string | 文件搜索调用的唯一 ID | +| `status` | string | 状态：`"in_progress"` / `"searching"` / `"incomplete"` / `"failed"` | +| `queries` | array | 搜索查询列表 | +| `results` | array | 搜索结果列表 | + +**搜索结果项：** + +| 字段 | 类型 | 说明 | +|------|------|------| +| `file_id` | string | 文件唯一 ID | +| `text` | string | 从文件中检索到的文本 | +| `score` | number | 相关性评分（0-1） | + +#### 2.3.5 computer_call — 计算机调用 + +```json +{ + "type": "computer_call", + "id": "cu_abc123", + "call_id": "call_abc123", + "action": { + "type": "click", + "x": 100, + "y": 200, + "button": "left" + }, + "pending_safety_checks": [], + "status": "completed" +} +``` + +**action 类型：** + +| 类型 | 字段 | 说明 | +|------|------|------| +| `click` | `x`, `y`, `button` | 鼠标点击。`button`: `"left"` / `"right"` / `"wheel"` / `"back"` / `"forward"` | +| `double_click` | `x`, `y` | 双击 | +| `drag` | `path` (坐标数组 `[{x, y}, ...]`) | 拖拽操作 | +| `keypress` | `keys` (字符串数组) | 按键操作 | +| `move` | `x`, `y` | 鼠标移动 | +| `scroll` | `x`, `y`, `scroll_x`, `scroll_y` | 滚动操作 | +| `type` | `text` | 输入文本 | +| `wait` | — | 等待 | +| `screenshot` | — | 截图 | + +| 字段 | 类型 | 说明 | +|------|------|------| +| `type` | string | 固定值 `"computer_call"` | +| `id` | string | 计算机调用的唯一 ID | +| `call_id` | string | 用于响应的标识符 | +| `action` | object | 要执行的动作 | +| `pending_safety_checks` | array | 待处理的安全检查列表 | +| `status` | string | 状态 | + +#### 2.3.6 reasoning — 推理 + +```json +{ + "type": "reasoning", + "id": "rs_abc123", + "summary": [ + { + "type": "summary_text", + "text": "I analyzed the input and determined..." + } + ], + "encrypted_content": "encrypted-data...", + "status": "completed" +} +``` + +| 字段 | 类型 | 说明 | +|------|------|------| +| `type` | string | 固定值 `"reasoning"` | +| `id` | string | 推理内容唯一标识符 | +| `summary` | array | 推理摘要内容数组，每项包含 `type: "summary_text"` 和 `text` | +| `encrypted_content` | string | 加密的推理内容（在 `include` 中包含 `reasoning.encrypted_content` 时填充） | +| `status` | string | 状态 | + +推理项还可以包含 `reasoning_text` 类型的内容： + +```json +{ + "type": "reasoning", + "id": "rs_abc123", + "content": [ + { + "type": "reasoning_text", + "text": "Let me think about this step by step..." + } + ] +} +``` + +#### 2.3.7 image_generation_call — 图像生成调用 + +```json +{ + "type": "image_generation_call", + "id": "ig_abc123", + "status": "completed", + "result": "base64-encoded-image-data" +} +``` + +| 字段 | 类型 | 说明 | +|------|------|------| +| `type` | string | 固定值 `"image_generation_call"` | +| `id` | string | 图像生成调用的唯一 ID | +| `status` | string | 状态 | +| `result` | string | Base64 编码的生成图像 | + +#### 2.3.8 code_interpreter_call — 代码解释器调用 + +```json +{ + "type": "code_interpreter_call", + "id": "ci_abc123", + "status": "completed", + "code": "print('Hello, World!')", + "container_id": "container_abc123", + "outputs": [ + { + "type": "logs", + "logs": "Hello, World!\n" + } + ] +} +``` + +| 字段 | 类型 | 说明 | +|------|------|------| +| `type` | string | 固定值 `"code_interpreter_call"` | +| `id` | string | 代码解释器调用的唯一 ID | +| `status` | string | 状态：`"in_progress"` / `"completed"` / `"incomplete"` / `"interpreting"` / `"failed"` | +| `code` | string | 要运行的代码，不可用时为 null | +| `container_id` | string | 运行代码的容器 ID | +| `outputs` | array | 输出列表，包含 `logs` 或 `image` 类型 | + +**outputs 类型：** + +| 类型 | 字段 | 说明 | +|------|------|------| +| `logs` | `type: "logs"`, `logs` | 日志输出 | +| `image` | `type: "image"`, `url` | 图像输出（URL） | + +#### 2.3.9 shell_call — Shell 调用 + +```json +{ + "type": "shell_call", + "id": "sh_abc123", + "call_id": "call_abc123", + "status": "completed", + "action": { + "type": "exec", + "command": ["ls", "-la"], + "max_output_length": 10000, + "timeout": 30000 + } +} +``` + +| 字段 | 类型 | 说明 | +|------|------|------| +| `type` | string | 固定值 `"shell_call"` | +| `id` | string | Shell 调用的唯一 ID | +| `call_id` | string | 模型生成的调用 ID | +| `status` | string | 状态 | +| `action` | object | Shell 命令和限制 | + +**action 字段：** + +| 字段 | 类型 | 说明 | +|------|------|------| +| `type` | string | 固定值 `"exec"` | +| `command` | array | 有序的 Shell 命令列表 | +| `max_output_length` | integer | 最大输出字符数（UTF-8） | +| `timeout` | integer | 最大运行时间（毫秒） | + +#### 2.3.10 local_shell_call — 本地 Shell 调用 + +```json +{ + "type": "local_shell_call", + "id": "lsh_abc123", + "call_id": "call_abc123", + "status": "completed", + "action": { + "type": "exec", + "command": ["npm", "test"], + "env": {"NODE_ENV": "test"}, + "timeout": 60000, + "user": "runner", + "working_directory": "/app" + } +} +``` + +与 `shell_call` 类似，但包含额外的本地执行环境参数。 + +#### 2.3.11 apply_patch_call — 应用补丁调用 + +```json +{ + "type": "apply_patch_call", + "id": "ap_abc123", + "call_id": "call_abc123", + "status": "completed", + "action": { + "type": "create_file", + "path": "src/newfile.js", + "patch": "--- /dev/null\n+++ src/newfile.js\n@@ -0,0 +1 @@\n+console.log('hello');" + } +} +``` + +**action 操作类型：** + +| 操作类型 | 字段 | 说明 | +|----------|------|------| +| `create_file` | `path`, `patch`, `type` | 创建新文件 | +| `delete_file` | `path`, `type` | 删除文件 | +| `update_file` | `path`, `patch`, `type` | 更新文件（unified diff） | + +#### 2.3.12 mcp_call — MCP 工具调用 + +```json +{ + "type": "mcp_call", + "id": "mcp_abc123", + "server_label": "deepwiki", + "name": "read_wiki", + "arguments": "{\"page\": \"Go_(programming_language)\"}", + "output": "Go is a statically typed...", + "status": "completed", + "approval_request_id": "apr_abc123", + "error": null +} +``` + +| 字段 | 类型 | 说明 | +|------|------|------| +| `type` | string | 固定值 `"mcp_call"` | +| `id` | string | 工具调用的唯一 ID | +| `server_label` | string | MCP 服务器标签 | +| `name` | string | 运行的工具名称 | +| `arguments` | string | JSON 字符串格式的参数 | +| `output` | string | 工具调用输出 | +| `error` | string | 错误信息（如果有） | +| `status` | string | 状态：`"in_progress"` / `"completed"` / `"incomplete"` / `"calling"` / `"failed"` | +| `approval_request_id` | string | MCP 工具调用审批请求的唯一标识符 | + +#### 2.3.13 mcp_list_tools — MCP 工具列表 + +```json +{ + "type": "mcp_list_tools", + "id": "mlt_abc123", + "server_label": "deepwiki", + "tools": [ + { + "name": "read_wiki", + "description": "Read a wiki page", + "inputSchema": {"type": "object", "properties": {...}}, + "annotations": {} + } + ], + "error": null +} +``` + +#### 2.3.14 mcp_approval_request — MCP 审批请求 + +```json +{ + "type": "mcp_approval_request", + "id": "apr_abc123", + "server_label": "deepwiki", + "name": "delete_page", + "arguments": "{\"page\": \"test\"}" +} +``` + +#### 2.3.15 custom_tool_call — 自定义工具调用 + +```json +{ + "type": "custom_tool_call", + "id": "ctc_abc123", + "call_id": "call_abc123", + "name": "my_tool", + "input": "some input data", + "status": "completed" +} +``` + +**custom_tool_call_output — 自定义工具调用输出：** + +```json +{ + "type": "custom_tool_call_output", + "call_id": "call_abc123", + "output": "tool output data" +} +``` + +#### 2.3.16 compaction — 压缩项 + +```json +{ + "type": "compaction", + "id": "cmp_001", + "encrypted_content": "gAAAAABpM0Yj-...=" +} +``` + +| 字段 | 类型 | 说明 | +|------|------|------| +| `type` | string | 固定值 `"compaction"` | +| `id` | string | 压缩项 ID | +| `encrypted_content` | string | 加密的压缩摘要内容 | + +### 2.4 工具定义（Tools） + +`tools` 参数是一个数组，支持以下工具类别： + +1. **内置工具**：OpenAI 提供的工具（Web 搜索、文件搜索等） +2. **MCP 工具**：通过 MCP 协议连接第三方系统 +3. **函数调用（自定义工具）**：用户自定义的函数 + +#### 2.4.1 function — 函数工具 + +```json +{ + "type": "function", + "name": "get_weather", + "description": "Get the current weather for a location", + "parameters": { + "type": "object", + "properties": { + "location": { + "type": "string", + "description": "City name" + } + }, + "required": ["location"] + }, + "strict": true +} +``` + +| 字段 | 类型 | 必填 | 说明 | +|------|------|------|------| +| `type` | string | 是 | 固定值 `"function"` | +| `name` | string | 是 | 函数名称 | +| `description` | string | 否 | 函数描述，帮助模型决定是否调用 | +| `parameters` | object | 否 | JSON Schema 格式的参数描述 | +| `strict` | boolean | 否 | 是否强制严格参数验证，默认 `true` | + +#### 2.4.2 web_search / web_search_preview — Web 搜索工具 + +```json +{ + "type": "web_search_preview", + "search_context_size": "medium", + "user_location": { + "type": "approximate", + "city": "San Francisco", + "region": "California" + } +} +``` + +| 字段 | 类型 | 说明 | +|------|------|------| +| `type` | string | `"web_search"` 或 `"web_search_preview"` 或 `"web_search_preview_2025_03_11"` | +| `search_context_size` | string | 上下文窗口使用量指导：`"low"` / `"medium"` / `"high"`，默认 `"medium"` | +| `user_location` | object | 用户的大致位置 | + +**user_location 对象：** + +| 字段 | 类型 | 说明 | +|------|------|------| +| `type` | string | 固定值 `"approximate"` | +| `city` | string | 城市名（自由文本） | +| `region` | string | 地区名（自由文本） | + +#### 2.4.3 web_search（2025-08-26 版本） + +```json +{ + "type": "web_search", + "search_context_size": "medium", + "allowed_domains": ["pubmed.ncbi.nlm.nih.gov"], + "user_location": { + "type": "approximate", + "city": "San Francisco", + "region": "California" + } +} +``` + +| 字段 | 类型 | 说明 | +|------|------|------| +| `type` | string | `"web_search"` 或 `"web_search_2025_08_26"` | +| `allowed_domains` | array | 允许的搜索域名。不提供则允许所有域名。子域名也被允许 | +| `search_context_size` | string | 上下文窗口使用量指导 | +| `user_location` | object | 用户的大致位置 | + +#### 2.4.4 file_search — 文件搜索工具 + +```json +{ + "type": "file_search", + "vector_store_ids": ["vs_abc123"], + "max_num_results": 10, + "ranking_options": { + "ranker": "auto", + "score_threshold": 0.5 + }, + "filters": { + "type": "and", + "filters": [ + { + "type": "eq", + "key": "category", + "value": "finance" + } + ] + } +} +``` + +| 字段 | 类型 | 说明 | +|------|------|------| +| `type` | string | 固定值 `"file_search"` | +| `vector_store_ids` | array | 要搜索的向量存储 ID 列表 | +| `max_num_results` | integer | 最大返回结果数（1-50） | +| `ranking_options` | object | 搜索排名选项 | +| `filters` | object | 搜索过滤器 | + +**ranking_options 对象：** + +| 字段 | 类型 | 说明 | +|------|------|------| +| `ranker` | string | 使用的排名器 | +| `score_threshold` | number | 分数阈值（0-1） | + +**filters 类型：** + +比较过滤器： + +| 字段 | 类型 | 说明 | +|------|------|------| +| `type` | string | 比较运算符：`"eq"` / `"ne"` / `"gt"` / `"gte"` / `"lt"` / `"lte"` / `"in"` / `"nin"` | +| `key` | string | 要比较的属性键 | +| `value` | string/number/boolean/array | 比较值 | + +复合过滤器： + +| 字段 | 类型 | 说明 | +|------|------|------| +| `type` | string | `"and"` 或 `"or"` | +| `filters` | array | 过滤器数组（可以是比较过滤器或复合过滤器） | + +#### 2.4.5 computer_use_preview — 计算机使用工具 + +```json +{ + "type": "computer_use_preview", + "display_width": 1024, + "display_height": 768, + "environment": "browser" +} +``` + +| 字段 | 类型 | 说明 | +|------|------|------| +| `type` | string | 固定值 `"computer_use_preview"` | +| `display_width` | integer | 显示器宽度 | +| `display_height` | integer | 显示器高度 | +| `environment` | string | 计算机环境类型 | + +#### 2.4.6 code_interpreter — 代码解释器工具 + +```json +{ + "type": "code_interpreter", + "container": { + "type": "auto", + "file_ids": ["file-abc123"], + "memory_limit": "512m" + } +} +``` + +| 字段 | 类型 | 说明 | +|------|------|------| +| `type` | string | 固定值 `"code_interpreter"` | +| `container` | string / object | 代码解释器容器 ID 或配置对象 | + +**container 配置对象：** + +| 字段 | 类型 | 说明 | +|------|------|------| +| `type` | string | `"auto"` | +| `file_ids` | array | 上传文件 ID 列表，供代码使用 | +| `memory_limit` | string | 容器内存限制 | + +#### 2.4.7 image_generation — 图像生成工具 + +```json +{ + "type": "image_generation", + "size": "1024x1024", + "quality": "high", + "output_format": "png", + "output_compression": 100, + "partial_images": 0, + "background": "auto", + "input_image_mask": { + "image_url": "base64-data-url" + } +} +``` + +| 字段 | 类型 | 默认值 | 说明 | +|------|------|--------|------| +| `type` | string | — | 固定值 `"image_generation"` | +| `size` | string | `"auto"` | 图像尺寸：`"1024x1024"` / `"1024x1536"` / `"1536x1024"` / `"auto"` | +| `quality` | string | `"auto"` | 图像质量：`"low"` / `"medium"` / `"high"` / `"auto"` | +| `output_format` | string | `"png"` | 输出格式：`"png"` / `"webp"` / `"jpeg"` | +| `output_compression` | integer | 100 | 压缩级别 | +| `partial_images` | integer | 0 | 流式模式下生成的部分图像数量（0-3） | +| `background` | string | `"auto"` | 背景类型：`"transparent"` / `"opaque"` / `"auto"` | +| `moderation` | string | `"auto"` | 内容审核级别 | +| `input_image_mask` | object | — | 编辑模式下的蒙版（含 `image_url` 和 `file_id`） | +| `style` | string | — | 仅 `gpt-image-1` 及以上版本支持，`"high"` / `"low"` | + +#### 2.4.8 mcp — MCP 工具 + +```json +{ + "type": "mcp", + "server_label": "deepwiki", + "server_url": "https://mcp.deepwiki.com/mcp", + "server_description": "DeepWiki MCP Server", + "allowed_tools": ["read_wiki", "search_wiki"], + "require_approval": "never", + "headers": { + "Authorization": "Bearer token123" + } +} +``` + +| 字段 | 类型 | 必填 | 说明 | +|------|------|------|------| +| `type` | string | 是 | 固定值 `"mcp"` | +| `server_label` | string | 是 | MCP 服务器标签，用于在工具调用中标识 | +| `server_url` | string | 条件 | MCP 服务器 URL。与 `connector_id` 二选一 | +| `connector_id` | string | 条件 | 服务连接器 ID。与 `server_url` 二选一 | +| `server_description` | string | 否 | MCP 服务器描述 | +| `allowed_tools` | array / object | 否 | 允许的工具名称列表或过滤对象 | +| `require_approval` | string / object | 否 | 需要审批的工具策略，默认 `"always"` | +| `headers` | object | 否 | 发送到 MCP 服务器的 HTTP 头 | +| `oauth_token` | string | 否 | OAuth 访问令牌 | + +**支持的 connector_id 值：** + +| 连接器 | connector_id | +|--------|-------------| +| Dropbox | `connector_dropbox` | +| Gmail | `connector_gmail` | +| Google Calendar | `connector_googlecalendar` | +| Google Drive | `connector_googledrive` | +| Microsoft Teams | `connector_microsoftteams` | +| Outlook Calendar | `connector_outlookcalendar` | +| Outlook Email | `connector_outlookemail` | +| SharePoint | `connector_sharepoint` | + +**allowed_tools 过滤对象：** + +| 字段 | 类型 | 说明 | +|------|------|------| +| `type` | string | 工具过滤器类型 | +| `tool_names` | array | 允许的工具名称列表 | +| `readOnlyHint` | boolean | 匹配只读工具 | + +**require_approval 选项：** + +- `"always"`：所有工具都需要审批 +- `"never"`：所有工具都不需要审批 +- 过滤对象：指定哪些工具需要审批 + +#### 2.4.9 shell — Shell 工具 + +```json +{ + "type": "shell" +} +``` + +允许模型在托管环境中执行 Shell 命令。 + +#### 2.4.10 local_shell — 本地 Shell 工具 + +```json +{ + "type": "local_shell" +} +``` + +允许模型在本地环境中执行 Shell 命令。 + +#### 2.4.11 custom — 自定义工具 + +```json +{ + "type": "custom", + "name": "my_tool", + "description": "My custom tool description", + "input_format": { + "type": "text" + } +} +``` + +| 字段 | 类型 | 说明 | +|------|------|------| +| `type` | string | 固定值 `"custom"` | +| `name` | string | 自定义工具名称 | +| `description` | string | 工具描述 | +| `input_format` | object | 输入格式。`{"type": "text"}` 或 `{"type": "grammar", "syntax": "lark"|"regex", "definition": "..."}` | + +#### 2.4.12 apply_patch — 应用补丁工具 + +```json +{ + "type": "apply_patch" +} +``` + +允许使用 unified diff 创建、删除或更新文件。 + +### 2.5 工具选择（Tool Choice） + +`tool_choice` 参数控制模型如何选择工具： + +| 值 | 类型 | 说明 | +|----|------|------| +| `"auto"` | string | 模型自动选择生成消息或调用工具 | +| `"required"` | string | 模型必须调用一个或多个工具 | +| `"none"` | string | 模型不会调用任何工具，只生成消息 | + +**特定工具选择（对象形式）：** + +| 格式 | 说明 | +|------|------| +| `{"type": "function", "name": "get_weather"}` | 强制调用指定函数 | +| `{"type": "mcp", "server_label": "srv", "name": "tool"}` | 强制调用指定 MCP 工具 | +| `{"type": "custom", "name": "my_tool"}` | 强制调用指定自定义工具 | +| `{"type": "apply_patch"}` | 强制调用 apply_patch 工具 | +| `{"type": "shell"}` | 强制调用 shell 工具 | + +**allowed_tools 约束（对象形式）：** + +```json +{ + "type": "allowed_tools", + "tools": [ + {"type": "file_search"}, + {"type": "web_search_preview"} + ], + "mode": "auto" +} +``` + +| 字段 | 类型 | 说明 | +|------|------|------| +| `type` | string | 固定值 `"allowed_tools"` | +| `tools` | array | 允许的内置工具定义列表 | +| `mode` | string | `"auto"` 允许生成消息或调用工具，`"required"` 必须调用工具 | + +允许的内置工具类型：`file_search`、`web_search_preview`、`computer_use_preview`、`code_interpreter`、`image_generation` + +### 2.6 推理配置（Reasoning） + +```json +{ + "reasoning": { + "effort": "high", + "summary": "auto" + } +} +``` + +| 字段 | 类型 | 默认值 | 说明 | +|------|------|--------|------| +| `effort` | string | `"medium"` | 推理力度约束 | +| `summary` | string | — | 推理摘要设置 | + +**effort 支持的值：** + +| 值 | 说明 | +|----|------| +| `none` | 不执行推理（gpt-5.1 默认） | +| `minimal` | 最小推理 | +| `low` | 低推理 | +| `medium` | 中等推理（gpt-5.1 之前模型的默认值） | +| `high` | 高推理（gpt-5-pro 的默认且唯一支持的值） | +| `xhigh` | 超高推理（gpt-5.1-codex-max 之后模型支持） | + +**summary 支持的值：** + +| 值 | 说明 | +|----|------| +| `"auto"` | 自动生成推理摘要 | +| `"concise"` | 简洁摘要（支持 computer-use-preview 和 gpt-5 之后模型） | +| `"detailed"` | 详细摘要 | + +> 注意：`generate_summary` 参数已弃用，请使用 `summary` 替代。 + +### 2.7 文本输出格式（Text Format） + +通过 `text.format` 参数控制输出格式： + +```json +{ + "text": { + "format": { + "type": "json_schema", + "name": "my_schema", + "schema": {...}, + "strict": true + }, + "verbosity": "medium" + } +} +``` + +**格式类型：** + +| 类型 | 格式对象 | 说明 | +|------|----------|------| +| `text` | `{"type": "text"}` | 默认文本格式（默认值） | +| `json_schema` | 见下方 | 结构化 JSON 输出（推荐） | +| `json_object` | `{"type": "json_object"}` | 旧版 JSON 模式（不推荐用于 gpt-4o 及更新模型） | + +**json_schema 格式对象：** + +| 字段 | 类型 | 必填 | 说明 | +|------|------|------|------| +| `type` | string | 是 | 固定值 `"json_schema"` | +| `name` | string | 是 | 格式名称（a-z, A-Z, 0-9, _-，最长 64 字符） | +| `schema` | object | 是 | JSON Schema 对象 | +| `description` | string | 否 | 格式描述 | +| `strict` | boolean | 否 | 是否启用严格模式，默认 `false` | + +**verbosity（详细程度）：** + +| 值 | 说明 | +|----|------| +| `"low"` | 更简洁 | +| `"medium"` | 中等（默认） | +| `"high"` | 更详细 | + +### 2.8 上下文管理（Context Management） + +```json +{ + "context_management": { + "type": "compaction", + "compact_threshold": 80000 + } +} +``` + +| 字段 | 类型 | 说明 | +|------|------|------| +| `type` | string | 上下文管理条目类型，目前仅支持 `"compaction"` | +| `compact_threshold` | integer | 触发压缩的 Token 阈值 | + +### 2.9 对话管理（Conversation） + +```json +{ + "conversation": { + "id": "conv_abc123" + } +} +``` + +| 字段 | 类型 | 说明 | +|------|------|------| +| `id` | string | 对话的唯一 ID | + +对话对象中的项会预置到此响应请求的 `input_items` 前面。此响应完成后，输入项和输出项会自动添加到此对话中。 + +> 注意：`conversation` 不能与 `previous_response_id` 同时使用。 + +### 2.10 流式选项（Stream Options） + +```json +{ + "stream": true, + "stream_options": { + "include_obfuscation": true + } +} +``` + +| 字段 | 类型 | 默认值 | 说明 | +|------|------|--------|------| +| `include_obfuscation` | boolean | — | 是否启用流混淆。混淆在流式 delta 事件的 `obfuscation` 字段中添加随机字符，标准化载荷大小以缓解侧信道攻击 | + +### 2.11 其他请求参数 + +**prompt — 提示模板引用：** + +```json +{ + "prompt": { + "id": "pmpt_abc123", + "version": "1", + "variables": { + "topic": "AI safety", + "image": {"type": "input_image", "image_url": "..."} + } + } +} +``` + +| 字段 | 类型 | 说明 | +|------|------|------| +| `id` | string | 提示模板 ID | +| `version` | string | 模板版本（可选） | +| `variables` | object | 变量映射，值可以是字符串或其他 Response 输入类型 | + +**service_tier — 服务等级：** + +| 值 | 说明 | +|----|------| +| `"auto"` | 使用项目设置中配置的服务等级（默认） | +| `"default"` | 标准定价和性能 | +| `"flex"` | Flex 服务等级 | +| `"priority"` | Priority 服务等级 | + +**truncation — 截断策略：** + +| 值 | 说明 | +|----|------| +| `"disabled"` | 如果输入超过上下文窗口大小，请求失败返回 400 错误（默认） | +| `"auto"` | 自动截断对话开头的内容以适应上下文窗口 | + +--- + +## 3. 响应对象（Response Object） + +### 3.1 完整响应结构 + +| 字段 | 类型 | 说明 | +|------|------|------| +| `id` | string | 响应唯一标识符，格式 `resp_*` | +| `object` | string | 对象类型，固定值 `"response"` | +| `created_at` | integer | 创建时间（Unix 时间戳，秒） | +| `status` | string | 响应状态 | +| `completed_at` | integer / null | 完成时间（Unix 时间戳，秒），仅 `completed` 状态时存在 | +| `error` | object / null | 错误对象 | +| `incomplete_details` | object / null | 不完整详情 | +| `instructions` | string / null | 系统/开发者消息 | +| `max_output_tokens` | integer / null | 最大输出 Token 数 | +| `max_tool_calls` | integer / null | 最大工具调用次数 | +| `model` | string | 实际使用的模型 ID | +| `output` | array | 输出项数组 | +| `output_text` | string | （仅 SDK）聚合所有 `output_text` 项的文本内容 | +| `parallel_tool_calls` | boolean | 是否允许并行工具调用 | +| `previous_response_id` | string / null | 上一个响应 ID | +| `prompt` | object / null | 提示模板引用 | +| `prompt_cache_key` | string / null | 缓存键 | +| `prompt_cache_retention` | string / null | 缓存保留策略 | +| `reasoning` | object | 推理配置 | +| `safety_identifier` | string / null | 安全标识符 | +| `service_tier` | string / null | 实际使用的服务等级 | +| `store` | boolean | 是否已存储 | +| `temperature` | number / null | 使用的采样温度 | +| `text` | object | 文本格式配置 | +| `tool_choice` | string / object | 工具选择配置 | +| `tools` | array | 可用工具列表 | +| `top_p` | number / null | 使用的 top_p | +| `truncation` | string | 截断策略 | +| `usage` | object / null | Token 使用量统计 | +| `user` | string / null | 用户标识（已弃用） | +| `metadata` | object | 元数据 | +| `background` | boolean / null | 是否后台运行 | +| `conversation` | object / null | 对话对象 | + +### 3.2 响应示例 + +**基本文本响应：** + +```json +{ + "id": "resp_67ccd2bed1ec8190b14f964abc0542670bb6a6b452d3795b", + "object": "response", + "created_at": 1741476542, + "status": "completed", + "completed_at": 1741476543, + "error": null, + "incomplete_details": null, + "instructions": null, + "max_output_tokens": null, + "model": "gpt-4.1-2025-04-14", + "output": [ + { + "type": "message", + "id": "msg_67ccd2bf17f0819081ff3bb2cf6508e60bb6a6b452d3795b", + "status": "completed", + "role": "assistant", + "content": [ + { + "type": "output_text", + "text": "In a peaceful grove beneath a silver moon, a unicorn named Lumina discovered a hidden pool that reflected the stars.", + "annotations": [] + } + ] + } + ], + "parallel_tool_calls": true, + "previous_response_id": null, + "reasoning": { + "effort": null, + "summary": null + }, + "store": true, + "temperature": 1.0, + "text": { + "format": { + "type": "text" + } + }, + "tool_choice": "auto", + "tools": [], + "top_p": 1.0, + "truncation": "disabled", + "usage": { + "input_tokens": 36, + "input_tokens_details": { + "cached_tokens": 0 + }, + "output_tokens": 87, + "output_tokens_details": { + "reasoning_tokens": 0 + }, + "total_tokens": 123 + }, + "user": null, + "metadata": {} +} +``` + +### 3.3 状态值（Status） + +| 状态 | 说明 | +|------|------| +| `completed` | 响应已完成 | +| `failed` | 响应生成失败 | +| `in_progress` | 响应正在生成中 | +| `cancelled` | 响应已被取消 | +| `queued` | 响应排队中（后台模式） | +| `incomplete` | 响应不完整 | + +### 3.4 Usage 统计 + +```json +{ + "usage": { + "input_tokens": 36, + "input_tokens_details": { + "cached_tokens": 0 + }, + "output_tokens": 87, + "output_tokens_details": { + "reasoning_tokens": 0 + }, + "total_tokens": 123 + } +} +``` + +| 字段 | 类型 | 说明 | +|------|------|------| +| `input_tokens` | integer | 输入 Token 数 | +| `input_tokens_details` | object | 输入 Token 明细 | +| `input_tokens_details.cached_tokens` | integer | 缓存命中的 Token 数 | +| `output_tokens` | integer | 输出 Token 数 | +| `output_tokens_details` | object | 输出 Token 明细 | +| `output_tokens_details.reasoning_tokens` | integer | 推理 Token 数 | +| `total_tokens` | integer | 总 Token 数 | + +### 3.5 错误对象 + +```json +{ + "error": { + "code": "rate_limit_exceeded", + "message": "You have exceeded your rate limit." + } +} +``` + +| 字段 | 类型 | 说明 | +|------|------|------| +| `code` | string | 错误码 | +| `message` | string | 人类可读的错误描述 | + +### 3.6 不完整详情 + +```json +{ + "incomplete_details": { + "reason": "max_output_tokens" + } +} +``` + +| 字段 | 类型 | 说明 | +|------|------|------| +| `reason` | string | 不完整的原因，如 `"max_output_tokens"` | + +--- + +## 4. 其他端点 + +### 4.1 获取响应 — GET /v1/responses/{response_id} + +检索具有给定 ID 的模型响应。 + +**路径参数：** + +| 参数 | 类型 | 说明 | +|------|------|------| +| `response_id` | string | 要检索的响应 ID | + +**查询参数：** + +| 参数 | 类型 | 说明 | +|------|------|------| +| `include` | array | 要包含的附加字段 | +| `include_obfuscation` | boolean | 是否启用流混淆 | +| `starting_after` | integer | 从指定序号后开始流式传输 | + +**响应：** 返回完整的 Response 对象。 + +**请求示例：** + +```bash +curl https://api.openai.com/v1/responses/resp_123 \ + -H "Authorization: Bearer $OPENAI_API_KEY" +``` + +### 4.2 删除响应 — DELETE /v1/responses/{response_id} + +删除具有给定 ID 的模型响应。 + +**路径参数：** + +| 参数 | 类型 | 说明 | +|------|------|------| +| `response_id` | string | 要删除的响应 ID | + +**响应：** + +```json +{ + "id": "resp_6786a1bec27481909a17d673315b29f6", + "object": "response", + "deleted": true +} +``` + +### 4.3 取消响应 — POST /v1/responses/{response_id}/cancel + +取消具有给定 ID 的模型响应。仅 `background` 参数设置为 `true` 创建的响应可以取消。 + +**路径参数：** + +| 参数 | 类型 | 说明 | +|------|------|------| +| `response_id` | string | 要取消的响应 ID | + +**响应：** 返回 Response 对象，`status` 字段为 `"cancelled"`。注意取消的响应 `completed_at` 为 `null`，`usage` 为 `null`。 + +**请求示例：** + +```bash +curl -X POST https://api.openai.com/v1/responses/resp_123/cancel \ + -H "Authorization: Bearer $OPENAI_API_KEY" +``` + +### 4.4 压缩对话 — POST /v1/responses/compact + +对对话执行压缩处理。压缩返回加密的不透明项，底层逻辑可能随时间演变。 + +**请求体：** + +| 参数 | 类型 | 必填 | 说明 | +|------|------|------|------| +| `model` | string | 是 | 模型 ID | +| `input` | string / array | 是 | 输入项 | +| `instructions` | string | 否 | 系统消息 | +| `previous_response_id` | string | 否 | 上一个响应 ID | + +**响应 — 压缩响应对象（response.compaction）：** + +```json +{ + "id": "resp_001", + "object": "response.compaction", + "created_at": 1764967971, + "output": [ + { + "id": "msg_000", + "type": "message", + "status": "completed", + "content": [ + { + "type": "input_text", + "text": "Create a simple landing page for a dog petting cafe." + } + ], + "role": "user" + }, + { + "id": "cmp_001", + "type": "compaction", + "encrypted_content": "gAAAAABpM0Yj-...=" + } + ], + "usage": { + "input_tokens": 139, + "input_tokens_details": { + "cached_tokens": 0 + }, + "output_tokens": 438, + "output_tokens_details": { + "reasoning_tokens": 64 + }, + "total_tokens": 577 + } +} +``` + +**压缩响应对象字段：** + +| 字段 | 类型 | 说明 | +|------|------|------| +| `id` | string | 压缩响应唯一标识符 | +| `object` | string | 固定值 `"response.compaction"` | +| `created_at` | integer | 创建时间戳 | +| `output` | array | 压缩后的输出项列表（包含原始消息和 compaction 项） | +| `usage` | object | Token 使用量 | + +**使用方式：** 将 `compactedResponse.output` 作为下一次请求的 `input` 传入。 + +**请求示例：** + +```bash +curl -X POST https://api.openai.com/v1/responses/compact \ + -H "Content-Type: application/json" \ + -H "Authorization: Bearer $OPENAI_API_KEY" \ + -d '{ + "model": "gpt-5.1-codex-max", + "input": [ + {"role": "user", "content": "Create a landing page."}, + {"id": "msg_001", "type": "message", "status": "completed", + "content": [{"type": "output_text", "text": "...", "annotations": []}], + "role": "assistant"} + ] + }' +``` + +### 4.5 列出输入项 — GET /v1/responses/{response_id}/input_items + +返回给定响应的输入项列表。 + +**路径参数：** + +| 参数 | 类型 | 说明 | +|------|------|------| +| `response_id` | string | 响应 ID | + +**查询参数：** + +| 参数 | 类型 | 默认值 | 说明 | +|------|------|--------|------| +| `after` | string | — | 分页游标，列出此 ID 之后的项 | +| `include` | array | — | 要包含的附加字段 | +| `limit` | integer | 20 | 返回对象数量限制（1-100） | +| `order` | string | `"desc"` | 排序顺序：`"asc"` / `"desc"` | + +**响应 — 输入项列表对象：** + +```json +{ + "object": "list", + "data": [ + { + "id": "msg_abc123", + "type": "message", + "role": "user", + "content": [ + { + "type": "input_text", + "text": "Tell me a three sentence bedtime story about a unicorn." + } + ] + } + ], + "first_id": "msg_abc123", + "last_id": "msg_abc123", + "has_more": false +} +``` + +| 字段 | 类型 | 说明 | +|------|------|------| +| `object` | string | 固定值 `"list"` | +| `data` | array | 输入项数组 | +| `first_id` | string | 列表中第一项的 ID | +| `last_id` | string | 列表中最后一项的 ID | +| `has_more` | boolean | 是否有更多项 | + +### 4.6 计算输入 Token — POST /v1/responses/input_tokens + +返回请求的输入 Token 数量，不实际生成响应。 + +**请求体：** 与创建响应相同的参数子集。 + +| 参数 | 类型 | 必填 | 说明 | +|------|------|------|------| +| `model` | string | 是 | 模型 ID | +| `input` | string / array | 是 | 输入项 | +| `instructions` | string | 否 | 系统消息 | +| `tools` | array | 否 | 工具定义 | +| `tool_choice` | string / object | 否 | 工具选择 | +| `previous_response_id` | string | 否 | 上一个响应 ID | +| `conversation` | object | 否 | 对话对象 | +| `parallel_tool_calls` | boolean | 否 | 是否并行工具调用 | +| `truncation` | string | 否 | 截断策略 | + +**响应：** + +```json +{ + "object": "response.input_tokens", + "input_tokens": 11 +} +``` + +| 字段 | 类型 | 说明 | +|------|------|------| +| `object` | string | 固定值 `"response.input_tokens"` | +| `input_tokens` | integer | 输入 Token 数 | + +**请求示例：** + +```bash +curl -X POST https://api.openai.com/v1/responses/input_tokens \ + -H "Content-Type: application/json" \ + -H "Authorization: Bearer $OPENAI_API_KEY" \ + -d '{ + "model": "gpt-4.1", + "input": "Tell me a joke." + }' +``` + +--- + +## 5. 与 Chat Completions API 的对比 + +| 特性 | Chat Completions API | Responses API | +|------|---------------------|---------------| +| 端点 | `POST /v1/chat/completions` | `POST /v1/responses` | +| 输入格式 | `messages` 数组 | `input` 字符串或数组 | +| 输出格式 | `choices` 数组 | `output` 数组 | +| 系统消息 | `messages` 中 `role: "system"` | `instructions` 参数 | +| 多轮对话 | 客户端维护完整消息历史 | `previous_response_id` 或 `conversation` 服务端管理 | +| 工具调用 | `tools` + `tool_choice` | `tools` + `tool_choice`（更多工具类型） | +| 内置工具 | 无 | Web 搜索、文件搜索、代码解释器等 | +| 流式响应 | SSE `data: {...}\n\n` + `data: [DONE]` | SSE 事件流 | +| 推理控制 | 无 | `reasoning.effort` + `reasoning.summary` | +| 结构化输出 | `response_format` | `text.format` | +| 后台模式 | 不支持 | `background: true` | +| 响应存储 | 不支持 | `store: true` + 响应检索 API | +| 上下文管理 | 不支持 | `context_management` + `compact` | +| MCP 集成 | 不支持 | `type: "mcp"` 工具 | +| 自定义工具 | 不支持 | `type: "custom"` 工具 | +| Shell 执行 | 不支持 | `type: "shell"` / `type: "local_shell"` | +| 文件操作 | 不支持 | `type: "apply_patch"` | +| 图像生成 | 不支持 | `type: "image_generation"` | +| 计算机控制 | 不支持 | `type: "computer_use_preview"` | +| Token 计数 | 不支持（需使用单独端点） | `POST /v1/responses/input_tokens` | + +**关键转换映射（协议转换器参考）：** + +| Chat Completions | Responses API | +|------------------|---------------| +| `messages[].role: "system"` | `instructions` | +| `messages[].role: "user"` + `content: "text"` | `{"type": "message", "role": "user", "content": "text"}` 或简写 `{"role": "user", "content": "text"}` | +| `messages[].role: "user"` + `content: [{"type": "text", ...}]` | `{"type": "message", "role": "user", "content": [{"type": "input_text", ...}]}` | +| `messages[].role: "user"` + `content: [{"type": "image_url", ...}]` | `{"type": "message", "role": "user", "content": [{"type": "input_image", ...}]}` | +| `messages[].role: "assistant"` | `{"type": "message", "role": "assistant", ...}` | +| `messages[].role: "tool"` | `{"type": "function_call_output", ...}` | +| `choices[0].message.content` | `output[].content[].text`（type 为 `output_text`） | +| `choices[0].message.tool_calls` | `output[]`（type 为 `function_call`） | +| `finish_reason: "stop"` | `status: "completed"` | +| `finish_reason: "tool_calls"` | `output` 中包含 `function_call` 项 | +| `usage.prompt_tokens` | `usage.input_tokens` | +| `usage.completion_tokens` | `usage.output_tokens` | +| `response_format` | `text.format` | diff --git a/docs/协议文档/OpenAI/OpenAI协议文档.md b/docs/协议文档/OpenAI/OpenAI协议文档.md new file mode 100644 index 0000000..f8c2de3 --- /dev/null +++ b/docs/协议文档/OpenAI/OpenAI协议文档.md @@ -0,0 +1,1072 @@ +# OpenAI Chat Completions API 协议文档 + +> 基于官方 API Reference (https://platform.openai.com/docs/api-reference/chat/create) 整理 +> +> 本文档用于 team-api 协议转换模块的开发参考，涵盖 Chat Completions API 的完整请求/响应规范。 + +--- + +## 目录 + +- [1. API 概览](#1-api-概览) +- [2. 认证方式](#2-认证方式) +- [3. 请求参数](#3-请求参数) +- [4. 消息类型](#4-消息类型) +- [5. 响应格式](#5-响应格式) +- [6. 流式响应 (SSE)](#6-流式响应-sse) +- [7. Tool Calling 工作流](#7-tool-calling-工作流) +- [8. Structured Outputs](#8-structured-outputs) +- [9. 错误格式](#9-错误格式) +- [10. finish_reason 取值说明](#10-finish_reason-取值说明) +- [11. 协议转换注意事项](#11-协议转换注意事项) + +--- + +## 1. API 概览 + +| 项目 | 说明 | +|------|------| +| **端点** | `POST https://api.openai.com/v1/chat/completions` | +| **功能** | 根据给定的对话消息，生成模型响应 | +| **Content-Type** | `application/json` | +| **协议版本** | Chat Completions API（OpenAI 建议新项目使用 Responses API，但 Chat Completions 仍是主流） | + +### 基本请求示例 + +```bash +curl https://api.openai.com/v1/chat/completions \ + -H "Content-Type: application/json" \ + -H "Authorization: Bearer $OPENAI_API_KEY" \ + -d '{ + "model": "gpt-4o", + "messages": [ + {"role": "system", "content": "You are a helpful assistant."}, + {"role": "user", "content": "Hello!"} + ] + }' +``` + +--- + +## 2. 认证方式 + +请求头中携带 Bearer Token： + +``` +Authorization: Bearer sk-xxxxxxxxxxxxxxxx +``` + +| 认证方式 | 说明 | +|---------|------| +| API Key | `Authorization: Bearer sk-xxx`，在平台 API Keys 页面创建 | +| Organization（可选） | `OpenAI-Organization: org-xxx`，指定组织 | +| Project（可选） | `OpenAI-Project: proj-xxx`，指定项目 | + +--- + +## 3. 请求参数 + +### 3.1 完整参数表 + +| 参数名 | 类型 | 必填 | 默认值 | 说明 | +|--------|------|------|--------|------| +| `model` | string | **是** | — | 模型 ID，如 `gpt-4o`、`gpt-4.1`、`o3`、`o4-mini` | +| `messages` | array | **是** | — | 对话消息列表，详见 [第 4 节](#4-消息类型) | +| `max_completion_tokens` | integer | 否 | — | 生成 token 数上限（包含可见输出和推理 token）。推荐使用，替代已弃用的 `max_tokens` | +| `temperature` | number | 否 | `1` | 采样温度，范围 0-2。值越高越随机，越低越确定。建议与 `top_p` 二选一调整 | +| `top_p` | number | 否 | `1` | 核采样概率质量。如 0.1 表示只考虑前 10% 概率的 token。建议与 `temperature` 二选一调整 | +| `n` | integer | 否 | `1` | 为每条输入消息生成的候选响应数量。注意：按所有候选的总 token 数计费 | +| `stream` | boolean | 否 | `false` | 是否以流式（SSE）方式返回响应 | +| `stream_options` | object | 否 | `null` | 流式响应选项，仅在 `stream: true` 时有效。见下方详细说明 | +| `stop` | string / array | 否 | `null` | 最多 4 个停止序列。返回文本不包含停止序列。**注意：不支持最新推理模型 o3、o4-mini** | +| `presence_penalty` | number | 否 | `0` | 存在惩罚，范围 -2.0 到 2.0。正值增加模型谈论新话题的概率 | +| `frequency_penalty` | number | 否 | `0` | 频率惩罚，范围 -2.0 到 2.0。正值降低模型逐字重复相同内容的概率 | +| `logit_bias` | object | 否 | `null` | 修改指定 token 的生成概率。键为 token ID，值为 -100 到 100 的偏置值 | +| `logprobs` | boolean | 否 | `false` | 是否返回输出 token 的对数概率 | +| `top_logprobs` | integer | 否 | — | 每个位置返回的最可能 token 数量（0-20），需 `logprobs: true` | +| `response_format` | object | 否 | — | 指定输出格式，支持 `text`、`json_object`、`json_schema`。详见 [第 8 节](#8-structured-outputs) | +| `seed` | integer | 否 | — | 随机种子。尽可能确定性采样，配合 `system_fingerprint` 监控后端变更 | +| `service_tier` | string | 否 | `auto` | 处理层级：`auto`、`default`、`flex`、`priority` | +| `tools` | array | 否 | — | 工具定义列表。详见 [第 7 节](#7-tool-calling-工作流) | +| `tool_choice` | string / object | 否 | — | 控制工具调用行为。详见 [第 7 节](#7-tool-calling-工作流) | +| `parallel_tool_calls` | boolean | 否 | `true` | 是否允许并行调用多个工具 | +| `reasoning_effort` | string | 否 | `medium` | 推理模型的推理力度：`none`、`minimal`、`low`、`medium`、`high`、`xhigh` | +| `user` | string | 否 | — | **已弃用**。终端用户标识符，建议使用 `prompt_cache_key` 替代 | +| `prompt_cache_key` | string | 否 | — | 用于缓存类似请求的响应，优化缓存命中率。替代 `user` 字段 | +| `prompt_cache_retention` | string | 否 | — | 提示缓存保留策略，可设为 `24h` 启用扩展缓存 | +| `metadata` | object | 否 | — | 附加元数据，最多 16 个键值对。键最长 64 字符，值最长 512 字符 | +| `modalities` | array | 否 | `["text"]` | 输出模态类型，如 `["text"]` 或 `["text", "audio"]` | +| `audio` | object | 否 | — | 音频输出参数。当 `modalities` 包含 `"audio"` 时必填 | +| `store` | boolean | 否 | `false` | 是否存储此请求的输出，用于模型蒸馏和评估产品 | +| `web_search_options` | object | 否 | — | 网页搜索工具选项 | +| `safety_identifier` | string | 否 | — | 用户安全标识符（Beta），用于检测违反使用策略的用户 | +| `verbosity` | string | 否 | `medium` | 控制模型回复的详细程度：`low`、`medium`、`high` | +| `prediction` | object | 否 | — | 预测输出配置，当大部分响应内容已知时可显著提升响应速度 | +| `max_tokens` | integer | 否 | — | **已弃用**。被 `max_completion_tokens` 替代，不兼容 o 系列模型 | +| `function_call` | string / object | 否 | — | **已弃用**。被 `tool_choice` 替代 | +| `functions` | array | 否 | — | **已弃用**。被 `tools` 替代 | + +### 3.2 stream_options 参数 + +| 字段 | 类型 | 说明 | +|------|------|------| +| `include_usage` | boolean | 是否在最后一个 chunk 中包含 usage 统计信息。默认 `false`。仅在 `stream: true` 时有效 | + +示例： + +```json +{ + "stream": true, + "stream_options": { + "include_usage": true + } +} +``` + +### 3.3 audio 参数 + +当 `modalities` 包含 `"audio"` 时，需要指定音频输出参数： + +| 字段 | 类型 | 必填 | 说明 | +|------|------|------|------| +| `voice` | string | 是 | 语音类型：`alloy`、`ash`、`ballad`、`coral`、`echo`、`sage`、`shimmer`、`verse` | +| `format` | string | 是 | 音频格式：`wav`、`mp3`、`flac`、`opus`、`pcm16` | + +示例： + +```json +{ + "modalities": ["text", "audio"], + "audio": { + "voice": "alloy", + "format": "mp3" + } +} +``` + +### 3.4 prediction 参数（Predicted Output） + +当已知大部分响应内容（如重新生成仅做少量修改的文件）时，可使用预测输出加速： + +```json +{ + "prediction": { + "type": "content", + "content": "已知的预期输出内容..." + } +} +``` + +### 3.5 web_search_options 参数 + +```json +{ + "web_search_options": { + "search_context_size": "medium" + } +} +``` + +`search_context_size` 取值：`low`、`medium`、`high`。 + +### 3.6 reasoning_effort 说明 + +| 模型 | 默认值 | 支持的值 | 说明 | +|------|--------|---------|------| +| `gpt-5.1` | `none` | `none`、`low`、`medium`、`high` | `none` 不执行推理；所有级别均支持工具调用 | +| `gpt-5.1` 之前的模型 | `medium` | `minimal`、`low`、`medium`、`high` | 不支持 `none` | +| `gpt-5-pro` | `high` | `high` | 仅支持 `high` | +| `gpt-5.1-codex-max` 之后的模型 | `medium` | 含 `xhigh` | 支持最高 `xhigh` | + +--- + +## 4. 消息类型 + +`messages` 数组中的每条消息都包含 `role` 字段，不同角色对应不同的内容结构。 + +### 4.1 system / developer 消息 + +设定模型的行为和上下文。`developer` 是新版名称（推荐用于新模型如 gpt-4.1+），`system` 是传统名称。 + +**纯文本系统消息：** + +```json +{ + "role": "system", + "content": "You are a helpful assistant that translates English to French." +} +``` + +**新版 developer 消息：** + +```json +{ + "role": "developer", + "content": "You are a helpful assistant." +} +``` + +> 在协议转换时，`system` 和 `developer` 角色通常映射为等价的系统指令。部分供应商只支持 `system`，需要将 `developer` 转换为 `system`。 + +### 4.2 user 消息 + +用户输入消息，支持纯文本和多模态内容。 + +#### 纯文本 + +```json +{ + "role": "user", + "content": "What is the capital of France?" +} +``` + +#### 多模态内容（文本 + 图片） + +```json +{ + "role": "user", + "content": [ + { + "type": "text", + "text": "What's in this image?" + }, + { + "type": "image_url", + "image_url": { + "url": "https://example.com/image.png", + "detail": "high" + } + } + ] +} +``` + +**image_url.detail 取值：** + +| 值 | 说明 | +|---|------| +| `auto` | 自动选择（默认） | +| `low` | 低分辨率，更快更省 token | +| `high` | 高分辨率，更详细但消耗更多 token | + +**image_url.url 支持格式：** +- HTTPS URL：`https://example.com/image.png` +- Base64 Data URI：`data:image/png;base64,iVBORw0KGgo...` + +#### 音频输入 + +```json +{ + "role": "user", + "content": [ + { + "type": "input_audio", + "input_audio": { + "data": "base64编码的音频数据", + "format": "wav" + } + }, + { + "type": "text", + "text": "What is being said in this audio?" + } + ] +} +``` + +**input_audio.format 取值：** `wav`、`mp3`、`flac`、`opus`、`pcm16` + +#### 文件附件 + +```json +{ + "role": "user", + "content": [ + { + "type": "file", + "file": { + "file_id": "file-abc123", + "file_data": "base64编码的文件数据", + "filename": "document.pdf" + } + } + ] +} +``` + +### 4.3 assistant 消息 + +模型生成的响应消息。 + +#### 纯文本响应 + +```json +{ + "role": "assistant", + "content": "The capital of France is Paris." +} +``` + +#### 带工具调用的响应 + +```json +{ + "role": "assistant", + "content": null, + "tool_calls": [ + { + "id": "call_abc123", + "type": "function", + "function": { + "name": "get_weather", + "arguments": "{\"location\": \"Paris\", \"unit\": \"celsius\"}" + } + } + ] +} +``` + +**tool_calls 数组中每个元素的结构：** + +| 字段 | 类型 | 说明 | +|------|------|------| +| `id` | string | 工具调用的唯一 ID，格式为 `call_xxxxx` | +| `type` | string | 固定为 `"function"` | +| `function.name` | string | 要调用的函数名 | +| `function.arguments` | string | JSON 格式的函数参数（**字符串类型**，非对象） | +| `index` | integer | 可选，在流式响应中用于标识调用位置 | + +#### 推理内容（部分推理模型） + +```json +{ + "role": "assistant", + "content": "最终回答内容", + "reasoning_content": "这是模型的思考过程..." +} +``` + +> `reasoning_content` 字段在 DeepSeek、o1 等推理模型的响应中出现，包含模型的内部推理过程。 + +### 4.4 tool 消息 + +将工具执行结果返回给模型。 + +```json +{ + "role": "tool", + "tool_call_id": "call_abc123", + "content": "{\"temperature\": 22, \"unit\": \"celsius\"}" +} +``` + +| 字段 | 类型 | 必填 | 说明 | +|------|------|------|------| +| `role` | string | 是 | 固定为 `"tool"` | +| `tool_call_id` | string | 是 | 对应的 assistant 消息中 `tool_calls[].id` | +| `content` | string | 是 | 工具执行结果，通常是 JSON 字符串 | + +### 4.5 消息角色汇总 + +| role | 发起方 | content 类型 | 特殊字段 | +|------|--------|-------------|---------| +| `system` | 客户端 | string | — | +| `developer` | 客户端 | string | — | +| `user` | 客户端 | string 或 array（多模态） | — | +| `assistant` | 模型 | string 或 null | `tool_calls`、`reasoning_content` | +| `tool` | 客户端 | string | `tool_call_id`（必填） | + +--- + +## 5. 响应格式 + +### 5.1 完整响应结构 + +```json +{ + "id": "chatcmpl-B9MBs8CjcvOU2jLn4n570S5qMJKcT", + "object": "chat.completion", + "created": 1741569952, + "model": "gpt-4o", + "choices": [ + { + "index": 0, + "message": { + "role": "assistant", + "content": "Hello! How can I assist you today?", + "refusal": null, + "annotations": [] + }, + "logprobs": null, + "finish_reason": "stop" + } + ], + "usage": { + "prompt_tokens": 19, + "completion_tokens": 10, + "total_tokens": 29, + "prompt_tokens_details": { + "cached_tokens": 0, + "audio_tokens": 0 + }, + "completion_tokens_details": { + "reasoning_tokens": 0, + "audio_tokens": 0, + "accepted_prediction_tokens": 0, + "rejected_prediction_tokens": 0 + } + }, + "service_tier": "default", + "system_fingerprint": "fp_fc9f1d7035" +} +``` + +### 5.2 响应字段说明 + +| 字段 | 类型 | 说明 | +|------|------|------| +| `id` | string | 完成请求的唯一标识符，格式 `chatcmpl-xxx` | +| `object` | string | 对象类型，固定为 `"chat.completion"` | +| `created` | integer | 创建时间（Unix 时间戳，秒） | +| `model` | string | 实际使用的模型 ID（可能与请求中的 `model` 不同） | +| `choices` | array | 响应选择项数组，见下方 | +| `usage` | object | Token 使用量统计，见下方 | +| `service_tier` | string | 实际使用的处理层级：`default`、`flex`、`priority` | +| `system_fingerprint` | string | **已弃用**。后端配置指纹，配合 `seed` 用于监控后端变更 | + +### 5.3 choice 对象 + +| 字段 | 类型 | 说明 | +|------|------|------| +| `index` | integer | 选择项索引（当 `n > 1` 时有用） | +| `message` | object | 模型生成的消息对象 | +| `message.role` | string | 固定为 `"assistant"` | +| `message.content` | string / null | 模型生成的文本内容，当模型调用工具时可能为 `null` | +| `message.refusal` | string / null | 模型拒绝生成时的拒绝原因（仅当使用 structured outputs 时） | +| `message.annotations` | array | 注解列表（如 URL 引用来源等） | +| `message.tool_calls` | array / null | 工具调用列表，见 [第 7 节](#7-tool-calling-工作流) | +| `logprobs` | object / null | 对数概率信息（当 `logprobs: true` 时） | +| `finish_reason` | string | 完成原因，见 [第 10 节](#10-finish_reason-取值说明) | + +### 5.4 usage 对象 + +| 字段 | 类型 | 说明 | +|------|------|------| +| `prompt_tokens` | integer | 输入 token 总数 | +| `completion_tokens` | integer | 输出 token 总数 | +| `total_tokens` | integer | token 总数 | +| `prompt_tokens_details` | object | 输入 token 细分（可选） | +| `prompt_tokens_details.cached_tokens` | integer | 缓存命中的 token 数 | +| `prompt_tokens_details.audio_tokens` | integer | 音频输入 token 数 | +| `completion_tokens_details` | object | 输出 token 细分（可选） | +| `completion_tokens_details.reasoning_tokens` | integer | 推理 token 数（o 系列模型） | +| `completion_tokens_details.audio_tokens` | integer | 音频输出 token 数 | +| `completion_tokens_details.accepted_prediction_tokens` | integer | 被 `prediction` 参数接受的预测 token 数 | +| `completion_tokens_details.rejected_prediction_tokens` | integer | 被 `prediction` 参数拒绝的预测 token 数 | + +### 5.5 带工具调用的响应示例 + +```json +{ + "id": "chatcmpl-abc123", + "object": "chat.completion", + "created": 1741569952, + "model": "gpt-4o", + "choices": [ + { + "index": 0, + "message": { + "role": "assistant", + "content": null, + "tool_calls": [ + { + "id": "call_abc123", + "type": "function", + "function": { + "name": "get_weather", + "arguments": "{\"location\": \"Paris\", \"unit\": \"celsius\"}" + } + } + ] + }, + "logprobs": null, + "finish_reason": "tool_calls" + } + ], + "usage": { + "prompt_tokens": 82, + "completion_tokens": 17, + "total_tokens": 99 + } +} +``` + +--- + +## 6. 流式响应 (SSE) + +### 6.1 基本格式 + +当 `stream: true` 时，响应使用 Server-Sent Events (SSE) 格式： + +- **Content-Type**: `text/event-stream` +- 每个 chunk 格式：`data: {JSON}\n\n` +- 流结束标记：`data: [DONE]\n\n` +- 每个chunk的 `object` 类型为 `"chat.completion.chunk"` + +### 6.2 完整流式示例 + +``` +data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1741569952,"model":"gpt-4o","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]} + +data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1741569952,"model":"gpt-4o","choices":[{"index":0,"delta":{"content":"Hello"},"finish_reason":null}]} + +data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1741569952,"model":"gpt-4o","choices":[{"index":0,"delta":{"content":"!"},"finish_reason":null}]} + +data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1741569952,"model":"gpt-4o","choices":[{"index":0,"delta":{"content":" How"},"finish_reason":null}]} + +data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1741569952,"model":"gpt-4o","choices":[{"index":0,"delta":{"content":" can"},"finish_reason":null}]} + +data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1741569952,"model":"gpt-4o","choices":[{"index":0,"delta":{"content":" I"},"finish_reason":null}]} + +data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1741569952,"model":"gpt-4o","choices":[{"index":0,"delta":{"content":" help"},"finish_reason":null}]} + +data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1741569952,"model":"gpt-4o","choices":[{"index":0,"delta":{"content":" you"},"finish_reason":null}]} + +data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1741569952,"model":"gpt-4o","choices":[{"index":0,"delta":{"content":"?"},"finish_reason":null}]} + +data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1741569952,"model":"gpt-4o","choices":[{"index":0,"delta":{},"finish_reason":"stop"}],"usage":{"prompt_tokens":19,"completion_tokens":10,"total_tokens":29,"prompt_tokens_details":{"cached_tokens":0,"audio_tokens":0},"completion_tokens_details":{"reasoning_tokens":0,"audio_tokens":0,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}}} + +data: [DONE] +``` + +### 6.3 流式 chunk 结构 + +| 字段 | 类型 | 说明 | +|------|------|------| +| `id` | string | 与非流式响应相同的 completion ID | +| `object` | string | 固定为 `"chat.completion.chunk"` | +| `created` | integer | Unix 时间戳 | +| `model` | string | 实际使用的模型 | +| `choices` | array | 流式选择项数组 | +| `choices[].index` | integer | 选择项索引 | +| `choices[].delta` | object | 增量内容 | +| `choices[].delta.role` | string | 仅在第一个 chunk 中出现 | +| `choices[].delta.content` | string | 增量文本内容 | +| `choices[].delta.tool_calls` | array | 增量工具调用（流式工具调用场景） | +| `choices[].finish_reason` | string / null | 结束原因，最后一个 chunk 非 null | +| `choices[].logprobs` | object / null | 对数概率 | +| `usage` | object / null | 仅在最后一个 chunk 中出现（需 `stream_options.include_usage: true`） | +| `system_fingerprint` | string | 后端指纹 | + +### 6.4 流式中的 tool_calls + +当模型在流式响应中调用工具时，`delta.tool_calls` 增量拼接： + +``` +data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","created":1741569952,"model":"gpt-4o","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"id":"call_abc123","type":"function","function":{"name":"get_weather","arguments":""}}]},"finish_reason":null}]} + +data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","created":1741569952,"model":"gpt-4o","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"{\"lo"}}]},"finish_reason":null}]} + +data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","created":1741569952,"model":"gpt-4o","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"cation"}}]},"finish_reason":null}]} + +data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","created":1741569952,"model":"gpt-4o","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"\": \"Paris\"}"}}]},"finish_reason":null}]} + +data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","created":1741569952,"model":"gpt-4o","choices":[{"index":0,"delta":{},"finish_reason":"tool_calls"}]} +``` + +**关键点**： +- 第一个 chunk 包含 `id`、`type`、`function.name` +- 后续 chunk 通过 `function.arguments` 增量拼接参数字符串 +- `index` 字段用于标识多个并行工具调用的位置 +- 最后一个 chunk 的 `finish_reason` 为 `"tool_calls"` + +### 6.5 include_usage 的使用 + +当设置 `stream_options.include_usage: true` 时，在 `data: [DONE]` 之前会发送一个包含完整 usage 的 chunk： + +```json +{ + "id": "chatcmpl-abc123", + "object": "chat.completion.chunk", + "created": 1741569952, + "model": "gpt-4o", + "choices": [], + "usage": { + "prompt_tokens": 19, + "completion_tokens": 10, + "total_tokens": 29, + "prompt_tokens_details": { + "cached_tokens": 0, + "audio_tokens": 0 + }, + "completion_tokens_details": { + "reasoning_tokens": 0, + "audio_tokens": 0, + "accepted_prediction_tokens": 0, + "rejected_prediction_tokens": 0 + } + } +} +``` + +> 此 chunk 的 `choices` 为空数组，仅用于传递 usage 信息。 + +--- + +## 7. Tool Calling 工作流 + +### 7.1 工具定义格式 + +```json +{ + "tools": [ + { + "type": "function", + "function": { + "name": "get_weather", + "description": "Get current weather for a location", + "parameters": { + "type": "object", + "properties": { + "location": { + "type": "string", + "description": "City name, e.g. Paris" + }, + "unit": { + "type": "string", + "enum": ["celsius", "fahrenheit"] + } + }, + "required": ["location"] + }, + "strict": true + } + } + ] +} +``` + +**function 对象字段：** + +| 字段 | 类型 | 必填 | 说明 | +|------|------|------|------| +| `name` | string | 是 | 函数名称 | +| `description` | string | 否 | 函数描述，帮助模型决定何时调用 | +| `parameters` | object | 否 | JSON Schema 格式的参数定义 | +| `strict` | boolean | 否 | 是否启用严格模式（确保参数严格匹配 schema）。仅 `json_schema` 模式下有效 | + +### 7.2 tool_choice 取值 + +| 值 | 说明 | +|---|------| +| `"none"` | 模型不调用任何工具，始终生成消息（无工具时的默认值） | +| `"auto"` | 模型自行决定生成消息或调用工具（有工具时的默认值） | +| `"required"` | 模型必须调用一个或多个工具 | +| `{"type": "function", "function": {"name": "my_func"}}` | 强制调用指定函数 | + +### 7.3 完整工具调用流程 + +``` +步骤 1: 客户端发送带工具定义的请求 +──────────────────────────────────────── +POST /v1/chat/completions +{ + "model": "gpt-4o", + "messages": [ + {"role": "user", "content": "What's the weather in Paris?"} + ], + "tools": [ + { + "type": "function", + "function": { + "name": "get_weather", + "description": "Get weather for a location", + "parameters": { + "type": "object", + "properties": { + "location": {"type": "string"}, + "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]} + }, + "required": ["location"] + } + } + } + ] +} + +步骤 2: 模型返回工具调用请求 +──────────────────────────────────────── +{ + "choices": [{ + "message": { + "role": "assistant", + "content": null, + "tool_calls": [{ + "id": "call_abc123", + "type": "function", + "function": { + "name": "get_weather", + "arguments": "{\"location\":\"Paris\",\"unit\":\"celsius\"}" + } + }] + }, + "finish_reason": "tool_calls" + }] +} + +步骤 3: 客户端执行工具，将结果追加到消息列表 +──────────────────────────────────────── +POST /v1/chat/completions +{ + "model": "gpt-4o", + "messages": [ + {"role": "user", "content": "What's the weather in Paris?"}, + {"role": "assistant", "content": null, "tool_calls": [ + { + "id": "call_abc123", + "type": "function", + "function": { + "name": "get_weather", + "arguments": "{\"location\":\"Paris\",\"unit\":\"celsius\"}" + } + } + ]}, + {"role": "tool", "tool_call_id": "call_abc123", "content": "{\"temperature\": 22, \"condition\": \"sunny\"}"} + ], + "tools": [ ... 同上 ... ] +} + +步骤 4: 模型根据工具结果生成最终响应 +──────────────────────────────────────── +{ + "choices": [{ + "message": { + "role": "assistant", + "content": "The current weather in Paris is sunny with a temperature of 22 degrees Celsius." + }, + "finish_reason": "stop" + }] +} +``` + +### 7.4 并行工具调用 + +当 `parallel_tool_calls: true`（默认）时，模型可能在一个响应中调用多个工具： + +```json +{ + "role": "assistant", + "content": null, + "tool_calls": [ + { + "id": "call_001", + "type": "function", + "function": {"name": "get_weather", "arguments": "{\"location\":\"Paris\"}"} + }, + { + "id": "call_002", + "type": "function", + "function": {"name": "get_weather", "arguments": "{\"location\":\"Tokyo\"}"} + } + ] +} +``` + +客户端需要为每个工具调用分别返回结果： + +```json +[ + {"role": "tool", "tool_call_id": "call_001", "content": "{\"temperature\": 22}"}, + {"role": "tool", "tool_call_id": "call_002", "content": "{\"temperature\": 18}"} +] +``` + +--- + +## 8. Structured Outputs + +### 8.1 response_format 取值 + +#### 纯文本模式（默认） + +```json +{ + "response_format": { + "type": "text" + } +} +``` + +#### JSON 模式（旧版） + +```json +{ + "response_format": { + "type": "json_object" + } +} +``` + +> 确保 `messages` 中包含要求输出 JSON 的指令，否则模型可能输出空白。 + +#### JSON Schema 模式（推荐，Structured Outputs） + +```json +{ + "response_format": { + "type": "json_schema", + "json_schema": { + "name": "weather_response", + "strict": true, + "schema": { + "type": "object", + "properties": { + "location": {"type": "string"}, + "temperature": {"type": "number"}, + "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]} + }, + "required": ["location", "temperature", "unit"], + "additionalProperties": false + } + } + } +} +``` + +**json_schema 对象字段：** + +| 字段 | 类型 | 必填 | 说明 | +|------|------|------|------| +| `name` | string | 是 | Schema 名称 | +| `strict` | boolean | 否 | 设为 `true` 启用严格模式（推荐） | +| `schema` | object | 是 | JSON Schema 定义 | +| `description` | string | 否 | Schema 描述 | + +**严格模式要求：** +- `additionalProperties` 必须设为 `false` +- 所有字段必须在 `required` 中列出（如需可选，使用单元素 `anyOf`） +- 嵌套对象也必须遵循以上规则 + +### 8.2 拒绝行为 + +当使用 Structured Outputs 时，如果模型拒绝生成，`message.refusal` 字段非 null： + +```json +{ + "message": { + "role": "assistant", + "content": null, + "refusal": "I'm sorry, but I cannot assist with that request." + } +} +``` + +--- + +## 9. 错误格式 + +### 9.1 错误响应结构 + +```json +{ + "error": { + "message": "Incorrect API key provided: sk-xxxx...xxxx.", + "type": "invalid_request_error", + "param": null, + "code": "invalid_api_key" + } +} +``` + +| 字段 | 类型 | 说明 | +|------|------|------| +| `error.message` | string | 人类可读的错误描述 | +| `error.type` | string | 错误类型 | +| `error.param` | string / null | 导致错误的参数名 | +| `error.code` | string / null | 错误码 | + +### 9.2 错误类型映射 + +| error.type | HTTP 状态码 | 说明 | +|-----------|------------|------| +| `invalid_request_error` | 400 | 请求参数错误、模型不存在 | +| `authentication_error` | 401 | API Key 无效、缺失 | +| `permission_error` | 403 | 无权访问该模型或资源 | +| `insufficient_quota` | 402 | 余额不足、额度耗尽 | +| `rate_limit_error` | 429 | 请求频率超限 | +| `server_error` | 500 / 503 | 服务器内部错误、服务不可用 | + +### 9.3 常见错误示例 + +**认证错误（401）：** + +```json +{ + "error": { + "message": "Incorrect API key provided.", + "type": "authentication_error", + "param": null, + "code": "invalid_api_key" + } +} +``` + +**频率限制（429）：** + +```json +{ + "error": { + "message": "You exceeded your current quota, please check your plan and billing details.", + "type": "rate_limit_error", + "param": null, + "code": "insufficient_quota" + } +} +``` + +**请求参数错误（400）：** + +```json +{ + "error": { + "message": "Unsupported value: 'messages[0].role' does not support 'foo' value.", + "type": "invalid_request_error", + "param": "messages[0].role", + "code": null + } +} +``` + +**模型不存在（404）：** + +```json +{ + "error": { + "message": "The model `gpt-5-nonexistent` does not exist.", + "type": "invalid_request_error", + "param": null, + "code": "model_not_found" + } +} +``` + +**内容过滤：** + +```json +{ + "error": { + "message": "The content was filtered due to policy violations.", + "type": "invalid_request_error", + "param": null, + "code": "content_filter" + } +} +``` + +--- + +## 10. finish_reason 取值说明 + +| 值 | 说明 | +|---|------| +| `stop` | 模型自然结束输出或命中了 `stop` 序列 | +| `length` | 输出达到 `max_completion_tokens` / `max_tokens` 上限而被截断 | +| `tool_calls` | 模型决定调用一个或多个工具 | +| `content_filter` | 输出因内容过滤策略被截断 | + +--- + +## 11. 协议转换注意事项 + +> 本节针对 team-api 的协议转换模块开发，总结在实现 OpenAI 与其他供应商协议互转时需要注意的要点。 + +### 11.1 请求参数映射 + +| OpenAI 参数 | Claude Messages | Gemini | 注意事项 | +|------------|----------------|--------|---------| +| `model` | `model` | `model` | 需做模型名称映射 | +| `messages` | `messages` | `contents` | 角色和内容结构完全不同 | +| `max_completion_tokens` | `max_tokens` | `generationConfig.maxOutputTokens` | 字段名不同 | +| `temperature` | `temperature` | `generationConfig.temperature` | 语义相同 | +| `top_p` | `top_p` | `generationConfig.topP` | 语义相同 | +| `stream` | `stream` | `stream` | 流式协议格式不同 | +| `tools` | `tools` | `tools` | 结构差异较大 | +| `tool_choice` | `tool_choice` | `tool_config` | 值的格式不同 | +| `stop` | `stop_sequences` | `generationConfig.stopSequences` | 数组格式 | +| `response_format` | N/A | `responseMimeType` | Claude 不支持 JSON Schema | +| `reasoning_effort` | `thinking.budget_tokens` | N/A | 推理控制方式完全不同 | +| `presence_penalty` | N/A | N/A | Claude/Gemini 不支持 | +| `frequency_penalty` | N/A | N/A | Claude/Gemini 不支持 | +| `logprobs` | N/A | N/A | Claude/Gemini 不支持 | + +### 11.2 消息角色映射 + +| OpenAI role | Claude role | Gemini role | +|------------|------------|-------------| +| `system` | `system` | `systemInstruction` | +| `developer` | `system` | `systemInstruction` | +| `user` | `user` | `user` | +| `assistant` | `assistant` | `model` | +| `tool` | `tool_result` | `functionCall` / `functionResponse` | + +### 11.3 usage 字段映射 + +| OpenAI Chat Completions | OpenAI Responses API | 说明 | +|------------------------|---------------------|------| +| `prompt_tokens` | `input_tokens` | 字段名不同 | +| `completion_tokens` | `output_tokens` | 字段名不同 | +| `total_tokens` | `total_tokens` | 相同 | +| `prompt_tokens_details` | `input_tokens_details` | 字段名不同 | +| `completion_tokens_details` | `output_tokens_details` | 字段名不同 | + +### 11.4 流式协议差异 + +| 特性 | OpenAI | Claude | +|------|--------|--------| +| Content-Type | `text/event-stream` | `text/event-stream` | +| 数据格式 | `data: {JSON}\n\n` | `event: xxx\ndata: {JSON}\n\n` | +| 结束标记 | `data: [DONE]` | `event: message_stop` | +| 内容字段 | `choices[].delta.content` | `delta.text` | +| 事件类型 | 无 event 行 | `message_start`、`content_block_start`、`content_block_delta`、`message_delta`、`message_stop` | + +### 11.5 工具调用差异 + +| 特性 | OpenAI | Claude | +|------|--------|--------| +| 工具定义位置 | `tools` 数组 | `tools` 数组（结构不同） | +| 调用结果返回 | `role: "tool"` 消息 | `role: "user"` 消息内嵌 `tool_result` content block | +| 调用 ID 字段 | `tool_call_id`（消息级） | `tool_use_id`（content block 级） | +| 并行调用 | `parallel_tool_calls` 参数控制 | 天然支持多个 tool_use content block | + +### 11.6 关键实现提醒 + +1. **`function.arguments` 是字符串**：OpenAI 的 `tool_calls[].function.arguments` 是 JSON 字符串而非 JSON 对象，解析时需要先 `json.Unmarshal`。 + +2. **`content` 可以是字符串或数组**：user 消息的 `content` 字段有两种形态 —— 纯文本时为 `string`，多模态时为 `[]ContentPart` 数组。协议转换时需要判断类型。 + +3. **流式 tool_calls 增量拼接**：流式响应中的 tool_calls 是增量发送的，需要按 `index` 拼接 `function.arguments` 字符串。 + +4. **`system` 与 `developer` 角色**：新模型（gpt-4.1+）推荐使用 `developer`，旧模型使用 `system`。转换到其他供应商时统一映射为系统指令。 + +5. **`max_tokens` 与 `max_completion_tokens`**：`max_tokens` 已弃用但仍有大量客户端使用，实现时需兼容两个字段，优先使用 `max_completion_tokens`。 + +6. **推理模型的差异**：o 系列推理模型不支持 `temperature`、`top_p`、`presence_penalty`、`frequency_penalty`、`logprobs`、`logit_bias` 等参数，转换时需过滤。 + +7. **`stop` 参数的限制**：o3 和 o4-mini 不支持 `stop` 参数，转发时需忽略。 + +8. **`stream_options.include_usage`**：部分客户端依赖此选项获取流式响应的 usage 信息，实现时需要确保最后一个 chunk 包含 usage。 + +--- + +## 附录：项目代码参考 + +本项目中 OpenAI 协议相关的数据结构定义在以下文件中： + +| 文件 | 内容 | +|------|------| +| `relay/dto/openai.go` | Chat Completions 请求/响应 DTO（`GeneralOpenAIRequest`、`ChatCompletionResponse` 等） | +| `relay/dto/openai_responses.go` | Responses API 请求/响应 DTO | +| `relay/dto/usage.go` | Token 使用量 DTO、文本补全/嵌入/图像生成 DTO | +| `relay/channel/openai/` | OpenAI 适配器实现 | diff --git a/docs/协议文档/OpenAI/OpenAI图像生成文档.md b/docs/协议文档/OpenAI/OpenAI图像生成文档.md new file mode 100644 index 0000000..4ff06e9 --- /dev/null +++ b/docs/协议文档/OpenAI/OpenAI图像生成文档.md @@ -0,0 +1,637 @@ +# 图像生成 API 协议文档 + +> 基于 OpenAI Images API 官方文档 (https://platform.openai.com/docs/api-reference/images) 整理 +> +> 本文档用于 team-api 协议转换模块的开发参考，涵盖图像生成（Generations）、图像编辑（Edits）、图像变体（Variations）三个端点的完整请求/响应规范，以及各供应商适配器的协议差异。 + +--- + +## 目录 + +- [1. API 概览](#1-api-概览) +- [2. 认证方式](#2-认证方式) +- [3. 图像生成 (Create Image)](#3-图像生成-create-image) +- [4. 图像编辑 (Edit Image)](#4-图像编辑-edit-image) +- [5. 图像变体 (Create Image Variation)](#5-图像变体-create-image-variation) +- [6. 响应格式](#6-响应格式) +- [7. 流式响应 (GPT Image 模型)](#7-流式响应-gpt-image-模型) +- [8. 各模型参数差异对比](#8-各模型参数差异对比) +- [9. 错误格式](#9-错误格式) +- [10. 供应商适配器协议差异](#10-供应商适配器协议差异) +- [11. 协议转换注意事项](#11-协议转换注意事项) + +--- + +## 1. API 概览 + +OpenAI 图像 API 提供三个端点，支持 DALL-E 系列和 GPT Image 系列模型： + +| 端点 | 方法 | 路径 | 功能 | 支持模型 | +|------|------|------|------|----------| +| 图像生成 | POST | `/v1/images/generations` | 根据文本提示生成图像 | `dall-e-2`, `dall-e-3`, `gpt-image-1`, `gpt-image-1-mini`, `gpt-image-1.5` | +| 图像编辑 | POST | `/v1/images/edits` | 编辑现有图像（支持多图） | `dall-e-2`, `gpt-image-1` | +| 图像变体 | POST | `/v1/images/variations` | 基于现有图像生成变体 | `dall-e-2` | + +### 基本请求示例（图像生成） + +```bash +curl https://api.openai.com/v1/images/generations \ + -H "Content-Type: application/json" \ + -H "Authorization: Bearer $OPENAI_API_KEY" \ + -d '{ + "model": "gpt-image-1", + "prompt": "A cute baby sea otter", + "n": 1, + "size": "1024x1024" + }' +``` + +### 基本请求示例（图像编辑，multipart/form-data） + +```bash +curl https://api.openai.com/v1/images/edits \ + -H "Authorization: Bearer $OPENAI_API_KEY" \ + -F image="@otter.png" \ + -F prompt="A cute baby sea otter wearing a beret" \ + -F model="gpt-image-1" +``` + +--- + +## 2. 认证方式 + +请求头中携带 Bearer Token，与 Chat Completions API 相同： + +``` +Authorization: Bearer sk-xxxxxxxxxxxxxxxx +``` + +| 认证方式 | 说明 | +|---------|------| +| API Key | `Authorization: Bearer sk-xxx`，在平台 API Keys 页面创建 | +| Organization（可选） | `OpenAI-Organization: org-xxx`，指定组织 | +| Project（可选） | `OpenAI-Project: proj-xxx`，指定项目 | + +--- + +## 3. 图像生成 (Create Image) + +**端点**：`POST /v1/images/generations` + +**Content-Type**：`application/json` + +### 3.1 完整参数表 + +| 参数名 | 类型 | 必填 | 默认值 | 适用模型 | 说明 | +|--------|------|------|--------|----------|------| +| `prompt` | string | **是** | — | 全部 | 图像描述文本。GPT Image 最长 32000 字符，dall-e-2 最长 1000 字符，dall-e-3 最长 4000 字符 | +| `model` | string | **是** | `dall-e-2` | — | 模型 ID：`dall-e-2`、`dall-e-3`、`gpt-image-1`、`gpt-image-1-mini`、`gpt-image-1.5` | +| `n` | integer | 否 | `1` | 全部 | 生成图像数量。dall-e-2/dall-e-3: 1；GPT Image: 1-10 | +| `size` | string | 否 | `1024x1024` | 全部 | 图像尺寸，各模型支持值不同，详见下方 | +| `quality` | string | 否 | `auto` | GPT Image; dall-e-3 | 图像质量。GPT Image: `auto`/`high`/`medium`/`low`；dall-e-3: `standard`/`hd` | +| `response_format` | string | 否 | `url` | 全部 | 返回格式：`url` 或 `b64_json`。GPT Image 默认 `b64_json` | +| `background` | string | 否 | `auto` | GPT Image | 背景透明度：`transparent`、`opaque`、`auto`。仅支持 PNG/WebP 输出格式 | +| `output_format` | string | 否 | `png` | GPT Image | 输出图像格式：`png`、`jpeg`、`webp` | +| `output_compression` | integer | 否 | — | GPT Image | 输出压缩率（0-100），仅 `jpeg` 和 `webp` 格式有效。100 = 无压缩 | +| `partial_images` | integer | 否 | `0` | GPT Image | 流式中间图数量（0-3）。设为非 0 时自动启用流式模式 | +| `stream` | boolean | 否 | `false` | GPT Image | 是否启用流式响应。也可通过 `partial_images > 0` 隐式启用 | +| `style` | string | 否 | `vivid` | dall-e-3 | 图像风格：`vivid`（生动）或 `natural`（自然） | +| `moderation` | string | 否 | `auto` | GPT Image | 内容审核严格度：`auto` 或 `low` | +| `user` | string | 否 | — | 全部 | 终端用户标识符，用于监控滥用 | + +### 3.2 各模型支持的尺寸 + +**dall-e-2**： + +| 尺寸 | +|------| +| `256x256` | +| `512x512` | +| `1024x1024` | + +**dall-e-3**： + +| 尺寸 | +|------| +| `1024x1024` | +| `1024x1792`（纵向） | +| `1792x1024`（横向） | + +**gpt-image-1 / gpt-image-1-mini / gpt-image-1.5**： + +| 尺寸 | +|------| +| `1024x1024`（正方形） | +| `1536x1024`（横向） | +| `1024x1536`（纵向） | + +### 3.3 请求示例 + +**DALL-E 3 高质量生成**： + +```json +{ + "model": "dall-e-3", + "prompt": "A white Siamese cat wearing a tiny top hat, digital art", + "n": 1, + "size": "1792x1024", + "quality": "hd", + "style": "vivid", + "response_format": "url" +} +``` + +**GPT Image 透明背景 PNG**： + +```json +{ + "model": "gpt-image-1", + "prompt": "A red logo of a rocket ship, minimalist, clean design", + "n": 1, + "size": "1024x1024", + "quality": "high", + "background": "transparent", + "output_format": "png" +} +``` + +**GPT Image 流式生成（含中间图）**： + +```json +{ + "model": "gpt-image-1", + "prompt": "A sunset over a mountain lake", + "n": 1, + "size": "1024x1024", + "partial_images": 2 +} +``` + +--- + +## 4. 图像编辑 (Edit Image) + +**端点**：`POST /v1/images/edits` + +**Content-Type**：`multipart/form-data` 或 `application/json`（GPT Image 模型支持 JSON） + +### 4.1 完整参数表 + +| 参数名 | 类型 | 必填 | 默认值 | 适用模型 | 说明 | +|--------|------|------|--------|----------|------| +| `image` | file / array | **是** | — | 全部 | 源图像文件。dall-e-2: 单个 PNG（正方形，<4MB）；GPT Image: 最多 16 张图（PNG/JPEG/WebP，每张 <20MB） | +| `prompt` | string | **是** | — | 全部 | 编辑指令文本 | +| `model` | string | 否 | `dall-e-2` | — | 模型 ID：`dall-e-2` 或 `gpt-image-1` | +| `mask` | file | 否 | — | dall-e-2 | 遮罩图像（PNG，与源图同尺寸）。透明区域表示需要编辑的部分 | +| `n` | integer | 否 | `1` | 全部 | 生成图像数量。dall-e-2: 1；GPT Image: 1-10 | +| `size` | string | 否 | `1024x1024` | 全部 | 输出尺寸。dall-e-2: 256x256/512x512/1024x1024；GPT Image: 同生成端点 | +| `response_format` | string | 否 | `url` | 全部 | 返回格式：`url` 或 `b64_json` | +| `quality` | string | 否 | `auto` | GPT Image | 图像质量：`auto`/`high`/`medium`/`low` | +| `background` | string | 否 | `auto` | GPT Image | 背景透明度：`transparent`/`opaque`/`auto` | +| `output_format` | string | 否 | `png` | GPT Image | 输出格式：`png`/`jpeg`/`webp` | +| `output_compression` | integer | 否 | — | GPT Image | 压缩率 0-100 | +| `partial_images` | integer | 否 | `0` | GPT Image | 流式中间图数量 0-3 | +| `stream` | boolean | 否 | `false` | GPT Image | 是否流式 | +| `user` | string | 否 | — | 全部 | 终端用户标识 | + +### 4.2 GPT Image 的 JSON 请求格式 + +GPT Image 模型支持以 JSON 传递图像 URL 或 base64 数据（而非 multipart/form-data）： + +```json +{ + "model": "gpt-image-1", + "prompt": "Add a red bow tie to the cat", + "image": [ + { + "type": "image_url", + "url": "https://example.com/cat.png" + } + ], + "n": 1, + "size": "1024x1024" +} +``` + +或使用 base64 内联： + +```json +{ + "model": "gpt-image-1", + "prompt": "Add a red bow tie to the cat", + "image": [ + { + "type": "image_url", + "url": "data:image/png;base64,iVBORw0KGgo..." + } + ] +} +``` + +### 4.3 DALL-E 2 的遮罩编辑 + +DALL-E 2 的编辑使用 mask 参数标记需要重新生成的区域。mask 必须满足： +- 格式为 PNG +- 尺寸与源图相同 +- 透明（alpha = 0）区域表示需要编辑的部分 +- 非透明区域保留原样 + +```bash +curl https://api.openai.com/v1/images/edits \ + -H "Authorization: Bearer $OPENAI_API_KEY" \ + -F image="@original.png" \ + -F mask="@mask.png" \ + -F prompt="Replace the background with a beach" \ + -F model="dall-e-2" +``` + +--- + +## 5. 图像变体 (Create Image Variation) + +**端点**：`POST /v1/images/variations` + +**Content-Type**：`multipart/form-data` + +**仅支持 `dall-e-2` 模型**。 + +### 5.1 完整参数表 + +| 参数名 | 类型 | 必填 | 默认值 | 说明 | +|--------|------|------|--------|------| +| `image` | file | **是** | — | 源图像（PNG，正方形，<4MB） | +| `model` | string | 否 | `dall-e-2` | 必须为 `dall-e-2` | +| `n` | integer | 否 | `1` | 生成变体数量（1-10） | +| `size` | string | 否 | `1024x1024` | 输出尺寸：`256x256`/`512x512`/`1024x1024` | +| `response_format` | string | 否 | `url` | 返回格式：`url` 或 `b64_json` | +| `user` | string | 否 | — | 终端用户标识 | + +### 5.2 请求示例 + +```bash +curl https://api.openai.com/v1/images/variations \ + -H "Authorization: Bearer $OPENAI_API_KEY" \ + -F image="@otter.png" \ + -F n=2 \ + -F size="1024x1024" +``` + +--- + +## 6. 响应格式 + +### 6.1 非流式响应（所有模型通用） + +```json +{ + "created": 1706734328, + "data": [ + { + "url": "https://oaidalleapiprodscus.blob.core.windows.net/private/...", + "revised_prompt": "A white Siamese cat wearing a tiny top hat, rendered in digital art style with vibrant colors" + } + ] +} +``` + +或 base64 格式（`response_format: "b64_json"`）： + +```json +{ + "created": 1706734328, + "data": [ + { + "b64_json": "iVBORw0KGgoAAAANSUhEUgAA..." + } + ] +} +``` + +### 6.2 GPT Image 增强响应 + +GPT Image 模型在响应中包含额外的元数据字段： + +```json +{ + "created": 1706734328, + "data": [ + { + "b64_json": "iVBORw0KGgoAAAANSUhEUgAA..." + } + ], + "background": "transparent", + "output_format": "png", + "size": "1024x1024", + "quality": "high", + "usage": { + "total_tokens": 100, + "input_tokens": 50, + "output_tokens": 50, + "input_tokens_details": { + "text_tokens": 20, + "image_tokens": 30 + } + } +} +``` + +### 6.3 响应字段说明 + +| 字段 | 类型 | 说明 | +|------|------|------| +| `created` | integer | 创建时间戳（Unix 时间） | +| `data` | array | 图像数据数组 | +| `data[].url` | string | 图像下载 URL（`response_format: "url"` 时返回，有效期 1 小时） | +| `data[].b64_json` | string | 图像 Base64 编码（`response_format: "b64_json"` 时返回） | +| `data[].revised_prompt` | string | 模型修改后的 prompt（dall-e-3 和 GPT Image 可能返回） | +| `background` | string | 输出背景（GPT Image 专属） | +| `output_format` | string | 输出格式（GPT Image 专属） | +| `size` | string | 输出尺寸（GPT Image 专属） | +| `quality` | string | 输出质量（GPT Image 专属） | +| `usage` | object | Token 使用量（GPT Image 专属） | + +### 6.4 Usage 字段说明（GPT Image） + +| 字段 | 类型 | 说明 | +|------|------|------| +| `usage.total_tokens` | integer | 总 token 数 | +| `usage.input_tokens` | integer | 输入 token 数 | +| `usage.output_tokens` | integer | 输出 token 数（图像 token） | +| `usage.input_tokens_details.text_tokens` | integer | 文本输入 token 数 | +| `usage.input_tokens_details.image_tokens` | integer | 图像输入 token 数（编辑场景） | + +--- + +## 7. 流式响应 (GPT Image 模型) + +仅 GPT Image 模型（`gpt-image-1`、`gpt-image-1-mini`、`gpt-image-1.5`）支持流式响应。通过设置 `stream: true` 或 `partial_images > 0` 启用。 + +**Content-Type**：`text/event-stream` + +### 7.1 流式事件格式 + +流式响应通过 SSE 返回，每个事件包含一个 JSON 对象： + +``` +data: {"type": "generation.partial", "data": [{"b64_json": "...", "index": 0}]} + +data: {"type": "generation.partial", "data": [{"b64_json": "...", "index": 0}]} + +data: {"type": "generation.completed", "data": [{"b64_json": "...", "index": 0}]} +``` + +### 7.2 事件类型 + +| 事件类型 | 说明 | +|---------|------| +| `generation.partial` | 中间过程图，数量取决于 `partial_images` 参数 | +| `generation.completed` | 最终完成的图像 | + +### 7.3 流式请求示例 + +```json +{ + "model": "gpt-image-1", + "prompt": "A beautiful mountain landscape at sunset", + "n": 1, + "size": "1024x1024", + "partial_images": 2, + "stream": true +} +``` + +### 7.4 流式响应完整示例 + +``` +data: {"type":"generation.partial","data":[{"b64_json":"iVBORw0KGgo...partial1","index":0}],"usage":{"total_tokens":0,"input_tokens":50,"output_tokens":0,"input_tokens_details":{"text_tokens":50,"image_tokens":0}}} + +data: {"type":"generation.partial","data":[{"b64_json":"iVBORw0KGgo...partial2","index":0}],"usage":{"total_tokens":0,"input_tokens":50,"output_tokens":0,"input_tokens_details":{"text_tokens":50,"image_tokens":0}}} + +data: {"type":"generation.completed","data":[{"b64_json":"iVBORw0KGgo...final","index":0}],"usage":{"total_tokens":100,"input_tokens":50,"output_tokens":50,"input_tokens_details":{"text_tokens":50,"image_tokens":0}}} +``` + +--- + +## 8. 各模型参数差异对比 + +| 参数 | dall-e-2 | dall-e-3 | gpt-image-1 / mini / 1.5 | +|------|----------|----------|--------------------------| +| **生成端点** | 支持 | 支持 | 支持 | +| **编辑端点** | 支持（mask） | 不支持 | 支持（多图，无 mask） | +| **变体端点** | 支持 | 不支持 | 不支持 | +| **prompt 最大长度** | 1000 字符 | 4000 字符 | 32000 字符 | +| **n（生成数量）** | 1 | 1 | 1-10 | +| **尺寸** | 256/512/1024 正方形 | 1024 正方形 + 横纵 1792 | 1024 正方形 + 1536 横纵 | +| **quality** | 不支持 | `standard`/`hd` | `auto`/`high`/`medium`/`low` | +| **style** | 不支持 | `vivid`/`natural` | 不支持 | +| **background** | 不支持 | 不支持 | `transparent`/`opaque`/`auto` | +| **output_format** | 不支持 | 不支持 | `png`/`jpeg`/`webp` | +| **output_compression** | 不支持 | 不支持 | 0-100 | +| **流式响应** | 不支持 | 不支持 | 支持（`partial_images`） | +| **moderation** | 不支持 | 不支持 | `auto`/`low` | +| **response_format 默认值** | `url` | `url` | `b64_json` | +| **Usage 统计** | 不返回 | 不返回 | 返回 | + +--- + +## 9. 错误格式 + +图像 API 使用与 Chat Completions 相同的 OpenAI 错误格式： + +```json +{ + "error": { + "type": "invalid_request_error", + "message": "Invalid size for model dall-e-3", + "param": "size", + "code": null + } +} +``` + +### 常见错误 + +| HTTP 状态码 | error.type | 说明 | +|------------|-----------|------| +| 400 | `invalid_request_error` | 参数错误（不支持的尺寸、无效的 prompt 等） | +| 401 | `authentication_error` | API Key 无效或缺失 | +| 403 | `permission_error` | 账号无权使用该模型 | +| 429 | `rate_limit_error` | 请求频率超限 | +| 500 | `server_error` | 服务器内部错误 | + +### 特殊错误场景 + +| 场景 | 说明 | +|------|------| +| prompt 被安全系统拒绝 | `"type": "invalid_request_error"`, `"message": "Your prompt was rejected by the safety system"` | +| 图像过大 | `"message": "Image file is too large. Maximum file size is 4MB."` | +| 遮罩尺寸不匹配 | `"message": "Mask must have the same dimensions as the image."` | + +--- + +## 10. 供应商适配器协议差异 + +team-api Relay 层已实现以下供应商的图像生成适配器。各供应商使用不同的上游端点和参数格式： + +### 10.1 端点映射表 + +| 供应商 | RelayModeImagesGenerations 对应的上游端点 | 说明 | +|--------|------------------------------------------|------| +| **OpenAI** | `POST {baseURL}/v1/images/generations` | 原生 OpenAI 格式 | +| **Ali（阿里云百炼）** | `POST {baseURL}/api/v1/services/aigc/text2image/image-synthesis` | DashScope 原生端点 | +| **Baidu V2（百度文心）** | `POST {baseURL}/v2/images/generations` | 百度兼容格式 | +| **Zhipu（智谱 AI）** | `POST {baseURL}/api/paas/v4/images/generations` | 智谱 CogView 端点 | +| **SiliconFlow** | `POST {baseURL}/v1/images/generations` | 兼容 OpenAI 格式 | +| **xAI（Grok）** | `POST {baseURL}/v1/images/generations` | 兼容 OpenAI 格式 | +| **Mistral** | `POST {baseURL}/v1/images/generations` | 兼容 OpenAI 格式 | +| **Cloudflare（Workers AI）** | `POST {baseURL}/accounts/{account_id}/ai/models/@cf/.../images/generations` | Cloudflare AI 端点 | +| **Replicate** | `POST {baseURL}/v1/images/generations` | 兼容 OpenAI 格式 | +| **Jimeng（即梦）** | `POST {baseURL}/v2/images/generations` | 字节跳动即梦专属端点，仅支持图像生成 | + +### 10.2 参数转换注意事项 + +| 供应商 | 参数差异 | +|--------|---------| +| **Ali** | 请求体需要转换为 DashScope 格式（`input.prompt`、`parameters.size` 等），响应需要从 DashScope 格式转回 OpenAI 格式 | +| **SiliconFlow** | `size` 字段需重命名为 `image_size` | +| **Jimeng** | 仅支持 `RelayModeImagesGenerations`，不支持其他模式 | +| **Replicate** | 仅支持 `RelayModeImagesGenerations`，不支持其他模式 | + +--- + +## 11. 协议转换注意事项 + +### 11.1 当前实现状态 + +team-api 中图像生成的处理链路： + +``` +客户端请求 POST /v1/images/generations + → Path2RelayMode 识别为 RelayModeImagesGenerations + → HandleImagesGenerations 入口 + → RelayHandler 统一调度 + → 渠道适配器 GetRequestURL（构建上游 URL） + → 渠道适配器 ConvertRequest（转换请求参数） + → 渠道适配器 DoRequest（发送上游请求） + → 渠道适配器 DoResponse（处理上游响应） +``` + +### 11.2 DTO 定义（当前） + +当前 `relay/dto/usage.go` 中的图像相关结构体： + +```go +// ImageRequest 图像生成请求 +type ImageRequest struct { + Model string `json:"model"` + Prompt string `json:"prompt"` + N *int `json:"n,omitempty"` + Size string `json:"size,omitempty"` + Quality string `json:"quality,omitempty"` + ResponseFormat string `json:"response_format,omitempty"` + Style string `json:"style,omitempty"` + User string `json:"user,omitempty"` +} + +// ImageResponse 图像生成响应 +type ImageResponse struct { + Created int64 `json:"created"` + Data []ImageData `json:"data"` +} + +// ImageData 单个图像数据 +type ImageData struct { + URL string `json:"url,omitempty"` + B64JSON string `json:"b64_json,omitempty"` + RevisedPrompt string `json:"revised_prompt,omitempty"` +} +``` + +### 11.3 待完善项 + +1. **GPT Image 新参数未覆盖**：当前 `ImageRequest` 缺少 `background`、`output_format`、`output_compression`、`partial_images`、`stream`、`moderation` 字段 +2. **GPT Image 增强响应未覆盖**：当前 `ImageResponse` 缺少 `background`、`output_format`、`size`、`quality`、`usage` 字段 +3. **图像编辑端点未注册**：`/v1/images/edits` 路径尚未在 `Path2RelayMode` 中注册 +4. **图像变体端点未注册**：`/v1/images/variations` 路径尚未在 `Path2RelayMode` 中注册 +5. **流式图像响应未实现**：GPT Image 的 SSE 流式响应需要专门的流处理器 +6. **Usage 统计缺失**：当前 `handleImageResponse` 返回空的 `Usage{}`，GPT Image 模型应返回实际的 token 用量 +7. **计费缺失**：DALL-E 和 GPT Image 的定价模型不同（DALL-E 按次计费，GPT Image 按 token 计费），需要区分 + +### 11.4 DTO 扩展建议 + +为支持 GPT Image 的完整功能，建议扩展 DTO： + +```go +// ImageRequest 图像生成请求（扩展版） +type ImageRequest struct { + Model string `json:"model"` + Prompt string `json:"prompt"` + N *int `json:"n,omitempty"` + Size string `json:"size,omitempty"` + Quality string `json:"quality,omitempty"` + ResponseFormat string `json:"response_format,omitempty"` + Style string `json:"style,omitempty"` + Background string `json:"background,omitempty"` // GPT Image: transparent/opaque/auto + OutputFormat string `json:"output_format,omitempty"` // GPT Image: png/jpeg/webp + OutputCompression *int `json:"output_compression,omitempty"` // GPT Image: 0-100 + PartialImages *int `json:"partial_images,omitempty"` // GPT Image: 0-3 + Stream *bool `json:"stream,omitempty"` // GPT Image: 流式 + Moderation string `json:"moderation,omitempty"` // GPT Image: auto/low + User string `json:"user,omitempty"` +} + +// ImageResponse 图像生成响应（扩展版） +type ImageResponse struct { + Created int64 `json:"created"` + Data []ImageData `json:"data"` + Background string `json:"background,omitempty"` + OutputFormat string `json:"output_format,omitempty"` + Size string `json:"size,omitempty"` + Quality string `json:"quality,omitempty"` + Usage *ImageUsage `json:"usage,omitempty"` +} + +// ImageUsage GPT Image token 用量 +type ImageUsage struct { + TotalTokens int `json:"total_tokens"` + InputTokens int `json:"input_tokens"` + OutputTokens int `json:"output_tokens"` + InputTokenDetails *ImageTokenDetails `json:"input_tokens_details,omitempty"` +} + +// ImageTokenDetails GPT Image 输入 token 细分 +type ImageTokenDetails struct { + TextTokens int `json:"text_tokens"` + ImageTokens int `json:"image_tokens"` +} +``` + +### 11.5 新增 RelayMode 建议 + +为支持图像编辑和变体端点，建议在 `relay/constant/relay_mode.go` 中新增： + +```go +RelayModeImagesEdits // /v1/images/edits +RelayModeImagesVariations // /v1/images/variations +``` + +并在 `Path2RelayMode` 中添加路径匹配： + +```go +case strings.HasSuffix(path, "/images/edits"): + result = RelayModeImagesEdits +case strings.HasSuffix(path, "/images/variations"): + result = RelayModeImagesVariations +``` + +### 11.6 计费差异 + +| 模型 | 计费方式 | 说明 | +|------|---------|------| +| `dall-e-2` | 按次计费 | 根据尺寸和数量定价，与 token 无关 | +| `dall-e-3` | 按次计费 | quality=hd 价格高于 standard | +| `gpt-image-1` | 按 token 计费 | 使用 `usage` 字段中的 token 数计费 | +| `gpt-image-1-mini` | 按 token 计费 | 价格低于 gpt-image-1 | +| `gpt-image-1.5` | 按 token 计费 | 价格低于 gpt-image-1 | + +Relay 层的 `BillingProvider` 需要根据模型类型选择不同的计费策略：DALL-E 模型使用固定价格计费，GPT Image 模型使用 token 计费。当前 `handleImageResponse` 返回空 Usage，需要区分处理。 diff --git a/docs/协议文档/Stability-AI图像生成API文档.md b/docs/协议文档/Stability-AI图像生成API文档.md new file mode 100644 index 0000000..fef82e1 --- /dev/null +++ b/docs/协议文档/Stability-AI图像生成API文档.md @@ -0,0 +1,745 @@ +# Stability AI 图像生成 API 文档 + +## 概述 + +Stability AI 是 Stable Diffusion 模型的创建者，提供图像生成 REST API（v2beta）。API 基于 Stable Image 品牌提供服务，支持文生图、图生图、局部重绘、超分辨率、背景移除等功能。 + +- **官方文档**：https://platform.stability.ai/docs +- **API Base URL**：`https://api.stability.ai` +- **API 版本**：v2beta（当前最新） +- **认证方式**：Bearer Token（API Key） +- **协议格式**：REST + multipart/form-data（部分接口支持 JSON） + +## 认证 + +### 获取 API Key + +1. 注册 https://platform.stability.ai 账号 +2. 在 Account Settings → API Keys 页面创建 Key +3. 使用 Bearer Token 认证 + +### 认证头 + +``` +Authorization: Bearer sk-xxxxxxxxxxxxxxxxxxxxxxxx +``` + +### 计费方式 + +- 按 Credit 消耗计费，不同模型/服务消耗不同 Credit +- 免费账户有初始 Credit +- Credit 可在平台购买 + +## 核心概念 + +### Stable Image 服务 + +Stability AI 的图像生成服务按功能分为以下类别： + +| 服务 | 说明 | 核心端点路径 | +|------|------|-------------| +| Text to Image | 文生图 | `/v2beta/stable-image/generate/{model}` | +| Image to Image | 图生图 | `/v2beta/stable-image/generate/{model}` | +| Inpaint / Outpaint | 局部重绘/外扩 | `/v2beta/stable-image/generate/{model}` | +| Upscale | 超分辨率放大 | `/v2beta/stable-image/upscale/{model}` | +| Remove Background | 背景移除 | `/v2beta/stable-image/remove-bg` | +| Search and Replace | 搜索替换 | `/v2beta/stable-image/generate/{model}` | + +### 可用模型 + +| 模型 ID | 说明 | 特点 | +|---------|------|------| +| `sd3` | Stable Diffusion 3 | 高质量文生图，支持 T5 文本编码器 | +| `sd3-turbo` | SD3 Turbo | 快速生成，适合高吞吐场景 | +| `sd3.5-large` | Stable Diffusion 3.5 Large | 最新旗舰模型，最高质量 | +| `sd3.5-large-turbo` | SD3.5 Large Turbo | 旗舰快速版 | +| `sd3.5-medium` | Stable Diffusion 3.5 Medium | 质量与速度平衡 | +| `sdxl` | Stable Diffusion XL | 上代旗舰，广泛支持 | +| `sdxl-turbo` | SDXL Turbo | SDXL 快速版 | +| `core` | Stable Image Core | 通用高质量模型 | +| `ultra` | Stable Image Ultra | 最高质量，4 步生成 | + +### 模型选择建议 + +| 场景 | 推荐模型 | 原因 | +|------|---------|------| +| 最高质量 | `sd3.5-large` 或 `ultra` | 旗舰级输出 | +| 质量与速度平衡 | `sd3.5-medium` 或 `core` | 性价比最优 | +| 快速批量生成 | `sd3-turbo` 或 `sd3.5-large-turbo` | 最低延迟 | +| 兼容性最佳 | `sdxl` | 社区生态最丰富 | + +## API 端点详解 + +### 1. 文生图（Text to Image） + +根据文本提示词生成图像。 + +**请求** + +``` +POST https://api.stability.ai/v2beta/stable-image/generate/{model} +``` + +**Headers** + +| Header | 值 | 必填 | 说明 | +|--------|---|------|------| +| `Authorization` | `Bearer sk-xxx` | 是 | API Key 认证 | +| `Content-Type` | `multipart/form-data` | 是 | 请求体格式 | +| `Accept` | `image/*` 或 `application/json` | 是 | 响应格式。`image/*` 直接返回图像二进制；`application/json` 返回 base64 编码 | + +**Body 参数（multipart/form-data）** + +| 参数 | 类型 | 必填 | 默认值 | 说明 | +|------|------|------|--------|------| +| `prompt` | string | 是 | — | 正向提示词，描述期望生成的图像内容 | +| `negative_prompt` | string | 否 | `""` | 反向提示词，描述不希望出现的内容（部分模型支持） | +| `aspect_ratio` | string | 否 | `"1:1"` | 宽高比。可选值：`"1:1"`, `"16:9"`, `"21:9"`, `"2:3"`, `"3:2"`, `"4:5"`, `"5:4"`, `"9:16"`, `"9:21"` | +| `seed` | integer | 否 | 随机 | 随机种子，相同种子+参数可复现结果。范围：0 ~ 4294967294 | +| `output_format` | string | 否 | `"png"` | 输出格式。可选值：`"png"`, `"jpeg"`, `"webp"` | +| `mode` | string | 否 | `"text-to-image"` | 生成模式。文生图使用 `"text-to-image"` | +| `model` | string | 否 | URL 中的模型 | 覆盖 URL 路径中指定的模型（部分场景可用） | +| `style_preset` | string | 否 | 无 | 预设风格（见下方风格列表） | + +**示例请求** + +```bash +curl -X POST \ + https://api.stability.ai/v2beta/stable-image/generate/sd3.5-large \ + -H "Authorization: Bearer sk-xxxxxxxx" \ + -H "Accept: image/*" \ + -F "prompt=A majestic lion standing on a cliff at sunset, cinematic lighting, 8k" \ + -F "output_format=png" \ + -F "aspect_ratio=16:9" \ + -F "seed=42" \ + -o lion.png +``` + +**示例请求（JSON 响应）** + +```bash +curl -X POST \ + https://api.stability.ai/v2beta/stable-image/generate/sd3.5-large \ + -H "Authorization: Bearer sk-xxxxxxxx" \ + -H "Accept: application/json" \ + -F "prompt=A majestic lion standing on a cliff at sunset" \ + -F "output_format=png" +``` + +**JSON 响应格式** + +```json +{ + "image": "iVBORw0KGgoAAAANSUhEUgAA...", + "seed": 42, + "finish_reason": "SUCCESS" +} +``` + +| 字段 | 类型 | 说明 | +|------|------|------| +| `image` | string | Base64 编码的图像数据 | +| `seed` | integer | 使用的随机种子 | +| `finish_reason` | string | 完成原因。`"SUCCESS"` 表示成功 | + +**二进制响应** + +当 `Accept: image/*` 时，直接返回图像二进制数据，Content-Type 为 `image/png`、`image/jpeg` 或 `image/webp`。 + +--- + +### 2. 图生图（Image to Image） + +基于输入图像和提示词生成新图像。 + +**请求** + +``` +POST https://api.stability.ai/v2beta/stable-image/generate/{model} +``` + +**Headers** + +与文生图相同。 + +**Body 参数（multipart/form-data）** + +| 参数 | 类型 | 必填 | 默认值 | 说明 | +|------|------|------|--------|------| +| `prompt` | string | 是 | — | 提示词，描述期望的变化 | +| `image` | file | 是 | — | 输入图像文件（PNG/JPEG/WebP） | +| `negative_prompt` | string | 否 | `""` | 反向提示词 | +| `strength` | float | 否 | `0.5` | 图像变化强度。0.0 = 不变，1.0 = 完全重新生成。范围：0.0 ~ 1.0 | +| `seed` | integer | 否 | 随机 | 随机种子 | +| `output_format` | string | 否 | `"png"` | 输出格式 | +| `mode` | string | 否 | `"image-to-image"` | 生成模式 | +| `style_preset` | string | 否 | 无 | 预设风格 | + +**示例请求** + +```bash +curl -X POST \ + https://api.stability.ai/v2beta/stable-image/generate/sd3.5-large \ + -H "Authorization: Bearer sk-xxxxxxxx" \ + -H "Accept: image/*" \ + -F "prompt=Transform into a watercolor painting style" \ + -F "image=@input.jpg" \ + -F "strength=0.7" \ + -F "output_format=png" \ + -o output.png +``` + +--- + +### 3. 局部重绘（Inpaint） + +使用遮罩图像指定需要重绘的区域。 + +**请求** + +``` +POST https://api.stability.ai/v2beta/stable-image/generate/{model} +``` + +**Headers** + +与文生图相同。 + +**Body 参数（multipart/form-data）** + +| 参数 | 类型 | 必填 | 默认值 | 说明 | +|------|------|------|--------|------| +| `prompt` | string | 是 | — | 描述重绘区域的内容 | +| `image` | file | 是 | — | 原始图像文件 | +| `mask` | file | 是 | — | 遮罩图像（白色区域为需要重绘的部分，PNG 格式推荐） | +| `negative_prompt` | string | 否 | `""` | 反向提示词 | +| `seed` | integer | 否 | 随机 | 随机种子 | +| `output_format` | string | 否 | `"png"` | 输出格式 | +| `mode` | string | 否 | `"inpaint"` | 生成模式 | + +**示例请求** + +```bash +curl -X POST \ + https://api.stability.ai/v2beta/stable-image/generate/sd3.5-large \ + -H "Authorization: Bearer sk-xxxxxxxx" \ + -H "Accept: image/*" \ + -F "prompt=A beautiful blue sky with fluffy clouds" \ + -F "image=@original.png" \ + -F "mask=@mask.png" \ + -F "mode=inpaint" \ + -o inpainted.png +``` + +--- + +### 4. 外扩（Outpaint） + +扩展图像边界，在原始图像外部生成新内容。 + +**请求** + +``` +POST https://api.stability.ai/v2beta/stable-image/generate/{model} +``` + +**Body 参数** + +与局部重绘类似，但 `mode` 设置为 `"outpaint"`。可通过 `left`、`right`、`up`、`down` 参数指定各方向的扩展像素数。 + +| 参数 | 类型 | 必填 | 默认值 | 说明 | +|------|------|------|--------|------| +| `prompt` | string | 是 | — | 描述扩展区域的内容 | +| `image` | file | 是 | — | 原始图像 | +| `left` | integer | 否 | `0` | 左侧扩展像素数 | +| `right` | integer | 否 | `0` | 右侧扩展像素数 | +| `up` | integer | 否 | `0` | 上方扩展像素数 | +| `down` | integer | 否 | `0` | 下方扩展像素数 | +| `seed` | integer | 否 | 随机 | 随机种子 | +| `output_format` | string | 否 | `"png"` | 输出格式 | +| `mode` | string | 否 | `"outpaint"` | 生成模式 | + +--- + +### 5. 超分辨率（Upscale） + +放大图像并增强细节。 + +**请求** + +``` +POST https://api.stability.ai/v2beta/stable-image/upscale/{model} +``` + +**可用 Upscale 模型** + +| 模型 ID | 说明 | +|---------|------| +| `creative` | 创意增强（添加新细节） | +| `fast` | 快速放大 | +| `conservative` | 保守放大（保持原始风格） | + +**Body 参数** + +| 参数 | 类型 | 必填 | 默认值 | 说明 | +|------|------|------|--------|------| +| `image` | file | 是 | — | 需要放大的图像 | +| `prompt` | string | 否 | `""` | 提示词，引导放大时的细节增强方向（`creative` 模式推荐） | +| `negative_prompt` | string | 否 | `""` | 反向提示词 | +| `seed` | integer | 否 | 随机 | 随机种子 | +| `output_format` | string | 否 | `"png"` | 输出格式 | +| `creativity` | float | 否 | `0.5` | 创意度（仅 `creative` 模式）。范围：0.0 ~ 1.0 | + +**示例请求** + +```bash +curl -X POST \ + https://api.stability.ai/v2beta/stable-image/upscale/creative \ + -H "Authorization: Bearer sk-xxxxxxxx" \ + -H "Accept: image/*" \ + -F "image=@low_res.png" \ + -F "prompt=Enhance details, add texture" \ + -F "creativity=0.7" \ + -o upscaled.png +``` + +**注意**：Upscale 结果可能是异步的。如果图像较大，API 可能返回 `202 Accepted`，需要轮询获取结果。 + +--- + +### 6. 背景移除（Remove Background） + +自动识别并移除图像背景。 + +**请求** + +``` +POST https://api.stability.ai/v2beta/stable-image/remove-bg +``` + +**Body 参数** + +| 参数 | 类型 | 必填 | 默认值 | 说明 | +|------|------|------|--------|------| +| `image` | file | 是 | — | 输入图像 | +| `output_format` | string | 否 | `"png"` | 输出格式（推荐 PNG 以保留透明度） | + +**示例请求** + +```bash +curl -X POST \ + https://api.stability.ai/v2beta/stable-image/remove-bg \ + -H "Authorization: Bearer sk-xxxxxxxx" \ + -H "Accept: image/*" \ + -F "image=@photo.jpg" \ + -F "output_format=png" \ + -o no_bg.png +``` + +--- + +### 7. 搜索替换（Search and Replace） + +在图像中查找特定内容并替换为提示词描述的新内容。 + +**请求** + +``` +POST https://api.stability.ai/v2beta/stable-image/generate/{model} +``` + +**Body 参数** + +| 参数 | 类型 | 必填 | 默认值 | 说明 | +|------|------|------|--------|------| +| `prompt` | string | 是 | — | 替换后的内容描述 | +| `image` | file | 是 | — | 原始图像 | +| `search_prompt` | string | 是 | — | 要查找/替换的内容描述 | +| `negative_prompt` | string | 否 | `""` | 反向提示词 | +| `seed` | integer | 否 | 随机 | 随机种子 | +| `output_format` | string | 否 | `"png"` | 输出格式 | +| `mode` | string | 否 | `"search-and-replace"` | 生成模式 | + +## 风格预设（Style Presets） + +以下风格预设可用于 `style_preset` 参数： + +| 类别 | 预设名称 | +|------|---------| +| 3D | `3d-model` | +| 摄影风格 | `analog-film`, `cinematic`, `photographic` | +| 动漫 | `anime`, `comic-book`, `line-art` | +| 数字艺术 | `digital-art`, `enhance`, `isometric`, `low-poly`, `pixel-art` | +| 绘画 | `craft-clay`, `fantasy-art`, `neon-punk`, `origami`, `stained-glass`, `watercolor`, `tile-texture` | +| 其他 | `anime`, `aztec-jungle`, `blockprint`, `collage`, `conceptual-art`, `diagram`, `fashion`, `filmic`, ` hdr`, `tile-texture`, `thriller`, `ukiyo-e`, `vintage`, `vivid-colors` | + +## SDK 参数参考 + +Stability AI 提供官方 Python SDK（`stability-sdk`），以下参数在 SDK 和 REST API 中通用： + +### 通用参数 + +| 参数 | CLI 参数 | 类型 | 默认值 | 说明 | +|------|---------|------|--------|------| +| 宽度 | `--width` | integer | 1024 | 输出图像宽度（像素） | +| 高度 | `--height` | integer | 1024 | 输出图像高度（像素） | +| CFG Scale | `--cfg_scale` | float | 7.0 | 提示词相关性/分类器自由引导尺度。范围：1 ~ 35 | +| 采样器 | `--sampler` | string | 自动 | 采样算法（见下方采样器列表） | +| 步数 | `--steps` | integer | 30 | 扩散步数。范围：10 ~ 50（Turbo 模型推荐 4） | +| 种子 | `--seed` | integer | 随机 | 随机种子。范围：0 ~ 4294967294 | +| 初始图像 | `--init_image` | file | 无 | 图生图输入图像 | +| 遮罩图像 | `--mask_image` | file | 无 | 局部重绘遮罩 | +| 引擎 | `--engine` | string | 自动 | 模型/引擎 ID | +| 样式预设 | `--style_preset` | string | 无 | 风格预设 | + +### 采样器（Sampler） + +| 采样器 ID | 说明 | +|-----------|------| +| `ddim` | DDIM 采样器 | +| `plms` | PLMS 采样器 | +| `k_euler` | Euler 采样器 | +| `k_euler_ancestral` | Euler Ancestral 采样器 | +| `k_heun` | Heun 采样器 | +| `k_dpm_2` | DPM2 采样器 | +| `k_dpm_2_ancestral` | DPM2 Ancestral 采样器 | +| `k_dpmpp_2s_ancestral` | DPM++ 2S Ancestral 采样器 | +| `k_dpmpp_sde` | DPM++ SDE 采样器 | +| `k_dpmpp_2m` | DPM++ 2M 采样器 | +| `k_dpmpp_2m_sde` | DPM++ 2M SDE 采样器 | +| `k_dpmpp_3m_sde` | DPM++ 3M SDE 采样器 | + +## 错误处理 + +### HTTP 状态码 + +| 状态码 | 含义 | +|--------|------| +| `200` | 成功 | +| `202` | 已接受（异步任务，需轮询） | +| `400` | 请求参数错误 | +| `401` | 认证失败（API Key 无效） | +| `402` | Credit 不足 | +| `403` | 无权限 | +| `404` | 模型/端点不存在 | +| `413` | 请求体过大 | +| `429` | 请求频率超限 | +| `500` | 服务器内部错误 | + +### 错误响应格式 + +```json +{ + "id": "6a6c3a9e-ee82-4e94-b36e-5b3f14e4c631", + "message": "Invalid request: prompt is required", + "name": "bad_request" +} +``` + +| 字段 | 类型 | 说明 | +|------|------|------| +| `id` | string | 错误唯一标识 | +| `message` | string | 错误描述 | +| `name` | string | 错误类型名称 | + +### 常见错误类型 + +| 错误类型 name | HTTP 状态码 | 说明 | +|--------------|------------|------| +| `bad_request` | 400 | 请求参数错误 | +| `unauthorized` | 401 | API Key 无效或缺失 | +| `insufficient_credits` | 402 | Credit 余额不足 | +| `permission_denied` | 403 | 无权访问该模型/功能 | +| `not_found` | 404 | 资源不存在 | +| `rate_limit_exceeded` | 429 | 请求频率超限 | +| `server_error` | 500 | 服务器内部错误 | + +## 异步操作 + +### 提交异步任务 + +对于耗时操作（如 Upscale），API 可能返回 `202 Accepted`： + +```json +{ + "id": "abc123", + "status": "in-progress" +} +``` + +### 轮询结果 + +``` +GET https://api.stability.ai/v2beta/stable-image/results/{id} +``` + +**Headers** + +``` +Authorization: Bearer sk-xxx +Accept: image/* 或 application/json +``` + +### 响应状态 + +| 状态 | 说明 | +|------|------| +| `in-progress` | 正在处理 | +| `complete` | 已完成（返回图像） | +| `failed` | 处理失败 | + +### 轮询建议 + +- 使用指数退避策略 +- 建议初始间隔 1 秒，最大间隔 10 秒 +- 总超时建议 5 分钟 + +## 速率限制 + +| 计划 | 并发请求数 | 每秒请求数 | +|------|-----------|-----------| +| Free | 2 | 2 | +| Basic | 4 | 4 | +| Standard | 8 | 8 | +| Premium | 16 | 16 | +| Enterprise | 自定义 | 自定义 | + +速率限制响应头： + +``` +X-RateLimit-Limit: 100 +X-RateLimit-Remaining: 95 +X-RateLimit-Reset: 1677652288 +``` + +## 图像尺寸与限制 + +### 支持的宽高比 + +| 宽高比 | 尺寸（像素） | 说明 | +|--------|-------------|------| +| `1:1` | 1024 × 1024 | 正方形（默认） | +| `16:9` | 1344 × 768 | 横屏宽幅 | +| `21:9` | 1536 × 640 | 超宽屏 | +| `2:3` | 768 × 1152 | 竖屏 | +| `3:2` | 1152 × 768 | 横屏 | +| `4:5` | 896 × 1088 | 近正方形竖屏 | +| `5:4` | 1088 × 896 | 近正方形横屏 | +| `9:16` | 768 × 1344 | 竖屏长幅 | +| `9:21` | 640 × 1536 | 超长竖屏 | + +### 文件大小限制 + +- 输入图像：最大 10MB +- 输出图像：最大 20MB + +## Python SDK 使用示例 + +### 安装 + +```bash +pip install stability-sdk +``` + +### 文生图 + +```python +import io +import requests +from PIL import Image + +response = requests.post( + "https://api.stability.ai/v2beta/stable-image/generate/sd3.5-large", + headers={ + "Authorization": f"Bearer {api_key}", + "Accept": "image/*" + }, + files={"none": ""}, + data={ + "prompt": "A beautiful mountain landscape at golden hour", + "output_format": "png", + "aspect_ratio": "16:9", + } +) + +if response.status_code == 200: + with open("output.png", "wb") as f: + f.write(response.content) +else: + print(f"Error: {response.status_code}") + print(response.json()) +``` + +### 图生图 + +```python +response = requests.post( + "https://api.stability.ai/v2beta/stable-image/generate/sd3.5-large", + headers={ + "Authorization": f"Bearer {api_key}", + "Accept": "image/*" + }, + files={ + "image": open("input.jpg", "rb") + }, + data={ + "prompt": "Transform into oil painting style", + "strength": 0.7, + "output_format": "png", + } +) +``` + +### 局部重绘 + +```python +response = requests.post( + "https://api.stability.ai/v2beta/stable-image/generate/sd3.5-large", + headers={ + "Authorization": f"Bearer {api_key}", + "Accept": "image/*" + }, + files={ + "image": open("original.png", "rb"), + "mask": open("mask.png", "rb"), + }, + data={ + "prompt": "A clear blue sky with white clouds", + "output_format": "png", + } +) +``` + +## Go HTTP 调用示例 + +```go +package main + +import ( + "bytes" + "fmt" + "io" + "mime/multipart" + "net/http" + "os" +) + +func generateImage(apiKey, prompt, model, outputPath string) error { + var buf bytes.Buffer + writer := multipart.NewWriter(&buf) + + _ = writer.WriteField("prompt", prompt) + _ = writer.WriteField("output_format", "png") + _ = writer.WriteField("aspect_ratio", "1:1") + + // 必须有一个 none 字段（当没有文件上传时） + part, _ := writer.CreateFormFile("none", "") + part.Write([]byte{}) + + writer.Close() + + url := fmt.Sprintf("https://api.stability.ai/v2beta/stable-image/generate/%s", model) + req, _ := http.NewRequest("POST", url, &buf) + req.Header.Set("Authorization", "Bearer "+apiKey) + req.Header.Set("Accept", "image/*") + req.Header.Set("Content-Type", writer.FormDataContentType()) + + resp, err := http.DefaultClient.Do(req) + if err != nil { + return err + } + defer resp.Body.Close() + + if resp.StatusCode != 200 { + body, _ := io.ReadAll(resp.Body) + return fmt.Errorf("API error %d: %s", resp.StatusCode, body) + } + + file, _ := os.Create(outputPath) + defer file.Close() + io.Copy(file, resp.Body) + + return nil +} +``` + +## 与 OpenAI Images API 的对比 + +| 特性 | Stability AI | OpenAI Images | +|------|-------------|---------------| +| 认证方式 | Bearer Token | Bearer Token | +| 请求格式 | multipart/form-data | JSON | +| 文生图 | `/v2beta/stable-image/generate/{model}` | `/v1/images/generations` | +| 图生图 | 同端点 + image 文件 | `/v1/images/edits` | +| 局部重绘 | 同端点 + mask 文件 | `/v1/images/edits` + mask | +| 超分辨率 | `/v2beta/stable-image/upscale/{model}` | 不支持 | +| 背景移除 | `/v2beta/stable-image/remove-bg` | 不支持 | +| 响应格式 | 二进制或 JSON（base64） | JSON（URL 或 base64） | +| 流式支持 | 不支持 | gpt-image-1 支持 SSE | +| 模型数量 | 10+ | 5（DALL-E 2/3, GPT Image 系列） | +| 风格控制 | style_preset + sampler + steps | quality + style | +| 免费额度 | 初始 Credit | 无（按次付费） | + +## team-api 适配器实现参考 + +### 端点映射 + +| team-api 入口 | Stability AI 上游 | +|--------------|------------------| +| `POST /v1/images/generations` | `POST /v2beta/stable-image/generate/{model}` | +| `POST /v1/images/edits` | `POST /v2beta/stable-image/generate/{model}` (mode=image-to-image) | +| `POST /v1/images/variations` | `POST /v2beta/stable-image/generate/{model}` (mode=image-to-image) | + +### 请求转换要点 + +1. **认证转换**：OpenAI 格式 `Bearer sk-xxx` → Stability 格式 `Bearer sk-xxx`（直接透传 API Key） +2. **请求体转换**： + - OpenAI JSON body → multipart/form-data + - `size` → `aspect_ratio`（需要尺寸映射逻辑） + - `n` 参数：Stability API 不支持批量生成，需要循环调用 + - `response_format` → `Accept` header（`url` → 不支持，`b64_json` → `application/json`） +3. **响应体转换**： + - Stability 二进制响应 → OpenAI JSON 格式（base64 编码） + - Stability JSON 响应 → 提取 `image` 字段 → 放入 OpenAI `data[].b64_json` + +### DTO 参考 + +```go +// Stability AI 请求参数（转换后） +type StabilityImageRequest struct { + Prompt string `json:"prompt"` + NegativePrompt string `json:"negative_prompt,omitempty"` + AspectRatio string `json:"aspect_ratio,omitempty"` + Seed uint32 `json:"seed,omitempty"` + OutputFormat string `json:"output_format,omitempty"` + Mode string `json:"mode,omitempty"` + Strength float64 `json:"strength,omitempty"` + StylePreset string `json:"style_preset,omitempty"` +} + +// Stability AI JSON 响应 +type StabilityImageResponse struct { + Image string `json:"image"` + Seed uint32 `json:"seed"` + FinishReason string `json:"finish_reason"` +} + +// Stability AI 错误响应 +type StabilityError struct { + ID string `json:"id"` + Message string `json:"message"` + Name string `json:"name"` +} +``` + +### 尺寸映射表 + +| OpenAI size | Stability aspect_ratio | +|-------------|----------------------| +| `256x256` | `1:1`（SDXL 不支持 256，降质） | +| `512x512` | `1:1`（SDXL 不支持 512，降质） | +| `1024x1024` | `1:1` | +| `1792x1024` | `16:9` | +| `1024x1792` | `9:16` | +| `1536x1024` | `3:2` | +| `1024x1536` | `2:3` | diff --git a/docs/协议文档/智谱GLM/GLM图像生成（异步）.md b/docs/协议文档/智谱GLM/GLM图像生成（异步）.md new file mode 100644 index 0000000..71ef88b --- /dev/null +++ b/docs/协议文档/智谱GLM/GLM图像生成（异步）.md @@ -0,0 +1,167 @@ +> ## Documentation Index +> Fetch the complete documentation index at: https://docs.bigmodel.cn/llms.txt +> Use this file to discover all available pages before exploring further. + +# 图像生成(异步) + +> 使用 [GLM-Image](/cn/guide/models/image-generation/glm-image) 系列模型从文本提示生成高质量图像。通过对用户文字描述快速、精准的理解，让 `AI` 的图像表达更加精确和个性化。仅支持 `GLM-Image` 模型。 + + + +## OpenAPI + +````yaml /openapi/openapi.json post /paas/v4/async/images/generations +openapi: 3.0.1 +info: + title: ZHIPU AI API + description: ZHIPU AI 接口提供强大的 AI 能力，包括聊天对话、工具调用和视频生成。 + license: + name: ZHIPU AI 开发者协议和政策 + url: https://chat.z.ai/legal-agreement/terms-of-service + version: 1.0.0 + contact: + name: Z.AI 开发者 + url: https://chat.z.ai/legal-agreement/privacy-policy + email: user_feedback@z.ai +servers: + - url: https://open.bigmodel.cn/api/ + description: 开放平台服务 +security: + - bearerAuth: [] +tags: + - name: 模型 API + description: Chat API + - name: 工具 API + description: Web Search API + - name: Agent API + description: Agent API + - name: 文件 API + description: File API + - name: 知识库 API + description: Knowledge API + - name: 实时 API + description: Realtime API + - name: 批处理 API + description: Batch API + - name: 助理 API + description: Assistant API + - name: 智能体 API（旧） + description: QingLiu Agent API +paths: + /paas/v4/async/images/generations: + post: + tags: + - 模型 API + summary: 图像生成(异步) + description: >- + 使用 [GLM-Image](/cn/guide/models/image-generation/glm-image) + 系列模型从文本提示生成高质量图像。通过对用户文字描述快速、精准的理解，让 `AI` 的图像表达更加精确和个性化。仅支持 `GLM-Image` + 模型。 + requestBody: + content: + application/json: + schema: + $ref: '#/components/schemas/AsyncCreateImageRequest' + examples: + 图像生成示例: + value: + model: glm-image + prompt: 一只可爱的小猫咪，坐在阳光明媚的窗台上，背景是蓝天白云. + size: 1280x1280 + required: true + responses: + '200': + description: 业务处理成功 + content: + application/json: + schema: + $ref: '#/components/schemas/AsyncResponse' + default: + description: 请求失败。 + content: + application/json: + schema: + $ref: '#/components/schemas/Error' +components: + schemas: + AsyncCreateImageRequest: + type: object + required: + - model + - prompt + properties: + model: + type: string + description: 模型编码 + enum: + - glm-image + example: glm-image + prompt: + type: string + description: 所需图像的文本描述 + example: 一只可爱的小猫咪 + quality: + type: string + description: '生成图像的质量。`hd`: 生成更精细、细节更丰富的图像，整体一致性更高，耗时约`20`秒；' + enum: + - hd + default: hd + size: + type: string + description: >- + 图片尺寸，推荐枚举值：`1280x1280` (默认), `1568×1056`, `1056×1568`, `1472×1088`, + `1088×1472`, `1728×960`, `960×1728`。 + + 自定义参数:长宽推荐设置在`1024px-2048px`范围内,并保证最大像素数不超过`2^22px`;长宽均需为`32`的整数倍。 + default: 1280x1280 + example: 1280x1280 + watermark_enabled: + type: boolean + description: |- + 控制`AI`生成图片时是否添加水印。 + - `true`: 默认启用`AI`生成的显式水印及隐式数字水印，符合政策要求。 + - `false`: 关闭所有水印，仅允许已签署免责声明的客户使用，签署路径：个人中心-安全管理-去水印管理 + example: true + user_id: + type: string + description: >- + 终端用户的唯一`ID`，协助平台对终端用户的违规行为、生成违法及不良信息或其他滥用行为进行干预。`ID`长度要求：最少`6`个字符，最多`128`个字符。 + minLength: 6 + maxLength: 128 + AsyncResponse: + type: object + properties: + model: + description: 此次调用使用的名称。 + type: string + id: + description: 生成的任务`ID`，调用请求结果接口时使用此`ID`。 + type: string + request_id: + description: 用户在客户端请求期间提交的任务编号或平台生成的任务编号。 + type: string + task_status: + description: 处理状态，`PROCESSING (处理中)`、`SUCCESS (成功)`、`FAIL (失败)`。结果需要通过查询获取。 + type: string + Error: + type: object + properties: + error: + required: + - code + - message + type: object + properties: + code: + type: string + message: + type: string + securitySchemes: + bearerAuth: + type: http + scheme: bearer + description: >- + 使用以下格式进行身份验证：Bearer [](https://bigmodel.cn/usercenter/proj-mgmt/apikeys) + +```` \ No newline at end of file diff --git a/docs/协议文档/智谱GLM/GLM对话补全.md b/docs/协议文档/智谱GLM/GLM对话补全.md new file mode 100644 index 0000000..203beea --- /dev/null +++ b/docs/协议文档/智谱GLM/GLM对话补全.md @@ -0,0 +1,1700 @@ +> ## Documentation Index +> Fetch the complete documentation index at: https://docs.bigmodel.cn/llms.txt +> Use this file to discover all available pages before exploring further. + +# 对话补全 + +> 和 [指定模型](/cn/guide/start/model-overview) 对话，模型根据请求给出响应。支持多种模型，支持多模态（文本、图片、音频、视频、文件），流式和非流式输出，可配置采样，温度，最大令牌数，工具调用等。 + + + +## OpenAPI + +````yaml /openapi/openapi.json post /paas/v4/chat/completions +openapi: 3.0.1 +info: + title: ZHIPU AI API + description: ZHIPU AI 接口提供强大的 AI 能力，包括聊天对话、工具调用和视频生成。 + license: + name: ZHIPU AI 开发者协议和政策 + url: https://chat.z.ai/legal-agreement/terms-of-service + version: 1.0.0 + contact: + name: Z.AI 开发者 + url: https://chat.z.ai/legal-agreement/privacy-policy + email: user_feedback@z.ai +servers: + - url: https://open.bigmodel.cn/api/ + description: 开放平台服务 +security: + - bearerAuth: [] +tags: + - name: 模型 API + description: Chat API + - name: 工具 API + description: Web Search API + - name: Agent API + description: Agent API + - name: 文件 API + description: File API + - name: 知识库 API + description: Knowledge API + - name: 实时 API + description: Realtime API + - name: 批处理 API + description: Batch API + - name: 助理 API + description: Assistant API + - name: 智能体 API（旧） + description: QingLiu Agent API +paths: + /paas/v4/chat/completions: + post: + tags: + - 模型 API + summary: 对话补全 + description: >- + 和 [指定模型](/cn/guide/start/model-overview) + 对话，模型根据请求给出响应。支持多种模型，支持多模态（文本、图片、音频、视频、文件），流式和非流式输出，可配置采样，温度，最大令牌数，工具调用等。 + requestBody: + content: + application/json: + schema: + oneOf: + - $ref: '#/components/schemas/ChatCompletionTextRequest' + title: 文本模型 + - $ref: '#/components/schemas/ChatCompletionVisionRequest' + title: 视觉模型 + - $ref: '#/components/schemas/ChatCompletionAudioRequest' + title: 音频模型 + - $ref: '#/components/schemas/ChatCompletionHumanOidRequest' + title: 角色模型 + examples: + 基础调用示例: + value: + model: glm-5.1 + messages: + - role: system + content: 你是一个有用的AI助手。 + - role: user + content: 请介绍一下人工智能的发展历程。 + temperature: 1 + stream: false + 流式调用示例: + value: + model: glm-5.1 + messages: + - role: user + content: 写一首关于春天的诗。 + temperature: 1 + stream: true + 深度思考示例: + value: + model: glm-5.1 + messages: + - role: user + content: 写一首关于春天的诗。 + thinking: + type: enabled + stream: true + 多轮对话示例: + value: + model: glm-5.1 + messages: + - role: system + content: 你是一个专业的编程助手 + - role: user + content: 什么是递归？ + - role: assistant + content: 递归是一种编程技术，函数调用自身来解决问题... + - role: user + content: 能给我一个 Python 递归的例子吗？ + stream: true + 图片理解示例: + value: + model: glm-5v-turbo + messages: + - role: user + content: + - type: image_url + image_url: + url: https://cdn.bigmodel.cn/static/logo/register.png + - type: image_url + image_url: + url: https://cdn.bigmodel.cn/static/logo/api-key.png + - type: text + text: What are the pics talk about? + 视频理解示例: + value: + model: glm-5v-turbo + messages: + - role: user + content: + - type: video_url + video_url: + url: >- + https://cdn.bigmodel.cn/agent-demos/lark/113123.mov + - type: text + text: What are the video show about? + 文件理解示例: + value: + model: glm-5v-turbo + messages: + - role: user + content: + - type: file_url + file_url: + url: https://cdn.bigmodel.cn/static/demo/demo2.txt + - type: file_url + file_url: + url: https://cdn.bigmodel.cn/static/demo/demo1.pdf + - type: text + text: What are the files show about? + 音频对话示例: + value: + model: glm-4-voice + messages: + - role: user + content: + - type: text + text: 你好，这是我的语音输入测试，请慢速复述一遍 + - type: input_audio + input_audio: + data: base64_voice_xxx + format: wav + Function Call 示例: + value: + model: glm-5.1 + messages: + - role: user + content: 今天北京的天气怎么样？ + tools: + - type: function + function: + name: get_weather + description: 获取指定城市的天气信息 + parameters: + type: object + properties: + city: + type: string + description: 城市名称 + required: + - city + tool_choice: auto + temperature: 0.3 + required: true + responses: + '200': + description: 业务处理成功 + content: + application/json: + schema: + $ref: '#/components/schemas/ChatCompletionResponse' + text/event-stream: + schema: + $ref: '#/components/schemas/ChatCompletionChunk' + default: + description: 请求失败 + content: + application/json: + schema: + $ref: '#/components/schemas/Error' +components: + schemas: + ChatCompletionTextRequest: + required: + - model + - messages + type: object + description: 普通对话模型请求，支持纯文本对话和工具调用 + properties: + model: + type: string + description: >- + 调用的普通对话模型代码。`GLM-5.1` 和 `GLM-5-Turbo` 是最新的旗舰模型系列。`GLM-5` + 系列提供了复杂推理、超长上下文、极快推理速度等多款模型。 + example: glm-5.1 + default: glm-5.1 + enum: + - glm-5.1 + - glm-5-turbo + - glm-5 + - glm-4.7 + - glm-4.7-flash + - glm-4.7-flashx + - glm-4.6 + - glm-4.5-air + - glm-4.5-airx + - glm-4.5-flash + - glm-4-flash-250414 + - glm-4-flashx-250414 + messages: + type: array + description: >- + 对话消息列表，包含当前对话的完整上下文信息。每条消息都有特定的角色和内容，模型会根据这些消息生成回复。消息按时间顺序排列，支持四种角色：`system`（系统消息，用于设定`AI`的行为和角色）、`user`（用户消息，来自用户的输入）、`assistant`（助手消息，来自`AI`的回复）、`tool`（工具消息，工具调用的结果）。普通对话模型主要支持纯文本内容。注意不能只包含系统消息或助手消息。 + items: + oneOf: + - title: 用户消息 + type: object + properties: + role: + type: string + enum: + - user + description: 消息作者的角色 + default: user + content: + type: string + description: 文本消息内容 + example: >- + What opportunities and challenges will the Chinese large + model industry face in 2025? + required: + - role + - content + - title: 系统消息 + type: object + properties: + role: + type: string + enum: + - system + description: 消息作者的角色 + default: system + content: + type: string + description: 消息文本内容 + example: You are a helpful assistant. + required: + - role + - content + - title: 助手消息 + type: object + description: 可包含工具调用 + properties: + role: + type: string + enum: + - assistant + description: 消息作者的角色 + default: assistant + content: + type: string + description: 文本消息内容 + example: I'll help you with that analysis. + tool_calls: + type: array + description: 模型生成的工具调用消息。当提供此字段时，`content`通常为空。 + items: + type: object + properties: + id: + type: string + description: 工具调用ID + type: + type: string + description: 工具类型，支持 `web_search、retrieval、function` + enum: + - function + - web_search + - retrieval + function: + type: object + description: 函数调用信息，当`type`为`function`时不为空 + properties: + name: + type: string + description: 函数名称 + arguments: + type: string + description: 函数参数，`JSON`格式字符串 + required: + - name + - arguments + required: + - id + - type + required: + - role + - title: 工具消息 + type: object + properties: + role: + type: string + enum: + - tool + description: 消息作者的角色 + default: tool + content: + type: string + description: 消息文本内容 + example: 'Function executed successfully with result: ...' + tool_call_id: + type: string + description: 指示此消息对应的工具调用 `ID` + required: + - role + - content + minItems: 1 + stream: + type: boolean + example: false + default: false + description: >- + 是否启用流式输出模式。默认值为 `false`。当设置为 `false` + 时，模型会在生成完整响应后一次性返回所有内容，适合短文本生成和批处理场景。当设置为 `true` 时，模型会通过`Server-Sent + Events + (SSE)`流式返回生成的内容，用户可以实时看到文本生成过程，适合聊天对话和长文本生成场景，能提供更好的用户体验。流式输出结束时会返回 + `data: [DONE]` 消息。 + thinking: + $ref: '#/components/schemas/ChatThinking' + do_sample: + type: boolean + example: true + default: true + description: >- + 是否启用采样策略来生成文本。默认值为 `true`。当设置为 `true` 时，模型会使用 `temperature、top_p` + 等参数进行随机采样，生成更多样化的输出；当设置为 `false` 时，模型总是选择概率最高的词汇，生成更确定性的输出，此时 + `temperature` 和 `top_p` 参数将被忽略。对于需要一致性和可重复性的任务（如代码生成、翻译），建议设置为 + `false`。 + temperature: + type: number + description: >- + 采样温度，控制输出的随机性和创造性，取值范围为 `[0.0, 1.0]`，限两位小数。对于`GLM-5.1` `GLM-5` + `GLM-4.7` `GLM-4.6`系列默认值为 `1.0`，`GLM-4.5`系列默认值为 `0.6`，`GLM-4`系列默认值为 + `0.75`。较高的值（如`0.8`）会使输出更随机、更具创造性，适合创意写作和头脑风暴；较低的值（如`0.2`）会使输出更稳定、更确定，适合事实性问答和代码生成。建议根据应用场景调整 + `top_p` 或 `temperature` 参数，但不要同时调整两个参数。 + format: float + example: 1 + default: 1 + minimum: 0 + maximum: 1 + top_p: + type: number + description: >- + 核采样（`nucleus sampling`）参数，是`temperature`采样的替代方法，取值范围为 `[0.01, + 1.0]`，限两位小数。对于`GLM-5.1` `GLM-5` `GLM-4.7` `GLM-4.6` `GLM-4.5`系列默认值为 + `0.95`，`GLM-4`系列默认值为 + `0.9`。模型只考虑累积概率达到`top_p`的候选词汇。例如：`0.1`表示只考虑前`10%`概率的词汇，`0.9`表示考虑前`90%`概率的词汇。较小的值会产生更集中、更一致的输出；较大的值会增加输出的多样性。建议根据应用场景调整 + `top_p` 或 `temperature` 参数，但不建议同时调整两个参数。 + format: float + example: 0.95 + default: 0.95 + minimum: 0.01 + maximum: 1 + max_tokens: + type: integer + description: >- + 模型输出的最大令牌`token`数量限制。`GLM-5.1` `GLM-5` `GLM-4.7` + `GLM-4.6`系列最大支持`128K`输出长度，`GLM-4.5`系列最大支持`96K`输出长度，建议设置不小于`1024`。令牌是文本的基本单位，通常`1`个令牌约等于`0.75`个英文单词或`1.5`个中文字符。设置合适的`max_tokens`可以控制响应长度和成本，避免过长的输出。如果模型在达到`max_tokens`限制前完成回答，会自然结束；如果达到限制，输出可能被截断。 + + 默认值和最大值等更多详见 [max_tokens + 文档](/cn/guide/start/concept-param#max_tokens) + example: 1024 + minimum: 1 + maximum: 131072 + tool_stream: + type: boolean + example: false + default: false + description: >- + 是否开启流式响应`Function Calls`，仅限`GLM-5.1` `GLM-5` `GLM-5-Turbo` `GLM-4.7` + `GLM-4.6`系列支持此参数，默认值`false`。参考 + [工具流式输出](/cn/guide/capabilities/stream-tool) + tools: + type: array + description: >- + 模型可以调用的工具列表。支持函数调用、知识库检索和网络搜索。使用此参数提供模型可以生成 `JSON` + 输入的函数列表或配置其他工具。最多支持 `128` 个函数。目前 `GLM-4` 系列已支持所有 `tools`，`GLM-4.5` + 已支持 `web search` 和 `retrieval`。 + anyOf: + - items: + $ref: '#/components/schemas/FunctionToolSchema' + - items: + $ref: '#/components/schemas/RetrievalToolSchema' + - items: + $ref: '#/components/schemas/WebSearchToolSchema' + - items: + $ref: '#/components/schemas/MCPToolSchema' + tool_choice: + oneOf: + - type: string + enum: + - auto + description: 用于控制模型选择调用哪个函数的方式，仅在工具类型为`function`时补充。默认`auto`且仅支持`auto`。 + description: 控制模型如何选择工具。 + stop: + type: array + description: >- + 停止词列表，当模型生成的文本中遇到这些指定的字符串时会立即停止生成。目前仅支持单个停止词，格式为["stop_word1"]。停止词不会包含在返回的文本中。这对于控制输出格式、防止模型生成不需要的内容非常有用，例如在对话场景中可以设置["Human:"]来防止模型模拟用户发言。 + items: + type: string + maxItems: 1 + response_format: + type: object + description: >- + 指定模型的响应输出格式，默认为`text`，仅文本模型支持此字段。支持两种格式：{ "type": "text" } + 表示普通文本输出模式，模型返回自然语言文本；{ "type": "json_object" } + 表示`JSON`输出模式，模型会返回有效的`JSON`格式数据，适用于结构化数据提取、`API`响应生成等场景。使用`JSON`模式时，建议在提示词中明确说明需要`JSON`格式输出。 + properties: + type: + type: string + enum: + - text + - json_object + default: text + description: 输出格式类型：`text`表示普通文本输出，`json_object`表示`JSON`格式输出 + required: + - type + request_id: + type: string + description: 请求唯一标识符。由用户端传递，建议使用`UUID`格式确保唯一性，若未提供平台将自动生成。 + user_id: + type: string + description: 终端用户的唯一标识符。`ID`长度要求：最少`6`个字符，最多`128`个字符，建议使用不包含敏感信息的唯一标识。 + minLength: 6 + maxLength: 128 + ChatCompletionVisionRequest: + required: + - model + - messages + type: object + description: 视觉模型请求，支持多模态内容（文本、图片、视频、文件） + properties: + model: + type: string + description: >- + 调用的视觉模型代码。`GLM-5V-Turbo` + 系列支持视觉理解，具备卓越的多模态理解能力和工具调用能力。`AutoGLM-Phone` 是手机智能助理模型。 + example: glm-5v-turbo + default: glm-5v-turbo + enum: + - glm-5v-turbo + - glm-4.6v + - autoglm-phone + - glm-4.6v-flash + - glm-4.6v-flashx + - glm-4v-flash + - glm-4.1v-thinking-flashx + - glm-4.1v-thinking-flash + messages: + type: array + description: >- + 对话消息列表，包含当前对话的完整上下文信息。每条消息都有特定的角色和内容，模型会根据这些消息生成回复。消息按时间顺序排列，支持角色：`system`（系统消息，用于设定`AI`的行为和角色）、`user`（用户消息，来自用户的输入）、`assistant`（助手消息，来自`AI`的回复）。视觉模型支持纯文本和多模态内容（文本、图片、视频、文件）。注意不能只包含系统或助手消息。 + items: + oneOf: + - title: 用户消息 + type: object + properties: + role: + type: string + enum: + - user + description: 消息作者的角色 + default: user + content: + oneOf: + - type: array + description: 多模态消息内容，支持文本、图片、文件、视频（可从上方切换至文本消息） + items: + $ref: '#/components/schemas/VisionMultimodalContentItem' + - type: string + description: 文本消息内容（可从上方切换至多模态消息） + example: >- + What opportunities and challenges will the Chinese + large model industry face in 2025? + required: + - role + - content + - title: 系统消息 + type: object + properties: + role: + type: string + enum: + - system + description: 消息作者的角色 + default: system + content: + oneOf: + - type: string + description: 消息文本内容 + example: You are a helpful assistant. + required: + - role + - content + - title: 助手消息 + type: object + properties: + role: + type: string + enum: + - assistant + description: 消息作者的角色 + default: assistant + content: + oneOf: + - type: string + description: 文本消息内容 + example: I'll help you with that analysis. + required: + - role + minItems: 1 + stream: + type: boolean + example: false + default: false + description: >- + 是否启用流式输出模式。默认值为 `false`。当设置为 `false` + 时，模型会在生成完整响应后一次性返回所有内容，适合短文本生成和批处理场景。当设置为 `true` 时，模型会通过`Server-Sent + Events + (SSE)`流式返回生成的内容，用户可以实时看到文本生成过程，适合聊天对话和长文本生成场景，能提供更好的用户体验。流式输出结束时会返回 + `data: [DONE]` 消息。 + thinking: + $ref: '#/components/schemas/ChatThinking' + do_sample: + type: boolean + example: true + default: true + description: >- + 是否启用采样策略来生成文本。默认值为 `true`。当设置为 `true` 时，模型会使用 `temperature、top_p` + 等参数进行随机采样，生成更多样化的输出；当设置为 `false` 时，模型总是选择概率最高的词汇，生成更确定性的输出，此时 + `temperature` 和 `top_p` 参数将被忽略。对于需要一致性和可重复性的任务（如代码生成、翻译），建议设置为 + `false`。 + temperature: + type: number + description: >- + 采样温度，控制输出的随机性和创造性，取值范围为 `[0.0, + 1.0]`，限两位小数。对于`GLM-5V-Turbo`，`GLM-4.6V`, `GLM-4.5V`系列默认值为 + `0.8`，`AutoGLM-Phone`默认值为 `0.0`，`GLM-4.1v`系列默认值为 + `0.8`。较高的值（如`0.8`）会使输出更随机、更具创造性，适合创意写作和头脑风暴；较低的值（如`0.2`）会使输出更稳定、更确定，适合事实性问答和代码生成。建议根据应用场景调整 + `top_p` 或 `temperature` 参数，但不要同时调整两个参数。 + format: float + example: 0.8 + default: 0.8 + minimum: 0 + maximum: 1 + top_p: + type: number + description: >- + 核采样（`nucleus sampling`）参数，是`temperature`采样的替代方法，取值范围为 `[0.01, + 1.0]`，限两位小数。对于`GLM-5V-Turbo`，`GLM-4.6V`, `GLM-4.5V`系列默认值为 + `0.6`，`AutoGLM-Phone`默认值为 `0.85`，`GLM-4.1v`系列默认值为 + `0.6`。模型只考虑累积概率达到`top_p`的候选词汇。例如：`0.1`表示只考虑前`10%`概率的词汇，`0.9`表示考虑前`90%`概率的词汇。较小的值会产生更集中、更一致的输出；较大的值会增加输出的多样性。建议根据应用场景调整 + `top_p` 或 `temperature` 参数，但不要同时调整两个参数。 + format: float + example: 0.6 + default: 0.6 + minimum: 0.01 + maximum: 1 + max_tokens: + type: integer + description: >- + 模型输出的最大令牌`token`数量限制。`GLM-5V-Turbo`最大支持`128K`输出长度，`GLM-4.6V`最大支持`32K`输出长度，`GLM-4.5V`最大支持`16K`输出长度，`AutoGLM-Phone`最大支持`4K`输出长度，`GLM-4.1v`系列最大支持`16K`输出长度，建议设置不小于`1024`。令牌是文本的基本单位，通常`1`个令牌约等于`0.75`个英文单词或`1.5`个中文字符。设置合适的`max_tokens`可以控制响应长度和成本，避免过长的输出。如果模型在达到`max_tokens`限制前完成回答，会自然结束；如果达到限制，输出可能被截断。 + + 默认值和最大值等更多详见 [max_tokens + 文档](/cn/guide/start/concept-param#max_tokens) + example: 1024 + minimum: 1 + maximum: 131072 + tools: + type: array + description: >- + 模型可以调用的工具列表。仅限`GLM-4.6V`和`AutoGLM-Phone`支持。使用此参数提供模型可以生成 `JSON` + 输入的函数列表或配置其他工具。最多支持 `128` 个函数。 + anyOf: + - items: + $ref: '#/components/schemas/FunctionToolSchema' + tool_choice: + oneOf: + - type: string + enum: + - auto + description: >- + 用于控制模型选择调用哪个函数的方式，仅在工具类型为`function`时补充，仅限`GLM-4.6V`支持此参数。默认`auto`且仅支持`auto`。 + description: 控制模型如何选择工具。 + stop: + type: array + description: >- + 停止词列表，当模型生成的文本中遇到这些指定的字符串时会立即停止生成。目前仅支持单个停止词，格式为["stop_word1"]。停止词不会包含在返回的文本中。这对于控制输出格式、防止模型生成不需要的内容非常有用，例如在对话场景中可以设置["Human:"]来防止模型模拟用户发言。 + items: + type: string + maxItems: 1 + request_id: + type: string + description: 请求唯一标识符。由用户端传递，建议使用`UUID`格式确保唯一性，若未提供平台将自动生成。 + user_id: + type: string + description: 终端用户的唯一标识符。`ID`长度要求：最少`6`个字符，最多`128`个字符，建议使用不包含敏感信息的唯一标识。 + minLength: 6 + maxLength: 128 + ChatCompletionAudioRequest: + required: + - model + - messages + type: object + description: 音频模型请求，支持语音理解、生成和识别功能 + properties: + model: + type: string + description: 调用的音频模型代码。`GLM-4-Voice` 支持语音理解和生成。 + example: glm-4-voice + default: glm-4-voice + enum: + - glm-4-voice + - 禁用仅占位 + messages: + type: array + description: >- + 对话消息列表，包含当前对话的完整上下文信息。每条消息都有特定的角色和内容，模型会根据这些消息生成回复。消息按时间顺序排列，支持角色：`system`（系统消息，用于设定`AI`的行为和角色）、`user`（用户消息，来自用户的输入）、`assistant`（助手消息，来自`AI`的回复）。音频模型支持文本和音频内容。注意不能只包含系统或助手消息。 + items: + oneOf: + - title: 用户消息 + type: object + properties: + role: + type: string + enum: + - user + description: 消息作者的角色 + default: user + content: + oneOf: + - type: array + description: 多模态消息内容，支持文本、音频 + items: + $ref: '#/components/schemas/AudioMultimodalContentItem' + - type: string + description: 消息文本内容 + example: You are a helpful assistant. + required: + - role + - content + - title: 系统消息 + type: object + properties: + role: + type: string + enum: + - system + description: 消息作者的角色 + default: system + content: + type: string + description: 消息文本内容 + example: 你是一个专业的语音助手，能够理解和生成自然语音。 + required: + - role + - content + - title: 助手消息 + type: object + properties: + role: + type: string + enum: + - assistant + description: 消息作者的角色 + default: assistant + content: + oneOf: + - type: string + description: 文本消息内容 + example: I'll help you with that analysis. + audio: + type: object + description: 语音消息 + properties: + id: + type: string + description: 语音消息`id`，用于多轮对话 + required: + - role + minItems: 1 + stream: + type: boolean + example: false + default: false + description: >- + 是否启用流式输出模式。默认值为 `false`。当设置为 `false` + 时，模型会在生成完整响应后一次性返回所有内容，适合语音识别和批处理场景。当设置为 `true` 时，模型会通过`Server-Sent + Events + (SSE)`流式返回生成的内容，用户可以实时看到文本生成过程，适合实时语音对话场景，能提供更好的用户体验。流式输出结束时会返回 + `data: [DONE]` 消息。 + do_sample: + type: boolean + example: true + default: true + description: >- + 是否启用采样策略来生成文本。默认值为 `true`。当设置为 `true` 时，模型会使用 `temperature、top_p` + 等参数进行随机采样，生成更多样化的输出；当设置为 `false` 时，模型总是选择概率最高的词汇，生成更确定性的输出，此时 + `temperature` 和 `top_p` 参数将被忽略。对于需要一致性和可重复性的任务（如语音识别、转录），建议设置为 + `false`。 + temperature: + type: number + description: >- + 采样温度，控制输出的随机性和创造性，取值范围为 `[0.0, 1.0]`，限两位小数。对于`GLM-4-Voice`默认值为 + `0.8`。较高的值（如`0.8`）会使输出更随机、更具创造性，适合语音生成和对话；较低的值（如`0.1`）会使输出更稳定、更确定，适合语音识别和转录。建议根据应用场景调整 + `top_p` 或 `temperature` 参数，但不要同时调整两个参数。 + format: float + example: 0.8 + default: 0.8 + minimum: 0 + maximum: 1 + top_p: + type: number + description: >- + 核采样（`nucleus sampling`）参数，是`temperature`采样的替代方法，取值范围为 `[0.01, + 1.0]`，限两位小数。对于`GLM-4-Voice`默认值为 + `0.6`。模型只考虑累积概率达到`top_p`的候选词汇。例如：`0.1`表示只考虑前`10%`概率的词汇，`0.9`表示考虑前`90%`概率的词汇。较小的值会产生更集中、更一致的输出；较大的值会增加输出的多样性。建议根据应用场景调整 + `top_p` 或 `temperature` 参数，但不要同时调整两个参数。 + format: float + example: 0.6 + default: 0.6 + minimum: 0.01 + maximum: 1 + max_tokens: + type: integer + description: 模型输出的最大令牌`token`数量限制。`GLM-4-Voice`最大支持`4K`输出长度，默认`1024`。令牌是文本的基本单位。 + example: 1024 + minimum: 1 + maximum: 4096 + watermark_enabled: + type: boolean + description: |- + 控制`AI`生成图片时是否添加水印。 + - `true`: 默认启用`AI`生成的显式水印及隐式数字水印，符合政策要求。 + - `false`: 关闭所有水印，仅允许已签署免责声明的客户使用，签署路径：个人中心-安全管理-去水印管理 + example: true + stop: + type: array + description: >- + 停止词列表，当模型生成的文本中遇到这些指定的字符串时会立即停止生成。目前仅支持单个停止词，格式为["stop_word1"]。停止词不会包含在返回的文本中。这对于控制输出格式、防止模型生成不需要的内容非常有用。 + items: + type: string + maxItems: 1 + request_id: + type: string + description: 请求唯一标识符。由用户端传递，建议使用`UUID`格式确保唯一性，若未提供平台将自动生成。 + user_id: + type: string + description: 终端用户的唯一标识符。`ID`长度要求：最少`6`个字符，最多`128`个字符，建议使用不包含敏感信息的唯一标识。 + minLength: 6 + maxLength: 128 + ChatCompletionHumanOidRequest: + required: + - model + - messages + type: object + description: 角色扮演，专业心理咨询专用模型 + properties: + model: + type: string + description: 调用的专用模型代码。`CharGLM-4` 是角色扮演专用模型，`Emohaa` 是专业心理咨询模型。 + example: charglm-4 + default: charglm-4 + enum: + - charglm-4 + - emohaa + meta: + type: object + description: 角色及用户信息数据(仅限 `Emohaa` 支持此参数) + required: + - user_info + - bot_info + - bot_name + - user_name + properties: + user_info: + type: string + description: 用户信息描述 + bot_info: + type: string + description: 角色信息描述 + bot_name: + type: string + description: 角色名称 + user_name: + type: string + description: 用户名称 + messages: + type: array + description: >- + 对话消息列表，包含当前对话的完整上下文信息。每条消息都有特定的角色和内容，模型会根据这些消息生成回复。消息按时间顺序排列，支持角色：`system`（系统消息，用于设定`AI`的行为和角色）、`user`（用户消息，来自用户的输入）、`assistant`（助手消息，来自`AI`的回复）。注意不能只包含系统消息或助手消息。 + items: + oneOf: + - title: 用户消息 + type: object + properties: + role: + type: string + enum: + - user + description: 消息作者的角色 + default: user + content: + type: string + description: 文本消息内容 + example: 我最近工作压力很大，经常感到焦虑，不知道该怎么办 + required: + - role + - content + - title: 系统消息 + type: object + properties: + role: + type: string + enum: + - system + description: 消息作者的角色 + default: system + content: + type: string + description: 消息文本内容 + example: >- + 你乃苏东坡。人生如梦，何不活得潇洒一些？在这忙碌纷繁的现代生活中，帮助大家找到那份属于自己的自在与豁达，共赏人生之美好 + required: + - role + - content + - title: 助手消息 + type: object + properties: + role: + type: string + enum: + - assistant + description: 消息作者的角色 + default: assistant + content: + type: string + description: 文本消息内容 + example: I'll help you with that analysis. + required: + - role + - content + minItems: 1 + stream: + type: boolean + example: false + default: false + description: >- + 是否启用流式输出模式。默认值为 `false`。当设置为 `fals`e + 时，模型会在生成完整响应后一次性返回所有内容，适合语音识别和批处理场景。当设置为 `true` 时，模型会通过`Server-Sent + Events + (SSE)`流式返回生成的内容，用户可以实时看到文本生成过程，适合实时语音对话场景，能提供更好的用户体验。流式输出结束时会返回 + `data: [DONE]` 消息。 + do_sample: + type: boolean + example: true + default: true + description: >- + 是否启用采样策略来生成文本。默认值为 `true`。当设置为 `true` 时，模型会使用 `temperature、top_p` + 等参数进行随机采样，生成更多样化的输出；当设置为 `false` 时，模型总是选择概率最高的词汇，生成更确定性的输出，此时 + `temperatur`e 和 `top_p` 参数将被忽略。对于需要一致性和可重复性的任务（如语音识别、转录），建议设置为 + `false`。 + temperature: + type: number + description: >- + 采样温度，控制输出的随机性和创造性，取值范围为 `[0.0, 1.0]`，限两位小数。`Charglm-4` 和 `Emohaa` + 默认值为 `0.95`。建议根据应用场景调整 `top_p` 或 `temperature` 参数，但不要同时调整两个参数。 + format: float + example: 0.8 + default: 0.8 + minimum: 0 + maximum: 1 + top_p: + type: number + description: >- + 核采样（`nucleus sampling`）参数，是`temperature`采样的替代方法，取值范围为 `[0.01, + 1.0]`，限两位小数。`Charglm-4` 和 `Emohaa` 默认值为 `0.7`。建议根据应用场景调整 `top_p` 或 + `temperature` 参数，但不要同时调整两个参数。 + format: float + example: 0.6 + default: 0.6 + minimum: 0.01 + maximum: 1 + max_tokens: + type: integer + description: >- + 模型输出的最大令牌`token`数量限制。`Charglm-4` 和 `Emohaa` + 最大支持`4K`输出长度，默认`1024`。令牌是文本的基本单位。 + example: 1024 + minimum: 1 + maximum: 4096 + stop: + type: array + description: >- + 停止词列表，当模型生成的文本中遇到这些指定的字符串时会立即停止生成。目前仅支持单个停止词，格式为["stop_word1"]。停止词不会包含在返回的文本中。这对于控制输出格式、防止模型生成不需要的内容非常有用。 + items: + type: string + maxItems: 1 + request_id: + type: string + description: 请求唯一标识符。由用户端传递，建议使用`UUID`格式确保唯一性，若未提供平台将自动生成。 + user_id: + type: string + description: 终端用户的唯一标识符。`ID`长度要求：最少`6`个字符，最多`128`个字符，建议使用不包含敏感信息的唯一标识。 + minLength: 6 + maxLength: 128 + ChatCompletionResponse: + type: object + properties: + id: + description: 任务 `ID` + type: string + request_id: + description: 请求 `ID` + type: string + created: + description: 请求创建时间，`Unix` 时间戳（秒） + type: integer + model: + description: 模型名称 + type: string + choices: + type: array + description: 模型响应列表 + items: + type: object + properties: + index: + type: integer + description: 结果索引 + message: + $ref: '#/components/schemas/ChatCompletionResponseMessage' + finish_reason: + type: string + description: >- + 推理终止原因。'stop’表示自然结束或触发stop词，'tool_calls’表示模型命中函数，'length’表示达到token长度限制，'sensitive’表示内容被安全审核接口拦截（用户应判断并决定是否撤回公开内容），'network_error’表示模型推理异常，'model_context_window_exceeded'表示超出模型上下文窗口。 + usage: + type: object + description: 调用结束时返回的 `Token` 使用统计。 + properties: + prompt_tokens: + type: number + description: 用户输入的 `Token` 数量。 + completion_tokens: + type: number + description: 输出的 `Token` 数量 + prompt_tokens_details: + type: object + properties: + cached_tokens: + type: number + description: 命中的缓存 `Token` 数量 + total_tokens: + type: integer + description: '`Token` 总数，对于 `glm-4-voice` 模型，`1`秒音频=`12.5 Tokens`，向上取整' + video_result: + type: array + description: 视频生成结果。 + items: + type: object + properties: + url: + type: string + description: 视频链接。 + cover_image_url: + type: string + description: 视频封面链接。 + web_search: + type: array + description: 返回与网页搜索相关的信息，使用`WebSearchToolSchema`时返回 + items: + type: object + properties: + icon: + type: string + description: 来源网站的图标 + title: + type: string + description: 搜索结果的标题 + link: + type: string + description: 搜索结果的网页链接 + media: + type: string + description: 搜索结果网页的媒体来源名称 + publish_date: + type: string + description: 网站发布时间 + content: + type: string + description: 搜索结果网页引用的文本内容 + refer: + type: string + description: 角标序号 + content_filter: + type: array + description: 返回内容安全的相关信息 + items: + type: object + properties: + role: + type: string + description: >- + 安全生效环节，包括 `role = assistant` 模型推理，`role = user` 用户输入，`role = + history` 历史上下文 + level: + type: integer + description: 严重程度 `level 0-3`，`level 0`表示最严重，`3`表示轻微 + ChatCompletionChunk: + type: object + properties: + id: + type: string + description: 任务 ID + created: + description: 请求创建时间，`Unix` 时间戳（秒） + type: integer + model: + description: 模型名称 + type: string + choices: + type: array + description: 模型响应列表 + items: + type: object + properties: + index: + type: integer + description: 结果索引 + delta: + type: object + description: 模型增量返回的文本信息 + properties: + role: + type: string + description: 当前对话的角色，目前默认为 `assistant`（模型） + content: + oneOf: + - type: string + description: >- + 当前对话文本内容。如果调用函数则为 `null`，否则返回推理结果。 + + 对于`GLM-4.5V`系列模型，返回内容可能包含思考过程标签 ` + `，文本边界标签 `<|begin_of_box|> <|end_of_box|>`。 + - type: array + description: 当前对话的多模态内容（适用于`GLM-4V`系列） + items: + type: object + properties: + type: + type: string + enum: + - text + description: 内容类型，目前为文本 + text: + type: string + description: 文本内容 + - type: string + nullable: true + description: 当使用`tool_calls`时，`content`可能为`null` + audio: + type: object + description: 当使用 `glm-4-voice` 模型时返回的音频内容 + properties: + id: + type: string + description: 当前对话的音频内容`id`，可用于多轮对话输入 + data: + type: string + description: 当前对话的音频内容`base64`编码 + expires_at: + type: string + description: 当前对话的音频内容过期时间 + reasoning_content: + type: string + description: 思维链内容, 仅 `glm-4.5` 系列支持 + tool_calls: + type: array + description: 生成的应该被调用的工具信息，流式返回时会逐步生成 + items: + type: object + properties: + index: + type: integer + description: 工具调用索引 + id: + type: string + description: 工具调用的唯一标识符 + type: + type: string + description: 工具类型，目前支持`function` + enum: + - function + function: + type: object + properties: + name: + type: string + description: 函数名称 + arguments: + type: string + description: 函数参数，`JSON`格式字符串 + finish_reason: + type: string + description: >- + 模型推理终止的原因。`stop` 表示自然结束或触发stop词，`tool_calls` 表示模型命中函数，`length` + 表示达到 `token` 长度限制，`sensitive` + 表示内容被安全审核接口拦截（用户应判断并决定是否撤回公开内容），`network_error` + 表示模型推理异常，'model_context_window_exceeded'表示超出模型上下文窗口。 + enum: + - stop + - length + - tool_calls + - sensitive + - network_error + usage: + type: object + description: 本次模型调用的 `tokens` 数量统计 + properties: + prompt_tokens: + type: integer + description: 用户输入的 `tokens` 数量。对于 `glm-4-voice`，`1`秒音频=`12.5 Tokens`，向上取整。 + completion_tokens: + type: integer + description: 模型输出的 `tokens` 数量 + total_tokens: + type: integer + description: 总 `tokens` 数量，对于 `glm-4-voice` 模型，`1`秒音频=`12.5 Tokens`，向上取整 + content_filter: + type: array + description: 返回内容安全的相关信息 + items: + type: object + properties: + role: + type: string + description: >- + 安全生效环节，包括：`role = assistant` 模型推理，`role = user` 用户输入，`role = + history` 历史上下文 + level: + type: integer + description: 严重程度 `level 0-3`，`level 0` 表示最严重，`3` 表示轻微 + Error: + type: object + properties: + error: + required: + - code + - message + type: object + properties: + code: + type: string + message: + type: string + ChatThinking: + type: object + description: 仅 `GLM-4.5` 及以上模型支持此参数配置. 控制大模型是否开启思维链。 + properties: + type: + type: string + description: >- + 是否开启思维链(当开启后 `GLM-5.1` `GLM-5` `GLM-5v-Turbo` `GLM-4.7` `GLM-4.5V` + 为强制思考，`GLM-4.6` `GLM-4.6V` `GLM-4.5` 为模型自动判断是否思考), 默认: `enabled`. + default: enabled + enum: + - enabled + - disabled + clear_thinking: + type: boolean + description: >- + 默认为 `True`。用于控制是否清除历史对话轮次（`previous turns`）中的 `reasoning_content`。详见 + [思考模式](/cn/guide/capabilities/thinking-mode) + - `true`（默认）：在本次请求中，系统会忽略/移除历史 `turns` 的 `reasoning_content`，仅使用非推理内容（如用户/助手可见文本、工具调用与结果等）作为上下文输入。适用于普通对话与轻量任务，可降低上下文长度与成本 + - `false`：保留历史 `turns` 的 `reasoning_content` 并随上下文一同提供给模型。若你希望启用 `Preserved Thinking`，必须在 `messages` 中完整、未修改、按原顺序透传历史 `reasoning_content`；缺失、裁剪、改写或重排会导致效果下降或无法生效。 + - 注意：该参数只影响跨 `turn` 的历史 `thinking blocks`；不改变模型在当前 `turn` 内是否产生/输出 `thinking` + default: true + example: true + FunctionToolSchema: + type: object + title: Function Call + properties: + type: + type: string + default: function + enum: + - function + function: + $ref: '#/components/schemas/FunctionObject' + required: + - type + - function + additionalProperties: false + RetrievalToolSchema: + type: object + title: Retrieval + properties: + type: + type: string + default: retrieval + enum: + - retrieval + retrieval: + $ref: '#/components/schemas/RetrievalObject' + required: + - type + - retrieval + additionalProperties: false + WebSearchToolSchema: + type: object + title: Web Search + properties: + type: + type: string + default: web_search + enum: + - web_search + web_search: + $ref: '#/components/schemas/WebSearchObject' + required: + - type + - web_search + additionalProperties: false + MCPToolSchema: + type: object + title: MCP + properties: + type: + type: string + default: mcp + enum: + - mcp + mcp: + $ref: '#/components/schemas/MCPObject' + required: + - type + - mcp + additionalProperties: false + VisionMultimodalContentItem: + oneOf: + - title: 文本 + type: object + properties: + type: + type: string + enum: + - text + description: 内容类型为文本 + default: text + text: + type: string + description: 文本内容 + required: + - type + - text + additionalProperties: false + - title: 图片 + type: object + properties: + type: + type: string + enum: + - image_url + description: 内容类型为图片`URL` + default: image_url + image_url: + type: object + description: 图片信息 + properties: + url: + type: string + description: >- + 图片的`URL`地址或`Base64`编码。图像大小上传限制为每张图像`5M`以下，且像素不超过`6000*6000`。支持`jpg、png、jpeg`格式。`GLM-5V-Turbo` + `GLM4.6V` `GLM4.5V` 系列限制`50`张。`GLM-4V-Plus-0111` + 限制`5`张。`GLM-4V-Flash`限制`1`张图像且不支持`Base64`编码。 + required: + - url + additionalProperties: false + required: + - type + - image_url + additionalProperties: false + - title: 视频 + type: object + properties: + type: + type: string + enum: + - video_url + description: 内容类型为视频输入 + default: video_url + video_url: + type: object + description: 视频信息。注意：`GLM-4V-Plus-0111` 的 `video_url` 参数必须在 `content` 数组的第一位。 + properties: + url: + type: string + description: >- + 视频的`URL`地址。`GLM-5V-Turbo` `GLM-4.6V` `GLM-4.5V` 系列视频大小限制为 + `200M` + 以内。`GLM-4V-Plus`视频大小限制为`20M`以内，视频时长不超过`30s`。对于其他多模态模型，视频大小限制为`200M`以内。视频类型：`mp4`，`mkv`，`mov`。 + required: + - url + additionalProperties: false + required: + - type + - video_url + additionalProperties: false + - title: 文件 + type: object + properties: + type: + type: string + enum: + - file_url + description: >- + 内容类型为文件输入(仅`GLM-5V-Turbo` `GLM-4.6V` `GLM-4.5V`支持，且不支持同时传入 + `file_url` 和 `image_url` 或 `video_url` 参数) + default: file_url + file_url: + type: object + description: 文件信息。 + properties: + url: + type: string + description: >- + 文件的`URL`地址，不支持`Base64`编码。支持`pdf、txt、word、jsonl、xlsx、pptx`等格式，最多支持`50`个。 + required: + - url + additionalProperties: false + required: + - type + - file_url + additionalProperties: false + AudioMultimodalContentItem: + oneOf: + - title: 文本 + type: object + properties: + type: + type: string + enum: + - text + description: 内容类型为文本 + default: text + text: + type: string + description: 文本内容 + required: + - type + - text + additionalProperties: false + - title: 音频 + type: object + properties: + type: + type: string + enum: + - input_audio + description: 内容类型为音频输入 + default: input_audio + input_audio: + type: object + description: 音频信息，仅`glm-4-voice`支持音频输入 + properties: + data: + type: string + description: 语音文件的`base64`编码。音频最长不超过 `10` 分钟。`1s`音频=`12.5 Tokens`，向上取整。 + format: + type: string + description: 语音文件的格式，支持`wav`和`mp3` + enum: + - wav + - mp3 + required: + - data + - format + additionalProperties: false + required: + - type + - input_audio + additionalProperties: false + ChatCompletionResponseMessage: + type: object + properties: + role: + type: string + description: 当前对话角色，默认为 `assistant` + example: assistant + content: + oneOf: + - type: string + description: >- + 当前对话文本内容。如果调用函数则为 `null`，否则返回推理结果。 + + 对于`GLM-4.5V`系列模型，返回内容可能包含思考过程标签 ` `，文本边界标签 + `<|begin_of_box|> <|end_of_box|>`。 + - type: array + description: 多模态回复内容，适用于`GLM-4V`系列模型 + items: + type: object + properties: + type: + type: string + enum: + - text + description: 回复内容类型，目前为文本 + text: + type: string + description: 文本内容 + - type: string + nullable: true + description: 当使用`tool_calls`时，`content`可能为`null` + reasoning_content: + type: string + description: 思维链内容，仅在使用 `glm-4.5` 系列, `glm-4.1v-thinking` 系列模型时返回。 + audio: + type: object + description: 当使用 `glm-4-voice` 模型时返回的音频内容 + properties: + id: + type: string + description: 当前对话的音频内容`id`，可用于多轮对话输入 + data: + type: string + description: 当前对话的音频内容`base64`编码 + expires_at: + type: string + description: 当前对话的音频内容过期时间 + tool_calls: + type: array + description: 生成的应该被调用的函数名称和参数。 + items: + $ref: '#/components/schemas/ChatCompletionResponseMessageToolCall' + FunctionObject: + type: object + properties: + name: + type: string + description: 要调用的函数名称。必须是 `a-z、A-Z、0-9`，或包含下划线和破折号，最大长度为 `64`。 + minLength: 1 + maxLength: 64 + pattern: ^[a-zA-Z0-9_-]+$ + description: + type: string + description: 函数功能的描述，供模型选择何时以及如何调用函数。 + parameters: + $ref: '#/components/schemas/FunctionParameters' + required: + - name + - description + - parameters + RetrievalObject: + type: object + properties: + knowledge_id: + type: string + description: 知识库 `ID`，从平台创建或获取 + prompt_template: + type: string + description: >- + 请求模型的提示模板，包含占位符 `{{ knowledge }}` 和 `{{ question }}` + 的自定义请求模板。默认模板：`在文档 `{{ knowledge }}` 中搜索问题 `{{question}}` + 的答案。如果找到答案，仅使用文档中的陈述进行回应；如果没有找到答案，使用你自己的知识回答并告知用户信息不来自文档。不要重复问题，直接开始答案。` + required: + - knowledge_id + WebSearchObject: + type: object + properties: + enable: + type: boolean + description: 是否启用搜索功能，默认值为 `false`，启用时设置为 `true` + search_engine: + type: string + description: >- + 搜索引擎类型，默认为 + `search_std`；支持`search_std、search_pro、search_pro_sogou、search_pro_quark`。 + enum: + - search_std + - search_pro + - search_pro_sogou + - search_pro_quark + search_query: + type: string + description: 强制触发搜索 + search_intent: + type: string + description: >- + 是否进行搜索意图识别，默认执行搜索意图识别。`true`：执行搜索意图识别，有搜索意图后执行搜索；`false`：跳过搜索意图识别，直接执行搜索 + count: + type: integer + description: >- + 返回结果的条数。可填范围：`1-50`，最大单次搜索返回`50`条，默认为`10`。支持的搜索引擎：`search_std、search_pro、search_pro_sogou`。对于`search_pro_sogou`: + 可选枚举值，`10、20、30、40、50` + minimum: 1 + maximum: 50 + search_domain_filter: + type: string + description: |- + 用于限定搜索结果的范围，仅返回指定白名单域名的内容。 + 白名单域名:（如 `www.example.com`）。 + 支持的搜索引擎：`search_std、search_pro、search_pro_sogou` + search_recency_filter: + type: string + description: >- + 搜索指定时间范围内的网页。默认为`noLimit`。可填值：`oneDay`（一天内）、`oneWeek`（一周内）、`oneMonth`（一个月内）、`oneYear`（一年内）、`noLimit`（不限，默认）。支持的搜索引擎：`search_std、search_pro、search_pro_sogou、search_pro_quark` + enum: + - oneDay + - oneWeek + - oneMonth + - oneYear + - noLimit + content_size: + type: string + description: >- + 控制网页摘要的字数。默认值为`medium`。`medium`：返回摘要信息，满足大模型的基础推理需求。`high`：最大化上下文，信息量较大但内容详细，适合需要信息细节的场景。 + enum: + - medium + - high + result_sequence: + type: string + description: 指定搜索结果返回的顺序是在模型回复结果之前还是之后，可选值：`before`、`after`，默认 `after` + enum: + - before + - after + search_result: + type: boolean + description: 是否返回搜索来源的详细信息，默认值 `false` + require_search: + type: boolean + description: 是否强制搜索结果才返回回答，默认值 `false` + search_prompt: + type: string + description: |- + 用于定制搜索结果处理的`Prompt`，默认`Prompt`： + + 你是一位智能问答专家，具备整合信息的能力，能够进行时间识别、语义理解与矛盾信息清洗处理。 + 当前日期是{{current_date}}，请以此时间为唯一基准，参考以下信息，全面、准确地回答用户问题。 + 仅提炼有价值的内容用于回答，确保答案具有实时性与权威性，直接陈述答案，无需说明数据来源或内部处理过程。 + required: + - search_engine + MCPObject: + type: object + properties: + server_label: + description: >- + `mcp server`标识，如果连接智谱的`mcp server`，以`mcp + code`填充该字段，且无需填写`server_url` + type: string + server_url: + description: '`mcp server`地址' + type: string + transport_type: + description: 传输类型 + type: string + default: streamable-http + enum: + - sse + - streamable-http + allowed_tools: + description: 允许的工具集合 + type: array + items: + type: string + headers: + description: '`mcp server` 需要的鉴权信息' + type: object + required: + - server_label + ChatCompletionResponseMessageToolCall: + type: object + properties: + function: + type: object + description: 包含生成的函数名称和 `JSON` 格式参数。 + properties: + name: + type: string + description: 生成的函数名称。 + arguments: + type: string + description: 生成的函数调用参数的 `JSON` 格式字符串。调用函数前请验证参数。 + required: + - name + - arguments + mcp: + type: object + description: '`MCP` 工具调用参数' + properties: + id: + description: '`mcp` 工具调用唯一标识' + type: string + type: + description: 工具调用类型, 例如 `mcp_list_tools, mcp_call` + type: string + enum: + - mcp_list_tools + - mcp_call + server_label: + description: '`MCP`服务器标签' + type: string + error: + description: 错误信息 + type: string + tools: + description: '`type = mcp_list_tools` 时的工具列表' + type: array + items: + type: object + properties: + name: + description: 工具名称 + type: string + description: + description: 工具描述 + type: string + annotations: + description: 工具注解 + type: object + input_schema: + description: 工具输入参数规范 + type: object + properties: + type: + description: 固定值 'object' + type: string + default: object + enum: + - object + properties: + description: 参数属性定义 + type: object + required: + description: 必填属性列表 + type: array + items: + type: string + additionalProperties: + description: 是否允许额外参数 + type: boolean + arguments: + description: 工具调用参数，参数为 `json` 字符串 + type: string + name: + description: 工具名称 + type: string + output: + description: 工具返回的结果输出 + type: object + id: + type: string + description: 命中函数的唯一标识符。 + type: + type: string + description: 调用的工具类型，目前仅支持 'function', 'mcp'。 + FunctionParameters: + type: object + description: 使用 `JSON Schema` 定义的参数。必须传递 `JSON Schema` 对象以准确定义接受的参数。如果调用函数时不需要参数，则省略。 + additionalProperties: true + securitySchemes: + bearerAuth: + type: http + scheme: bearer + description: >- + 使用以下格式进行身份验证：Bearer [](https://bigmodel.cn/usercenter/proj-mgmt/apikeys) + +```` \ No newline at end of file diff --git a/docs/协议文档/智谱GLM/GLM对话补全（异步）.md b/docs/协议文档/智谱GLM/GLM对话补全（异步）.md new file mode 100644 index 0000000..488ea32 --- /dev/null +++ b/docs/协议文档/智谱GLM/GLM对话补全（异步）.md @@ -0,0 +1,1276 @@ +> ## Documentation Index +> Fetch the complete documentation index at: https://docs.bigmodel.cn/llms.txt +> Use this file to discover all available pages before exploring further. + +# 对话补全(异步) + +> 和 [指定模型](/cn/guide/start/model-overview) 对话，通过查询异步结果获取模型响应。支持多种模型，支持多模态（文本、图片、音频、视频、文件），可配置采样，温度，最大令牌数，工具调用等。注意此为异步接口，通过 [查询异步结果](/api-reference/%E6%A8%A1%E5%9E%8B-api/%E6%9F%A5%E8%AF%A2%E5%BC%82%E6%AD%A5%E7%BB%93%E6%9E%9C) 获取生成结果。 + + + +## OpenAPI + +````yaml /openapi/openapi.json post /paas/v4/async/chat/completions +openapi: 3.0.1 +info: + title: ZHIPU AI API + description: ZHIPU AI 接口提供强大的 AI 能力，包括聊天对话、工具调用和视频生成。 + license: + name: ZHIPU AI 开发者协议和政策 + url: https://chat.z.ai/legal-agreement/terms-of-service + version: 1.0.0 + contact: + name: Z.AI 开发者 + url: https://chat.z.ai/legal-agreement/privacy-policy + email: user_feedback@z.ai +servers: + - url: https://open.bigmodel.cn/api/ + description: 开放平台服务 +security: + - bearerAuth: [] +tags: + - name: 模型 API + description: Chat API + - name: 工具 API + description: Web Search API + - name: Agent API + description: Agent API + - name: 文件 API + description: File API + - name: 知识库 API + description: Knowledge API + - name: 实时 API + description: Realtime API + - name: 批处理 API + description: Batch API + - name: 助理 API + description: Assistant API + - name: 智能体 API（旧） + description: QingLiu Agent API +paths: + /paas/v4/async/chat/completions: + post: + tags: + - 模型 API + summary: 对话补全(异步) + description: >- + 和 [指定模型](/cn/guide/start/model-overview) + 对话，通过查询异步结果获取模型响应。支持多种模型，支持多模态（文本、图片、音频、视频、文件），可配置采样，温度，最大令牌数，工具调用等。注意此为异步接口，通过 + [查询异步结果](/api-reference/%E6%A8%A1%E5%9E%8B-api/%E6%9F%A5%E8%AF%A2%E5%BC%82%E6%AD%A5%E7%BB%93%E6%9E%9C) + 获取生成结果。 + requestBody: + content: + application/json: + schema: + oneOf: + - $ref: '#/components/schemas/AsyncChatCompletionTextRequest' + title: 文本模型 + - $ref: '#/components/schemas/AsyncChatCompletionVisionRequest' + title: 视觉模型 + - $ref: '#/components/schemas/AsyncChatCompletionAudioRequest' + title: 音频模型 + - $ref: '#/components/schemas/AsyncChatCompletionHumanOidRequest' + title: 角色模型 + examples: + 基础调用示例: + value: + model: glm-5.1 + messages: + - role: system + content: 你是一个有用的AI助手。 + - role: user + content: 请介绍一下人工智能的发展历程。 + temperature: 1 + 深度思考示例: + value: + model: glm-5.1 + messages: + - role: user + content: 写一首关于春天的诗。 + thinking: + type: enabled + 多轮对话示例: + value: + model: glm-5.1 + messages: + - role: system + content: 你是一个专业的编程助手 + - role: user + content: 什么是递归？ + - role: assistant + content: 递归是一种编程技术，函数调用自身来解决问题... + - role: user + content: 能给我一个 Python 递归的例子吗？ + 图片理解示例: + value: + model: glm-5v-turbo + messages: + - role: user + content: + - type: image_url + image_url: + url: https://cdn.bigmodel.cn/static/logo/register.png + - type: image_url + image_url: + url: https://cdn.bigmodel.cn/static/logo/api-key.png + - type: text + text: What are the pics talk about? + 视频理解示例: + value: + model: glm-5v-turbo + messages: + - role: user + content: + - type: video_url + video_url: + url: >- + https://cdn.bigmodel.cn/agent-demos/lark/113123.mov + - type: text + text: What are the video show about? + 文件理解示例: + value: + model: glm-5v-turbo + messages: + - role: user + content: + - type: file_url + file_url: + url: https://cdn.bigmodel.cn/static/demo/demo2.txt + - type: file_url + file_url: + url: https://cdn.bigmodel.cn/static/demo/demo1.pdf + - type: text + text: What are the files show about? + 音频对话示例: + value: + model: glm-4-voice + messages: + - role: user + content: + - type: text + text: 你好，这是我的语音输入测试，请慢速复述一遍 + - type: input_audio + input_audio: + data: base64_voice_xxx + format: wav + Function Call 示例: + value: + model: glm-5.1 + messages: + - role: user + content: 今天北京的天气怎么样？ + tools: + - type: function + function: + name: get_weather + description: 获取指定城市的天气信息 + parameters: + type: object + properties: + city: + type: string + description: 城市名称 + required: + - city + tool_choice: auto + temperature: 0.3 + required: true + responses: + '200': + description: 业务处理成功 + content: + application/json: + schema: + $ref: '#/components/schemas/AsyncResponse' + default: + description: 请求失败 + content: + application/json: + schema: + $ref: '#/components/schemas/Error' +components: + schemas: + AsyncChatCompletionTextRequest: + required: + - model + - messages + type: object + description: 普通对话模型请求，支持纯文本对话和工具调用 + properties: + model: + type: string + description: >- + 调用的普通对话模型代码。`GLM-5.1` 是最新的旗舰模型系列。`GLM-5` + 系列提供了复杂推理、超长上下文、极快推理速度等多款模型。 + example: glm-5.1 + default: glm-5.1 + enum: + - glm-5.1 + - glm-5-turbo + - glm-5 + - glm-4.7 + - glm-4.6 + - glm-4.5-air + - glm-4.5-airx + - glm-4.5-flash + - glm-4-flash-250414 + - glm-4-flashx-250414 + messages: + type: array + description: >- + 对话消息列表，包含当前对话的完整上下文信息。每条消息都有特定的角色和内容，模型会根据这些消息生成回复。消息按时间顺序排列，支持四种角色：`system`（系统消息，用于设定`AI`的行为和角色）、`user`（用户消息，来自用户的输入）、`assistant`（助手消息，来自`AI`的回复）、`tool`（工具消息，工具调用的结果）。普通对话模型主要支持纯文本内容。注意不能只包含系统消息或助手消息。 + items: + oneOf: + - title: 用户消息 + type: object + properties: + role: + type: string + enum: + - user + description: 消息作者的角色 + default: user + content: + type: string + description: 文本消息内容 + example: >- + What opportunities and challenges will the Chinese large + model industry face in 2025? + required: + - role + - content + - title: 系统消息 + type: object + properties: + role: + type: string + enum: + - system + description: 消息作者的角色 + default: system + content: + type: string + description: 消息文本内容 + example: You are a helpful assistant. + required: + - role + - content + - title: 助手消息 + type: object + description: 可包含工具调用 + properties: + role: + type: string + enum: + - assistant + description: 消息作者的角色 + default: assistant + content: + type: string + description: 文本消息内容 + example: I'll help you with that analysis. + tool_calls: + type: array + description: 模型生成的工具调用消息。当提供此字段时，`content`通常为空。 + items: + type: object + properties: + id: + type: string + description: 工具调用ID + type: + type: string + description: 工具类型，支持 `web_search、retrieval、function` + enum: + - function + - web_search + - retrieval + function: + type: object + description: 函数调用信息，当`type`为`function`时不为空 + properties: + name: + type: string + description: 函数名称 + arguments: + type: string + description: 函数参数，`JSON`格式字符串 + required: + - name + - arguments + required: + - id + - type + required: + - role + - title: 工具消息 + type: object + properties: + role: + type: string + enum: + - tool + description: 消息作者的角色 + default: tool + content: + type: string + description: 消息文本内容 + example: 'Function executed successfully with result: ...' + tool_call_id: + type: string + description: 指示此消息对应的工具调用 `ID` + required: + - role + - content + minItems: 1 + thinking: + $ref: '#/components/schemas/ChatThinking' + do_sample: + type: boolean + example: true + default: true + description: >- + 是否启用采样策略来生成文本。默认值为 `true`。当设置为 `true` 时，模型会使用 `temperature、top_p` + 等参数进行随机采样，生成更多样化的输出；当设置为 `false` 时，模型总是选择概率最高的词汇，生成更确定性的输出，此时 + `temperature` 和 `top_p` 参数将被忽略。对于需要一致性和可重复性的任务（如代码生成、翻译），建议设置为 + `false`。 + temperature: + type: number + description: >- + 采样温度，控制输出的随机性和创造性，取值范围为 `(0.0, 1.0]`，限两位小数。对于`GLM-5.1` `GLM-5` + `GLM-5-Turbo` `GLM-4.7` `GLM-4.6`系列默认值为 `1.0`，`GLM-4.5`系列默认值为 + `0.6`，`GLM-4`系列默认值为 + `0.75`。较高的值（如`0.8`）会使输出更随机、更具创造性，适合创意写作和头脑风暴；较低的值（如`0.2`）会使输出更稳定、更确定，适合事实性问答和代码生成。建议根据应用场景调整 + `top_p` 或 `temperature` 参数，但不要同时调整两个参数。 + format: float + example: 1 + default: 1 + minimum: 0 + maximum: 1 + top_p: + type: number + description: >- + 核采样（`nucleus sampling`）参数，是`temperature`采样的替代方法，取值范围为 `[0.01, + 1.0]`，限两位小数。对于`GLM-5.1` `GLM-5` `GLM-5-Turbo` `GLM-4.7` `GLM-4.6` + `GLM-4.5`系列默认值为 `0.95`，`GLM-4`系列默认值为 + `0.9`。模型只考虑累积概率达到`top_p`的候选词汇。例如：`0.1`表示只考虑前`10%`概率的词汇，`0.9`表示考虑前`90%`概率的词汇。较小的值会产生更集中、更一致的输出；较大的值会增加输出的多样性。建议根据应用场景调整 + `top_p` 或 `temperature` 参数，但不建议同时调整两个参数。 + format: float + example: 0.95 + default: 0.95 + minimum: 0.01 + maximum: 1 + max_tokens: + type: integer + description: >- + 模型输出的最大令牌`token`数量限制。`GLM-5.1` `GLM-5` `GLM-5-Turbo` `GLM-4.7` + `GLM-4.6`系列最大支持`128K`输出长度，`GLM-4.5`最大支持`96K`输出长度，建议设置不小于`1024`。令牌是文本的基本单位，通常`1`个令牌约等于`0.75`个英文单词或`1.5`个中文字符。设置合适的`max_tokens`可以控制响应长度和成本，避免过长的输出。如果模型在达到`max_tokens`限制前完成回答，会自然结束；如果达到限制，输出可能被截断。 + + 默认值和最大值等更多详见 [max_tokens + 文档](/cn/guide/start/concept-param#max_tokens) + example: 1024 + minimum: 1 + maximum: 131072 + tools: + type: array + description: >- + 模型可以调用的工具列表。支持函数调用、知识库检索和网络搜索。使用此参数提供模型可以生成 `JSON` + 输入的函数列表或配置其他工具。最多支持 `128` 个函数。目前 `GLM-4` 系列已支持所有 `tools`，`GLM-4.5` + 已支持 `web search` 和 `retrieval`。 + anyOf: + - items: + $ref: '#/components/schemas/FunctionToolSchema' + - items: + $ref: '#/components/schemas/RetrievalToolSchema' + - items: + $ref: '#/components/schemas/WebSearchToolSchema' + - items: + $ref: '#/components/schemas/MCPToolSchema' + tool_choice: + oneOf: + - type: string + enum: + - auto + description: 用于控制模型选择调用哪个函数的方式，仅在工具类型为`function`时补充。默认`auto`且仅支持`auto`。 + description: 控制模型如何选择工具。 + stop: + type: array + description: >- + 停止词列表，当模型生成的文本中遇到这些指定的字符串时会立即停止生成。目前仅支持单个停止词，格式为["stop_word1"]。停止词不会包含在返回的文本中。这对于控制输出格式、防止模型生成不需要的内容非常有用，例如在对话场景中可以设置["Human:"]来防止模型模拟用户发言。 + items: + type: string + maxItems: 1 + response_format: + type: object + description: >- + 指定模型的响应输出格式，默认为`text`，仅文本模型支持此字段。支持两种格式：{ "type": "text" } + 表示普通文本输出模式，模型返回自然语言文本；{ "type": "json_object" } + 表示`JSON`输出模式，模型会返回有效的`JSON`格式数据，适用于结构化数据提取、`API`响应生成等场景。使用`JSON`模式时，建议在提示词中明确说明需要`JSON`格式输出。 + properties: + type: + type: string + enum: + - text + - json_object + default: text + description: 输出格式类型：`text`表示普通文本输出，`json_object`表示`JSON`格式输出 + required: + - type + request_id: + type: string + description: 请求唯一标识符。由用户端传递，建议使用`UUID`格式确保唯一性，若未提供平台将自动生成。 + user_id: + type: string + description: 终端用户的唯一标识符。`ID`长度要求：最少`6`个字符，最多`128`个字符，建议使用不包含敏感信息的唯一标识。 + minLength: 6 + maxLength: 128 + AsyncChatCompletionVisionRequest: + required: + - model + - messages + type: object + description: 视觉模型请求，支持多模态内容（文本、图片、视频、文件） + properties: + model: + type: string + description: 调用的视觉模型代码。`GLM-5V-Turbo` 系列支持视觉理解，具备卓越的多模态理解能力和工具调用能力。 + example: glm-5v-turbo + default: glm-5v-turbo + enum: + - glm-5v-turbo + - glm-4.6v + - glm-4.6v-flash + - glm-4.6v-flashx + - glm-4v-flash + - glm-4.1v-thinking-flashx + - glm-4.1v-thinking-flash + messages: + type: array + description: >- + 对话消息列表，包含当前对话的完整上下文信息。每条消息都有特定的角色和内容，模型会根据这些消息生成回复。消息按时间顺序排列，支持角色：`system`（系统消息，用于设定`AI`的行为和角色）、`user`（用户消息，来自用户的输入）、`assistant`（助手消息，来自`AI`的回复）。视觉模型支持纯文本和多模态内容（文本、图片、视频、文件）。注意不能只包含系统或助手消息。 + items: + oneOf: + - title: 用户消息 + type: object + properties: + role: + type: string + enum: + - user + description: 消息作者的角色 + default: user + content: + oneOf: + - type: array + description: 多模态消息内容，支持文本、图片、文件、视频（可从上方切换至文本消息） + items: + $ref: '#/components/schemas/VisionMultimodalContentItem' + - type: string + description: 文本消息内容（可从上方切换至多模态消息） + example: >- + What opportunities and challenges will the Chinese + large model industry face in 2025? + required: + - role + - content + - title: 系统消息 + type: object + properties: + role: + type: string + enum: + - system + description: 消息作者的角色 + default: system + content: + oneOf: + - type: string + description: 消息文本内容 + example: You are a helpful assistant. + required: + - role + - content + - title: 助手消息 + type: object + properties: + role: + type: string + enum: + - assistant + description: 消息作者的角色 + default: assistant + content: + oneOf: + - type: string + description: 文本消息内容 + example: I'll help you with that analysis. + required: + - role + minItems: 1 + thinking: + $ref: '#/components/schemas/ChatThinking' + do_sample: + type: boolean + example: true + default: true + description: >- + 是否启用采样策略来生成文本。默认值为 `true`。当设置为 `true` 时，模型会使用 `temperature、top_p` + 等参数进行随机采样，生成更多样化的输出；当设置为 `false` 时，模型总是选择概率最高的词汇，生成更确定性的输出，此时 + `temperature` 和 `top_p` 参数将被忽略。对于需要一致性和可重复性的任务（如代码生成、翻译），建议设置为 + `false`。 + temperature: + type: number + description: >- + 采样温度，控制输出的随机性和创造性，取值范围为 `[0.0, + 1.0]`，限两位小数。对于`GLM-5V-Turbo`，`GLM-4.6V`，`GLM-4.5V`系列默认值为 + `0.8`，`GLM-4.1v`系列默认值为 + `0.8`。较高的值（如`0.8`）会使输出更随机、更具创造性，适合创意写作和头脑风暴；较低的值（如`0.2`）会使输出更稳定、更确定，适合事实性问答和代码生成。建议根据应用场景调整 + `top_p` 或 `temperature` 参数，但不要同时调整两个参数。 + format: float + example: 0.8 + default: 0.8 + minimum: 0 + maximum: 1 + top_p: + type: number + description: >- + 核采样（`nucleus sampling`）参数，是`temperature`采样的替代方法，取值范围为 `[0.01, + 1.0]`，限两位小数。对于`GLM-5V-Turbo`，`GLM-4.6V`，`GLM-4.5V`系列默认值为 + `0.6`，`GLM-4.1v`系列默认值为 + `0.6`。模型只考虑累积概率达到`top_p`的候选词汇。例如：`0.1`表示只考虑前`10%`概率的词汇，`0.9`表示考虑前`90%`概率的词汇。较小的值会产生更集中、更一致的输出；较大的值会增加输出的多样性。建议根据应用场景调整 + `top_p` 或 `temperature` 参数，但不要同时调整两个参数。 + format: float + example: 0.6 + default: 0.6 + minimum: 0.01 + maximum: 1 + max_tokens: + type: integer + description: >- + 模型输出的最大令牌`token`数量限制。`GLM-5V-Turbo`最大支持`128K`输出长度，`GLM-4.6V`最大支持`32K`输出长度，`GLM-4.5V`最大支持`16K`输出长度，`GLM-4.1v`系列最大支持`16K`输出长度，建议设置不小于`1024`。令牌是文本的基本单位，通常`1`个令牌约等于`0.75`个英文单词或`1.5`个中文字符。设置合适的`max_tokens`可以控制响应长度和成本，避免过长的输出。如果模型在达到`max_tokens`限制前完成回答，会自然结束；如果达到限制，输出可能被截断。 + + 默认值和最大值等更多详见 [max_tokens + 文档](/cn/guide/start/concept-param#max_tokens) + example: 1024 + minimum: 1 + maximum: 131072 + tools: + type: array + description: >- + 模型可以调用的工具列表。仅限`GLM-5V-Turbo`, `GLM-4.6V`支持。使用此参数提供模型可以生成 `JSON` + 输入的函数列表或配置其他工具。最多支持 `128` 个函数。 + anyOf: + - items: + $ref: '#/components/schemas/FunctionToolSchema' + tool_choice: + oneOf: + - type: string + enum: + - auto + description: 用于控制模型选择调用哪个函数的方式，仅在工具类型为`function`时补充。默认`auto`且仅支持`auto`。 + description: 控制模型如何选择工具。 + stop: + type: array + description: >- + 停止词列表，当模型生成的文本中遇到这些指定的字符串时会立即停止生成。目前仅支持单个停止词，格式为["stop_word1"]。停止词不会包含在返回的文本中。这对于控制输出格式、防止模型生成不需要的内容非常有用，例如在对话场景中可以设置["Human:"]来防止模型模拟用户发言。 + items: + type: string + maxItems: 1 + request_id: + type: string + description: 请求唯一标识符。由用户端传递，建议使用`UUID`格式确保唯一性，若未提供平台将自动生成。 + user_id: + type: string + description: 终端用户的唯一标识符。`ID`长度要求：最少`6`个字符，最多`128`个字符，建议使用不包含敏感信息的唯一标识。 + minLength: 6 + maxLength: 128 + AsyncChatCompletionAudioRequest: + required: + - model + - messages + type: object + description: 音频模型请求，支持语音理解、生成和识别功能 + properties: + model: + type: string + description: 调用的音频模型代码。`GLM-4-Voice` 支持语音理解和生成。 + example: glm-4-voice + default: glm-4-voice + enum: + - glm-4-voice + - 禁用仅占位 + messages: + type: array + description: >- + 对话消息列表，包含当前对话的完整上下文信息。每条消息都有特定的角色和内容，模型会根据这些消息生成回复。消息按时间顺序排列，支持角色：`system`（系统消息，用于设定`AI`的行为和角色）、`user`（用户消息，来自用户的输入）、`assistant`（助手消息，来自`AI`的回复）。音频模型支持文本和音频内容。注意不能只包含系统或助手消息。 + items: + oneOf: + - title: 用户消息 + type: object + properties: + role: + type: string + enum: + - user + description: 消息作者的角色 + default: user + content: + oneOf: + - type: array + description: 多模态消息内容，支持文本、音频 + items: + $ref: '#/components/schemas/AudioMultimodalContentItem' + - type: string + description: 消息文本内容 + example: You are a helpful assistant. + required: + - role + - content + - title: 系统消息 + type: object + properties: + role: + type: string + enum: + - system + description: 消息作者的角色 + default: system + content: + type: string + description: 消息文本内容 + example: 你是一个专业的语音助手，能够理解和生成自然语音。 + required: + - role + - content + - title: 助手消息 + type: object + properties: + role: + type: string + enum: + - assistant + description: 消息作者的角色 + default: assistant + content: + oneOf: + - type: string + description: 文本消息内容 + example: I'll help you with that analysis. + audio: + type: object + description: 语音消息 + properties: + id: + type: string + description: 语音消息`id`，用于多轮对话 + required: + - role + minItems: 1 + do_sample: + type: boolean + example: true + default: true + description: >- + 是否启用采样策略来生成文本。默认值为 `true`。当设置为 `true` 时，模型会使用 `temperature、top_p` + 等参数进行随机采样，生成更多样化的输出；当设置为 `false` 时，模型总是选择概率最高的词汇，生成更确定性的输出，此时 + `temperature` 和 `top_p` 参数将被忽略。对于需要一致性和可重复性的任务（如语音识别、转录），建议设置为 + `false`。 + temperature: + type: number + description: >- + 采样温度，控制输出的随机性和创造性，取值范围为 `[0.0, 1.0]`，限两位小数。对于`GLM-4-Voice`默认值为 + `0.8`。较高的值（如`0.8`）会使输出更随机、更具创造性，适合语音生成和对话；较低的值（如`0.1`）会使输出更稳定、更确定，适合语音识别和转录。建议根据应用场景调整 + `top_p` 或 `temperature` 参数，但不要同时调整两个参数。 + format: float + example: 0.8 + default: 0.8 + minimum: 0 + maximum: 1 + top_p: + type: number + description: >- + 核采样（`nucleus sampling`）参数，是`temperature`采样的替代方法，取值范围为 `[0.01, + 1.0]`，限两位小数。对于`GLM-4-Voice`默认值为 + `0.6`。模型只考虑累积概率达到`top_p`的候选词汇。例如：`0.1`表示只考虑前`10%`概率的词汇，`0.9`表示考虑前`90%`概率的词汇。较小的值会产生更集中、更一致的输出；较大的值会增加输出的多样性。建议根据应用场景调整 + `top_p` 或 `temperature` 参数，但不要同时调整两个参数。 + format: float + example: 0.6 + default: 0.6 + minimum: 0.01 + maximum: 1 + max_tokens: + type: integer + description: 模型输出的最大令牌`token`数量限制。`GLM-4-Voice`最大支持`4K`输出长度，默认`1024`。令牌是文本的基本单位。 + example: 1024 + minimum: 1 + maximum: 4096 + watermark_enabled: + type: boolean + description: |- + 控制`AI`生成图片时是否添加水印。 + - `true`: 默认启用`AI`生成的显式水印及隐式数字水印，符合政策要求。 + - `false`: 关闭所有水印，仅允许已签署免责声明的客户使用，签署路径：个人中心-安全管理-去水印管理 + example: true + stop: + type: array + description: >- + 停止词列表，当模型生成的文本中遇到这些指定的字符串时会立即停止生成。目前仅支持单个停止词，格式为["stop_word1"]。停止词不会包含在返回的文本中。这对于控制输出格式、防止模型生成不需要的内容非常有用。 + items: + type: string + maxItems: 1 + request_id: + type: string + description: 请求唯一标识符。由用户端传递，建议使用`UUID`格式确保唯一性，若未提供平台将自动生成。 + user_id: + type: string + description: 终端用户的唯一标识符。`ID`长度要求：最少`6`个字符，最多`128`个字符，建议使用不包含敏感信息的唯一标识。 + minLength: 6 + maxLength: 128 + AsyncChatCompletionHumanOidRequest: + required: + - model + - messages + type: object + description: 角色扮演，专业心理咨询专用模型 + properties: + model: + type: string + description: 调用的专用模型代码。`CharGLM-4` 是角色扮演专用模型，`Emohaa` 是专业心理咨询模型。 + example: charglm-4 + default: charglm-4 + enum: + - charglm-4 + - emohaa + meta: + type: object + description: 角色及用户信息数据(仅限 `Emohaa` 支持此参数) + required: + - user_info + - bot_info + - bot_name + - user_name + properties: + user_info: + type: string + description: 用户信息描述 + bot_info: + type: string + description: 角色信息描述 + bot_name: + type: string + description: 角色名称 + user_name: + type: string + description: 用户名称 + messages: + type: array + description: >- + 对话消息列表，包含当前对话的完整上下文信息。每条消息都有特定的角色和内容，模型会根据这些消息生成回复。消息按时间顺序排列，支持角色：`system`（系统消息，用于设定`AI`的行为和角色）、`user`（用户消息，来自用户的输入）、`assistant`（助手消息，来自`AI`的回复）。注意不能只包含系统消息或助手消息。 + items: + oneOf: + - title: 用户消息 + type: object + properties: + role: + type: string + enum: + - user + description: 消息作者的角色 + default: user + content: + type: string + description: 文本消息内容 + example: 我最近工作压力很大，经常感到焦虑，不知道该怎么办 + required: + - role + - content + - title: 系统消息 + type: object + properties: + role: + type: string + enum: + - system + description: 消息作者的角色 + default: system + content: + type: string + description: 消息文本内容 + example: >- + 你乃苏东坡。人生如梦，何不活得潇洒一些？在这忙碌纷繁的现代生活中，帮助大家找到那份属于自己的自在与豁达，共赏人生之美好 + required: + - role + - content + - title: 助手消息 + type: object + properties: + role: + type: string + enum: + - assistant + description: 消息作者的角色 + default: assistant + content: + type: string + description: 文本消息内容 + example: I'll help you with that analysis. + required: + - role + - content + minItems: 1 + do_sample: + type: boolean + example: true + default: true + description: >- + 是否启用采样策略来生成文本。默认值为 `true`。当设置为 `true` 时，模型会使用 `temperature、top_p` + 等参数进行随机采样，生成更多样化的输出；当设置为 `false` 时，模型总是选择概率最高的词汇，生成更确定性的输出，此时 + `temperatur`e 和 `top_p` 参数将被忽略。对于需要一致性和可重复性的任务（如语音识别、转录），建议设置为 + `false`。 + temperature: + type: number + description: >- + 采样温度，控制输出的随机性和创造性，取值范围为 `[0.0, 1.0]`，限两位小数。`Charglm-4` 和 `Emohaa` + 默认值为 `0.95`。建议根据应用场景调整 `top_p` 或 `temperature` 参数，但不要同时调整两个参数。 + format: float + example: 0.8 + default: 0.8 + minimum: 0 + maximum: 1 + top_p: + type: number + description: >- + 核采样（`nucleus sampling`）参数，是`temperature`采样的替代方法，取值范围为 `[0.01, + 1.0]`，限两位小数。`Charglm-4` 和 `Emohaa` 默认值为 `0.7`。建议根据应用场景调整 `top_p` 或 + `temperature` 参数，但不要同时调整两个参数。 + format: float + example: 0.6 + default: 0.6 + minimum: 0.01 + maximum: 1 + max_tokens: + type: integer + description: >- + 模型输出的最大令牌`token`数量限制。`Charglm-4` 和 `Emohaa` + 最大支持`4K`输出长度，默认`1024`。令牌是文本的基本单位。 + example: 1024 + minimum: 1 + maximum: 4096 + stop: + type: array + description: >- + 停止词列表，当模型生成的文本中遇到这些指定的字符串时会立即停止生成。目前仅支持单个停止词，格式为["stop_word1"]。停止词不会包含在返回的文本中。这对于控制输出格式、防止模型生成不需要的内容非常有用。 + items: + type: string + maxItems: 1 + request_id: + type: string + description: 请求唯一标识符。由用户端传递，建议使用`UUID`格式确保唯一性，若未提供平台将自动生成。 + user_id: + type: string + description: 终端用户的唯一标识符。`ID`长度要求：最少`6`个字符，最多`128`个字符，建议使用不包含敏感信息的唯一标识。 + minLength: 6 + maxLength: 128 + AsyncResponse: + type: object + properties: + model: + description: 此次调用使用的名称。 + type: string + id: + description: 生成的任务`ID`，调用请求结果接口时使用此`ID`。 + type: string + request_id: + description: 用户在客户端请求期间提交的任务编号或平台生成的任务编号。 + type: string + task_status: + description: 处理状态，`PROCESSING (处理中)`、`SUCCESS (成功)`、`FAIL (失败)`。结果需要通过查询获取。 + type: string + Error: + type: object + properties: + error: + required: + - code + - message + type: object + properties: + code: + type: string + message: + type: string + ChatThinking: + type: object + description: 仅 `GLM-4.5` 及以上模型支持此参数配置. 控制大模型是否开启思维链。 + properties: + type: + type: string + description: >- + 是否开启思维链(当开启后 `GLM-5.1` `GLM-5` `GLM-5v-Turbo` `GLM-4.7` `GLM-4.5V` + 为强制思考，`GLM-4.6` `GLM-4.6V` `GLM-4.5` 为模型自动判断是否思考), 默认: `enabled`. + default: enabled + enum: + - enabled + - disabled + clear_thinking: + type: boolean + description: >- + 默认为 `True`。用于控制是否清除历史对话轮次（`previous turns`）中的 `reasoning_content`。详见 + [思考模式](/cn/guide/capabilities/thinking-mode) + - `true`（默认）：在本次请求中，系统会忽略/移除历史 `turns` 的 `reasoning_content`，仅使用非推理内容（如用户/助手可见文本、工具调用与结果等）作为上下文输入。适用于普通对话与轻量任务，可降低上下文长度与成本 + - `false`：保留历史 `turns` 的 `reasoning_content` 并随上下文一同提供给模型。若你希望启用 `Preserved Thinking`，必须在 `messages` 中完整、未修改、按原顺序透传历史 `reasoning_content`；缺失、裁剪、改写或重排会导致效果下降或无法生效。 + - 注意：该参数只影响跨 `turn` 的历史 `thinking blocks`；不改变模型在当前 `turn` 内是否产生/输出 `thinking` + default: true + example: true + FunctionToolSchema: + type: object + title: Function Call + properties: + type: + type: string + default: function + enum: + - function + function: + $ref: '#/components/schemas/FunctionObject' + required: + - type + - function + additionalProperties: false + RetrievalToolSchema: + type: object + title: Retrieval + properties: + type: + type: string + default: retrieval + enum: + - retrieval + retrieval: + $ref: '#/components/schemas/RetrievalObject' + required: + - type + - retrieval + additionalProperties: false + WebSearchToolSchema: + type: object + title: Web Search + properties: + type: + type: string + default: web_search + enum: + - web_search + web_search: + $ref: '#/components/schemas/WebSearchObject' + required: + - type + - web_search + additionalProperties: false + MCPToolSchema: + type: object + title: MCP + properties: + type: + type: string + default: mcp + enum: + - mcp + mcp: + $ref: '#/components/schemas/MCPObject' + required: + - type + - mcp + additionalProperties: false + VisionMultimodalContentItem: + oneOf: + - title: 文本 + type: object + properties: + type: + type: string + enum: + - text + description: 内容类型为文本 + default: text + text: + type: string + description: 文本内容 + required: + - type + - text + additionalProperties: false + - title: 图片 + type: object + properties: + type: + type: string + enum: + - image_url + description: 内容类型为图片`URL` + default: image_url + image_url: + type: object + description: 图片信息 + properties: + url: + type: string + description: >- + 图片的`URL`地址或`Base64`编码。图像大小上传限制为每张图像`5M`以下，且像素不超过`6000*6000`。支持`jpg、png、jpeg`格式。`GLM-5V-Turbo` + `GLM4.6V` `GLM4.5V` 系列限制`50`张。`GLM-4V-Plus-0111` + 限制`5`张。`GLM-4V-Flash`限制`1`张图像且不支持`Base64`编码。 + required: + - url + additionalProperties: false + required: + - type + - image_url + additionalProperties: false + - title: 视频 + type: object + properties: + type: + type: string + enum: + - video_url + description: 内容类型为视频输入 + default: video_url + video_url: + type: object + description: 视频信息。注意：`GLM-4V-Plus-0111` 的 `video_url` 参数必须在 `content` 数组的第一位。 + properties: + url: + type: string + description: >- + 视频的`URL`地址。`GLM-5V-Turbo` `GLM-4.6V` `GLM-4.5V` 系列视频大小限制为 + `200M` + 以内。`GLM-4V-Plus`视频大小限制为`20M`以内，视频时长不超过`30s`。对于其他多模态模型，视频大小限制为`200M`以内。视频类型：`mp4`，`mkv`，`mov`。 + required: + - url + additionalProperties: false + required: + - type + - video_url + additionalProperties: false + - title: 文件 + type: object + properties: + type: + type: string + enum: + - file_url + description: >- + 内容类型为文件输入(仅`GLM-5V-Turbo` `GLM-4.6V` `GLM-4.5V`支持，且不支持同时传入 + `file_url` 和 `image_url` 或 `video_url` 参数) + default: file_url + file_url: + type: object + description: 文件信息。 + properties: + url: + type: string + description: >- + 文件的`URL`地址，不支持`Base64`编码。支持`pdf、txt、word、jsonl、xlsx、pptx`等格式，最多支持`50`个。 + required: + - url + additionalProperties: false + required: + - type + - file_url + additionalProperties: false + AudioMultimodalContentItem: + oneOf: + - title: 文本 + type: object + properties: + type: + type: string + enum: + - text + description: 内容类型为文本 + default: text + text: + type: string + description: 文本内容 + required: + - type + - text + additionalProperties: false + - title: 音频 + type: object + properties: + type: + type: string + enum: + - input_audio + description: 内容类型为音频输入 + default: input_audio + input_audio: + type: object + description: 音频信息，仅`glm-4-voice`支持音频输入 + properties: + data: + type: string + description: 语音文件的`base64`编码。音频最长不超过 `10` 分钟。`1s`音频=`12.5 Tokens`，向上取整。 + format: + type: string + description: 语音文件的格式，支持`wav`和`mp3` + enum: + - wav + - mp3 + required: + - data + - format + additionalProperties: false + required: + - type + - input_audio + additionalProperties: false + FunctionObject: + type: object + properties: + name: + type: string + description: 要调用的函数名称。必须是 `a-z、A-Z、0-9`，或包含下划线和破折号，最大长度为 `64`。 + minLength: 1 + maxLength: 64 + pattern: ^[a-zA-Z0-9_-]+$ + description: + type: string + description: 函数功能的描述，供模型选择何时以及如何调用函数。 + parameters: + $ref: '#/components/schemas/FunctionParameters' + required: + - name + - description + - parameters + RetrievalObject: + type: object + properties: + knowledge_id: + type: string + description: 知识库 `ID`，从平台创建或获取 + prompt_template: + type: string + description: >- + 请求模型的提示模板，包含占位符 `{{ knowledge }}` 和 `{{ question }}` + 的自定义请求模板。默认模板：`在文档 `{{ knowledge }}` 中搜索问题 `{{question}}` + 的答案。如果找到答案，仅使用文档中的陈述进行回应；如果没有找到答案，使用你自己的知识回答并告知用户信息不来自文档。不要重复问题，直接开始答案。` + required: + - knowledge_id + WebSearchObject: + type: object + properties: + enable: + type: boolean + description: 是否启用搜索功能，默认值为 `false`，启用时设置为 `true` + search_engine: + type: string + description: >- + 搜索引擎类型，默认为 + `search_std`；支持`search_std、search_pro、search_pro_sogou、search_pro_quark`。 + enum: + - search_std + - search_pro + - search_pro_sogou + - search_pro_quark + search_query: + type: string + description: 强制触发搜索 + search_intent: + type: string + description: >- + 是否进行搜索意图识别，默认执行搜索意图识别。`true`：执行搜索意图识别，有搜索意图后执行搜索；`false`：跳过搜索意图识别，直接执行搜索 + count: + type: integer + description: >- + 返回结果的条数。可填范围：`1-50`，最大单次搜索返回`50`条，默认为`10`。支持的搜索引擎：`search_std、search_pro、search_pro_sogou`。对于`search_pro_sogou`: + 可选枚举值，`10、20、30、40、50` + minimum: 1 + maximum: 50 + search_domain_filter: + type: string + description: |- + 用于限定搜索结果的范围，仅返回指定白名单域名的内容。 + 白名单域名:（如 `www.example.com`）。 + 支持的搜索引擎：`search_std、search_pro、search_pro_sogou` + search_recency_filter: + type: string + description: >- + 搜索指定时间范围内的网页。默认为`noLimit`。可填值：`oneDay`（一天内）、`oneWeek`（一周内）、`oneMonth`（一个月内）、`oneYear`（一年内）、`noLimit`（不限，默认）。支持的搜索引擎：`search_std、search_pro、search_pro_sogou、search_pro_quark` + enum: + - oneDay + - oneWeek + - oneMonth + - oneYear + - noLimit + content_size: + type: string + description: >- + 控制网页摘要的字数。默认值为`medium`。`medium`：返回摘要信息，满足大模型的基础推理需求。`high`：最大化上下文，信息量较大但内容详细，适合需要信息细节的场景。 + enum: + - medium + - high + result_sequence: + type: string + description: 指定搜索结果返回的顺序是在模型回复结果之前还是之后，可选值：`before`、`after`，默认 `after` + enum: + - before + - after + search_result: + type: boolean + description: 是否返回搜索来源的详细信息，默认值 `false` + require_search: + type: boolean + description: 是否强制搜索结果才返回回答，默认值 `false` + search_prompt: + type: string + description: |- + 用于定制搜索结果处理的`Prompt`，默认`Prompt`： + + 你是一位智能问答专家，具备整合信息的能力，能够进行时间识别、语义理解与矛盾信息清洗处理。 + 当前日期是{{current_date}}，请以此时间为唯一基准，参考以下信息，全面、准确地回答用户问题。 + 仅提炼有价值的内容用于回答，确保答案具有实时性与权威性，直接陈述答案，无需说明数据来源或内部处理过程。 + required: + - search_engine + MCPObject: + type: object + properties: + server_label: + description: >- + `mcp server`标识，如果连接智谱的`mcp server`，以`mcp + code`填充该字段，且无需填写`server_url` + type: string + server_url: + description: '`mcp server`地址' + type: string + transport_type: + description: 传输类型 + type: string + default: streamable-http + enum: + - sse + - streamable-http + allowed_tools: + description: 允许的工具集合 + type: array + items: + type: string + headers: + description: '`mcp server` 需要的鉴权信息' + type: object + required: + - server_label + FunctionParameters: + type: object + description: 使用 `JSON Schema` 定义的参数。必须传递 `JSON Schema` 对象以准确定义接受的参数。如果调用函数时不需要参数，则省略。 + additionalProperties: true + securitySchemes: + bearerAuth: + type: http + scheme: bearer + description: >- + 使用以下格式进行身份验证：Bearer [](https://bigmodel.cn/usercenter/proj-mgmt/apikeys) + +```` \ No newline at end of file diff --git a/docs/协议文档/智谱GLM/GLM查询异步结果.md b/docs/协议文档/智谱GLM/GLM查询异步结果.md new file mode 100644 index 0000000..c400dbf --- /dev/null +++ b/docs/协议文档/智谱GLM/GLM查询异步结果.md @@ -0,0 +1,528 @@ +> ## Documentation Index +> Fetch the complete documentation index at: https://docs.bigmodel.cn/llms.txt +> Use this file to discover all available pages before exploring further. + +# 查询异步结果 + +> 查询对话补全和视频生成异步请求的处理结果和状态。 + + + +## OpenAPI + +````yaml /openapi/openapi.json get /paas/v4/async-result/{id} +openapi: 3.0.1 +info: + title: ZHIPU AI API + description: ZHIPU AI 接口提供强大的 AI 能力，包括聊天对话、工具调用和视频生成。 + license: + name: ZHIPU AI 开发者协议和政策 + url: https://chat.z.ai/legal-agreement/terms-of-service + version: 1.0.0 + contact: + name: Z.AI 开发者 + url: https://chat.z.ai/legal-agreement/privacy-policy + email: user_feedback@z.ai +servers: + - url: https://open.bigmodel.cn/api/ + description: 开放平台服务 +security: + - bearerAuth: [] +tags: + - name: 模型 API + description: Chat API + - name: 工具 API + description: Web Search API + - name: Agent API + description: Agent API + - name: 文件 API + description: File API + - name: 知识库 API + description: Knowledge API + - name: 实时 API + description: Realtime API + - name: 批处理 API + description: Batch API + - name: 助理 API + description: Assistant API + - name: 智能体 API（旧） + description: QingLiu Agent API +paths: + /paas/v4/async-result/{id}: + get: + tags: + - 模型 API + summary: 查询异步结果 + description: 查询对话补全和视频生成异步请求的处理结果和状态。 + parameters: + - name: id + in: path + required: true + schema: + type: string + description: 任务 ID。 + responses: + '200': + description: 业务处理成功 + content: + application/json: + schema: + oneOf: + - $ref: '#/components/schemas/ChatCompletionResponse' + title: 对话补全 + - $ref: '#/components/schemas/AsyncVideoGenerationResponse' + title: 视频生成 + - $ref: '#/components/schemas/AsyncImageGenerationResponse' + title: 图像生成 + text/event-stream: + schema: + $ref: '#/components/schemas/ChatCompletionChunk' + default: + description: 请求失败。 + content: + application/json: + schema: + $ref: '#/components/schemas/Error' +components: + schemas: + ChatCompletionResponse: + type: object + properties: + id: + description: 任务 `ID` + type: string + request_id: + description: 请求 `ID` + type: string + created: + description: 请求创建时间，`Unix` 时间戳（秒） + type: integer + model: + description: 模型名称 + type: string + choices: + type: array + description: 模型响应列表 + items: + type: object + properties: + index: + type: integer + description: 结果索引 + message: + $ref: '#/components/schemas/ChatCompletionResponseMessage' + finish_reason: + type: string + description: >- + 推理终止原因。'stop’表示自然结束或触发stop词，'tool_calls’表示模型命中函数，'length’表示达到token长度限制，'sensitive’表示内容被安全审核接口拦截（用户应判断并决定是否撤回公开内容），'network_error’表示模型推理异常，'model_context_window_exceeded'表示超出模型上下文窗口。 + usage: + type: object + description: 调用结束时返回的 `Token` 使用统计。 + properties: + prompt_tokens: + type: number + description: 用户输入的 `Token` 数量。 + completion_tokens: + type: number + description: 输出的 `Token` 数量 + prompt_tokens_details: + type: object + properties: + cached_tokens: + type: number + description: 命中的缓存 `Token` 数量 + total_tokens: + type: integer + description: '`Token` 总数，对于 `glm-4-voice` 模型，`1`秒音频=`12.5 Tokens`，向上取整' + video_result: + type: array + description: 视频生成结果。 + items: + type: object + properties: + url: + type: string + description: 视频链接。 + cover_image_url: + type: string + description: 视频封面链接。 + web_search: + type: array + description: 返回与网页搜索相关的信息，使用`WebSearchToolSchema`时返回 + items: + type: object + properties: + icon: + type: string + description: 来源网站的图标 + title: + type: string + description: 搜索结果的标题 + link: + type: string + description: 搜索结果的网页链接 + media: + type: string + description: 搜索结果网页的媒体来源名称 + publish_date: + type: string + description: 网站发布时间 + content: + type: string + description: 搜索结果网页引用的文本内容 + refer: + type: string + description: 角标序号 + content_filter: + type: array + description: 返回内容安全的相关信息 + items: + type: object + properties: + role: + type: string + description: >- + 安全生效环节，包括 `role = assistant` 模型推理，`role = user` 用户输入，`role = + history` 历史上下文 + level: + type: integer + description: 严重程度 `level 0-3`，`level 0`表示最严重，`3`表示轻微 + AsyncVideoGenerationResponse: + type: object + properties: + model: + type: string + description: 模型名称。 + task_status: + type: string + description: 任务处理状态，`PROCESSING`（处理中），`SUCCESS`（成功），`FAIL`（失败） 注：处理中状态需通过查询获取结果 + video_result: + type: array + description: 数组，包含生成的视频`URL`。 + items: + type: object + properties: + url: + type: string + description: 视频链接。 + cover_image_url: + type: string + description: 视频封面链接。 + request_id: + type: string + description: 标识此次请求的唯一`ID`，可由用户在客户端请求时提交或平台自动生成。 + AsyncImageGenerationResponse: + type: object + properties: + model: + type: string + description: 模型名称。 + task_status: + type: string + description: 任务处理状态，`PROCESSING`（处理中），`SUCCESS`（成功），`FAIL`（失败） 注：处理中状态需通过查询获取结果 + image_result: + type: array + description: 数组，包含生成的图片`URL`。 + items: + type: object + properties: + url: + type: string + description: 图片链接。图片的临时链接有效期为`30`天，请及时转存图片。 + request_id: + type: string + description: 标识此次请求的唯一`ID`，可由用户在客户端请求时提交或平台自动生成。 + ChatCompletionChunk: + type: object + properties: + id: + type: string + description: 任务 ID + created: + description: 请求创建时间，`Unix` 时间戳（秒） + type: integer + model: + description: 模型名称 + type: string + choices: + type: array + description: 模型响应列表 + items: + type: object + properties: + index: + type: integer + description: 结果索引 + delta: + type: object + description: 模型增量返回的文本信息 + properties: + role: + type: string + description: 当前对话的角色，目前默认为 `assistant`（模型） + content: + oneOf: + - type: string + description: >- + 当前对话文本内容。如果调用函数则为 `null`，否则返回推理结果。 + + 对于`GLM-4.5V`系列模型，返回内容可能包含思考过程标签 ` + `，文本边界标签 `<|begin_of_box|> <|end_of_box|>`。 + - type: array + description: 当前对话的多模态内容（适用于`GLM-4V`系列） + items: + type: object + properties: + type: + type: string + enum: + - text + description: 内容类型，目前为文本 + text: + type: string + description: 文本内容 + - type: string + nullable: true + description: 当使用`tool_calls`时，`content`可能为`null` + audio: + type: object + description: 当使用 `glm-4-voice` 模型时返回的音频内容 + properties: + id: + type: string + description: 当前对话的音频内容`id`，可用于多轮对话输入 + data: + type: string + description: 当前对话的音频内容`base64`编码 + expires_at: + type: string + description: 当前对话的音频内容过期时间 + reasoning_content: + type: string + description: 思维链内容, 仅 `glm-4.5` 系列支持 + tool_calls: + type: array + description: 生成的应该被调用的工具信息，流式返回时会逐步生成 + items: + type: object + properties: + index: + type: integer + description: 工具调用索引 + id: + type: string + description: 工具调用的唯一标识符 + type: + type: string + description: 工具类型，目前支持`function` + enum: + - function + function: + type: object + properties: + name: + type: string + description: 函数名称 + arguments: + type: string + description: 函数参数，`JSON`格式字符串 + finish_reason: + type: string + description: >- + 模型推理终止的原因。`stop` 表示自然结束或触发stop词，`tool_calls` 表示模型命中函数，`length` + 表示达到 `token` 长度限制，`sensitive` + 表示内容被安全审核接口拦截（用户应判断并决定是否撤回公开内容），`network_error` + 表示模型推理异常，'model_context_window_exceeded'表示超出模型上下文窗口。 + enum: + - stop + - length + - tool_calls + - sensitive + - network_error + usage: + type: object + description: 本次模型调用的 `tokens` 数量统计 + properties: + prompt_tokens: + type: integer + description: 用户输入的 `tokens` 数量。对于 `glm-4-voice`，`1`秒音频=`12.5 Tokens`，向上取整。 + completion_tokens: + type: integer + description: 模型输出的 `tokens` 数量 + total_tokens: + type: integer + description: 总 `tokens` 数量，对于 `glm-4-voice` 模型，`1`秒音频=`12.5 Tokens`，向上取整 + content_filter: + type: array + description: 返回内容安全的相关信息 + items: + type: object + properties: + role: + type: string + description: >- + 安全生效环节，包括：`role = assistant` 模型推理，`role = user` 用户输入，`role = + history` 历史上下文 + level: + type: integer + description: 严重程度 `level 0-3`，`level 0` 表示最严重，`3` 表示轻微 + Error: + type: object + properties: + error: + required: + - code + - message + type: object + properties: + code: + type: string + message: + type: string + ChatCompletionResponseMessage: + type: object + properties: + role: + type: string + description: 当前对话角色，默认为 `assistant` + example: assistant + content: + oneOf: + - type: string + description: >- + 当前对话文本内容。如果调用函数则为 `null`，否则返回推理结果。 + + 对于`GLM-4.5V`系列模型，返回内容可能包含思考过程标签 ` `，文本边界标签 + `<|begin_of_box|> <|end_of_box|>`。 + - type: array + description: 多模态回复内容，适用于`GLM-4V`系列模型 + items: + type: object + properties: + type: + type: string + enum: + - text + description: 回复内容类型，目前为文本 + text: + type: string + description: 文本内容 + - type: string + nullable: true + description: 当使用`tool_calls`时，`content`可能为`null` + reasoning_content: + type: string + description: 思维链内容，仅在使用 `glm-4.5` 系列, `glm-4.1v-thinking` 系列模型时返回。 + audio: + type: object + description: 当使用 `glm-4-voice` 模型时返回的音频内容 + properties: + id: + type: string + description: 当前对话的音频内容`id`，可用于多轮对话输入 + data: + type: string + description: 当前对话的音频内容`base64`编码 + expires_at: + type: string + description: 当前对话的音频内容过期时间 + tool_calls: + type: array + description: 生成的应该被调用的函数名称和参数。 + items: + $ref: '#/components/schemas/ChatCompletionResponseMessageToolCall' + ChatCompletionResponseMessageToolCall: + type: object + properties: + function: + type: object + description: 包含生成的函数名称和 `JSON` 格式参数。 + properties: + name: + type: string + description: 生成的函数名称。 + arguments: + type: string + description: 生成的函数调用参数的 `JSON` 格式字符串。调用函数前请验证参数。 + required: + - name + - arguments + mcp: + type: object + description: '`MCP` 工具调用参数' + properties: + id: + description: '`mcp` 工具调用唯一标识' + type: string + type: + description: 工具调用类型, 例如 `mcp_list_tools, mcp_call` + type: string + enum: + - mcp_list_tools + - mcp_call + server_label: + description: '`MCP`服务器标签' + type: string + error: + description: 错误信息 + type: string + tools: + description: '`type = mcp_list_tools` 时的工具列表' + type: array + items: + type: object + properties: + name: + description: 工具名称 + type: string + description: + description: 工具描述 + type: string + annotations: + description: 工具注解 + type: object + input_schema: + description: 工具输入参数规范 + type: object + properties: + type: + description: 固定值 'object' + type: string + default: object + enum: + - object + properties: + description: 参数属性定义 + type: object + required: + description: 必填属性列表 + type: array + items: + type: string + additionalProperties: + description: 是否允许额外参数 + type: boolean + arguments: + description: 工具调用参数，参数为 `json` 字符串 + type: string + name: + description: 工具名称 + type: string + output: + description: 工具返回的结果输出 + type: object + id: + type: string + description: 命中函数的唯一标识符。 + type: + type: string + description: 调用的工具类型，目前仅支持 'function', 'mcp'。 + securitySchemes: + bearerAuth: + type: http + scheme: bearer + description: >- + 使用以下格式进行身份验证：Bearer [](https://bigmodel.cn/usercenter/proj-mgmt/apikeys) + +```` \ No newline at end of file diff --git a/docs/协议文档/豆包系列/豆包图像生成API文档.md b/docs/协议文档/豆包系列/豆包图像生成API文档.md new file mode 100644 index 0000000..104a54a --- /dev/null +++ b/docs/协议文档/豆包系列/豆包图像生成API文档.md @@ -0,0 +1,1209 @@ +# 豆包图像生成 API 文档 + +> 火山引擎方舟（Volcengine Ark）平台 — 豆包图像生成模型 API 完整参考 +> +> 基于火山引擎方舟平台官方文档整理，覆盖 Seedream 文生图、Seedance 图生图、Seed Thinking 等豆包图像生成模型的完整 API 规范。 +> +> 官方文档入口：https://www.volcengine.com/docs/6791/1399418 + +--- + +## 目录 + +- [1. 概述](#1-概述) +- [2. 认证方式](#2-认证方式) +- [3. API 端点列表](#3-api-端点列表) +- [4. 图像生成（文生图）接口详细参数](#4-图像生成文生图接口详细参数) +- [5. 图像编辑（图生图）接口详细参数](#5-图像编辑图生图接口详细参数) +- [6. 支持模型与参数差异](#6-支持模型与参数差异) +- [7. 响应格式](#7-响应格式) +- [8. 错误码与错误处理](#8-错误码与错误处理) +- [9. 请求与响应示例](#9-请求与响应示例) +- [10. OpenAI 兼容接口说明](#10-openai-兼容接口说明) +- [11. team-api 适配器实现说明](#11-team-api-适配器实现说明) + +--- + +## 1. 概述 + +### 1.1 平台简介 + +豆包（Doubao）是字节跳动推出的 AI 大模型品牌，其图像生成能力通过**火山引擎方舟（Volcengine Ark）**平台提供 API 服务。方舟平台是火山引擎的一站式大模型推理服务平台，支持文本生成、图像生成、语音合成等多种 AI 能力。 + +豆包图像生成目前提供以下核心模型系列： + +| 模型系列 | 模型名 | 功能定位 | +|---------|--------|---------| +| Seedream | `doubao-seedream-*` | 文生图（Text-to-Image），根据文本描述生成图像 | +| Seedance | `doubao-seedance-*` | 图生图（Image-to-Image），基于参考图+文本描述生成新图像 | +| Seed Thinking | `doubao-seed-*-thinking-*` | 带推理增强的图像生成，先思考再生成 | + +### 1.2 API 基本信息总览 + +| 项目 | 值 | +|------|------| +| 平台名称 | 火山引擎方舟（Volcengine Ark） | +| 基础 URL | `https://ark.cn-beijing.volces.com` | +| API 版本 | v3 | +| 协议风格 | RESTful JSON | +| 兼容性 | 兼容 OpenAI API 格式 | +| 文档 ID | 6791（火山引擎文档中心） | + +### 1.3 核心特性 + +- **OpenAI 兼容**：方舟平台的图像生成接口完全兼容 OpenAI Images API 格式，客户端可直接使用 OpenAI SDK 调用 +- **多模型支持**：支持 Seedream（文生图）、Seedance（图生图）等多种图像生成模型 +- **多种输出格式**：支持 URL 返回和 Base64 编码返回 +- **多尺寸支持**：支持正方形、横向、纵向等多种尺寸 +- **流式生成**：部分模型支持流式返回中间结果 +- **Bot 模型**：支持 `bot-` 前缀的 Bot 模型，使用专用端点 + +--- + +## 2. 认证方式 + +### 2.1 API Key 认证 + +火山引擎方舟平台使用 Bearer Token 认证方式，在请求头中携带 API Key： + +```http +Authorization: Bearer +Content-Type: application/json +``` + +### 2.2 API Key 获取 + +1. 登录火山引擎控制台：https://console.volcengine.com/ark +2. 进入方舟平台，在「API Key 管理」中创建新的 API Key +3. 创建接入点（Endpoint），获取模型对应的 Endpoint ID +4. API Key 格式示例：`ark-xxxxxxxxxxxxxxxxxxxxxxxx` + +### 2.3 认证参数表 + +| 参数 | 位置 | 格式 | 必填 | 说明 | +|------|------|------|------|------| +| `Authorization` | Header | `Bearer ` | 是 | API Key 认证令牌 | +| `Content-Type` | Header | `application/json` | 是 | 请求体格式 | + +### 2.4 认证错误响应 + +当 API Key 无效或缺失时，返回以下错误： + +```json +{ + "error": { + "type": "authentication_error", + "message": "Incorrect API key provided", + "code": "invalid_api_key" + } +} +``` + +HTTP 状态码：`401 Unauthorized` + +### 2.5 注意事项 + +- API Key 必须妥善保管，不可暴露在客户端代码中 +- 每个 API Key 关联一个方舟平台账户，所有调用产生的费用由该账户承担 +- API Key 支持在方舟控制台进行轮换和吊销 +- 若使用 team-api 网关，API Key 存储在渠道配置中，格式为 `ark-xxxxx` + +--- + +## 3. API 端点列表 + +### 3.1 方舟平台原生端点 + +| 方法 | 端点路径 | 功能 | 支持模型 | 请求格式 | +|------|---------|------|---------|---------| +| POST | `/api/v3/images/generations` | 文生图 + 图生图 | Seedream, Seedance | JSON | +| POST | `/api/v3/chat/completions` | 文本对话 | Doubao-pro, Doubao-lite 等 | JSON | +| POST | `/api/v3/embeddings` | 文本向量化 | Doubao-embedding | JSON | +| POST | `/api/v3/bots/chat/completions` | Bot 对话 | bot-* 模型 | JSON | +| POST | `/api/v3/rerank` | 重排序 | 支持的模型 | JSON | +| POST | `/api/v3/responses` | Responses API | 支持的模型 | JSON | + +### 3.2 图像生成端点详解 + +方舟平台的图像生成统一使用 `/api/v3/images/generations` 端点，**文生图和图生图共用同一端点**。 + +**基础 URL**：`https://ark.cn-beijing.volces.com` + +**完整端点 URL**：`https://ark.cn-beijing.volces.com/api/v3/images/generations` + +### 3.3 端点功能矩阵 + +| 功能 | 端点 | 支持的模型系列 | 说明 | +|------|------|--------------|------| +| 文生图（Text-to-Image） | `/api/v3/images/generations` | Seedream | 仅提供 prompt，模型根据文本生成图像 | +| 图生图（Image-to-Image） | `/api/v3/images/generations` | Seedance | 提供 prompt + 参考图像，模型结合两者生成新图像 | +| 带推理图像生成 | `/api/v3/images/generations` | Seed Thinking | 带思维链的图像生成，先推理再出图 | + +--- + +## 4. 图像生成（文生图）接口详细参数 + +### 4.1 接口基本信息 + +| 项目 | 值 | +|------|------| +| 端点 | `POST /api/v3/images/generations` | +| Content-Type | `application/json` | +| 认证方式 | Bearer Token | +| 响应格式 | JSON | + +### 4.2 完整请求参数表 + +| 参数名 | 类型 | 必填 | 默认值 | 说明 | +|--------|------|------|--------|------| +| `model` | string | **是** | — | 模型 ID 或 Endpoint ID。例如 `doubao-seedream-4-0-250828`，也支持使用方舟平台的接入点 ID（`ep-xxxxxxxx`） | +| `prompt` | string | **是** | — | 图像描述文本，即文生图的提示词。支持中英文描述，建议使用详细具体的描述以获得更好的生成效果。Seedream 模型最长支持约 4000 字符 | +| `n` | integer | 否 | `1` | 生成图像数量。通常支持 1-4 张。部分模型仅支持 1 张 | +| `size` | string | 否 | `1024x1024` | 输出图像尺寸，格式为 `{width}x{height}`。支持的尺寸取决于模型，详见下方尺寸表 | +| `response_format` | string | 否 | `url` | 图像返回格式：`url`（返回临时下载链接）或 `b64_json`（返回 Base64 编码） | +| `seed` | integer | 否 | 随机 | 随机种子。相同种子 + 相同参数可复现近似结果 | +| `guidance_scale` | float | 否 | 模型默认 | 引导比例（CFG Scale），控制图像与提示词的一致性。值越高越严格遵循提示词，值越低创意自由度越高 | +| `user` | string | 否 | — | 终端用户标识符，用于监控和滥用检测 | + +### 4.3 Seedream 模型支持的尺寸 + +**doubao-seedream-4-0-250828（Seedream 4.0）**： + +| 尺寸 | 宽高比 | 说明 | +|------|--------|------| +| `1024x1024` | 1:1 | 正方形 | +| `1536x1024` | 3:2 | 横向 | +| `1024x1536` | 2:3 | 纵向 | +| `1280x720` | 16:9 | 宽屏横向 | +| `720x1280` | 9:16 | 宽屏纵向（手机竖屏） | + +**doubao-seedream-3-0-t2i-250415（Seedream 3.0 文生图）**： + +| 尺寸 | 宽高比 | 说明 | +|------|--------|------| +| `1024x1024` | 1:1 | 正方形 | +| `1024x1792` | 9:16 | 纵向 | +| `1792x1024` | 16:9 | 横向 | + +### 4.4 文生图请求示例 + +**基本文生图请求**： + +```json +{ + "model": "doubao-seedream-4-0-250828", + "prompt": "一只橘色的猫咪坐在窗台上，阳光从窗外照射进来，温暖的氛围，高清摄影风格", + "n": 1, + "size": "1024x1024", + "response_format": "url" +} +``` + +**使用 Endpoint ID 的请求**： + +```json +{ + "model": "ep-20250415xxxxx-xxxxx", + "prompt": "A beautiful sunset over a mountain lake, reflection in the water, photorealistic", + "n": 1, + "size": "1536x1024", + "response_format": "b64_json", + "seed": 42 +} +``` + +**多图生成请求**： + +```json +{ + "model": "doubao-seedream-4-0-250828", + "prompt": "四个季节的风景：春天樱花、夏天海滩、秋天枫叶、冬天雪景", + "n": 4, + "size": "1024x1024", + "response_format": "url" +} +``` + +### 4.5 Seedream 模型 prompt 技巧 + +| 技巧 | 示例 | 说明 | +|------|------|------| +| 具体描述主体 | "一只金色的拉布拉多犬在草地上奔跑" | 明确描述主体、动作、场景 | +| 指定风格 | "水彩画风格"、"赛博朋克风格"、"日本动漫风格" | 在 prompt 末尾添加风格关键词 | +| 指定视角 | "俯视角度"、"特写镜头"、"广角全景" | 控制拍摄/生成视角 | +| 指定光照 | "自然光"、"黄金时段"、"霓虹灯光" | 描述光照条件 | +| 指定色彩 | "暖色调"、"冷色调"、"莫兰迪色系" | 控制整体色彩倾向 | +| 中文支持 | 直接使用中文描述 | Seedream 原生支持中文 prompt | + +--- + +## 5. 图像编辑（图生图）接口详细参数 + +### 5.1 接口基本信息 + +| 项目 | 值 | +|------|------| +| 端点 | `POST /api/v3/images/generations` | +| Content-Type | `application/json` | +| 认证方式 | Bearer Token | +| 说明 | 图生图与文生图共用同一端点，通过传入参考图像区分 | + +### 5.2 完整请求参数表 + +| 参数名 | 类型 | 必填 | 默认值 | 说明 | +|--------|------|------|--------|------| +| `model` | string | **是** | — | 图生图模型 ID，如 `doubao-seedance-1-0-pro-250528` | +| `prompt` | string | **是** | — | 编辑指令文本，描述期望的编辑效果 | +| `image` | string 或 array | **是** | — | 参考图像。支持传入图像 URL 或 Base64 编码的图像数据。部分模型支持多张参考图（最多 16 张） | +| `n` | integer | 否 | `1` | 生成图像数量 | +| `size` | string | 否 | `1024x1024` | 输出图像尺寸 | +| `response_format` | string | 否 | `url` | 返回格式：`url` 或 `b64_json` | +| `strength` | float | 否 | 模型默认 | 编辑强度（0.0-1.0）。值越低越接近原图，值越高变化越大 | +| `guidance_scale` | float | 否 | 模型默认 | 引导比例，控制编辑结果与提示词的一致性 | +| `seed` | integer | 否 | 随机 | 随机种子 | +| `user` | string | 否 | — | 终端用户标识 | + +### 5.3 参考图像格式 + +参考图像（`image` 字段）支持以下格式： + +**方式一：图像 URL** + +```json +{ + "image": "https://example.com/photo.jpg" +} +``` + +**方式二：Base64 编码** + +```json +{ + "image": "data:image/jpeg;base64,/9j/4AAQSkZJRgABAQ..." +} +``` + +**方式三：多张参考图（数组格式）** + +```json +{ + "image": [ + { + "type": "image_url", + "url": "https://example.com/photo1.jpg" + }, + { + "type": "image_url", + "url": "data:image/png;base64,iVBORw0KGgo..." + } + ] +} +``` + +### 5.4 Seedance 模型支持的尺寸 + +**doubao-seedance-1-0-pro-250528（Seedance 1.0 Pro）**： + +| 尺寸 | 宽高比 | 说明 | +|------|--------|------| +| `1024x1024` | 1:1 | 正方形 | +| `1536x1024` | 3:2 | 横向 | +| `1024x1536` | 2:3 | 纵向 | + +### 5.5 图生图请求示例 + +**基本图生图请求**： + +```json +{ + "model": "doubao-seedance-1-0-pro-250528", + "prompt": "将背景替换为海滩日落场景，保持人物不变", + "image": "https://example.com/portrait.jpg", + "n": 1, + "size": "1024x1024", + "response_format": "url", + "strength": 0.7 +} +``` + +**使用 Base64 参考图的请求**： + +```json +{ + "model": "doubao-seedance-1-0-pro-250528", + "prompt": "Add a red scarf to the person in the image, winter style", + "image": "data:image/jpeg;base64,/9j/4AAQSkZJRgABAQ...", + "size": "1536x1024", + "response_format": "b64_json" +} +``` + +**多参考图请求**： + +```json +{ + "model": "doubao-seedance-1-0-pro-250528", + "prompt": "融合两张图片的风格，生成一幅新的艺术作品", + "image": [ + { + "type": "image_url", + "url": "https://example.com/style1.jpg" + }, + { + "type": "image_url", + "url": "https://example.com/style2.jpg" + } + ], + "n": 1, + "size": "1024x1024" +} +``` + +### 5.6 图生图使用场景 + +| 场景 | prompt 示例 | 说明 | +|------|-----------|------| +| 风格转换 | "将这张照片转换为梵高风格" | 保留内容，改变艺术风格 | +| 背景替换 | "将背景替换为星空" | 改变场景背景 | +| 细节编辑 | "给人物添加一顶红色帽子" | 局部修改添加元素 | +| 图像修复 | "修复图片左侧的划痕" | 修复图像缺陷 | +| 色彩调整 | "将整体色调调整为暖色调" | 改变色彩风格 | + +--- + +## 6. 支持模型与参数差异 + +### 6.1 模型列表 + +| 模型 ID | 模型系列 | 功能 | 发布日期（版本号） | 说明 | +|---------|---------|------|-----------------|------| +| `doubao-seedream-4-0-250828` | Seedream 4.0 | 文生图 | 2025-08-28 | 最新版本文生图模型，画质和文字渲染能力大幅提升 | +| `seedream-4-0-250828` | Seedream 4.0（短名） | 文生图 | 2025-08-28 | 同上，短名版本 | +| `doubao-seedream-3-0-t2i-250415` | Seedream 3.0 | 文生图 | 2025-04-15 | 第三代文生图模型 | +| `doubao-seedance-1-0-pro-250528` | Seedance 1.0 Pro | 图生图 | 2025-05-28 | 专业版图生图模型，支持多参考图 | +| `seedance-1-0-pro-250528` | Seedance 1.0 Pro（短名） | 图生图 | 2025-05-28 | 同上，短名版本 | +| `doubao-seed-1-6-thinking-250715` | Seed 1.6 Thinking | 带推理图像生成 | 2025-07-15 | 带思维链的图像生成模型 | +| `seed-1-6-thinking-250715` | Seed 1.6 Thinking（短名） | 带推理图像生成 | 2025-07-15 | 同上，短名版本 | + +### 6.2 模型命名规则 + +豆包图像生成模型遵循统一的命名规范： + +``` +doubao-{系列名}-{版本号}-{功能标识}-{发布日期} +``` + +| 部分 | 说明 | 示例 | +|------|------|------| +| `doubao-` | 品牌前缀（可选，短名不含此前缀） | `doubao-seedream-4-0-250828` | +| `seedream` | 模型系列：文生图 | seedream = see + dream | +| `seedance` | 模型系列：图生图 | seedance = see + dance | +| `seed` | 模型系列：通用 | 包含 thinking 能力 | +| `4-0` | 主版本号.次版本号 | 4.0 版本 | +| `t2i` | 功能标识：Text-to-Image | 仅文生图模型 | +| `pro` | 功能标识：专业版 | 图生图专业版 | +| `thinking` | 功能标识：推理增强 | 带思维链的模型 | +| `250828` | 发布日期 YYMMDD | 2025年8月28日 | + +### 6.3 各模型参数差异对比 + +| 参数 | Seedream 4.0 | Seedream 3.0 | Seedance 1.0 Pro | Seed 1.6 Thinking | +|------|-------------|-------------|-----------------|-------------------| +| **文生图** | 支持 | 支持 | 不支持 | 支持 | +| **图生图** | 不支持 | 不支持 | 支持 | 支持 | +| **prompt** | 必填 | 必填 | 必填 | 必填 | +| **image 参数** | 不支持 | 不支持 | 必填 | 可选 | +| **n（数量）** | 1-4 | 1 | 1-4 | 1 | +| **尺寸 1024x1024** | 支持 | 支持 | 支持 | 支持 | +| **尺寸 1536x1024** | 支持 | 不支持 | 支持 | 支持 | +| **尺寸 1024x1536** | 支持 | 不支持 | 支持 | 支持 | +| **尺寸 1792x1024** | 不支持 | 支持 | 不支持 | 不支持 | +| **尺寸 1280x720** | 支持 | 不支持 | 不支持 | 支持 | +| **尺寸 720x1280** | 支持 | 不支持 | 不支持 | 支持 | +| **response_format** | url/b64_json | url/b64_json | url/b64_json | url/b64_json | +| **seed** | 支持 | 支持 | 支持 | 支持 | +| **guidance_scale** | 支持 | 支持 | 支持 | 支持 | +| **strength** | 不支持 | 不支持 | 支持 | 支持 | +| **流式响应** | 不支持 | 不支持 | 不支持 | 不支持 | + +### 6.4 模型性能特征 + +| 模型 | 典型响应时间 | 图像质量 | 中文字体渲染 | 适用场景 | +|------|------------|---------|-------------|---------| +| Seedream 4.0 | 5-15 秒 | 极高 | 优秀 | 高质量文生图、商业海报、带文字的图片 | +| Seedream 3.0 | 3-10 秒 | 高 | 良好 | 通用文生图 | +| Seedance 1.0 Pro | 5-20 秒 | 极高 | 良好 | 图像编辑、风格迁移、细节修改 | +| Seed 1.6 Thinking | 10-30 秒 | 极高 | 优秀 | 复杂场景生成、需要推理的图像 | + +--- + +## 7. 响应格式 + +### 7.1 成功响应结构 + +#### URL 格式响应（response_format: "url"） + +```json +{ + "created": 1714051200, + "data": [ + { + "url": "https://ark-cn-beijing.volces.com/api/v3/files/xxxxxxxx" + } + ] +} +``` + +#### Base64 格式响应（response_format: "b64_json"） + +```json +{ + "created": 1714051200, + "data": [ + { + "b64_json": "iVBORw0KGgoAAAANSUhEUgAA..." + } + ] +} +``` + +### 7.2 响应字段说明 + +| 字段 | 类型 | 必返回 | 说明 | +|------|------|--------|------| +| `created` | integer | 是 | 响应创建时间，Unix 时间戳（秒） | +| `data` | array | 是 | 图像数据数组，长度等于请求中的 `n` | +| `data[].url` | string | 条件 | 图像下载 URL（当 `response_format: "url"` 时返回）。URL 有效期为 1 小时 | +| `data[].b64_json` | string | 条件 | 图像的 Base64 编码字符串（当 `response_format: "b64_json"` 时返回） | +| `data[].revised_prompt` | string | 否 | 模型修改后的 prompt（部分模型可能返回） | + +### 7.3 多图响应示例 + +当请求 `n > 1` 时，`data` 数组包含多个元素： + +```json +{ + "created": 1714051200, + "data": [ + { + "url": "https://ark-cn-beijing.volces.com/api/v3/files/image-001" + }, + { + "url": "https://ark-cn-beijing.volces.com/api/v3/files/image-002" + }, + { + "url": "https://ark-cn-beijing.volces.com/api/v3/files/image-003" + }, + { + "url": "https://ark-cn-beijing.volces.com/api/v3/files/image-004" + } + ] +} +``` + +### 7.4 URL 有效期 + +- 通过 `response_format: "url"` 返回的图像 URL 有效期为 **1 小时** +- 过期后需重新调用 API 获取新 URL +- 建议在获取 URL 后及时下载并存储到自有对象存储（S3/OSS/COS） +- 若需持久化使用，推荐使用 `b64_json` 格式或配合对象存储服务 + +### 7.5 图像格式与大小 + +| 属性 | 说明 | +|------|------| +| 输出格式 | PNG（默认） | +| 色彩空间 | RGB（不支持 CMYK） | +| 位深度 | 8-bit | +| 典型文件大小 | 512x512 约 200-500KB；1024x1024 约 500KB-2MB；1536x1024 约 800KB-3MB | + +--- + +## 8. 错误码与错误处理 + +### 8.1 错误响应格式 + +方舟平台的图像生成 API 使用 OpenAI 兼容的错误格式： + +```json +{ + "error": { + "type": "", + "message": "<错误描述>", + "param": "<相关参数（可选）>", + "code": "<错误代码（可选）>" + } +} +``` + +### 8.2 HTTP 状态码与错误类型 + +| HTTP 状态码 | error.type | 说明 | 可能原因 | +|------------|-----------|------|---------| +| 400 | `invalid_request_error` | 请求参数错误 | 缺少必填字段、参数格式不正确、不支持的尺寸等 | +| 401 | `authentication_error` | 认证失败 | API Key 无效、缺失或已过期 | +| 402 | `insufficient_quota` | 额度不足 | 账户余额不足或配额已用完 | +| 403 | `permission_error` | 权限不足 | 无权使用指定模型 | +| 404 | `not_found_error` | 资源不存在 | 模型 ID 或 Endpoint ID 不存在 | +| 429 | `rate_limit_error` | 请求频率超限 | 超过 API 调用频率限制 | +| 500 | `server_error` | 服务器内部错误 | 方舟平台内部异常 | +| 503 | `service_unavailable` | 服务不可用 | 模型服务暂时不可用 | + +### 8.3 常见业务错误 + +#### 8.3.1 认证相关错误 + +**API Key 无效**： + +```json +{ + "error": { + "type": "authentication_error", + "message": "Incorrect API key provided: ark-xxxx...xxxx. You can find your API key at https://console.volcengine.com/ark.", + "code": "invalid_api_key" + } +} +``` + +**API Key 缺失**： + +```json +{ + "error": { + "type": "authentication_error", + "message": "You didn't provide an API key. You need to provide your API key in the Authorization header.", + "code": "missing_api_key" + } +} +``` + +#### 8.3.2 参数相关错误 + +**不支持的尺寸**： + +```json +{ + "error": { + "type": "invalid_request_error", + "message": "Invalid size: 2048x2048. Supported sizes for doubao-seedream-4-0-250828 are: 1024x1024, 1536x1024, 1024x1536, 1280x720, 720x1280", + "param": "size", + "code": "invalid_size" + } +} +``` + +**模型不存在**： + +```json +{ + "error": { + "type": "invalid_request_error", + "message": "Model not found: doubao-seedream-xxx", + "param": "model", + "code": "model_not_found" + } +} +``` + +**prompt 过长**： + +```json +{ + "error": { + "type": "invalid_request_error", + "message": "Prompt length exceeds maximum limit of 4000 characters", + "param": "prompt", + "code": "prompt_too_long" + } +} +``` + +**缺少必填参数**： + +```json +{ + "error": { + "type": "invalid_request_error", + "message": "Required parameter 'prompt' is missing", + "param": "prompt", + "code": "missing_required_param" + } +} +``` + +#### 8.3.3 限流相关错误 + +**请求频率超限**： + +```json +{ + "error": { + "type": "rate_limit_error", + "message": "Rate limit reached. Please retry after 60 seconds.", + "code": "rate_limit_exceeded" + } +} +``` + +响应头会包含限流信息： + +```http +HTTP/1.1 429 Too Many Requests +X-RateLimit-Limit: 60 +X-RateLimit-Remaining: 0 +X-RateLimit-Reset: 1714051260 +Retry-After: 60 +``` + +#### 8.3.4 内容审核错误 + +**Prompt 被安全系统拒绝**： + +```json +{ + "error": { + "type": "invalid_request_error", + "message": "Your prompt was rejected by our safety system. It may contain content that is not allowed.", + "code": "content_filtered" + } +} +``` + +#### 8.3.5 服务端错误 + +**模型服务不可用**： + +```json +{ + "error": { + "type": "server_error", + "message": "The model service is temporarily unavailable. Please try again later.", + "code": "service_unavailable" + } +} +``` + +### 8.4 错误处理最佳实践 + +| 策略 | 说明 | +|------|------| +| **重试机制** | 对 429（限流）和 500/503（服务端错误）实现指数退避重试，建议最大重试 3 次 | +| **超时设置** | 图像生成请求建议设置 60-120 秒超时 | +| **错误日志** | 记录完整的 error.type、error.code 和 error.message 用于问题排查 | +| **降级策略** | 当主模型不可用时，可切换到备用模型（如从 Seedream 4.0 降级到 3.0） | +| **参数校验** | 在客户端预先校验参数，避免不必要的 API 调用失败 | + +--- + +## 9. 请求与响应示例 + +### 9.1 cURL 示例 + +#### 9.1.1 文生图（基本） + +```bash +curl -X POST https://ark.cn-beijing.volces.com/api/v3/images/generations \ + -H "Content-Type: application/json" \ + -H "Authorization: Bearer $ARK_API_KEY" \ + -d '{ + "model": "doubao-seedream-4-0-250828", + "prompt": "一只可爱的猫咪在雨中撑着小伞", + "n": 1, + "size": "1024x1024", + "response_format": "url" + }' +``` + +**响应**： + +```json +{ + "created": 1714051200, + "data": [ + { + "url": "https://ark-cn-beijing.volces.com/api/v3/files/fg-abc123def456" + } + ] +} +``` + +#### 9.1.2 文生图（Base64 返回） + +```bash +curl -X POST https://ark.cn-beijing.volces.com/api/v3/images/generations \ + -H "Content-Type: application/json" \ + -H "Authorization: Bearer $ARK_API_KEY" \ + -d '{ + "model": "doubao-seedream-4-0-250828", + "prompt": "A futuristic city skyline at night, neon lights, cyberpunk style, ultra detailed", + "n": 1, + "size": "1536x1024", + "response_format": "b64_json" + }' +``` + +**响应**： + +```json +{ + "created": 1714051200, + "data": [ + { + "b64_json": "iVBORw0KGgoAAAANSUhEUgAAB9AAAASwCAYAAAD..." + } + ] +} +``` + +#### 9.1.3 图生图 + +```bash +curl -X POST https://ark.cn-beijing.volces.com/api/v3/images/generations \ + -H "Content-Type: application/json" \ + -H "Authorization: Bearer $ARK_API_KEY" \ + -d '{ + "model": "doubao-seedance-1-0-pro-250528", + "prompt": "将这张照片变成油画风格", + "image": "https://example.com/landscape.jpg", + "n": 1, + "size": "1024x1024", + "response_format": "url", + "strength": 0.6 + }' +``` + +**响应**： + +```json +{ + "created": 1714051200, + "data": [ + { + "url": "https://ark-cn-beijing.volces.com/api/v3/files/fg-xyz789ghi012" + } + ] +} +``` + +#### 9.1.4 使用 Endpoint ID + +方舟平台支持通过接入点 ID（Endpoint ID）指定模型： + +```bash +curl -X POST https://ark.cn-beijing.volces.com/api/v3/images/generations \ + -H "Content-Type: application/json" \ + -H "Authorization: Bearer $ARK_API_KEY" \ + -d '{ + "model": "ep-20250415xxxxx-xxxxx", + "prompt": "中国水墨画风格的山水，远处有飞鸟", + "n": 1, + "size": "1024x1536", + "response_format": "url", + "seed": 12345 + }' +``` + +### 9.2 Python SDK 示例 + +#### 9.2.1 使用 OpenAI SDK（推荐） + +由于方舟平台兼容 OpenAI API 格式，可以直接使用 OpenAI Python SDK： + +```python +from openai import OpenAI + +# 创建客户端，指向方舟平台 +client = OpenAI( + api_key="ark-xxxxxxxxxxxxxxxx", + base_url="https://ark.cn-beijing.volces.com/api/v3" +) + +# 文生图 +response = client.images.generate( + model="doubao-seedream-4-0-250828", + prompt="一只在樱花树下读书的女孩，水彩画风格", + n=1, + size="1024x1024" +) + +# 获取生成的图像 URL +image_url = response.data[0].url +print(f"图像 URL: {image_url}") +``` + +#### 9.2.2 使用 Base64 格式 + +```python +import base64 +from openai import OpenAI + +client = OpenAI( + api_key="ark-xxxxxxxxxxxxxxxx", + base_url="https://ark.cn-beijing.volces.com/api/v3" +) + +response = client.images.generate( + model="doubao-seedream-4-0-250828", + prompt="A majestic dragon flying over a medieval castle at sunset", + n=1, + size="1536x1024", + response_format="b64_json" +) + +# 解码并保存图像 +image_data = base64.b64decode(response.data[0].b64_json) +with open("dragon.png", "wb") as f: + f.write(image_data) +print("图像已保存为 dragon.png") +``` + +#### 9.2.3 图生图示例 + +```python +from openai import OpenAI + +client = OpenAI( + api_key="ark-xxxxxxxxxxxxxxxx", + base_url="https://ark.cn-beijing.volces.com/api/v3" +) + +# 图生图请求 +response = client.images.generate( + model="doubao-seedance-1-0-pro-250528", + prompt="Convert this photo to Studio Ghibli animation style", + image="https://example.com/photo.jpg", + n=1, + size="1024x1024" +) + +print(f"生成图像 URL: {response.data[0].url}") +``` + +### 9.3 Go HTTP 请求示例 + +```go +package main + +import ( + "bytes" + "encoding/json" + "fmt" + "io" + "net/http" +) + +type ImageRequest struct { + Model string `json:"model"` + Prompt string `json:"prompt"` + N int `json:"n"` + Size string `json:"size"` + ResponseFormat string `json:"response_format"` +} + +type ImageResponse struct { + Created int64 `json:"created"` + Data []ImageData `json:"data"` +} + +type ImageData struct { + URL string `json:"url,omitempty"` + B64JSON string `json:"b64_json,omitempty"` +} + +func main() { + reqBody := ImageRequest{ + Model: "doubao-seedream-4-0-250828", + Prompt: "一只橘猫在秋天的落叶中玩耍", + N: 1, + Size: "1024x1024", + ResponseFormat: "url", + } + + body, _ := json.Marshal(reqBody) + + req, _ := http.NewRequest("POST", + "https://ark.cn-beijing.volces.com/api/v3/images/generations", + bytes.NewReader(body)) + req.Header.Set("Content-Type", "application/json") + req.Header.Set("Authorization", "Bearer ark-xxxxxxxxxxxxxxxx") + + client := &http.Client{} + resp, err := client.Do(req) + if err != nil { + panic(err) + } + defer resp.Body.Close() + + respBody, _ := io.ReadAll(resp.Body) + + var imgResp ImageResponse + json.Unmarshal(respBody, &imgResp) + + fmt.Printf("图像 URL: %s\n", imgResp.Data[0].URL) +} +``` + +### 9.4 Node.js 示例 + +```javascript +import OpenAI from 'openai'; + +const client = new OpenAI({ + apiKey: 'ark-xxxxxxxxxxxxxxxx', + baseURL: 'https://ark.cn-beijing.volces.com/api/v3', +}); + +async function generateImage() { + const response = await client.images.generate({ + model: 'doubao-seedream-4-0-250828', + prompt: 'A serene Japanese garden with cherry blossoms in full bloom', + n: 1, + size: '1024x1024', + }); + + console.log('图像 URL:', response.data[0].url); +} + +generateImage(); +``` + +--- + +## 10. OpenAI 兼容接口说明 + +### 10.1 兼容性概述 + +火山引擎方舟平台的图像生成 API **完全兼容 OpenAI Images API 格式**，这意味着： + +- **请求格式**：与 OpenAI `/v1/images/generations` 完全相同 +- **响应格式**：与 OpenAI 响应结构一致（`created` + `data` 数组） +- **错误格式**：与 OpenAI 错误结构一致（`error.type` + `error.message`） +- **SDK 兼容**：可直接使用 OpenAI 官方 SDK，只需修改 `base_url` 和 `api_key` + +### 10.2 端点映射 + +| OpenAI 格式 | 方舟平台格式 | 说明 | +|------------|------------|------| +| `POST /v1/images/generations` | `POST /api/v3/images/generations` | 文生图/图生图 | +| `POST /v1/chat/completions` | `POST /api/v3/chat/completions` | 文本对话 | +| `POST /v1/embeddings` | `POST /api/v3/embeddings` | 文本向量化 | +| `POST /v1/audio/speech` | `POST /api/v3/audio/speech` | 语音合成（方舟平台使用 WebSocket 方式） | + +### 10.3 使用 OpenAI SDK 的配置差异 + +| 配置项 | OpenAI 原生 | 方舟平台 | +|--------|-----------|---------| +| `base_url` | `https://api.openai.com/v1` | `https://ark.cn-beijing.volces.com/api/v3` | +| `api_key` | `sk-xxxxx` | `ark-xxxxx` | +| `model` | `dall-e-3`, `gpt-image-1` | `doubao-seedream-4-0-250828` 等 | + +### 10.4 team-api 网关集成 + +team-api 作为 API 网关，接收 OpenAI 格式的请求，转发至方舟平台： + +``` +客户端 → POST /v1/images/generations (OpenAI 格式) + → team-api Relay 层 + → Volcengine Adaptor + → GET RequestURL: {baseURL}/api/v3/images/generations + → SetupRequestHeader: Bearer {apiKey} + → ConvertRequest: 模型名映射 + OpenAI 格式直通 + → DoRequest: 发送 HTTP 请求 + → DoResponse: 委托 OpenAI 适配器处理响应 +``` + +### 10.5 不支持的 OpenAI 参数 + +以下 OpenAI Images API 参数在方舟平台图像生成中**不支持或行为不同**： + +| 参数 | OpenAI 支持 | 方舟平台支持 | 说明 | +|------|-----------|------------|------| +| `quality` | 支持 | 不支持 | 方舟平台不区分图像质量等级 | +| `style` | 支持（dall-e-3） | 不支持 | 方舟平台不区分 vivid/natural 风格 | +| `background` | 支持（gpt-image-1） | 不支持 | 方舟平台不支持透明背景 | +| `output_format` | 支持（gpt-image-1） | 不支持 | 方舟平台默认输出 PNG | +| `output_compression` | 支持（gpt-image-1） | 不支持 | 不支持压缩率设置 | +| `partial_images` | 支持（gpt-image-1） | 不支持 | 不支持流式中间图 | +| `stream` | 支持（gpt-image-1） | 不支持 | 不支持 SSE 流式响应 | +| `moderation` | 支持（gpt-image-1） | 不支持 | 不支持审核严格度设置 | +| `mask` | 支持（dall-e-2 编辑） | 不支持 | 不支持遮罩编辑 | + +### 10.6 方舟平台特有参数 + +方舟平台支持以下 OpenAI 标准中没有的参数： + +| 参数 | 说明 | +|------|------| +| `seed` | 随机种子，用于可复现生成 | +| `guidance_scale` | 引导比例（CFG Scale） | +| `strength` | 图生图编辑强度 | +| `image` | 参考图像（图生图模式） | + +--- + +## 11. team-api 适配器实现说明 + +### 11.1 当前实现状态 + +team-api 已实现火山引擎（Volcengine）适配器，位于 `relay/channel/volcengine/` 目录。 + +#### 文件结构 + +``` +relay/channel/volcengine/ +├── adaptor.go # 适配器主文件 +└── constants.go # 常量定义（ChannelName） +``` + +#### 适配器核心代码 + +```go +// Adaptor 火山引擎（豆包）供应商适配器。 +// OpenAI 兼容格式，bot- 前缀模型使用 bots 端点。 +type Adaptor struct { + info *common.RelayInfo +} +``` + +#### 当前支持的 RelayMode + +| RelayMode | 上游端点 | 状态 | +|-----------|---------|------| +| `RelayModeChatCompletions` | `/api/v3/chat/completions` 或 `/api/v3/bots/chat/completions` | 已实现 | +| `RelayModeClaudeMessages` | 转换为 OpenAI 格式后走 chat completions | 已实现 | +| `RelayModeEmbeddings` | `/api/v3/embeddings` | 已实现 | +| `RelayModeImagesGenerations` | `/api/v3/images/generations` | 待完善 | +| `RelayModeImagesEdits` | `/api/v3/images/generations`（共用） | 待实现 | + +### 11.2 图像生成适配流程 + +``` +客户端请求 POST /v1/images/generations + → Path2RelayMode 识别为 RelayModeImagesGenerations + → Volcengine Adaptor.GetRequestURL() + → 构建: {baseURL}/api/v3/images/generations + → Volcengine Adaptor.SetupRequestHeader() + → 设置: Authorization: Bearer {apiKey} + → Volcengine Adaptor.ConvertRequest() + → 非 OpenAI 格式先转换 + → 模型名映射（如需要） + → Volcengine Adaptor.DoRequest() + → 发送 HTTP POST 请求 + → Volcengine Adaptor.DoResponse() + → 委托 openai.Adaptor 处理响应 +``` + +### 11.3 已注册模型列表 + +在 `new-api/relay/channel/volcengine/constants.go` 中定义了支持模型的完整列表： + +```go +var ModelList = []string{ + "Doubao-pro-128k", + "Doubao-pro-32k", + "Doubao-pro-4k", + "Doubao-lite-128k", + "Doubao-lite-32k", + "Doubao-lite-4k", + "Doubao-embedding", + "doubao-seedream-4-0-250828", + "seedream-4-0-250828", + "doubao-seedance-1-0-pro-250528", + "seedance-1-0-pro-250528", + "doubao-seed-1-6-thinking-250715", + "seed-1-6-thinking-250715", +} +``` + +### 11.4 关键实现细节 + +#### URL 构建 + +```go +// 文生图和图生图共用同一端点 +// 参考: https://www.volcengine.com/docs/82379/1824121 +case constant.RelayModeImagesGenerations, constant.RelayModeImagesEdits: + return fmt.Sprintf("%s/api/v3/images/generations", baseUrl), nil +``` + +#### 认证方式 + +```go +// 标准 Bearer Token 认证 +header.Set("Authorization", "Bearer "+info.ChannelMeta.ApiKey) +``` + +#### 请求体处理 + +```go +// 火山引擎兼容 OpenAI 格式，只需做模型名映射 +// 非 OpenAI 格式先转换为 OpenAI +if info.InboundFormat != "" && info.InboundFormat != constant.RelayFormatOpenAI { + converted, err := openai.ConvertToOpenAI(requestBody, info) + // ... +} +// 模型名映射 +if info.ChannelMeta.IsModelMapped { + // 替换 model 字段 +} +``` + +#### 响应处理 + +```go +// 响应格式与 OpenAI 一致，委托 OpenAI 适配器处理 +func (a *Adaptor) DoResponse(...) (*common.Usage, error) { + delegate := &openai.Adaptor{} + delegate.Init(info) + return delegate.DoResponse(ctx, resp, info, writer) +} +``` + +### 11.5 与即梦（Jimeng）适配器的区别 + +| 特性 | Volcengine 适配器 | Jimeng 适配器 | +|------|------------------|--------------| +| 上游端点 | `/api/v3/images/generations` | `/v2/images/generations` | +| 基础 URL | `https://ark.cn-beijing.volces.com` | 自定义 Base URL | +| API Key 格式 | `ark-xxxxx`（单一 Token） | `accessKey\|secretKey`（竖线分隔） | +| 响应处理 | OpenAI 兼容格式 | 原生即梦格式需转换为 OpenAI | +| 支持模型 | Seedream, Seedance 等 | 即梦专属模型 | +| 功能范围 | 文生图 + 图生图 + 文本对话 + 嵌入等 | 仅图像生成 | + +### 11.6 待完善项 + +1. **图像生成模式完善**：当前 `team-api/relay/channel/volcengine/adaptor.go` 的 `GetRequestURL` 未覆盖 `RelayModeImagesGenerations`，需要添加图像生成路径支持 +2. **图生图参数支持**：需要在 `ConvertRequest` 中处理 `image` 字段的传递 +3. **模型名映射扩展**：确保所有豆包图像生成模型的名称在渠道配置中正确映射 +4. **计费适配**：豆包图像生成按次计费，需要在计费模块中添加对应的价格配置 +5. **错误格式适配**：确认方舟平台的错误格式与 OpenAI 完全一致 + +### 11.7 计费方式 + +| 模型 | 计费方式 | 说明 | +|------|---------|------| +| Seedream 系列 | 按次计费 | 每次生成按固定价格计费，与 token 无关 | +| Seedance 系列 | 按次计费 | 图生图按次计费，价格通常高于文生图 | +| Seed Thinking | 按次计费 | 带推理的图像生成价格最高 | + +team-api 的 Relay 层 `BillingProvider` 需要根据模型类型选择适当的计费策略。图像生成模型使用固定价格计费，而非 token 计费。在 `handleImageResponse` 中需要正确返回计费信息。 + +### 11.8 参考链接 + +| 资源 | URL | +|------|-----| +| 方舟平台控制台 | https://console.volcengine.com/ark | +| API 文档首页 | https://www.volcengine.com/docs/6791/1399418 | +| 图像生成文档 | https://www.volcengine.com/docs/82379/1824121 | +| API Key 管理 | https://console.volcengine.com/ark/region:ark+cn-beijing/apiKey | +| 模型广场 | https://console.volcengine.com/ark/region:ark+cn-beijing/model | +| SDK 文档 | https://www.volcengine.com/docs/6791/1399820 | +| 定价说明 | https://www.volcengine.com/docs/6791/1361753 | + +--- + +> **文档版本**：v1.0 +> +> **最后更新**：2026-04-19 +> +> **适用范围**：team-api 项目 Relay 层豆包图像生成适配器开发参考 diff --git a/docs/协议文档/阿里系列/通义万相图像生成API文档.md b/docs/协议文档/阿里系列/通义万相图像生成API文档.md new file mode 100644 index 0000000..e022106 --- /dev/null +++ b/docs/协议文档/阿里系列/通义万相图像生成API文档.md @@ -0,0 +1,1586 @@ +# 通义万相图像生成 API 文档 + +> 阿里云百炼（DashScope）平台 — 通义万相（Wanx）图像生成模型 API 完整参考 +> +> 基于阿里云大模型服务平台百炼官方文档整理，覆盖万相文生图、涂鸦作画、图像局部重绘、人像风格重绘、图像编辑等通义万相系列图像生成模型的完整 API 规范。 +> +> 官方文档入口：https://help.aliyun.com/zh/model-studio/developer-reference/image-generation + +--- + +## 目录 + +- [1. 概述](#1-概述) +- [2. 认证方式](#2-认证方式) +- [3. API 端点列表](#3-api-端点列表) +- [4. 文生图接口详细参数](#4-文生图接口详细参数) +- [5. 涂鸦作画接口详细参数](#5-涂鸦作画接口详细参数) +- [6. 图像局部重绘接口详细参数](#6-图像局部重绘接口详细参数) +- [7. 人像风格重绘接口详细参数](#7-人像风格重绘接口详细参数) +- [8. 通用图像编辑接口详细参数](#8-通用图像编辑接口详细参数) +- [9. 同步与异步调用方式](#9-同步与异步调用方式) +- [10. 响应格式](#10-响应格式) +- [11. 错误码与错误处理](#11-错误码与错误处理) +- [12. 请求与响应示例](#12-请求与响应示例) +- [13. OpenAI 兼容接口说明](#13-openai-兼容接口说明) +- [14. team-api 适配器实现说明](#14-team-api-适配器实现说明) + +--- + +## 1. 概述 + +### 1.1 平台简介 + +通义万相（Tongyi Wanxiang / Wanx）是阿里巴巴推出的 AI 图像生成模型系列，通过**阿里云大模型服务平台百炼（DashScope）**提供 API 服务。百炼平台是阿里云的一站式大模型推理服务平台，支持文本生成、图像生成、语音合成、视频生成等多种 AI 能力。 + +通义万相系列覆盖从文生图、图生图、涂鸦作画、人像风格重绘、图像局部重绘到通用图像编辑等多种图像生成和编辑场景，是目前国内功能最全面的 AI 图像生成模型系列之一。 + +### 1.2 API 基本信息总览 + +| 项目 | 值 | +|------|------| +| 平台名称 | 阿里云大模型服务平台百炼（DashScope） | +| 基础 URL | `https://dashscope.aliyuncs.com` | +| API 版本 | v1 | +| 协议风格 | RESTful JSON | +| 兼容模式 | 支持 OpenAI 兼容接口（`/compatible-mode/v1/`） | +| 控制台地址 | https://bailian.console.aliyun.com | + +### 1.3 支持模型完整列表 + +通义万相系列及百炼平台支持的全部图像生成模型如下： + +| 模型分类 | 模型名称（model 参数值） | 功能定位 | +|---------|------------------------|---------| +| **万相-文生图** | `wanx-v2` | 万相文生图 V2 版本，根据文本描述生成高质量图像 | +| **万相-文生图** | `wanx-v1` | 万相文生图 V1 版本，根据文本描述生成图像（基础版） | +| **万相-图像生成与编辑** | `wanx2.1-t2i-ediff` | 万相图像生成与编辑 2.6 版本 | +| **万相-通用图像编辑** | `wanx2.5-image-edit` | 万相通用图像编辑 2.5 版本 | +| **万相-通用图像编辑** | `wanx2.1-image-edit` | 万相通用图像编辑 2.1 版本 | +| **万相-涂鸦作画** | `wanx-sketch-to-image-v2` | 涂鸦作画，将涂鸦草图转化为精美图像 | +| **万相-涂鸦作画** | `wanx-sketch-to-image` | 涂鸦作画 V1 版本 | +| **万相-图像局部重绘** | `wanx-image-inpainting` | 图像局部重绘，对图像指定区域进行内容替换 | +| **万相-人像风格重绘** | `wanx-style-repaint` | 人像风格重绘，将人像照片转换为指定风格 | +| **万相-图像画面扩展** | `wanx-image-outpainting` | 图像画面扩展，扩展图像边界内容 | +| **千问-文生图** | `qwen-image-generation` | 千问文生图模型，千问多模态能力 | +| **千问-图像编辑** | `qwen-image-edit` | 千问图像编辑模型 | +| **千问-图像翻译** | `qwen-image-translate` | 千问图像翻译模型 | +| **文生图 Z-Image** | `zimage` | Z-Image 文生图模型 | +| **虚拟模特** | `virtualmodel` | 虚拟模特试衣展示 | +| **鞋靴模特** | `shoes-model` | 鞋靴模特展示 | +| **创意海报生成** | `poster-generation` | 创意海报自动生成 | +| **人物实例分割** | `person-instance-segmentation` | 人物实例分割 | +| **图像背景生成** | `image-background-generation` | 图像背景生成 | +| **图像擦除补全** | `image-erase-supplement` | 图像擦除补全 | +| **AI 试衣** | `OutfitAnyone` | AI 虚拟试衣 OutfitAnyone | +| **人物写真** | `FaceChain` | 人物写真 FaceChain | +| **创意文字** | `WordArt` | 创意文字生成（锦书） | +| **文生图 SD** | `stable-diffusion-xl` | Stable Diffusion XL 文生图 | +| **文生图 FLUX** | `flux-dev` / `flux-schnell` | FLUX 文生图模型 | + +### 1.4 核心特性 + +- **多模型覆盖**：从基础文生图到高级图像编辑，覆盖图像生成全场景 +- **同步/异步双模式**：支持同步调用（直接返回结果）和异步调用（提交任务后轮询结果） +- **OpenAI 兼容**：通过 `/compatible-mode/v1/` 路径提供 OpenAI 格式兼容接口 +- **多种输出格式**：支持 URL 返回和 Base64 编码返回 +- **多尺寸支持**：支持 512x512、720x1280、1280x720 等多种分辨率 +- **风格化生成**：支持指定摄影风格、艺术风格等 +- **中文优化**：对中文提示词有良好支持 + +--- + +## 2. 认证方式 + +### 2.1 API Key 认证 + +百炼平台使用 Bearer Token 认证方式，在请求头中携带 API Key： + +```http +Authorization: Bearer +Content-Type: application/json +``` + +### 2.2 API Key 获取 + +1. 登录阿里云控制台：https://bailian.console.aliyun.com +2. 进入百炼平台，在左侧导航栏选择「API-KEY 管理」 +3. 点击「创建 API Key」，系统自动生成 API Key +4. API Key 格式示例：`sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx` + +### 2.3 API Key 权限说明 + +| 权限项 | 说明 | +|--------|------| +| 工作空间绑定 | API Key 绑定到特定工作空间，只能访问该空间下的资源 | +| 模型访问 | 需要在工作空间中开通对应模型的访问权限 | +| 免费额度 | 部分模型提供免费额度，超出后按量计费 | +| 并发限制 | 不同 API Key 有不同的并发请求数限制 | + +### 2.4 请求头完整示例 + +```http +POST /api/v1/services/aigc/text2image/image-synthesis HTTP/1.1 +Host: dashscope.aliyuncs.com +Authorization: Bearer sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx +Content-Type: application/json +X-DashScope-Async: enable +``` + +| 请求头 | 说明 | +|--------|------| +| `Authorization` | 必填，Bearer Token 认证，格式为 `Bearer ` | +| `Content-Type` | 必填，请求体格式，固定为 `application/json` | +| `X-DashScope-Async` | 可选，设置为 `enable` 时启用异步调用模式 | + +--- + +## 3. API 端点列表 + +### 3.1 DashScope 原生端点 + +通义万相图像生成使用 DashScope 原生 API 端点，按功能分为以下几类： + +#### 3.1.1 文生图（Text-to-Image） + +| 方法 | 路径 | 功能 | 支持模型 | +|------|------|------|---------| +| POST | `/api/v1/services/aigc/text2image/image-synthesis` | 文本生成图像 | `wanx-v1`, `wanx-v2`, `wanx2.1-t2i-ediff` | +| POST | `/api/v1/services/aigc/text2image/image-synthesis` | 千问文生图 | `qwen-image-generation` | +| POST | `/api/v1/services/aigc/text2image/image-synthesis` | Z-Image 文生图 | `zimage` | + +#### 3.1.2 涂鸦作画（Sketch-to-Image） + +| 方法 | 路径 | 功能 | 支持模型 | +|------|------|------|---------| +| POST | `/api/v1/services/aigc/image2image/image-synthesis` | 涂鸦作画 | `wanx-sketch-to-image`, `wanx-sketch-to-image-v2` | + +#### 3.1.3 图像局部重绘（Image Inpainting） + +| 方法 | 路径 | 功能 | 支持模型 | +|------|------|------|---------| +| POST | `/api/v1/services/aigc/image-inpainting/image-synthesis` | 图像局部重绘 | `wanx-image-inpainting` | + +#### 3.1.4 人像风格重绘（Portrait Style Repaint） + +| 方法 | 路径 | 功能 | 支持模型 | +|------|------|------|---------| +| POST | `/api/v1/services/aigc/image-generation/image-synthesis` | 人像风格重绘 | `wanx-style-repaint` | + +#### 3.1.5 通用图像编辑（Image Edit） + +| 方法 | 路径 | 功能 | 支持模型 | +|------|------|------|---------| +| POST | `/api/v1/services/aigc/image-edit/image-synthesis` | 通用图像编辑 | `wanx2.1-image-edit`, `wanx2.5-image-edit` | + +#### 3.1.6 图像画面扩展（Image Outpainting） + +| 方法 | 路径 | 功能 | 支持模型 | +|------|------|------|---------| +| POST | `/api/v1/services/aigc/image-outpainting/image-synthesis` | 图像画面扩展 | `wanx-image-outpainting` | + +#### 3.1.7 异步任务管理端点 + +| 方法 | 路径 | 功能 | +|------|------|------| +| GET | `/api/v1/tasks/{task_id}` | 查询异步任务状态和结果 | + +### 3.2 OpenAI 兼容端点 + +百炼平台同时提供 OpenAI 兼容接口，基础路径为 `/compatible-mode/v1/`： + +| 方法 | 路径 | 功能 | 说明 | +|------|------|------|------| +| POST | `/compatible-mode/v1/images/generations` | 文本生成图像 | 兼容 OpenAI Images API 格式 | + +--- + +## 4. 文生图接口详细参数 + +### 4.1 万相文生图 V2（wanx-v2） + +**端点**：`POST /api/v1/services/aigc/text2image/image-synthesis` + +#### 请求体参数 + +| 参数名 | 类型 | 必填 | 默认值 | 说明 | +|--------|------|------|--------|------| +| `model` | string | 是 | — | 模型名称，固定为 `wanx-v2` | +| `input.prompt` | string | 是 | — | 文本提示词，描述希望生成的图像内容，支持中英文 | +| `input.negative_prompt` | string | 否 | `""` | 反向提示词，描述不希望出现在图像中的内容 | +| `parameters.size` | string | 否 | `"1024*1024"` | 图像尺寸，格式为 `宽*高` | +| `parameters.n` | integer | 否 | `1` | 生成图像数量，取值范围 1-4 | +| `parameters.seed` | integer | 否 | 随机 | 随机种子，相同种子和参数可复现结果 | +| `parameters.ref_strength` | float | 否 | `0.5` | 参考图影响强度，取值范围 0.0-1.0 | +| `parameters.ref_img` | string | 否 | `""` | 参考图片 URL，用于风格参考 | +| `parameters.style` | string | 否 | `""` | 图像风格，可选值见下方风格列表 | +| `parameters.num_inference_steps` | integer | 否 | `50` | 推理步数，影响生成质量和速度 | + +#### 支持的图像尺寸（wanx-v2） + +| 尺寸 | 格式 | 说明 | +|------|------|------| +| 1024x1024 | `"1024*1024"` | 正方形 | +| 720x1280 | `"720*1280"` | 竖版（手机壁纸） | +| 1280x720 | `"1280*720"` | 横版（桌面壁纸） | +| 960x1280 | `"960*1280"` | 竖版（海报） | +| 1280x960 | `"1280*960"` | 横版（海报） | + +#### 支持的风格（wanx-v2） + +| 风格值 | 说明 | +|--------|------| +| `""` | 摄影风格 | +| `""` | 人像摄影 | +| `"<3d-cartoon>"` | 3D 卡通 | +| `""` | 动漫风格 | +| `""` | 油画风格 | +| `""` | 水彩风格 | +| `""` | 素描风格 | +| `""` | 中国画风格 | +| `""` | 自动风格（由模型决定） | +| `""` (默认) | 自动选择最佳风格 | + +#### 请求体示例 + +```json +{ + "model": "wanx-v2", + "input": { + "prompt": "一只可爱的橘猫坐在窗台上，阳光洒在身上，温暖的色调", + "negative_prompt": "低质量, 模糊, 变形" + }, + "parameters": { + "size": "1024*1024", + "n": 1, + "seed": 42, + "style": "" + } +} +``` + +### 4.2 万相文生图 V1（wanx-v1） + +**端点**：`POST /api/v1/services/aigc/text2image/image-synthesis` + +#### 请求体参数 + +| 参数名 | 类型 | 必填 | 默认值 | 说明 | +|--------|------|------|--------|------| +| `model` | string | 是 | — | 模型名称，固定为 `wanx-v1` | +| `input.prompt` | string | 是 | — | 文本提示词，描述希望生成的图像内容 | +| `input.negative_prompt` | string | 否 | `""` | 反向提示词 | +| `parameters.size` | string | 否 | `"1024*1024"` | 图像尺寸 | +| `parameters.n` | integer | 否 | `1` | 生成图像数量，取值范围 1-4 | +| `parameters.seed` | integer | 否 | 随机 | 随机种子 | + +### 4.3 千问文生图（qwen-image-generation） + +**端点**：`POST /api/v1/services/aigc/text2image/image-synthesis` + +#### 请求体参数 + +| 参数名 | 类型 | 必填 | 默认值 | 说明 | +|--------|------|------|--------|------| +| `model` | string | 是 | — | 模型名称，固定为 `qwen-image-generation` | +| `input.prompt` | string | 是 | — | 文本提示词，详细描述希望生成的图像 | +| `input.negative_prompt` | string | 否 | `""` | 反向提示词 | +| `parameters.size` | string | 否 | `"1024*1024"` | 图像尺寸，支持 `"1024*1024"`、`"720*1280"`、`"1280*720"` | +| `parameters.n` | integer | 否 | `1` | 生成图像数量 | +| `parameters.seed` | integer | 否 | 随机 | 随机种子 | + +### 4.4 万相图像生成与编辑 2.6（wanx2.1-t2i-ediff） + +**端点**：`POST /api/v1/services/aigc/text2image/image-synthesis` + +#### 请求体参数 + +| 参数名 | 类型 | 必填 | 默认值 | 说明 | +|--------|------|------|--------|------| +| `model` | string | 是 | — | 模型名称，固定为 `wanx2.1-t2i-ediff` | +| `input.prompt` | string | 是 | — | 文本提示词 | +| `input.negative_prompt` | string | 否 | `""` | 反向提示词 | +| `parameters.size` | string | 否 | `"1024*1024"` | 图像尺寸 | +| `parameters.n` | integer | 否 | `1` | 生成图像数量 | +| `parameters.seed` | integer | 否 | 随机 | 随机种子 | +| `parameters.ref_img` | string | 否 | `""` | 参考图片 URL | +| `parameters.ref_strength` | float | 否 | `0.5` | 参考图影响强度，0.0-1.0 | + +--- + +## 5. 涂鸦作画接口详细参数 + +### 5.1 万相涂鸦作画 V2（wanx-sketch-to-image-v2） + +**端点**：`POST /api/v1/services/aigc/image2image/image-synthesis` + +#### 请求体参数 + +| 参数名 | 类型 | 必填 | 默认值 | 说明 | +|--------|------|------|--------|------| +| `model` | string | 是 | — | 模型名称，固定为 `wanx-sketch-to-image-v2` | +| `input.prompt` | string | 是 | — | 文本提示词，描述希望生成的图像内容 | +| `input.negative_prompt` | string | 否 | `""` | 反向提示词 | +| `input.sketch.image_url` | string | 是 | — | 涂鸦草图的 URL 地址 | +| `input.sketch.image_format` | string | 否 | `"png"` | 涂鸦草图格式，支持 `png`、`jpg`、`jpeg` | +| `input.base_image.image_url` | string | 否 | `""` | 底图 URL，提供后涂鸦将叠加在底图上进行风格化 | +| `parameters.size` | string | 否 | `"1024*1024"` | 输出图像尺寸 | +| `parameters.n` | integer | 否 | `1` | 生成图像数量，取值范围 1-4 | +| `parameters.seed` | integer | 否 | 随机 | 随机种子 | +| `parameters.strength` | float | 否 | `0.5` | 涂鸦影响强度，取值范围 0.0-1.0，值越大越贴近涂鸦 | +| `parameters.style` | string | 否 | `""` | 图像风格，与文生图风格列表相同 | + +#### 请求体示例 + +```json +{ + "model": "wanx-sketch-to-image-v2", + "input": { + "prompt": "一只彩色的蝴蝶在花丛中飞舞，水彩画风格", + "sketch": { + "image_url": "https://example.com/sketch.png" + } + }, + "parameters": { + "size": "1024*1024", + "n": 1, + "strength": 0.6, + "style": "" + } +} +``` + +### 5.2 万相涂鸦作画 V1（wanx-sketch-to-image） + +**端点**：`POST /api/v1/services/aigc/image2image/image-synthesis` + +#### 请求体参数 + +| 参数名 | 类型 | 必填 | 默认值 | 说明 | +|--------|------|------|--------|------| +| `model` | string | 是 | — | 模型名称，固定为 `wanx-sketch-to-image` | +| `input.prompt` | string | 是 | — | 文本提示词 | +| `input.sketch.image_url` | string | 是 | — | 涂鸦草图的 URL 地址 | +| `parameters.size` | string | 否 | `"1024*1024"` | 输出图像尺寸 | +| `parameters.n` | integer | 否 | `1` | 生成图像数量 | +| `parameters.seed` | integer | 否 | 随机 | 随机种子 | +| `parameters.strength` | float | 否 | `0.5` | 涂鸦影响强度 | + +--- + +## 6. 图像局部重绘接口详细参数 + +### 6.1 万相图像局部重绘（wanx-image-inpainting） + +**端点**：`POST /api/v1/services/aigc/image-inpainting/image-synthesis` + +#### 请求体参数 + +| 参数名 | 类型 | 必填 | 默认值 | 说明 | +|--------|------|------|--------|------| +| `model` | string | 是 | — | 模型名称，固定为 `wanx-image-inpainting` | +| `input.prompt` | string | 是 | — | 文本提示词，描述重绘区域希望生成的内容 | +| `input.negative_prompt` | string | 否 | `""` | 反向提示词 | +| `input.image_url` | string | 是 | — | 原始图片的 URL 地址 | +| `input.mask_image_url` | string | 是 | — | 遮罩图片 URL，白色区域为需要重绘的部分 | +| `parameters.size` | string | 否 | 与原图相同 | 输出图像尺寸，默认保持与原图一致 | +| `parameters.n` | integer | 否 | `1` | 生成图像数量，取值范围 1-4 | +| `parameters.seed` | integer | 否 | 随机 | 随机种子 | +| `parameters.strength` | float | 否 | `0.8` | 重绘强度，取值范围 0.0-1.0 | +| `parameters.ref_img` | string | 否 | `""` | 参考图片 URL，重绘区域风格将参考此图 | + +#### 遮罩图片说明 + +遮罩图片（mask）用于标记需要重绘的区域： + +| 颜色 | 含义 | +|------|------| +| 白色（RGB 255,255,255） | 需要重绘的区域 | +| 黑色（RGB 0,0,0） | 保持不变的区域 | + +- 遮罩图片尺寸必须与原始图片尺寸一致 +- 遮罩图片格式支持 PNG、JPG +- 支持多个不连续的重绘区域 + +#### 请求体示例 + +```json +{ + "model": "wanx-image-inpainting", + "input": { + "prompt": "把背景替换为日落时分的海边", + "image_url": "https://example.com/original.png", + "mask_image_url": "https://example.com/mask.png" + }, + "parameters": { + "n": 1, + "strength": 0.8, + "seed": 42 + } +} +``` + +--- + +## 7. 人像风格重绘接口详细参数 + +### 7.1 万相人像风格重绘（wanx-style-repaint） + +**端点**：`POST /api/v1/services/aigc/image-generation/image-synthesis` + +#### 请求体参数 + +| 参数名 | 类型 | 必填 | 默认值 | 说明 | +|--------|------|------|--------|------| +| `model` | string | 是 | — | 模型名称，固定为 `wanx-style-repaint` | +| `input.prompt` | string | 否 | `""` | 文本提示词，补充描述风格化方向 | +| `input.image_url` | string | 是 | — | 原始人像图片的 URL 地址 | +| `parameters.style` | string | 是 | — | 目标风格，见下方支持的风格列表 | +| `parameters.n` | integer | 否 | `1` | 生成图像数量，取值范围 1-4 | +| `parameters.seed` | integer | 否 | 随机 | 随机种子 | +| `parameters.strength` | float | 否 | `0.6` | 风格化强度，取值范围 0.0-1.0，值越大风格越明显 | +| `parameters.size` | string | 否 | 与原图相同 | 输出图像尺寸 | + +#### 支持的风格（人像风格重绘） + +| 风格值 | 说明 | +|--------|------| +| `"cartoon"` | 卡通风格 | +| `"3d_cartoon"` | 3D 卡通风格 | +| `"anime"` | 动漫风格 | +| `"oil_painting"` | 油画风格 | +| `"watercolor"` | 水彩风格 | +| `"sketch"` | 素描风格 | +| `"cyberpunk"` | 赛博朋克风格 | +| `"chinese_painting"` | 国画风格 | +| `"id_photo"` | 证件照风格 | +| `"template"` | 模板风格（需配合 template_id 使用） | + +#### 输入图片要求 + +| 要求项 | 说明 | +|--------|------| +| 图片格式 | PNG、JPG、JPEG | +| 图片大小 | 不超过 10MB | +| 图片尺寸 | 建议不超过 4096x4096 | +| 图片内容 | 必须包含清晰可识别的人像面部 | +| URL 可达性 | 图片 URL 必须公网可访问 | + +#### 请求体示例 + +```json +{ + "model": "wanx-style-repaint", + "input": { + "image_url": "https://example.com/portrait.jpg" + }, + "parameters": { + "style": "anime", + "strength": 0.7, + "n": 1, + "seed": 100 + } +} +``` + +--- + +## 8. 通用图像编辑接口详细参数 + +### 8.1 万相通用图像编辑 2.5（wanx2.5-image-edit） + +**端点**：`POST /api/v1/services/aigc/image-edit/image-synthesis` + +#### 请求体参数 + +| 参数名 | 类型 | 必填 | 默认值 | 说明 | +|--------|------|------|--------|------| +| `model` | string | 是 | — | 模型名称，固定为 `wanx2.5-image-edit` | +| `input.prompt` | string | 是 | — | 文本提示词，描述希望对图像进行的编辑操作 | +| `input.negative_prompt` | string | 否 | `""` | 反向提示词 | +| `input.image_url` | string | 是 | — | 原始图片的 URL 地址 | +| `input.mask_image_url` | string | 否 | `""` | 遮罩图片 URL，指定编辑区域 | +| `parameters.size` | string | 否 | 与原图相同 | 输出图像尺寸 | +| `parameters.n` | integer | 否 | `1` | 生成图像数量 | +| `parameters.seed` | integer | 否 | 随机 | 随机种子 | +| `parameters.strength` | float | 否 | `0.7` | 编辑强度，取值范围 0.0-1.0 | +| `parameters.ref_img` | string | 否 | `""` | 参考图片 URL | + +### 8.2 万相通用图像编辑 2.1（wanx2.1-image-edit） + +**端点**：`POST /api/v1/services/aigc/image-edit/image-synthesis` + +#### 请求体参数 + +| 参数名 | 类型 | 必填 | 默认值 | 说明 | +|--------|------|------|--------|------| +| `model` | string | 是 | — | 模型名称，固定为 `wanx2.1-image-edit` | +| `input.prompt` | string | 是 | — | 文本提示词 | +| `input.image_url` | string | 是 | — | 原始图片的 URL 地址 | +| `input.mask_image_url` | string | 否 | `""` | 遮罩图片 URL | +| `parameters.size` | string | 否 | 与原图相同 | 输出图像尺寸 | +| `parameters.n` | integer | 否 | `1` | 生成图像数量 | +| `parameters.seed` | integer | 否 | 随机 | 随机种子 | +| `parameters.strength` | float | 否 | `0.7` | 编辑强度 | + +### 8.3 千问图像编辑（qwen-image-edit） + +**端点**：`POST /api/v1/services/aigc/image-edit/image-synthesis` + +#### 请求体参数 + +| 参数名 | 类型 | 必填 | 默认值 | 说明 | +|--------|------|------|--------|------| +| `model` | string | 是 | — | 模型名称，固定为 `qwen-image-edit` | +| `input.prompt` | string | 是 | — | 文本提示词，描述编辑操作 | +| `input.image_url` | string | 是 | — | 原始图片 URL | +| `input.mask_image_url` | string | 否 | `""` | 遮罩图片 URL | +| `parameters.size` | string | 否 | `"1024*1024"` | 输出图像尺寸 | +| `parameters.n` | integer | 否 | `1` | 生成图像数量 | +| `parameters.seed` | integer | 否 | 随机 | 随机种子 | + +#### 请求体示例（通用图像编辑） + +```json +{ + "model": "wanx2.5-image-edit", + "input": { + "prompt": "将天空变为晴天蓝天白云的效果", + "image_url": "https://example.com/photo.jpg", + "mask_image_url": "https://example.com/sky_mask.png" + }, + "parameters": { + "strength": 0.7, + "n": 1 + } +} +``` + +--- + +## 9. 同步与异步调用方式 + +### 9.1 调用模式概述 + +通义万相 API 支持两种调用模式： + +| 模式 | 适用场景 | 响应方式 | +|------|---------|---------| +| 同步调用 | 生成速度快的小图、简单场景 | 请求阻塞等待，直接返回生成结果 | +| 异步调用 | 高分辨率图像、批量生成、耗时操作 | 立即返回任务 ID，通过轮询获取结果 | + +### 9.2 同步调用 + +同步调用时，不设置 `X-DashScope-Async` 请求头，API 会阻塞等待图像生成完成后直接返回结果。 + +**请求示例（同步）**： + +```bash +curl -X POST 'https://dashscope.aliyuncs.com/api/v1/services/aigc/text2image/image-synthesis' \ + -H 'Authorization: Bearer sk-xxxxxxxx' \ + -H 'Content-Type: application/json' \ + -d '{ + "model": "wanx-v2", + "input": { + "prompt": "一只可爱的橘猫坐在窗台上" + }, + "parameters": { + "size": "1024*1024", + "n": 1 + } + }' +``` + +**同步响应**： + +请求成功时，HTTP 状态码为 200，响应体中直接包含生成的图像 URL： + +```json +{ + "output": { + "task_status": "SUCCEEDED", + "results": [ + { + "url": "https://dashscope-result-bj.oss-cn-beijing.aliyuncs.com/xxx.png" + } + ] + }, + "request_id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" +} +``` + +### 9.3 异步调用 + +异步调用时，在请求头中设置 `X-DashScope-Async: enable`，API 立即返回任务 ID。 + +**步骤一：提交异步任务** + +```bash +curl -X POST 'https://dashscope.aliyuncs.com/api/v1/services/aigc/text2image/image-synthesis' \ + -H 'Authorization: Bearer sk-xxxxxxxx' \ + -H 'Content-Type: application/json' \ + -H 'X-DashScope-Async: enable' \ + -d '{ + "model": "wanx-v2", + "input": { + "prompt": "一只可爱的橘猫坐在窗台上" + }, + "parameters": { + "size": "1024*1024", + "n": 4 + } + }' +``` + +**异步任务提交响应**： + +```json +{ + "output": { + "task_id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", + "task_status": "PENDING" + }, + "request_id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" +} +``` + +**步骤二：轮询任务状态** + +使用返回的 `task_id` 查询任务状态： + +```bash +curl -X GET 'https://dashscope.aliyuncs.com/api/v1/tasks/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx' \ + -H 'Authorization: Bearer sk-xxxxxxxx' +``` + +**轮询响应（任务进行中）**： + +```json +{ + "output": { + "task_id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", + "task_status": "RUNNING" + }, + "request_id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" +} +``` + +**轮询响应（任务完成）**： + +```json +{ + "output": { + "task_id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", + "task_status": "SUCCEEDED", + "results": [ + { + "url": "https://dashscope-result-bj.oss-cn-beijing.aliyuncs.com/xxx_0.png" + }, + { + "url": "https://dashscope-result-bj.oss-cn-beijing.aliyuncs.com/xxx_1.png" + }, + { + "url": "https://dashscope-result-bj.oss-cn-beijing.aliyuncs.com/xxx_2.png" + }, + { + "url": "https://dashscope-result-bj.oss-cn-beijing.aliyuncs.com/xxx_3.png" + } + ], + "submit_time": "2026-04-19 10:00:00.000", + "scheduled_time": "2026-04-19 10:00:01.000", + "end_time": "2026-04-19 10:00:15.000" + }, + "usage": { + "image_count": 4 + }, + "request_id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" +} +``` + +### 9.4 任务状态流转 + +| 状态 | 说明 | 是否终态 | +|------|------|---------| +| `PENDING` | 任务已提交，排队等待中 | 否 | +| `RUNNING` | 任务正在执行中 | 否 | +| `SUCCEEDED` | 任务执行成功 | 是 | +| `FAILED` | 任务执行失败 | 是 | +| `UNKNOWN` | 任务状态未知 | 是 | +| `CANCELED` | 任务已取消 | 是 | + +### 9.5 轮询建议 + +| 建议项 | 说明 | +|--------|------| +| 轮询间隔 | 建议 2-5 秒一次，不宜过于频繁 | +| 最大轮询次数 | 建议 60 次（约 3-5 分钟） | +| 超时处理 | 超过最大轮询次数后视为超时 | +| 终态判断 | `task_status` 为 `SUCCEEDED`、`FAILED`、`UNKNOWN` 或 `CANCELED` 时停止轮询 | + +--- + +## 10. 响应格式 + +### 10.1 通用响应结构 + +所有 DashScope API 响应都遵循统一结构： + +```json +{ + "output": { ... }, + "usage": { ... }, + "request_id": "string" +} +``` + +| 字段 | 类型 | 说明 | +|------|------|------| +| `output` | object | 输出内容，包含任务状态和结果 | +| `usage` | object | 用量信息 | +| `request_id` | string | 请求唯一标识，用于问题排查 | + +### 10.2 同步调用成功响应 + +```json +{ + "output": { + "task_status": "SUCCEEDED", + "results": [ + { + "url": "https://dashscope-result-bj.oss-cn-beijing.aliyuncs.com/1a2b3c/image_0.png?Expires=1745088000&OSSAccessKeyId=xxx&Signature=xxx" + } + ] + }, + "usage": { + "image_count": 1 + }, + "request_id": "f1e2d3c4-b5a6-7890-abcd-ef1234567890" +} +``` + +### 10.3 异步调用任务提交响应 + +```json +{ + "output": { + "task_id": "f1e2d3c4-b5a6-7890-abcd-ef1234567890", + "task_status": "PENDING" + }, + "request_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890" +} +``` + +### 10.4 异步任务完成响应 + +```json +{ + "output": { + "task_id": "f1e2d3c4-b5a6-7890-abcd-ef1234567890", + "task_status": "SUCCEEDED", + "results": [ + { + "url": "https://dashscope-result-bj.oss-cn-beijing.aliyuncs.com/xxx/image_0.png?Expires=xxx&OSSAccessKeyId=xxx&Signature=xxx" + }, + { + "url": "https://dashscope-result-bj.oss-cn-beijing.aliyuncs.com/xxx/image_1.png?Expires=xxx&OSSAccessKeyId=xxx&Signature=xxx" + } + ], + "submit_time": "2026-04-19 10:00:00.000", + "scheduled_time": "2026-04-19 10:00:01.000", + "end_time": "2026-04-19 10:00:15.000" + }, + "usage": { + "image_count": 2 + }, + "request_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890" +} +``` + +### 10.5 异步任务失败响应 + +```json +{ + "output": { + "task_id": "f1e2d3c4-b5a6-7890-abcd-ef1234567890", + "task_status": "FAILED", + "code": "InvalidParameter", + "message": "Prompt content is empty." + }, + "request_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890" +} +``` + +### 10.6 结果 URL 说明 + +| 属性 | 说明 | +|------|------| +| 有效期 | 结果 URL 有效期为 **1 小时**，过期后无法访问 | +| 格式 | OSS 签名 URL，包含 Expires、OSSAccessKeyId、Signature 参数 | +| 建议处理 | 获取 URL 后尽快下载保存，避免过期失效 | +| 域名 | 通常为 `dashscope-result-bj.oss-cn-beijing.aliyuncs.com` | + +### 10.7 OpenAI 兼容模式响应 + +使用 `/compatible-mode/v1/images/generations` 端点时，返回 OpenAI 格式响应： + +```json +{ + "created": 1745088000, + "data": [ + { + "url": "https://dashscope-result-bj.oss-cn-beijing.aliyuncs.com/xxx/image_0.png" + } + ] +} +``` + +--- + +## 11. 错误码与错误处理 + +### 11.1 HTTP 状态码 + +| HTTP 状态码 | 说明 | +|-------------|------| +| 200 | 请求成功 | +| 400 | 请求参数错误 | +| 401 | 认证失败（API Key 无效或过期） | +| 403 | 权限不足（无权访问该模型或资源） | +| 404 | 资源不存在（模型不存在或任务不存在） | +| 429 | 请求频率超限或配额不足 | +| 500 | 服务器内部错误 | +| 503 | 模型服务不可用 | + +### 11.2 错误响应格式 + +DashScope 原生错误响应格式： + +```json +{ + "code": "InvalidParameter", + "message": "Prompt content is empty.", + "request_id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" +} +``` + +OpenAI 兼容模式错误响应格式： + +```json +{ + "error": { + "code": "invalid_request_error", + "message": "Prompt content is empty.", + "param": "prompt", + "type": "invalid_request_error" + } +} +``` + +### 11.3 完整错误码列表 + +#### 400 系列 — 请求参数错误 + +| 错误码 | 错误消息 | 说明 | +|--------|---------|------| +| `InvalidParameter` | Prompt content is empty | 提示词内容为空 | +| `InvalidParameter` | Image format is not supported | 不支持的图片格式 | +| `InvalidParameter` | Image size exceeds the limit | 图片尺寸超出限制 | +| `InvalidParameter` | The value of parameter `n` is invalid | 参数 n 的值无效 | +| `InvalidParameter` | The value of parameter `size` is invalid | 参数 size 的值无效 | +| `InvalidParameter` | The value of parameter `seed` is invalid | 参数 seed 的值无效 | +| `InvalidParameter` | The value of parameter `strength` is invalid | 参数 strength 的值无效 | +| `InvalidParameter` | The value of parameter `style` is invalid | 参数 style 的值无效 | +| `InvalidParameter` | Prompt content may contain inappropriate content | 提示词可能包含不适当内容 | +| `InvalidParameter` | The number of input images exceeds the limit | 输入图片数量超出限制 | +| `InvalidParameter` | Mask image size does not match original image | 遮罩图片尺寸与原图不匹配 | +| `InvalidParameter` | Image URL is not accessible | 图片 URL 不可访问 | +| `InvalidParameter` | Model is not found | 模型不存在 | + +#### 401 系列 — 认证错误 + +| 错误码 | 错误消息 | 说明 | +|--------|---------|------| +| `InvalidApiKey` | Invalid API-key provided | 提供的 API Key 无效 | +| `invalid_api_key` | Invalid API key | API Key 无效（OpenAI 兼容模式） | +| `InvalidApiKey` | API key is not valid or has expired | API Key 已过期或无效 | +| `InvalidAuthorization` | Authorization header is empty | 认证头为空 | + +#### 403 系列 — 权限错误 + +| 错误码 | 错误消息 | 说明 | +|--------|---------|------| +| `AccessDenied` | Access denied | 访问被拒绝 | +| `Model.AccessDenied` | You are not authorized to use this model | 无权使用该模型 | +| `AllocationQuota.FreeTierOnly` | Free tier quota exceeded | 免费额度已用完 | +| `AccessDenied` | Workspace does not have permission to access this resource | 工作空间无权访问该资源 | + +#### 404 系列 — 资源不存在 + +| 错误码 | 错误消息 | 说明 | +|--------|---------|------| +| `ModelNotFound` | Model not found | 模型不存在 | +| `model_not_supported` | Model is not supported | 不支持的模型（OpenAI 兼容模式） | +| `TaskNotFound` | Task not found | 任务不存在（查询异步任务时） | + +#### 429 系列 — 限流错误 + +| 错误码 | 错误消息 | 说明 | +|--------|---------|------| +| `Throttling` | Request was throttled. Expected available in X seconds | 请求被限流，预期 X 秒后可用 | +| `Throttting` | Requests rate limit exceeded | 请求频率超限 | +| `Throttling` | Quota exceeded | 配额不足 | +| `Throttling` | Concurrent request limit exceeded | 并发请求限制超出 | +| `insufficient_quota` | Insufficient quota | 配额不足（OpenAI 兼容模式） | +| `rate_limit_error` | Rate limit reached | 速率限制（OpenAI 兼容模式） | + +#### 500 系列 — 服务器错误 + +| 错误码 | 错误消息 | 说明 | +|--------|---------|------| +| `InternalError` | An internal error has occurred | 内部错误 | +| `RequestTimeOut` | Request timed out | 请求超时 | +| `ModelUnavailable` | Model is temporarily unavailable | 模型暂时不可用 | +| `ModelServingError` | Model serving error | 模型推理服务错误 | +| `InternalError` | Image generation failed | 图像生成失败 | + +### 11.4 错误处理最佳实践 + +| 实践 | 说明 | +|------|------| +| 重试策略 | 500/503 错误建议指数退避重试，最多 3 次 | +| 限流处理 | 429 错误根据 `Retry-After` 响应头等待后重试 | +| 参数校验 | 400 错误检查请求参数，修正后重新提交 | +| 认证检查 | 401 错误检查 API Key 是否正确且未过期 | +| 权限确认 | 403 错误确认已在百炼平台开通对应模型权限 | +| 超时设置 | 建议同步调用设置 60 秒超时，异步调用设置 10 秒超时 | +| URL 及时下载 | 生成结果 URL 有效期 1 小时，建议获取后立即下载 | +| 任务超时 | 异步任务建议最长等待 5 分钟，超时视为失败 | + +--- + +## 12. 请求与响应示例 + +### 12.1 文生图 — 同步调用 + +**请求**： + +```bash +curl -X POST 'https://dashscope.aliyuncs.com/api/v1/services/aigc/text2image/image-synthesis' \ + -H 'Authorization: Bearer sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' \ + -H 'Content-Type: application/json' \ + -d '{ + "model": "wanx-v2", + "input": { + "prompt": "一座古色古香的中式庭院，小桥流水，假山翠竹，傍晚的暖色调光线", + "negative_prompt": "低质量, 模糊, 变形, 水印" + }, + "parameters": { + "size": "1280*720", + "n": 1, + "seed": 42, + "style": "" + } + }' +``` + +**响应**： + +```json +{ + "output": { + "task_status": "SUCCEEDED", + "results": [ + { + "url": "https://dashscope-result-bj.oss-cn-beijing.aliyuncs.com/a1b2c3d4/e5f6g7h8.png?Expires=1745091600&OSSAccessKeyId=LTAI5txxx&Signature=xxxxx" + } + ] + }, + "usage": { + "image_count": 1 + }, + "request_id": "f1e2d3c4-b5a6-7890-abcd-ef1234567890" +} +``` + +### 12.2 文生图 — 异步调用 + +**步骤一：提交任务** + +```bash +curl -X POST 'https://dashscope.aliyuncs.com/api/v1/services/aigc/text2image/image-synthesis' \ + -H 'Authorization: Bearer sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' \ + -H 'Content-Type: application/json' \ + -H 'X-DashScope-Async: enable' \ + -d '{ + "model": "wanx-v2", + "input": { + "prompt": "未来城市的天际线，赛博朋克风格，霓虹灯光" + }, + "parameters": { + "size": "1280*720", + "n": 4, + "style": "" + } + }' +``` + +**提交任务响应**： + +```json +{ + "output": { + "task_id": "abcd1234-5678-efgh-9012-ijkl34567890", + "task_status": "PENDING" + }, + "request_id": "1234abcd-5678-efgh-9012-ijkl34567890" +} +``` + +**步骤二：查询任务结果** + +```bash +curl -X GET 'https://dashscope.aliyuncs.com/api/v1/tasks/abcd1234-5678-efgh-9012-ijkl34567890' \ + -H 'Authorization: Bearer sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' +``` + +**任务完成响应**： + +```json +{ + "output": { + "task_id": "abcd1234-5678-efgh-9012-ijkl34567890", + "task_status": "SUCCEEDED", + "results": [ + { + "url": "https://dashscope-result-bj.oss-cn-beijing.aliyuncs.com/xxx/img_0.png?Expires=1745091600&OSSAccessKeyId=xxx&Signature=xxx" + }, + { + "url": "https://dashscope-result-bj.oss-cn-beijing.aliyuncs.com/xxx/img_1.png?Expires=1745091600&OSSAccessKeyId=xxx&Signature=xxx" + }, + { + "url": "https://dashscope-result-bj.oss-cn-beijing.aliyuncs.com/xxx/img_2.png?Expires=1745091600&OSSAccessKeyId=xxx&Signature=xxx" + }, + { + "url": "https://dashscope-result-bj.oss-cn-beijing.aliyuncs.com/xxx/img_3.png?Expires=1745091600&OSSAccessKeyId=xxx&Signature=xxx" + } + ], + "submit_time": "2026-04-19 10:00:00.000", + "scheduled_time": "2026-04-19 10:00:01.500", + "end_time": "2026-04-19 10:00:20.000" + }, + "usage": { + "image_count": 4 + }, + "request_id": "1234abcd-5678-efgh-9012-ijkl34567890" +} +``` + +### 12.3 涂鸦作画 + +**请求**： + +```bash +curl -X POST 'https://dashscope.aliyuncs.com/api/v1/services/aigc/image2image/image-synthesis' \ + -H 'Authorization: Bearer sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' \ + -H 'Content-Type: application/json' \ + -d '{ + "model": "wanx-sketch-to-image-v2", + "input": { + "prompt": "一栋温馨的小木屋，周围是绿色的草地和鲜花，童话风格", + "sketch": { + "image_url": "https://example.com/my-sketch.png" + } + }, + "parameters": { + "size": "1024*1024", + "n": 1, + "strength": 0.5, + "style": "" + } + }' +``` + +**响应**： + +```json +{ + "output": { + "task_status": "SUCCEEDED", + "results": [ + { + "url": "https://dashscope-result-bj.oss-cn-beijing.aliyuncs.com/xxx/sketch_result.png?Expires=1745091600&OSSAccessKeyId=xxx&Signature=xxx" + } + ] + }, + "usage": { + "image_count": 1 + }, + "request_id": "c3d4e5f6-a7b8-9012-cdef-345678901234" +} +``` + +### 12.4 图像局部重绘 + +**请求**： + +```bash +curl -X POST 'https://dashscope.aliyuncs.com/api/v1/services/aigc/image-inpainting/image-synthesis' \ + -H 'Authorization: Bearer sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' \ + -H 'Content-Type: application/json' \ + -d '{ + "model": "wanx-image-inpainting", + "input": { + "prompt": "一只金色的花瓶放在桌子上", + "image_url": "https://example.com/room.png", + "mask_image_url": "https://example.com/room_mask.png" + }, + "parameters": { + "n": 1, + "strength": 0.8 + } + }' +``` + +**响应**： + +```json +{ + "output": { + "task_status": "SUCCEEDED", + "results": [ + { + "url": "https://dashscope-result-bj.oss-cn-beijing.aliyuncs.com/xxx/inpaint_result.png?Expires=1745091600&OSSAccessKeyId=xxx&Signature=xxx" + } + ] + }, + "usage": { + "image_count": 1 + }, + "request_id": "e5f6a7b8-c9d0-1234-ef56-789012345678" +} +``` + +### 12.5 人像风格重绘 + +**请求**： + +```bash +curl -X POST 'https://dashscope.aliyuncs.com/api/v1/services/aigc/image-generation/image-synthesis' \ + -H 'Authorization: Bearer sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' \ + -H 'Content-Type: application/json' \ + -d '{ + "model": "wanx-style-repaint", + "input": { + "image_url": "https://example.com/my-photo.jpg" + }, + "parameters": { + "style": "anime", + "strength": 0.7, + "n": 1 + } + }' +``` + +**响应**： + +```json +{ + "output": { + "task_status": "SUCCEEDED", + "results": [ + { + "url": "https://dashscope-result-bj.oss-cn-beijing.aliyuncs.com/xxx/style_result.png?Expires=1745091600&OSSAccessKeyId=xxx&Signature=xxx" + } + ] + }, + "usage": { + "image_count": 1 + }, + "request_id": "f6a7b8c9-d0e1-2345-fa67-890123456789" +} +``` + +### 12.6 通用图像编辑 + +**请求**： + +```bash +curl -X POST 'https://dashscope.aliyuncs.com/api/v1/services/aigc/image-edit/image-synthesis' \ + -H 'Authorization: Bearer sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' \ + -H 'Content-Type: application/json' \ + -d '{ + "model": "wanx2.5-image-edit", + "input": { + "prompt": "将图片中的猫咪替换为一只小狗", + "image_url": "https://example.com/cat_photo.jpg", + "mask_image_url": "https://example.com/cat_mask.png" + }, + "parameters": { + "strength": 0.8, + "n": 1, + "seed": 42 + } + }' +``` + +**响应**： + +```json +{ + "output": { + "task_status": "SUCCEEDED", + "results": [ + { + "url": "https://dashscope-result-bj.oss-cn-beijing.aliyuncs.com/xxx/edit_result.png?Expires=1745091600&OSSAccessKeyId=xxx&Signature=xxx" + } + ] + }, + "usage": { + "image_count": 1 + }, + "request_id": "a7b8c9d0-e1f2-3456-ab78-901234567890" +} +``` + +### 12.7 错误响应示例 + +#### 参数错误（400） + +```json +{ + "code": "InvalidParameter", + "message": "The value of parameter `size` is invalid. Supported values: 1024*1024, 720*1280, 1280*720, 960*1280, 1280*960", + "request_id": "b8c9d0e1-f2a3-4567-bc89-012345678901" +} +``` + +#### 认证失败（401） + +```json +{ + "code": "InvalidApiKey", + "message": "Invalid API-key provided.", + "request_id": "c9d0e1f2-a3b4-5678-cd90-123456789012" +} +``` + +#### 权限不足（403） + +```json +{ + "code": "Model.AccessDenied", + "message": "You are not authorized to use this model. Please activate the model in the Bailian console first.", + "request_id": "d0e1f2a3-b4c5-6789-de01-234567890123" +} +``` + +#### 模型不存在（404） + +```json +{ + "code": "ModelNotFound", + "message": "Model not found: wanx-v99", + "request_id": "e1f2a3b4-c5d6-7890-ef12-345678901234" +} +``` + +#### 请求限流（429） + +```json +{ + "code": "Throttling", + "message": "Request was throttled. Expected available in 5 seconds.", + "request_id": "f2a3b4c5-d6e7-8901-fa23-456789012345" +} +``` + +#### 服务器错误（500） + +```json +{ + "code": "InternalError", + "message": "An internal error has occurred. Please try again later.", + "request_id": "a3b4c5d6-e7f8-9012-ab34-567890123456" +} +``` + +#### 内容审核拦截（400） + +```json +{ + "output": { + "task_id": "b4c5d6e7-f8a9-0123-bc45-678901234567", + "task_status": "FAILED", + "code": "InvalidParameter", + "message": "Prompt content may contain inappropriate content." + }, + "request_id": "c5d6e7f8-a9b0-1234-cd56-789012345678" +} +``` + +--- + +## 13. OpenAI 兼容接口说明 + +### 13.1 概述 + +百炼平台提供 OpenAI 兼容接口，允许使用 OpenAI SDK 或兼容 OpenAI 格式的客户端直接调用通义万相图像生成模型。 + +### 13.2 兼容端点 + +| 方法 | 路径 | 功能 | +|------|------|------| +| POST | `/compatible-mode/v1/images/generations` | 文本生成图像（兼容 OpenAI Images API） | + +### 13.3 认证方式 + +与 OpenAI 相同的 Bearer Token 认证： + +```http +Authorization: Bearer sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx +Content-Type: application/json +``` + +### 13.4 请求参数映射 + +OpenAI 兼容模式的请求参数与 DashScope 原生参数映射关系： + +| OpenAI 参数 | DashScope 参数 | 说明 | +|-------------|---------------|------| +| `model` | `model` | 模型名称，直接映射 | +| `prompt` | `input.prompt` | 提示词 | +| `n` | `parameters.n` | 生成数量 | +| `size` | `parameters.size` | 图像尺寸（格式需转换） | +| `response_format` | — | `url` 或 `b64_json` | + +**尺寸格式差异**： + +| OpenAI 格式 | DashScope 格式 | +|-------------|---------------| +| `"1024x1024"` | `"1024*1024"` | +| `"1024x1792"` | — | +| `"1792x1024"` | — | + +注意：OpenAI 使用 `x` 分隔宽高，DashScope 使用 `*` 分隔宽高。 + +### 13.5 请求示例 + +```bash +curl -X POST 'https://dashscope.aliyuncs.com/compatible-mode/v1/images/generations' \ + -H 'Authorization: Bearer sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' \ + -H 'Content-Type: application/json' \ + -d '{ + "model": "wanx-v2", + "prompt": "一只可爱的橘猫坐在窗台上，阳光洒在身上", + "n": 1, + "size": "1024x1024" + }' +``` + +### 13.6 响应格式 + +OpenAI 兼容模式返回 OpenAI 标准响应格式： + +```json +{ + "created": 1745088000, + "data": [ + { + "url": "https://dashscope-result-bj.oss-cn-beijing.aliyuncs.com/xxx/image_0.png" + } + ] +} +``` + +当 `response_format` 为 `b64_json` 时： + +```json +{ + "created": 1745088000, + "data": [ + { + "b64_json": "iVBORw0KGgoAAAANSUhEUgAABAAAAAQACAIAAADwf7zUAAAABGdBTUEAALGPC..." + } + ] +} +``` + +### 13.7 OpenAI 兼容模式支持的模型 + +以下模型支持通过 OpenAI 兼容端点调用： + +| 模型名称 | 说明 | +|---------|------| +| `wanx-v1` | 万相文生图 V1 | +| `wanx-v2` | 万相文生图 V2 | +| `wanx2.1-t2i-ediff` | 万相图像生成与编辑 2.6 | +| `qwen-image-generation` | 千问文生图 | +| `zimage` | Z-Image 文生图 | +| `flux-dev` | FLUX Dev 文生图 | +| `flux-schnell` | FLUX Schnell 文生图 | + +--- + +## 14. team-api 适配器实现说明 + +### 14.1 协议转换要点 + +在 team-api 的 relay 层实现通义万相适配器时，需要注意以下关键转换： + +| 转换方向 | 转换内容 | +|---------|---------| +| 统一入口 → DashScope | OpenAI `/v1/images/generations` 格式转为 DashScope 原生格式 | +| DashScope → 统一出口 | DashScope 响应转为 OpenAI Images API 响应格式 | +| 尺寸格式 | `宽x高` → `宽*高`（OpenAI 格式转 DashScope 格式） | +| 响应结构 | `output.results[].url` → `data[].url` | + +### 14.2 异步任务处理 + +由于通义万相高分辨率图像生成耗时较长，适配器需要处理异步模式： + +1. **同步模式**：对于简单请求（低分辨率、单图），使用同步调用直接转发响应 +2. **异步模式**：对于耗时请求（高分辨率、多图），使用异步调用： + - 提交任务获取 task_id + - 内部轮询任务状态（对客户端透明） + - 任务完成后转换为 OpenAI 格式响应返回 + +### 14.3 错误格式转换 + +| DashScope 错误码 | OpenAI 错误类型 | HTTP 状态码 | +|-----------------|----------------|-------------| +| `InvalidApiKey` | `authentication_error` | 401 | +| `Model.AccessDenied` | `permission_error` | 403 | +| `InvalidParameter` | `invalid_request_error` | 400 | +| `ModelNotFound` | `invalid_request_error` | 404 | +| `Throttling` | `rate_limit_error` | 429 | +| `InternalError` | `server_error` | 500 | +| `ModelServingError` | `server_error` | 503 | + +### 14.4 适配器代码结构建议 + +在 `relay/channel/ali/` 目录下实现通义万相适配器： + +``` +relay/channel/ali/ +├── adaptor.go # Adaptor 接口实现 +├── image.go # 图像生成请求/响应转换 +├── image_types.go # DashScope 图像 API 类型定义 +└── async_task.go # 异步任务轮询处理 +``` + +### 14.5 渠道配置说明 + +| 配置项 | 说明 | 示例值 | +|--------|------|--------| +| Base URL | DashScope API 地址 | `https://dashscope.aliyuncs.com` | +| 模型映射 | 统一模型名 → DashScope 模型名 | `wanx-v2` → `wanx-v2` | +| 超时设置 | 同步调用建议 60s，异步提交建议 10s | — | +| 重试策略 | 500/503 错误指数退避重试，最多 3 次 | — | +| 默认模式 | 建议默认使用异步模式，避免同步超时 | `X-DashScope-Async: enable` | + +### 14.6 模型能力矩阵 + +| 模型 | 文生图 | 图生图 | 图像编辑 | 局部重绘 | 风格重绘 | 画面扩展 | +|------|--------|--------|---------|---------|---------|---------| +| `wanx-v2` | 是 | — | — | — | — | — | +| `wanx-v1` | 是 | — | — | — | — | — | +| `wanx2.1-t2i-ediff` | 是 | — | — | — | — | — | +| `wanx-sketch-to-image-v2` | — | 是 | — | — | — | — | +| `wanx-image-inpainting` | — | — | — | 是 | — | — | +| `wanx-style-repaint` | — | — | — | — | 是 | — | +| `wanx2.5-image-edit` | — | — | 是 | 是 | — | — | +| `wanx2.1-image-edit` | — | — | 是 | 是 | — | — | +| `wanx-image-outpainting` | — | — | — | — | — | 是 | +| `qwen-image-generation` | 是 | — | — | — | — | — | +| `qwen-image-edit` | — | — | 是 | — | — | — | + +### 14.7 计费相关说明 + +| 模型 | 计费单位 | 说明 | +|------|---------|------| +| `wanx-v1` | 每张 | 按生成图片数量计费 | +| `wanx-v2` | 每张 | 按生成图片数量计费 | +| `wanx-sketch-to-image-v2` | 每张 | 按生成图片数量计费 | +| `wanx-image-inpainting` | 每张 | 按生成图片数量计费 | +| `wanx-style-repaint` | 每张 | 按生成图片数量计费 | +| `wanx2.5-image-edit` | 每张 | 按生成图片数量计费 | + +在 team-api 的计费引擎中，需要将 DashScope 的 `usage.image_count` 字段转换为统一的 token 用量或次数计量，以便纳入平台的预扣-结算-退款流程。 + +--- + +## 附录 A：各端点完整 URL 汇总 + +| 功能 | 完整 URL | +|------|---------| +| 文生图 | `https://dashscope.aliyuncs.com/api/v1/services/aigc/text2image/image-synthesis` | +| 涂鸦作画 | `https://dashscope.aliyuncs.com/api/v1/services/aigc/image2image/image-synthesis` | +| 图像局部重绘 | `https://dashscope.aliyuncs.com/api/v1/services/aigc/image-inpainting/image-synthesis` | +| 人像风格重绘 | `https://dashscope.aliyuncs.com/api/v1/services/aigc/image-generation/image-synthesis` | +| 通用图像编辑 | `https://dashscope.aliyuncs.com/api/v1/services/aigc/image-edit/image-synthesis` | +| 图像画面扩展 | `https://dashscope.aliyuncs.com/api/v1/services/aigc/image-outpainting/image-synthesis` | +| 异步任务查询 | `https://dashscope.aliyuncs.com/api/v1/tasks/{task_id}` | +| OpenAI 兼容文生图 | `https://dashscope.aliyuncs.com/compatible-mode/v1/images/generations` | + +## 附录 B：常用 cURL 模板 + +### B.1 文生图同步模板 + +```bash +curl -X POST 'https://dashscope.aliyuncs.com/api/v1/services/aigc/text2image/image-synthesis' \ + -H 'Authorization: Bearer YOUR_API_KEY' \ + -H 'Content-Type: application/json' \ + -d '{ + "model": "wanx-v2", + "input": { + "prompt": "YOUR_PROMPT_HERE", + "negative_prompt": "" + }, + "parameters": { + "size": "1024*1024", + "n": 1, + "seed": 42, + "style": "" + } + }' +``` + +### B.2 文生图异步模板 + +```bash +curl -X POST 'https://dashscope.aliyuncs.com/api/v1/services/aigc/text2image/image-synthesis' \ + -H 'Authorization: Bearer YOUR_API_KEY' \ + -H 'Content-Type: application/json' \ + -H 'X-DashScope-Async: enable' \ + -d '{ + "model": "wanx-v2", + "input": { + "prompt": "YOUR_PROMPT_HERE" + }, + "parameters": { + "size": "1024*1024", + "n": 4 + } + }' +``` + +### B.3 查询异步任务模板 + +```bash +curl -X GET 'https://dashscope.aliyuncs.com/api/v1/tasks/YOUR_TASK_ID' \ + -H 'Authorization: Bearer YOUR_API_KEY' +``` + +### B.4 OpenAI 兼容模板 + +```bash +curl -X POST 'https://dashscope.aliyuncs.com/compatible-mode/v1/images/generations' \ + -H 'Authorization: Bearer YOUR_API_KEY' \ + -H 'Content-Type: application/json' \ + -d '{ + "model": "wanx-v2", + "prompt": "YOUR_PROMPT_HERE", + "n": 1, + "size": "1024x1024" + }' +```