Compare commits

...

28 Commits

Author SHA1 Message Date
b47be9dda4 main:新增健康检查支持和服务优化
- 在 Worker 中引入轻量级 HTTP 服务器,支持健康检查和就绪检查端点。
- 在 Kubernetes 和 Docker 配置中新增健康检查探针,提升服务稳定性。
- 更新依赖,引入 `aiohttp` 用于健康检查服务。
- 优化部署配置,调整 Redis 主机配置和镜像地址以适配新环境。
2026-02-04 12:00:30 +08:00
55419443cd main:新增健康检查支持和服务优化
- 在 Worker 中引入轻量级 HTTP 服务器,支持健康检查和就绪检查端点。
- 在 Kubernetes 和 Docker 配置中新增健康检查探针,提升服务稳定性。
- 更新依赖,引入 `aiohttp` 用于健康检查服务。
- 优化部署配置,调整 Redis 主机配置和镜像地址以适配新环境。
2026-02-04 11:58:56 +08:00
e0138d5531 main:新增阿里云 FC 部署文档及相关配置
- 更新 README,添加阿里云 FC 部署文档的链接。
- 新增 `docs/fc-deploy.md`,提供 FC 服务部署指南,包括环境准备与操作步骤。
- 优化文档表格格式,增加内容的可读性与完整性。
2026-02-04 11:36:01 +08:00
c92cac6ebb main:完善 Redis 密码配置支持
- 在函数计算配置文件中新增 `redis_password` 字段。
- 更新 API 和 Worker 环境变量以传递 Redis 密码。
- 提升服务安全性,支持连接受保护的 Redis 实例。
2026-02-04 11:24:29 +08:00
c76ece8f48 main:移除无效 Docker 镜像配置
- 从 `docker-compose.yml` 中删除无效的镜像配置,以简化服务环境设置。
2026-02-04 10:39:40 +08:00
d211074576 main:更新阿里云函数计算配置为 FC 3.0
变更内容:
- 重构函数计算配置文件,移除旧版 aliyun-fc.yaml,新增符合 FC 3.0 标准的 s.yaml。
- 引入 Serverless Devs 工具支持,添加部署、验证、日志查看等命令指引。
- 调整 API 和 Worker 函数配置,支持更灵活的资源分配及自动化管理。
- 更新文档,提供 FC 3.0 部署指南及优化建议。
2026-02-04 10:27:01 +08:00
a4d2ad1e93 main:采用异步 Redis 客户端优化指标管理模块
变更内容:
- 将 `redis` 客户端替换为 `redis.asyncio` 实现。
- 系统中同步方法调整为异步方法,提升事件循环效率。
- 在 `MetricsManager` 中添加异步初始化及关闭逻辑,避免阻塞问题。
- 更新便捷函数以支持异步上下文,并添加同步模式的兼容方法。
- 调整 Worker、JobManager、API 路由等模块,适配异步指标操作。
- 扩展单元测试,覆盖新增的异步方法及 Redis 操作逻辑。
- 简化 Dockerfile,取消开发依赖安装命令。
2026-02-03 19:54:22 +08:00
b5ca0e0593 Initial commit 2026-02-03 19:54:20 +08:00
7b627090f3 main:优化任务管理及队列监控性能
变更内容:
- 优化任务出队逻辑,采用 BLMOVE 提升队列操作的原子性和可靠性。
- 在 JobManager 中新增任务锁续租、超时任务回收、ACK/NACK 状态管理功能。
- 实现任务队列和死信队列监控指标收集,为系统性能分析提供数据支持。
- 扩展 Worker 模块,增加锁续租逻辑及任务回收调度。
- 更新测试用例,覆盖任务管理和队列指标的新增逻辑。
- 补充 metrics.yaml 文件,添加队列相关的监控指标定义。
- 更新依赖,补充 Redis 支持及相关库版本规范。
2026-02-03 18:18:02 +08:00
73bd66813c main:新增 Kubernetes 部署配置及文档
变更内容:
- 添加 Kubernetes 部署配置文件,包括 API Deployment、Worker Deployment 和 Redis Deployment。
- 新增 Service 定义,支持 API、Metrics 和 Redis 的集群访问。
- 配置 ConfigMap,用于全局共享环境变量。
- 编写 Kubernetes 部署指南文档,包含快速部署步骤、建议配置及故障排查方法。
- 提升系统的可扩展性和容器编排能力,适配生产环境使用。
2026-02-03 16:30:48 +08:00
6341cdf8ea main:新增 LICENSE 文件
变更内容:
- 添加 MIT License 文件,明确代码许可协议。
2026-02-03 16:06:37 +08:00
bf933b20f1 main:补充阿里FC平台兼容性说明
变更内容:
- 更新 README.md,添加阿里FC无法识别 `Platform:unknown/unknown` 问题的解决方法。
- 提供 `DOCKER_DEFAULT_PLATFORM` 和 `BUILDX_NO_DEFAULT_ATTESTATIONS` 环境变量设置及相关打包命令示例。
2026-02-03 15:48:07 +08:00
c3e16dcad3 main:更新 Docker 配置文件,添加镜像及平台支持
变更内容:
- 在 `Dockerfile` 中指定 `--platform=linux/amd64`,确保跨平台兼容性。
- 在 `docker-compose.yml` 中新增镜像配置及平台设置,贴合部署需求。
- 优化服务配置以匹配目标环境。
2026-02-03 15:44:42 +08:00
c0bd4760b1 main:更新 .env.example,翻译并补充配置项
变更内容:
- 将原配置项注释翻译为中文,提升可读性。
- 补充 Redis、异步任务、Worker 等相关配置项,为后续功能扩展做好准备。
- 调整文件结构和注释风格,规范化配置文件说明。
2026-02-03 15:27:05 +08:00
f2a164b82c main:新增 Worker 支持及任务管理优化
变更内容:
- 添加 Worker 进程模块,支持基于 Redis 的任务管理及分布式锁。
- 增加 `entrypoint.sh` 启动脚本,支持根据 `RUN_MODE` 自动运行 API 或 Worker。
- 优化 `docker-compose.yml` 配置,添加镜像及平台支持。
- 在 JobManager 中集成 `request_id` 上下文传递,改进日志追踪功能。
- 扩展单元测试,提升测试覆盖率。
2026-02-03 15:13:11 +08:00
bad3a34a82 main:支持 Worker 模式运行并优化任务管理
变更内容:
- 在 `Dockerfile` 和 `docker-compose.yml` 中添加 Worker 模式支持,包含运行模式 `RUN_MODE` 的配置。
- 更新 API 路由,改为将任务入队处理,并由 Worker 执行。
- 在 JobManager 中新增任务队列及分布式锁功能,支持任务的入队、出队、执行控制以及重试机制。
- 添加全局并发控制逻辑,避免任务超额运行。
- 扩展单元测试,覆盖任务队列、锁机制和并发控制的各类场景。
- 在 Serverless 配置中分别为 API 和 Worker 添加独立服务定义。

提升任务调度灵活性,增强系统可靠性与扩展性。
2026-02-03 13:29:32 +08:00
8ca2f64f7e main:移除 src 目录结构,更新模块引用路径
变更内容:
- 删除 `src` 子目录,将模块引用路径从 `src.functional_scaffold` 更新为 `functional_scaffold`。
- 修改相关代码、文档、测试用例及配置文件中的路径引用,包括 `README.md`、`Dockerfile`、`uvicorn` 启动命令等。
- 优化项目目录结构,提升代码维护性和可读性。
2026-02-03 11:29:37 +08:00
545616a5fe main:新增 AGENTS.md 文档
变更内容:
- 添加 AGENTS.md 文档,指导智能体开发与协作流程。
- 详细说明项目概述、技术架构、代码结构及关键设计约定。
- 提供常用命令与开发规范,明确算法开发与交付标准。
- 优化文档内容,方便团队阅读与高效协作。
2026-02-03 10:38:09 +08:00
9ffde9b842 main:新增 Docker 开发模式命令说明
变更内容:
- 在 `getting-started.md` 文档中新增 Docker 开发模式的命令和使用示例,提高开发环境配置便捷性。
2026-02-02 19:35:51 +08:00
baec3da7c1 main:新增 CLAUDE.md 验证脚本及日志收集文档
变更内容:
- 编写 `verify_claude_md.sh` 脚本,用于验证 CLAUDE.md 文档完整性及相关配置文件。
- 增加 Loki 日志收集系统的详细文档说明,包括架构、收集模式及查询方法。
- 更新 CLAUDE.md,完善日志追踪功能及 Loki 集成相关细节。
- 提升项目文档的完备度,方便日志分析与运维操作。
2026-02-02 19:25:08 +08:00
31d12ce4cb main:新增 Request ID 过滤文档及测试脚本
变更内容:
- 编写《Grafana 日志仪表板使用说明》,详细介绍 Request ID 过滤功能及使用方法。
- 新增 `test_request_id_filter.sh` 脚本,用于验证 Request ID 过滤功能的正确性。
- 提升文档完备性,方便使用该功能进行日志调试和性能分析。
2026-02-02 18:57:13 +08:00
fde946d3f0 main:更新 Grafana 仪表板查询条件
变更内容:
- 在日志仪表板查询中增加 `$request_id` 条件以优化日志筛选。
- 修改多处查询表达式,支持按请求 ID 过滤日志记录。
- 提升仪表板的可用性和查询精度。
2026-02-02 18:53:31 +08:00
3a6567ec4c main:删除 CI/CD 文档条目
更新内容:
- 从 README 中移除 CI/CD 功能条目,精简文档内容。
2026-02-02 18:41:06 +08:00
9e0ba8e74f main:删除 Grafana 仪表板配置文件
更新内容:
- 移除 `dashboard.json` 文件,清理不再需要的 Grafana 仪表板配置。
- 简化项目目录结构,删除多余的监控配置以优化维护。
2026-02-02 18:40:16 +08:00
8afff21fae main:新增并发控制文档及快速参考指南
更新内容:
- 编写《并发控制》详细文档,说明任务并发限制的配置、使用和最佳实践。
- 完成《并发控制实现总结》文档,记录设计决策和开发细节。
- 添加《并发控制快速参考》文档,提供配置和常见问题的快速解决方案。
2026-02-02 17:15:11 +08:00
9b6642635b main:新增中间件测试用例
变更内容:
- 为路径规范化函数添加单元测试,验证 /jobs 等路径的行为。
- 为指标中间件编写测试,包括健康检查端点跳过和普通端点的指标记录。
- 检查任务路径规范化逻辑并验证规范化后的路径是否正确。
2026-02-02 17:12:07 +08:00
87ed8c071c main:新增并发控制功能
变更内容:
- 增加 `max_concurrent_jobs` 配置项,支持设置最大并发任务数。
- 为 `JobManager` 添加信号量控制实现任务并发限制。
- 新增获取任务并发状态的接口 `/jobs/concurrency/status`。
- 编写并发控制功能相关的测试用
2026-02-02 17:11:52 +08:00
57b276d038 main:删除指标脚本并优化指标记录逻辑
变更内容:
- 删除 `start_metrics.sh` 脚本,精简项目结构,移除不再需要的启动逻辑。
- 优化 HTTP 请求指标记录,新增健康检查端点过滤和路径参数规范化功能。
- 更新文档,添加指标过滤及路径规范化的详细说明。
- 提高 Prometheus 指标的性能和可维护性,避免标签基数爆炸。
2026-02-02 15:53:00 +08:00
58 changed files with 6837 additions and 522 deletions

View File

@@ -1,32 +1,87 @@
# Environment Configuration
# Copy this file to .env and fill in your values
# Application
# =============================================================================
# 应用信息
# =============================================================================
APP_NAME=FunctionalScaffold
APP_VERSION=1.0.0
APP_ENV=development
# Server
# =============================================================================
# 服务器配置
# =============================================================================
HOST=0.0.0.0
PORT=8000
WORKERS=4
# Logging
# =============================================================================
# 日志配置
# =============================================================================
LOG_LEVEL=INFO
LOG_FORMAT=json
# 日志文件配置(可选,默认禁用)
LOG_FILE_ENABLED=false
LOG_FILE_PATH=/var/log/app/app.log
# Metrics
# =============================================================================
# 指标配置
# =============================================================================
METRICS_ENABLED=true
METRICS_CONFIG_PATH=config/metrics.yaml
# 指标实例 ID可选默认使用 hostname
# METRICS_INSTANCE_ID=my-instance
# Tracing
# =============================================================================
# 追踪配置
# =============================================================================
TRACING_ENABLED=false
JAEGER_ENDPOINT=http://localhost:14268/api/traces
# JAEGER_ENDPOINT=http://localhost:14268/api/traces
# External Services (examples)
# =============================================================================
# Redis 配置
# =============================================================================
REDIS_HOST=localhost
REDIS_PORT=6379
REDIS_DB=0
REDIS_PASSWORD=your_redis_password
# =============================================================================
# 异步任务配置
# =============================================================================
# 任务结果缓存时间(秒),默认 30 分钟
JOB_RESULT_TTL=1800
# Webhook 最大重试次数
WEBHOOK_MAX_RETRIES=3
# Webhook 超时时间(秒)
WEBHOOK_TIMEOUT=10
# 最大并发任务数
MAX_CONCURRENT_JOBS=10
# =============================================================================
# Worker 配置
# =============================================================================
# Worker 轮询间隔(秒)
WORKER_POLL_INTERVAL=1.0
# 任务队列 Redis Key
JOB_QUEUE_KEY=job:queue
# 全局并发计数器 Redis Key
JOB_CONCURRENCY_KEY=job:concurrency
# 任务锁 TTL
JOB_LOCK_TTL=300
# 任务最大重试次数
JOB_MAX_RETRIES=3
# 任务执行超时(秒)
JOB_EXECUTION_TIMEOUT=300
# =============================================================================
# 外部服务配置(示例)
# =============================================================================
# OSS 配置
# OSS_ENDPOINT=https://oss-cn-hangzhou.aliyuncs.com
# OSS_ACCESS_KEY_ID=your_access_key
# OSS_ACCESS_KEY_SECRET=your_secret_key
# OSS_BUCKET_NAME=your_bucket
# Database (if needed)
# DATABASE_URL=mysql://user:password@localhost:5432/dbname
# 数据库配置
# DATABASE_URL=mysql://user:password@localhost:3306/dbname

102
AGENTS.md Normal file
View File

@@ -0,0 +1,102 @@
# Agent.md
本文件为本仓库内各类智能体/助手提供工作指导,内容参考 `CLAUDE.md`,并针对日常开发与协作做了简化归纳。
## 项目概述
**FunctionalScaffold函数式脚手架** 是一个算法工程化 Serverless 解决方案的脚手架生成器。
- 为了方便团队交流,项目自然语言使用中文,包括代码注释和文档
- 核心目标:解决算力弹性、算法工程化门槛与后端集成复杂度问题
## 技术与架构
采用 **Docker 封装的 Serverless API 服务**方案:
- 算法代码 + 运行环境打包为 Docker 镜像
- 部署到云厂商 Serverless 平台实现自动扩缩容
- FastAPI 作为 HTTP 接口层
- 算法逻辑保持独立和专注
架构流程概览:
```
用户请求 → API网关 → 容器实例(冷/热启动)→ FastAPI → 算法程序 → 返回结果
外部服务OSS/数据库)
```
## 代码结构src layout
```
src/functional_scaffold/
├── algorithms/ # 算法层 - 所有算法必须继承 BaseAlgorithm
│ ├── base.py # execute() 包装器(埋点、错误处理)
│ └── prime_checker.py # 示例:质数判断算法
├── api/ # API 层 - FastAPI 路由和模型
│ ├── models.py # Pydantic 数据模型ConfigDict
│ ├── routes.py # 路由定义(/invoke, /healthz, /readyz, /jobs
│ └── dependencies.py # 依赖注入request_id 生成)
├── core/ # 核心功能 - 横切关注点
│ ├── errors.py # 异常类层次结构
│ ├── logging.py # 结构化日志JSON
│ ├── metrics.py # Prometheus 指标和装饰器
│ └── tracing.py # 分布式追踪ContextVar
├── utils/ # 工具函数
│ └── validators.py # 输入验证
├── config.py # 配置管理pydantic-settings
└── main.py # FastAPI 应用入口
```
## 关键设计约定
1. **算法抽象层**:所有算法继承 `BaseAlgorithm`,只实现 `process()``execute()` 负责埋点、日志和错误包装。
2. **依赖注入**FastAPI `Depends()` 注入 `request_id`,通过 `ContextVar` 透传。
3. **配置管理**`pydantic-settings` 读取环境变量或 `.env`,支持类型校验。
4. **可观测性**JSON 结构化日志、Prometheus 指标、Request ID 追踪。
5. **Pydantic V2**:使用 `ConfigDict``model_config`,不使用 `class Config`
## 常用命令
环境设置:
```bash
python -m venv venv
source venv/bin/activate
pip install -e ".[dev]"
```
运行服务:
```bash
./scripts/run_dev.sh
uvicorn functional_scaffold.main:app --reload --port 8000
```
测试与质量:
```bash
pytest tests/ -v
black src/ tests/
ruff check src/ tests/
```
## 添加新算法(简版步骤)
1.`src/functional_scaffold/algorithms/` 新建算法类,继承 `BaseAlgorithm` 并实现 `process()`
2.`algorithms/__init__.py` 导出新算法类。
3.`api/routes.py` 添加端点,在 `api/models.py` 添加请求/响应模型。
4.`tests/` 编写对应测试。
## 交付标准
必须包含以下组件与规范:
- `/invoke`, `/jobs`, `/healthz`, `/readyz`, `/metrics` 端点
- 统一的请求/响应 Schema 与错误格式
- 可观测性支持(日志、指标、追踪)
## 开发理念
算法同学只需关注 `process()` 的核心逻辑,其余基础设施能力由脚手架提供。

267
CLAUDE.md
View File

@@ -2,12 +2,12 @@
本文件为 Claude Code (claude.ai/code) 在此代码仓库中工作时提供指导。
为了方便团队交流,项目的自然语言使用中文,包括代码注释和文档等
## 项目概述
**FunctionalScaffold函数式脚手架** 是一个算法工程化 Serverless 解决方案的脚手架生成器。
- 为了方便团队交流,项目的自然语言使用中文,包括代码注释和文档等
### 核心目标
解决三大痛点:
@@ -65,9 +65,10 @@ src/functional_scaffold/
3. **配置管理**:使用 `pydantic-settings` 从环境变量或 `.env` 文件加载配置,支持类型验证。
4. **可观测性**
- 日志:结构化 JSON 日志pythonjsonlogger
- 日志:结构化 JSON 日志pythonjsonlogger,自动包含 request_id
- 指标Prometheus 格式request_counter, request_latency, algorithm_counter
- 追踪request_id 关联所有日志和指标
- 日志收集Loki + Promtail 自动收集和查询日志
## 开发命令
@@ -87,10 +88,10 @@ pip install -e ".[dev]"
./scripts/run_dev.sh
# 方式2直接运行开发模式自动重载
uvicorn src.functional_scaffold.main:app --reload --port 8000
uvicorn functional_scaffold.main:app --reload --port 8000
# 方式3生产模式
uvicorn src.functional_scaffold.main:app --host 0.0.0.0 --port 8000 --workers 4
uvicorn functional_scaffold.main:app --host 0.0.0.0 --port 8000 --workers 4
```
访问地址:
@@ -150,11 +151,12 @@ docker build -f deployment/Dockerfile -t functional-scaffold:latest .
# 运行容器
docker run -p 8000:8000 functional-scaffold:latest
# 使用 docker-compose包含 Prometheus + Grafana
# 使用 docker-compose包含 Prometheus + Grafana + Loki
cd deployment
docker-compose up
# Grafana: http://localhost:3000 (admin/admin)
# Prometheus: http://localhost:9090
# Loki: http://localhost:3100
```
### 文档
@@ -273,10 +275,52 @@ from functional_scaffold.core.logging import setup_logging
# 设置日志
logger = setup_logging(level="INFO", format_type="json")
# 记录日志
# 记录日志(自动包含 request_id
logger.info("处理请求", extra={"user_id": "123"})
```
**日志特性:**
- 结构化 JSON 格式
- 自动包含 request_id从 ContextVar 中提取)
- 支持文件日志(可选,通过环境变量启用)
- 日志轮转100MB保留 5 个备份)
### 日志收集Loki
项目集成了 Grafana Loki 日志收集系统,支持两种收集模式:
**模式 1: Docker stdio 收集(默认,推荐)**
- 自动收集容器标准输出/错误
- 无需修改应用代码
- 性能影响极小
**模式 2: 文件收集(备用)**
- 日志持久化到文件
- 支持日志轮转
- 需要设置 `LOG_FILE_ENABLED=true`
**查询日志:**
```bash
# 使用 Loki API
curl -G -s "http://localhost:3100/loki/api/v1/query_range" \
--data-urlencode 'query={job="functional-scaffold-app"}'
# 按 request_id 过滤
curl -G -s "http://localhost:3100/loki/api/v1/query_range" \
--data-urlencode 'query={job="functional-scaffold-app"} |= "request-id-here"'
```
**Grafana 仪表板:**
- 访问 http://localhost:3000
- 进入 "日志监控" 仪表板
- 使用 Request ID 输入框过滤特定请求的日志
**相关文档:**
- 完整文档:`docs/loki-integration.md`
- 使用说明:`docs/grafana-dashboard-usage.md`
- 快速参考:`docs/loki-quick-reference.md`
### 指标
使用 `core/metrics.py` 的装饰器:
@@ -298,7 +342,7 @@ def my_function():
### 追踪
Request ID 自动注入到所有请求:
Request ID 自动注入到所有请求和日志
```python
from functional_scaffold.core.tracing import get_request_id
@@ -307,6 +351,13 @@ from functional_scaffold.core.tracing import get_request_id
request_id = get_request_id()
```
**Request ID 特性:**
- 自动生成或从请求头 `X-Request-ID` 获取
- 通过 ContextVar 在异步上下文中传递
- 自动添加到所有日志记录中
- 可用于追踪单个请求的完整生命周期
- 在 Grafana 仪表板中可按 request_id 过滤日志
## 部署
### Kubernetes
@@ -321,10 +372,23 @@ kubectl apply -f deployment/kubernetes/service.yaml
- 资源限制256Mi-512Mi 内存250m-500m CPU
- 健康检查:存活探针 (/healthz),就绪探针 (/readyz)
### 阿里云函数计算
### 阿里云函数计算FC 3.0
```bash
fun deploy -t deployment/serverless/aliyun-fc.yaml
# 安装 Serverless Devs如未安装
npm install -g @serverless-devs/s
# 配置阿里云凭证(首次使用)
s config add
# 部署到阿里云函数计算
cd deployment/serverless && s deploy
# 验证配置语法
cd deployment/serverless && s plan
# 查看函数日志
cd deployment/serverless && s logs --tail
```
### AWS Lambda
@@ -355,7 +419,8 @@ sam deploy --template-file deployment/serverless/aws-lambda.yaml
- ✅ 参数校验Pydantic + utils/validators.py
- ✅ 错误包装和标准化core/errors.py
- ✅ 埋点core/metrics.py - 延迟、失败率)
- ✅ 分布式追踪的关联 IDcore/tracing.py
- ✅ 分布式追踪的关联 IDcore/tracing.py + RequestIdFilter
- ✅ 日志收集和查询Loki + Promtail
- ⏳ Worker 运行时重试、超时、DLQ - 待实现)
### 3. 脚手架生成器
@@ -365,8 +430,9 @@ sam deploy --template-file deployment/serverless/aws-lambda.yaml
- ✅ Dockerfiledeployment/Dockerfile
- ✅ CI/CD 流水线配置(.github/workflows/
- ✅ Serverless 平台部署 YAMLdeployment/serverless/
- ✅ Grafana 仪表板模板monitoring/grafana/dashboard.json
- ✅ Grafana 仪表板模板monitoring/grafana/dashboards/
- ✅ 告警规则配置monitoring/alerts/rules.yaml
- ✅ Loki 日志收集配置monitoring/loki.yaml, monitoring/promtail.yaml
## 开发理念
@@ -397,3 +463,180 @@ sam deploy --template-file deployment/serverless/aws-lambda.yaml
4. **Docker 构建**Dockerfile 使用非 root 用户appuser包含健康检查。
5. **配置优先级**:环境变量 > .env 文件 > 默认值。
6. **Promtail 版本**:使用 Promtail 3.0.0 或更高版本,以支持较新的 Docker API1.44+)。如果遇到 "client version too old" 错误,需要升级 Promtail 版本。
## 日志收集系统Loki
项目集成了 Grafana Loki 日志收集系统,提供强大的日志查询和分析能力。
### 架构
```
应用容器 (stdout/stderr)
Docker Engine
Promtail (日志采集器)
Loki (日志存储)
Grafana (可视化)
```
### 服务组件
**docker-compose 包含以下服务:**
- **app**: 应用服务(端口 8111
- **loki**: 日志存储服务(端口 3100
- **promtail**: 日志采集服务(端口 9080
- **grafana**: 可视化服务(端口 3000
- **prometheus**: 指标收集服务(端口 9090
- **redis**: 缓存服务(端口 6380
### 日志收集模式
#### 模式 1: Docker stdio 收集(默认)
**特点:**
- ✅ 无需修改应用代码
- ✅ 自动收集容器标准输出/错误
- ✅ 性能影响极小
- ✅ 推荐用于生产环境
**配置:**
应用容器需要添加标签(已配置):
```yaml
labels:
logging: "promtail"
logging_jobname: "functional-scaffold-app"
```
#### 模式 2: 文件收集(备用)
**特点:**
- ✅ 日志持久化到文件
- ✅ 支持日志轮转100MB5个备份
- ✅ 适合需要本地日志文件的场景
**启用方式:**
```yaml
# docker-compose.yml
environment:
- LOG_FILE_ENABLED=true
- LOG_FILE_PATH=/var/log/app/app.log
```
### 日志格式
所有日志使用 JSON 格式,自动包含以下字段:
- `asctime`: 时间戳
- `name`: 日志器名称
- `levelname`: 日志级别INFO, WARNING, ERROR
- `message`: 日志消息
- `request_id`: 请求 ID自动添加
- `timestamp`: ISO 格式时间戳
### 查询日志
#### 使用 Loki API
```bash
# 查询所有日志
curl -G -s "http://localhost:3100/loki/api/v1/query_range" \
--data-urlencode 'query={job="functional-scaffold-app"}'
# 按 request_id 过滤
curl -G -s "http://localhost:3100/loki/api/v1/query_range" \
--data-urlencode 'query={job="functional-scaffold-app"} |= "request-id-here"'
# 查询错误日志
curl -G -s "http://localhost:3100/loki/api/v1/query_range" \
--data-urlencode 'query={job="functional-scaffold-app", level="ERROR"}'
```
#### 使用 Grafana 仪表板
1. 访问 http://localhost:3000admin/admin
2. 进入 "日志监控" 仪表板
3. 使用 Request ID 输入框过滤特定请求的日志
**仪表板面板:**
- **日志流(实时)**: 实时日志流
- **日志量趋势**: 按时间和级别统计
- **日志级别分布**: INFO/WARNING/ERROR 分布
- **错误日志**: 只显示 ERROR 级别
#### 使用 Grafana Explore
1. 访问 http://localhost:3000/explore
2. 选择 Loki 数据源
3. 使用 LogQL 查询语言
**常用查询:**
```logql
# 查询所有日志
{job="functional-scaffold-app"}
# 查询错误日志
{job="functional-scaffold-app", level="ERROR"}
# 按 request_id 过滤
{job="functional-scaffold-app"} |= "request-id-here"
# 使用 JSON 解析
{job="functional-scaffold-app"} | json | request_id="request-id-here"
# 统计日志量
sum by (level) (count_over_time({job="functional-scaffold-app"}[5m]))
```
### 验证和测试
```bash
# 验证 Loki 集成
./scripts/verify_loki.sh
# 测试 Request ID 过滤
./scripts/test_request_id_filter.sh
```
### 配置文件
- **Loki 配置**: `monitoring/loki.yaml`
- 日志保留期: 7 天
- 摄入速率限制: 10MB/s
- 自动压缩和清理
- **Promtail 配置**: `monitoring/promtail.yaml`
- Docker stdio 收集配置
- 文件收集配置
- JSON 日志解析规则
- **Grafana Provisioning**: `monitoring/grafana/`
- 数据源自动配置datasources/
- 仪表板自动加载dashboards/
### 故障排查
**看不到日志:**
1. 检查服务状态: `docker-compose ps`
2. 查看 Promtail 日志: `docker-compose logs promtail`
3. 验证容器标签: `docker inspect <container> | grep Labels`
**Docker socket 权限问题:**
```bash
sudo chmod 666 /var/run/docker.sock
```
**日志延迟:**
- Promtail 每 5 秒刷新一次
- 建议等待 5-10 秒后再查询
### 相关文档
- **完整文档**: `docs/loki-integration.md` - 包含查询示例、故障排查、性能优化
- **快速参考**: `docs/loki-quick-reference.md` - 常用命令和 LogQL 查询
- **仪表板使用**: `docs/grafana-dashboard-usage.md` - Grafana 仪表板使用说明
- **实施总结**: `docs/loki-implementation-summary.md` - 架构和实施细节
- **监控目录**: `monitoring/README.md` - 配置文件说明

22
LICENSE Normal file
View File

@@ -0,0 +1,22 @@
MIT License
Copyright (c) 2026 Roog
Copyright (c) 2026 Guxinpei
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

View File

@@ -15,17 +15,19 @@
-**容器化部署** - Docker 和 Kubernetes 支持
-**Serverless 就绪** - 支持阿里云函数计算和 AWS Lambda
-**完整测试** - 单元测试和集成测试
-**CI/CD** - GitHub Actions 工作流
## 文档
| 文档 | 描述 |
|------|------|
|------------------------------------------------|--------------|
| [快速入门](docs/getting-started.md) | 10 分钟上手指南 |
| [算法开发指南](docs/algorithm-development.md) | 详细的算法开发教程 |
| [API 参考](docs/api-reference.md) | 完整的 API 文档 |
| [监控指南](docs/monitoring.md) | 监控和告警配置 |
| [API 规范](docs/api/README.md) | OpenAPI 规范说明 |
| [Kubernetes 部署](docs/kubernetes-deployment.md) | K8s 集群部署指南 |
| [日志集成(Loki)](docs/loki-quick-reference.md) | 日志收集部署说明 |
| [阿里云函数运算FC部署入门](docs/fc-deploy.md) | 阿里云FC部署入门 |
## 快速开始
@@ -58,7 +60,7 @@ pip install -e ".[dev]"
./scripts/run_dev.sh
# 方式2直接运行
uvicorn src.functional_scaffold.main:app --reload --port 8000
uvicorn functional_scaffold.main:app --reload --port 8000
```
4. 访问 API 文档
@@ -80,6 +82,12 @@ docker run -p 8000:8000 functional-scaffold:latest
# 或使用 docker-compose
cd deployment
docker-compose up
# 如果阿里FC无法识别 Platform:unknown/unknown 的情况时,请按下列执行打包:
export DOCKER_DEFAULT_PLATFORM=linux/amd64
export BUILDX_NO_DEFAULT_ATTESTATIONS=1
docker compose build
docker compose push
```
## API 端点

View File

@@ -94,6 +94,26 @@ custom_metrics:
type: counter
description: "Webhook 回调发送总数"
labels: [status]
# 队列监控指标
job_queue_length:
name: "job_queue_length"
type: gauge
description: "待处理任务队列长度"
labels: [queue]
job_oldest_waiting_seconds:
name: "job_oldest_waiting_seconds"
type: gauge
description: "最长任务等待时间(秒)"
labels: []
job_recovered_total:
name: "job_recovered_total"
type: counter
description: "回收的超时任务总数"
labels: []
prime_check_total:
name: "prime_check"
type: counter

View File

@@ -1,4 +1,4 @@
FROM python:3.11-slim
FROM --platform=linux/amd64 python:3.11-slim
WORKDIR /app
@@ -9,15 +9,21 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
# 复制依赖文件
COPY requirements.txt .
COPY requirements-dev.txt .
# 安装 Python 依赖
RUN pip install --no-cache-dir -r requirements.txt
RUN pip install --no-cache-dir -r requirements-dev.txt
# 安装dev依赖
#COPY requirements-dev.txt .
#RUN pip install --no-cache-dir -r requirements-dev.txt
# 复制应用代码和配置
COPY src/ ./src/
COPY config/ ./config/
COPY pyproject.toml .
# 安装包(使用 editable 模式)
RUN pip install --no-cache-dir -e .
# 创建非 root 用户
RUN useradd -m -u 1000 appuser && chown -R appuser:appuser /app
@@ -26,9 +32,15 @@ USER appuser
# 暴露端口
EXPOSE 8000
# 健康检查
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:8000/healthz')"
# 运行模式api默认或 worker
ENV RUN_MODE=api
# 启动命令
CMD ["uvicorn", "src.functional_scaffold.main:app", "--host", "0.0.0.0", "--port", "8000"]
# 健康检查(仅对 API 模式有效)
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
CMD if [ "$RUN_MODE" = "api" ]; then python -c "import urllib.request; urllib.request.urlopen('http://localhost:8000/healthz')"; else exit 0; fi
# 启动脚本
COPY --chown=appuser:appuser deployment/entrypoint.sh /app/entrypoint.sh
RUN chmod +x /app/entrypoint.sh
CMD ["/app/entrypoint.sh"]

View File

@@ -5,21 +5,30 @@ services:
build:
context: ..
dockerfile: deployment/Dockerfile
platform: linux/amd64
ports:
- "8111:8000"
environment:
- APP_ENV=development
- LOG_LEVEL=INFO
- METRICS_ENABLED=true
- RUN_MODE=api
# Redis 指标存储配置
- REDIS_HOST=redis
- REDIS_PORT=6379
- REDIS_DB=0
# 指标配置文件路径
- METRICS_CONFIG_PATH=config/metrics.yaml
# 日志文件配置
- LOG_FILE_ENABLED=false
- LOG_FILE_PATH=/var/log/app/app.log
volumes:
- ../src:/app/src
- ../config:/app/config
- app_logs:/var/log/app
labels:
logging: "promtail"
logging_jobname: "functional-scaffold-app"
restart: unless-stopped
depends_on:
redis:
@@ -31,6 +40,47 @@ services:
retries: 3
start_period: 5s
# Worker 服务 - 处理异步任务
worker:
build:
context: ..
dockerfile: deployment/Dockerfile
platform: linux/amd64
ports:
- "8112:8000"
environment:
- APP_ENV=development
- LOG_LEVEL=INFO
- METRICS_ENABLED=true
- RUN_MODE=worker
# Redis 配置
- REDIS_HOST=redis
- REDIS_PORT=6379
- REDIS_DB=0
# Worker 配置
- WORKER_POLL_INTERVAL=1.0
- MAX_CONCURRENT_JOBS=10
- JOB_MAX_RETRIES=3
- JOB_EXECUTION_TIMEOUT=300
volumes:
- ../src:/app/src
- ../config:/app/config
labels:
logging: "promtail"
logging_jobname: "functional-scaffold-worker"
restart: unless-stopped
depends_on:
redis:
condition: service_healthy
healthcheck:
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8000/healthz')"]
interval: 30s
timeout: 3s
retries: 3
start_period: 10s
deploy:
replicas: 2
# Redis - 用于集中式指标存储
redis:
image: redis:7-alpine
@@ -69,12 +119,47 @@ services:
- GF_SECURITY_ADMIN_PASSWORD=admin
volumes:
- grafana_data:/var/lib/grafana
- ../monitoring/grafana:/etc/grafana/provisioning
- ../monitoring/grafana/datasources:/etc/grafana/provisioning/datasources
- ../monitoring/grafana/dashboards:/etc/grafana/provisioning/dashboards
restart: unless-stopped
depends_on:
- prometheus
- loki
loki:
image: grafana/loki:2.9.3
ports:
- "3100:3100"
volumes:
- ../monitoring/loki.yaml:/etc/loki/local-config.yaml
- loki_data:/loki
command: -config.file=/etc/loki/local-config.yaml
restart: unless-stopped
healthcheck:
test: ["CMD", "wget", "--spider", "-q", "http://localhost:3100/ready"]
interval: 10s
timeout: 3s
retries: 3
promtail:
ports:
- "9080:9080"
image: grafana/promtail:3.0.0
volumes:
- ../monitoring/promtail.yaml:/etc/promtail/config.yml
# Docker stdio 收集
- /var/lib/docker/containers:/var/lib/docker/containers:ro
- /var/run/docker.sock:/var/run/docker.sock:ro
# Log 文件收集(备用)
- app_logs:/var/log/app:ro
command: -config.file=/etc/promtail/config.yml
restart: unless-stopped
depends_on:
- loki
volumes:
prometheus_data:
grafana_data:
redis_data:
loki_data:
app_logs:

12
deployment/entrypoint.sh Normal file
View File

@@ -0,0 +1,12 @@
#!/bin/bash
# 启动脚本:根据 RUN_MODE 环境变量选择启动 API 或 Worker
set -e
if [ "$RUN_MODE" = "worker" ]; then
echo "启动 Worker 模式..."
exec python -m functional_scaffold.worker
else
echo "启动 API 模式..."
exec uvicorn functional_scaffold.main:app --host 0.0.0.0 --port 8000
fi

View File

@@ -1,33 +1,70 @@
# Kubernetes 部署配置
# 包含ConfigMap、API Deployment、Worker Deployment、Redis Deployment
---
# ConfigMap - 共享配置
apiVersion: v1
kind: ConfigMap
metadata:
name: functional-scaffold-config
labels:
app: functional-scaffold
data:
APP_ENV: "production"
LOG_LEVEL: "INFO"
LOG_FORMAT: "json"
METRICS_ENABLED: "true"
# Redis 配置(指向集群内 Redis 服务)
REDIS_HOST: "functional-scaffold-redis"
REDIS_PORT: "6379"
REDIS_DB: "0"
# 异步任务配置
MAX_CONCURRENT_JOBS: "10"
JOB_RESULT_TTL: "1800"
WEBHOOK_MAX_RETRIES: "3"
WEBHOOK_TIMEOUT: "10"
# Worker 配置
WORKER_POLL_INTERVAL: "1.0"
JOB_QUEUE_KEY: "job:queue"
JOB_CONCURRENCY_KEY: "job:concurrency"
JOB_LOCK_TTL: "300"
JOB_MAX_RETRIES: "3"
JOB_EXECUTION_TIMEOUT: "300"
---
# API Deployment - HTTP 服务
apiVersion: apps/v1
kind: Deployment
metadata:
name: functional-scaffold
name: functional-scaffold-api
labels:
app: functional-scaffold
component: api
spec:
replicas: 3
selector:
matchLabels:
app: functional-scaffold
component: api
template:
metadata:
labels:
app: functional-scaffold
component: api
spec:
containers:
- name: functional-scaffold
- name: api
image: functional-scaffold:latest
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8000
name: http
env:
- name: APP_ENV
value: "production"
- name: LOG_LEVEL
value: "INFO"
- name: METRICS_ENABLED
value: "true"
- name: RUN_MODE
value: "api"
envFrom:
- configMapRef:
name: functional-scaffold-config
resources:
requests:
memory: "256Mi"
@@ -51,3 +88,125 @@ spec:
periodSeconds: 10
timeoutSeconds: 3
failureThreshold: 3
---
# Worker Deployment - 异步任务处理
apiVersion: apps/v1
kind: Deployment
metadata:
name: functional-scaffold-worker
labels:
app: functional-scaffold
component: worker
spec:
replicas: 2
selector:
matchLabels:
app: functional-scaffold
component: worker
template:
metadata:
labels:
app: functional-scaffold
component: worker
spec:
containers:
- name: worker
image: functional-scaffold:latest
imagePullPolicy: IfNotPresent
env:
- name: RUN_MODE
value: "worker"
envFrom:
- configMapRef:
name: functional-scaffold-config
resources:
requests:
memory: "256Mi"
cpu: "250m"
limits:
memory: "512Mi"
cpu: "500m"
# Worker 现在有 HTTP 健康检查端点
ports:
- containerPort: 8000
name: http
livenessProbe:
httpGet:
path: /healthz
port: 8000
initialDelaySeconds: 10
periodSeconds: 30
timeoutSeconds: 3
failureThreshold: 3
readinessProbe:
httpGet:
path: /readyz
port: 8000
initialDelaySeconds: 5
periodSeconds: 10
timeoutSeconds: 3
failureThreshold: 3
---
# Redis Deployment - 任务队列和状态存储
apiVersion: apps/v1
kind: Deployment
metadata:
name: functional-scaffold-redis
labels:
app: functional-scaffold
component: redis
spec:
replicas: 1
selector:
matchLabels:
app: functional-scaffold
component: redis
template:
metadata:
labels:
app: functional-scaffold
component: redis
spec:
containers:
- name: redis
image: redis:7-alpine
ports:
- containerPort: 6379
name: redis
command:
- redis-server
- --appendonly
- "yes"
resources:
requests:
memory: "128Mi"
cpu: "100m"
limits:
memory: "256Mi"
cpu: "200m"
livenessProbe:
exec:
command:
- redis-cli
- ping
initialDelaySeconds: 5
periodSeconds: 10
timeoutSeconds: 3
failureThreshold: 3
readinessProbe:
exec:
command:
- redis-cli
- ping
initialDelaySeconds: 5
periodSeconds: 5
timeoutSeconds: 3
failureThreshold: 3
volumeMounts:
- name: redis-data
mountPath: /data
volumes:
- name: redis-data
emptyDir: {}

View File

@@ -1,9 +1,15 @@
# Kubernetes Service 配置
# 包含API Service、Metrics Service、Redis Service
---
# API Service - 对外暴露 HTTP 服务
apiVersion: v1
kind: Service
metadata:
name: functional-scaffold
name: functional-scaffold-api
labels:
app: functional-scaffold
component: api
spec:
type: ClusterIP
ports:
@@ -13,13 +19,21 @@ spec:
name: http
selector:
app: functional-scaffold
component: api
---
# Metrics Service - Prometheus 抓取指标
apiVersion: v1
kind: Service
metadata:
name: functional-scaffold-metrics
labels:
app: functional-scaffold
component: api
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "8000"
prometheus.io/path: "/metrics"
spec:
type: ClusterIP
ports:
@@ -29,3 +43,24 @@ spec:
name: metrics
selector:
app: functional-scaffold
component: api
---
# Redis Service - 内部 Redis 服务
apiVersion: v1
kind: Service
metadata:
name: functional-scaffold-redis
labels:
app: functional-scaffold
component: redis
spec:
type: ClusterIP
ports:
- port: 6379
targetPort: 6379
protocol: TCP
name: redis
selector:
app: functional-scaffold
component: redis

View File

@@ -1,40 +0,0 @@
# 阿里云函数计算配置
ROSTemplateFormatVersion: '2015-09-01'
Transform: 'Aliyun::Serverless-2018-04-03'
Resources:
functional-scaffold:
Type: 'Aliyun::Serverless::Service'
Properties:
Description: '算法工程化 Serverless 脚手架'
LogConfig:
Project: functional-scaffold-logs
Logstore: function-logs
VpcConfig:
VpcId: 'vpc-xxxxx'
VSwitchIds:
- 'vsw-xxxxx'
SecurityGroupId: 'sg-xxxxx'
prime-checker:
Type: 'Aliyun::Serverless::Function'
Properties:
Description: '质数判断算法服务'
Runtime: custom-container
MemorySize: 512
Timeout: 60
InstanceConcurrency: 10
CAPort: 8000
CustomContainerConfig:
Image: 'registry.cn-hangzhou.aliyuncs.com/your-namespace/functional-scaffold:latest'
Command: '["uvicorn", "src.functional_scaffold.main:app", "--host", "0.0.0.0", "--port", "8000"]'
EnvironmentVariables:
APP_ENV: production
LOG_LEVEL: INFO
METRICS_ENABLED: 'true'
Events:
httpTrigger:
Type: HTTP
Properties:
AuthType: ANONYMOUS
Methods:
- GET
- POST

View File

@@ -0,0 +1,108 @@
# 阿里云函数计算 FC 3.0 配置
# 使用 Serverless Devs 部署: cd deployment/serverless && s deploy
edition: 3.0.0
name: functional-scaffold
access: default
vars:
region: cn-beijing
image: crpi-om2xd9y8cmaizszf-vpc.cn-beijing.personal.cr.aliyuncs.com/your-namespace/fc-test:test-v1
redis_host: 127.31.1.1
redis_port: "6379"
redis_password: "your-password"
resources:
# API 服务函数
prime-checker-api:
component: fc3
props:
region: ${vars.region}
functionName: prime-checker-api
description: 质数判断算法服务API
runtime: custom-container
cpu: 0.35
memorySize: 512
diskSize: 512
timeout: 60
instanceConcurrency: 10
handler: not-used
customContainerConfig:
image: ${vars.image}
port: 8000
command:
- /app/entrypoint.sh
healthCheckConfig:
httpGetUrl: /healthz
initialDelaySeconds: 3
periodSeconds: 5
timeoutSeconds: 3
failureThreshold: 3
successThreshold: 1
environmentVariables:
APP_ENV: production
LOG_LEVEL: INFO
METRICS_ENABLED: "true"
RUN_MODE: api
REDIS_HOST: ${vars.redis_host}
REDIS_PORT: ${vars.redis_port}
REDIS_PASSWORD: ${vars.redis_password}
vpcConfig: auto
logConfig: auto
triggers:
- triggerName: http-trigger
triggerType: http
triggerConfig:
authType: anonymous
methods:
- GET
- POST
- PUT
- DELETE
# 异步任务 Worker 函数
job-worker:
component: fc3
props:
region: ${vars.region}
functionName: job-worker
description: 异步任务 Worker
runtime: custom-container
cpu: 0.35
memorySize: 512
diskSize: 512
timeout: 900
instanceConcurrency: 1
handler: not-used
customContainerConfig:
image: ${vars.image}
port: 8000
command:
- /app/entrypoint.sh
healthCheckConfig:
httpGetUrl: /healthz
initialDelaySeconds: 5
periodSeconds: 10
timeoutSeconds: 3
failureThreshold: 3
successThreshold: 1
environmentVariables:
APP_ENV: production
LOG_LEVEL: INFO
METRICS_ENABLED: "true"
RUN_MODE: worker
REDIS_HOST: ${vars.redis_host}
REDIS_PORT: ${vars.redis_port}
REDIS_PASSWORD: ${vars.redis_password}
WORKER_POLL_INTERVAL: "1.0"
MAX_CONCURRENT_JOBS: "5"
JOB_MAX_RETRIES: "3"
JOB_EXECUTION_TIMEOUT: "300"
vpcConfig: auto
logConfig: auto
triggers:
- triggerName: timer-trigger
triggerType: timer
triggerConfig:
cronExpression: "0 */1 * * * *"
enable: true
payload: "{}"

View File

@@ -412,7 +412,7 @@ class MLPredictor(BaseAlgorithm):
```python
# tests/test_text_processor.py
import pytest
from src.functional_scaffold.algorithms.text_processor import TextProcessor
from functional_scaffold.algorithms.text_processor import TextProcessor
class TestTextProcessor:
"""文本处理算法测试"""

View File

@@ -0,0 +1,204 @@
# 异步任务并发控制实现总结
## 变更概述
为异步任务管理器添加了并发控制功能,使用 `asyncio.Semaphore` 限制同时运行的任务数量,防止系统资源耗尽。
## 修改的文件
### 1. `src/functional_scaffold/config.py`
**新增配置项:**
```python
max_concurrent_jobs: int = 10 # 最大并发任务数
```
### 2. `src/functional_scaffold/core/job_manager.py`
**新增属性:**
- `_semaphore: Optional[asyncio.Semaphore]` - 并发控制信号量
- `_max_concurrent_jobs: int` - 最大并发数(存储配置值)
**修改方法:**
- `__init__()` - 初始化 semaphore 和 max_concurrent_jobs 属性
- `initialize()` - 创建 Semaphore 实例
- `execute_job()` - 使用 `async with self._semaphore` 包裹执行逻辑
**新增方法:**
- `get_concurrency_status()` - 返回并发状态(最大并发数、可用槽位、运行中任务数)
### 3. `src/functional_scaffold/api/models.py`
**新增模型:**
```python
class ConcurrencyStatusResponse(BaseModel):
"""并发状态响应"""
max_concurrent: int
available_slots: int
running_jobs: int
```
### 4. `src/functional_scaffold/api/routes.py`
**新增端点:**
```python
GET /jobs/concurrency/status
```
返回当前并发执行状态。
### 5. `tests/test_job_manager.py`
**新增测试类:**
```python
class TestConcurrencyControl:
- test_get_concurrency_status()
- test_get_concurrency_status_without_semaphore()
- test_concurrency_limit()
- test_concurrency_status_api()
```
**修改测试:**
- `test_execute_job()` - 添加 semaphore 初始化
## 工作原理
### 并发控制流程
```
创建任务 (POST /jobs)
asyncio.create_task(execute_job)
检查 Redis 和 semaphore 可用性
async with self._semaphore: ← 获取槽位(阻塞直到有可用槽位)
├─ 更新状态为 running
├─ 执行算法
├─ 更新状态为 completed/failed
└─ 发送 webhook
自动释放槽位
```
### 关键设计决策
1. **使用 asyncio.Semaphore**
- 简单、高效、无需外部依赖
- 自动管理槽位获取和释放
- 支持异步等待
2. **在 execute_job 内部使用 semaphore**
- 快速失败的检查Redis 可用性、任务存在性)在 semaphore 外部
- 只有真正要执行的任务才占用槽位
- 任务完成后自动释放(即使发生异常)
3. **存储 _max_concurrent_jobs**
- Semaphore 不暴露最大值属性
- 需要单独存储以便 `get_concurrency_status()` 使用
## 测试覆盖
- ✅ 获取并发状态
- ✅ 未初始化时的并发状态
- ✅ 并发限制生效(创建超过限制的任务,验证只有限定数量在运行)
- ✅ API 端点测试
- ✅ 所有现有测试继续通过60/60
## 使用示例
### 配置并发限制
```bash
# 环境变量
export MAX_CONCURRENT_JOBS=20
# 或在 .env 文件
MAX_CONCURRENT_JOBS=20
```
### 查询并发状态
```bash
curl http://localhost:8000/jobs/concurrency/status
```
响应:
```json
{
"max_concurrent": 10,
"available_slots": 7,
"running_jobs": 3
}
```
### 测试并发控制
```bash
# 运行测试脚本
./scripts/test_concurrency.sh
```
## 性能影响
### 优点
1. **防止资源耗尽**:限制同时运行的任务数
2. **可预测的负载**:系统负载不会超过配置的限制
3. **自动排队**:超过限制的任务自动等待
4. **零开销**未达到限制时semaphore 几乎无性能开销
### 注意事项
1. **任务等待**:超过限制的任务会等待,可能导致响应延迟
2. **内存占用**:等待中的任务仍占用内存(协程对象)
3. **配置调优**:需要根据实际负载调整并发数
## 监控建议
### Prometheus 查询
```promql
# 任务创建速率
rate(jobs_created_total[5m])
# 任务完成速率
rate(jobs_completed_total[5m])
# 任务积压(创建 - 完成)
rate(jobs_created_total[5m]) - rate(jobs_completed_total[5m])
```
### Grafana 面板
建议添加以下面板:
1. 并发状态时间序列max_concurrent, available_slots, running_jobs
2. 任务创建/完成速率
3. 任务执行时间分布P50, P95, P99
## 未来改进
1. **任务超时机制**:为长时间运行的任务设置超时
2. **优先级队列**:支持高优先级任务优先执行
3. **动态调整**:根据系统负载动态调整并发数
4. **任务取消**:支持取消等待中或运行中的任务
5. **资源限制**:更细粒度的 CPU、内存限制
## 相关文档
- [并发控制详细文档](./concurrency-control.md)
- [异步任务接口实现计划](../plans/giggly-hatching-kite.md)
- [监控指南](./monitoring.md)
## 测试结果
```
======================== 60 passed, 7 warnings in 1.53s ========================
```
所有测试通过,包括 4 个新增的并发控制测试。

View File

@@ -0,0 +1,102 @@
# 并发控制快速参考
## 配置
```bash
# 设置最大并发数(默认 10
export MAX_CONCURRENT_JOBS=20
```
## API
### 查询并发状态
```bash
GET /jobs/concurrency/status
```
**响应:**
```json
{
"max_concurrent": 10, // 最大并发数
"available_slots": 7, // 可用槽位
"running_jobs": 3 // 运行中任务数
}
```
## 代码示例
### 在 JobManager 中使用
```python
# 并发控制自动生效,无需额外代码
job_manager = await get_job_manager()
job_id = await job_manager.create_job(...)
# 任务会自动排队,等待可用槽位
asyncio.create_task(job_manager.execute_job(job_id))
```
### 查询并发状态
```python
job_manager = await get_job_manager()
status = job_manager.get_concurrency_status()
print(f"运行中: {status['running_jobs']}/{status['max_concurrent']}")
print(f"可用槽位: {status['available_slots']}")
```
## 监控
### 实时监控
```bash
# 持续监控并发状态
watch -n 1 'curl -s http://localhost:8000/jobs/concurrency/status | jq'
```
### 测试脚本
```bash
# 运行并发控制测试
./scripts/test_concurrency.sh
```
## 推荐配置
| 任务类型 | 推荐并发数 |
|---------|-----------|
| CPU 密集型 | 核心数 × 1.5 |
| I/O 密集型 | 核心数 × 5-10 |
| 混合型 | 核心数 × 2-3 |
## 故障排查
### 任务一直 pending
```bash
# 检查并发状态
curl http://localhost:8000/jobs/concurrency/status
# 如果 available_slots = 0说明所有槽位被占用
# 解决方案:
# 1. 等待当前任务完成
# 2. 增加并发限制
# 3. 优化算法性能
```
### 系统资源耗尽
```bash
# 降低并发限制
export MAX_CONCURRENT_JOBS=5
# 重启服务
./scripts/run_dev.sh
```
## 相关文档
- [详细文档](./concurrency-control.md)
- [实现总结](./concurrency-control-changelog.md)

204
docs/concurrency-control.md Normal file
View File

@@ -0,0 +1,204 @@
# 异步任务并发控制
## 概述
为了防止系统资源耗尽和控制负载,任务管理器实现了基于 `asyncio.Semaphore` 的并发控制机制。
## 配置
`config.py` 或环境变量中设置最大并发任务数:
```python
# config.py
max_concurrent_jobs: int = 10 # 默认值
```
或通过环境变量:
```bash
export MAX_CONCURRENT_JOBS=20
```
## 工作原理
1. **信号量机制**:使用 `asyncio.Semaphore` 限制同时运行的任务数
2. **自动管理**:任务开始时获取槽位,完成后自动释放
3. **队列等待**:超过限制的任务会自动等待,直到有可用槽位
### 执行流程
```
POST /jobs 创建任务
asyncio.create_task(execute_job)
等待获取 semaphore 槽位
async with semaphore: ← 获取槽位
执行算法
更新状态
发送 webhook
自动释放槽位
```
## API 端点
### 查询并发状态
```bash
GET /jobs/concurrency/status
```
**响应示例:**
```json
{
"max_concurrent": 10,
"available_slots": 7,
"running_jobs": 3
}
```
**字段说明:**
- `max_concurrent`: 最大并发任务数(配置值)
- `available_slots`: 当前可用槽位数
- `running_jobs`: 当前正在运行的任务数
## 使用示例
### 1. 创建多个任务
```bash
# 创建 20 个任务
for i in {1..20}; do
curl -X POST http://localhost:8000/jobs \
-H "Content-Type: application/json" \
-d "{\"algorithm\": \"PrimeChecker\", \"params\": {\"number\": $i}}"
done
```
### 2. 监控并发状态
```bash
# 持续监控并发状态
watch -n 1 'curl -s http://localhost:8000/jobs/concurrency/status | jq'
```
输出示例:
```json
{
"max_concurrent": 10,
"available_slots": 0,
"running_jobs": 10
}
```
### 3. 调整并发限制
```bash
# 重启服务前设置环境变量
export MAX_CONCURRENT_JOBS=20
./scripts/run_dev.sh
```
## 性能考虑
### 选择合适的并发数
并发数应根据以下因素确定:
1. **CPU 核心数**CPU 密集型任务建议设置为核心数的 1-2 倍
2. **内存限制**:每个任务的内存占用 × 并发数 < 可用内存
3. **外部服务限制**如果调用外部 API考虑其速率限制
4. **Redis 连接池**确保 Redis 连接池大小 并发数
### 推荐配置
| 场景 | 推荐并发数 | 说明 |
|------|-----------|------|
| CPU 密集型如质数判断 | 核心数 × 1.5 | 充分利用 CPU |
| I/O 密集型如网络请求 | 核心数 × 5-10 | 等待 I/O 时可切换 |
| 混合型 | 核心数 × 2-3 | 平衡 CPU I/O |
| 内存受限 | 根据内存计算 | 避免 OOM |
### 示例计算
假设
- 服务器4 8GB 内存
- 任务类型I/O 密集型网络请求
- 单任务内存50MB
```
最大并发数 = min(
核心数 × 8 = 32,
可用内存 / 单任务内存 = 8000MB / 50MB = 160
) = 32
```
## 监控指标
相关 Prometheus 指标
```promql
# 任务创建速率
rate(jobs_created_total[5m])
# 任务完成速率
rate(jobs_completed_total[5m])
# 任务执行时间分布
histogram_quantile(0.95, job_execution_duration_seconds_bucket)
```
## 故障排查
### 问题:任务一直处于 pending 状态
**可能原因:**
1. 所有槽位都被占用
2. 某些任务执行时间过长
**解决方案:**
```bash
# 1. 检查并发状态
curl http://localhost:8000/jobs/concurrency/status
# 2. 如果 available_slots = 0说明所有槽位被占用
# 3. 检查是否有长时间运行的任务
# 4. 考虑增加并发限制或优化算法性能
```
### 问题:系统资源耗尽
**可能原因:**
并发数设置过高
**解决方案:**
```bash
# 降低并发限制
export MAX_CONCURRENT_JOBS=5
# 重启服务
```
## 最佳实践
1. **监控优先**部署后持续监控并发状态和系统资源
2. **逐步调整**从保守值开始逐步增加并发数
3. **压力测试**在生产环境前进行充分的压力测试
4. **设置告警** `available_slots = 0` 持续时间过长时告警
5. **任务超时**为长时间运行的任务设置超时机制待实现
## 未来改进
- [ ] 任务超时机制
- [ ] 优先级队列
- [ ] 动态调整并发数
- [ ] 任务取消功能
- [ ] 更细粒度的资源控制CPU内存限制

58
docs/fc-deploy.md Normal file
View File

@@ -0,0 +1,58 @@
# 阿里云 函数运算FC 部署入门
本指南帮助快速上手 FunctionalScaffold 脚手架,在 10 分钟内完成第一个算法服务的开发和部署。
## 环境准备
- 安装 [Serverless Devs CLI](https://serverless-devs.com/docs/overview)
1. 首先安装Node 环境在Node官网下载
- [Node.js 下载地址](https://nodejs.org/en/download/)
2. 安装 Serverless Devs CLI
```bash
npm install @serverless-devs/s -g
```
## 初始化 serverless dev cli 配置
执行以下命令初始化 serverless dev cli 配置
```bash
s config add
```
根据引导进行操作填入你的access key id 和 access key secret
## 部署算法服务
部署算法服务前,请确保已经完成环境准备和配置。
修改 `s.yaml` 文件中的 vars 部分
```yaml
# 阿里云函数计算 FC 3.0 配置
# 使用 Serverless Devs 部署: cd deployment/serverless && s deploy
edition: 3.0.0
name: functional-scaffold
access: default
vars:
region: cn-hangzhou # 换成你的区域
image: registry.cn-hangzhou.aliyuncs.com/your-namespace/functional-scaffold:latest # 换成你的docker 镜像
redis_host: r-xxxxx.redis.rds.aliyuncs.com # 换成你的redis连接
redis_port: "6379" # redis 端口号
redis_password: "your-password" #redis 密码,如果没有可留空
```
```bash
cd deployment && s deploy
```
部署完成后,可以在控制台查看服务的运行状态和日志。
## 删除算法服务
```bash
cd deployment && s remove
```

View File

@@ -34,7 +34,9 @@ pip install -e ".[dev]"
```bash
# 开发模式(自动重载)
uvicorn src.functional_scaffold.main:app --reload --port 8000
uvicorn functional_scaffold.main:app --reload --port 8000
# docker 开发者模式
cd deployment && docker compose up -d
```
### 3. 验证服务

View File

@@ -0,0 +1,182 @@
# Grafana 日志仪表板使用说明
## Request ID 过滤功能
日志监控仪表板现在支持按 request_id 过滤日志,可以追踪单个请求的完整生命周期。
### 如何使用
1. **访问仪表板**
- 打开 Grafana: http://localhost:3000
- 登录admin/admin
- 进入 "日志监控" 仪表板
2. **使用 Request ID 过滤**
- 在仪表板顶部找到 "Request ID" 输入框
- 输入完整的 request_id例如`59017bdd-5963-40b1-a325-5088593382c0`
- 所有面板会自动更新,只显示该 request_id 的日志
3. **查看所有日志**
- 清空 "Request ID" 输入框
- 所有面板会显示所有日志
### 示例
#### 获取 Request ID
从 API 响应中获取:
```bash
curl -X POST http://localhost:8111/invoke \
-H "Content-Type: application/json" \
-d '{"number": 17}' | jq -r '.request_id'
```
输出示例:
```
59017bdd-5963-40b1-a325-5088593382c0
```
#### 在仪表板中过滤
1. 复制上面的 request_id
2. 在 Grafana 仪表板顶部的 "Request ID" 输入框中粘贴
3. 按回车或点击刷新
#### 查看结果
过滤后,你会看到该请求的所有日志:
- **日志流面板**:显示该请求的所有日志条目
- **日志量趋势**:显示该请求的日志分布
- **日志级别分布**:显示该请求的日志级别统计
- **错误日志**:如果该请求有错误,会显示在这里
### 典型的请求日志流
一个成功的请求通常包含以下日志:
```
1. Request: POST /invoke
2. Processing request {request_id} with number=17
3. Starting algorithm: PrimeChecker
4. Algorithm PrimeChecker completed successfully in 0.001s
5. Response: 200
```
所有这些日志都有相同的 request_id可以通过过滤功能一起查看。
### 高级用法
#### 在 Explore 中使用
1. 进入 Grafana Explore: http://localhost:3000/explore
2. 选择 Loki 数据源
3. 使用以下查询:
```logql
# 查询特定 request_id
{job="functional-scaffold-app"} |= "59017bdd-5963-40b1-a325-5088593382c0"
# 使用 JSON 解析(更精确)
{job="functional-scaffold-app"} | json | request_id="59017bdd-5963-40b1-a325-5088593382c0"
# 查询特定 request_id 的错误日志
{job="functional-scaffold-app", level="ERROR"} |= "59017bdd-5963-40b1-a325-5088593382c0"
```
#### 组合过滤
可以结合其他过滤条件:
```logql
# 特定 request_id 的 ERROR 日志
{job="functional-scaffold-app", level="ERROR"} |= "59017bdd-5963-40b1-a325-5088593382c0"
# 特定 request_id 的特定 logger
{job="functional-scaffold-app", logger="functional_scaffold.algorithms.base"} |= "59017bdd-5963-40b1-a325-5088593382c0"
```
### 故障排查
#### Request ID 过滤不生效
1. **检查 request_id 格式**
- 确保输入的是完整的 UUID 格式
- 不要包含额外的空格或引号
2. **检查时间范围**
- 确保仪表板的时间范围包含该请求的时间
- 可以调整为 "Last 15 minutes" 或更长
3. **刷新仪表板**
- 点击右上角的刷新按钮
- 或者按 Ctrl+R (Cmd+R on Mac)
4. **验证日志是否存在**
- 在 Explore 中手动查询:
```logql
{job="functional-scaffold-app"} |= "your-request-id"
```
- 如果没有结果,说明日志还没有被收集
#### 日志延迟
- Promtail 每 5 秒刷新一次
- Loki 可能有几秒的延迟
- 建议等待 5-10 秒后再查询
### 最佳实践
1. **调试单个请求**
- 发送请求并记录 request_id
- 在仪表板中输入 request_id
- 查看完整的请求处理流程
2. **追踪错误**
- 当发现错误时,从错误日志中提取 request_id
- 使用 request_id 过滤查看完整的请求上下文
- 分析错误发生前后的日志
3. **性能分析**
- 使用 request_id 过滤慢请求
- 查看算法执行时间
- 分析性能瓶颈
4. **用户问题排查**
- 从用户报告中获取 request_id如果有
- 使用 request_id 重现问题场景
- 查看完整的请求处理过程
### 技术细节
#### 过滤实现
仪表板使用 LogQL 的文本匹配操作符 `|=`
```logql
{job="functional-scaffold-app"} |= "$request_id"
```
- 当 `$request_id` 为空时,`|= ""` 匹配所有日志
- 当 `$request_id` 有值时,只匹配包含该字符串的日志
#### 性能考虑
- 文本匹配 (`|=`) 比 JSON 解析更快
- 适合实时查询和仪表板
- 对于精确匹配,可以在 Explore 中使用 JSON 解析
#### 变量配置
Request ID 变量配置:
- 类型textbox文本输入框
- 名称request_id
- 标签Request ID
- 默认值:空字符串
## 相关文档
- [Loki 集成文档](loki-integration.md)
- [Loki 快速参考](loki-quick-reference.md)
- [LogQL 查询语言](https://grafana.com/docs/loki/latest/logql/)

View File

@@ -0,0 +1,307 @@
# Kubernetes 部署指南
本文档介绍如何在 Kubernetes 集群中部署 FunctionalScaffold 服务。
## 架构概览
```
┌─────────────────┐
│ Ingress/LB │
└────────┬────────┘
┌────────▼────────┐
│ API Service │
│ (ClusterIP) │
└────────┬────────┘
┌──────────────┼──────────────┐
│ │ │
┌──────▼──────┐ ┌─────▼─────┐ ┌─────▼─────┐
│ API Pod 1 │ │ API Pod 2 │ │ API Pod 3 │
└─────────────┘ └───────────┘ └───────────┘
┌────────▼────────┐
│ Redis Service │
└────────┬────────┘
┌──────────────┼──────────────┐
│ │ │
┌──────▼──────┐ ┌─────▼─────┐ │
│ Worker Pod 1│ │Worker Pod2│ │
└─────────────┘ └───────────┘ │
┌──────▼──────┐
│ Redis Pod │
└─────────────┘
```
## 组件说明
| 组件 | 副本数 | 说明 |
|------|--------|------|
| **API Deployment** | 3 | HTTP 服务,处理同步请求和任务创建 |
| **Worker Deployment** | 2 | 异步任务处理,从 Redis 队列消费任务 |
| **Redis Deployment** | 1 | 任务队列和状态存储 |
| **ConfigMap** | - | 共享配置管理 |
## 快速部署
```bash
# 部署所有资源
kubectl apply -f deployment/kubernetes/deployment.yaml
kubectl apply -f deployment/kubernetes/service.yaml
# 查看部署状态
kubectl get pods -l app=functional-scaffold
kubectl get svc -l app=functional-scaffold
```
## 配置文件说明
### deployment.yaml
包含以下资源:
#### ConfigMap
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: functional-scaffold-config
data:
APP_ENV: "production"
LOG_LEVEL: "INFO"
REDIS_HOST: "functional-scaffold-redis"
# ... 更多配置
```
主要配置项:
| 配置项 | 默认值 | 说明 |
|--------|--------|------|
| `APP_ENV` | production | 运行环境 |
| `LOG_LEVEL` | INFO | 日志级别 |
| `REDIS_HOST` | functional-scaffold-redis | Redis 服务地址 |
| `MAX_CONCURRENT_JOBS` | 10 | 最大并发任务数 |
| `JOB_EXECUTION_TIMEOUT` | 300 | 任务执行超时(秒) |
#### API Deployment
- **副本数**: 3
- **资源限制**: 256Mi-512Mi 内存250m-500m CPU
- **健康检查**: `/healthz`(存活)、`/readyz`(就绪)
- **环境变量**: `RUN_MODE=api`
#### Worker Deployment
- **副本数**: 2
- **资源限制**: 256Mi-512Mi 内存250m-500m CPU
- **健康检查**: exec 探针检查 Redis 连接
- **环境变量**: `RUN_MODE=worker`
#### Redis Deployment
- **副本数**: 1
- **资源限制**: 128Mi-256Mi 内存100m-200m CPU
- **持久化**: AOF 模式appendonly yes
- **存储**: emptyDir开发环境
### service.yaml
| Service | 类型 | 端口 | 说明 |
|---------|------|------|------|
| `functional-scaffold-api` | ClusterIP | 80 → 8000 | API 服务 |
| `functional-scaffold-metrics` | ClusterIP | 8000 | Prometheus 指标 |
| `functional-scaffold-redis` | ClusterIP | 6379 | Redis 服务 |
## 生产环境建议
### 1. 使用外部 Redis
生产环境建议使用托管 Redis 服务(如阿里云 Redis、AWS ElastiCache
```yaml
# 修改 ConfigMap
data:
REDIS_HOST: "r-xxxxx.redis.rds.aliyuncs.com"
REDIS_PORT: "6379"
REDIS_PASSWORD: "" # 使用 Secret 管理
```
### 2. 使用 Secret 管理敏感信息
```yaml
apiVersion: v1
kind: Secret
metadata:
name: functional-scaffold-secrets
type: Opaque
stringData:
REDIS_PASSWORD: "your-password"
DATABASE_URL: "postgresql://..."
```
在 Deployment 中引用:
```yaml
envFrom:
- configMapRef:
name: functional-scaffold-config
- secretRef:
name: functional-scaffold-secrets
```
### 3. 配置 HPA 自动扩缩容
```yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: functional-scaffold-api-hpa
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: functional-scaffold-api
minReplicas: 2
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70
```
### 4. 配置 PDB 保证可用性
```yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: functional-scaffold-api-pdb
spec:
minAvailable: 2
selector:
matchLabels:
app: functional-scaffold
component: api
```
### 5. 使用 PVC 持久化 Redis 数据
```yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: redis-data-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
```
## 监控集成
### Prometheus 抓取配置
`functional-scaffold-metrics` Service 已添加 Prometheus 注解:
```yaml
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "8000"
prometheus.io/path: "/metrics"
```
### ServiceMonitor如使用 Prometheus Operator
```yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: functional-scaffold
spec:
selector:
matchLabels:
app: functional-scaffold
component: api
endpoints:
- port: metrics
path: /metrics
interval: 30s
```
## 常用命令
```bash
# 查看所有资源
kubectl get all -l app=functional-scaffold
# 查看 Pod 日志
kubectl logs -l app=functional-scaffold,component=api -f
kubectl logs -l app=functional-scaffold,component=worker -f
# 扩缩容
kubectl scale deployment functional-scaffold-api --replicas=5
kubectl scale deployment functional-scaffold-worker --replicas=3
# 滚动更新
kubectl set image deployment/functional-scaffold-api \
api=functional-scaffold:v2.0.0
# 回滚
kubectl rollout undo deployment/functional-scaffold-api
# 查看部署历史
kubectl rollout history deployment/functional-scaffold-api
# 进入 Pod 调试
kubectl exec -it <pod-name> -- /bin/sh
# 端口转发(本地调试)
kubectl port-forward svc/functional-scaffold-api 8000:80
```
## 故障排查
### Pod 启动失败
```bash
# 查看 Pod 事件
kubectl describe pod <pod-name>
# 查看 Pod 日志
kubectl logs <pod-name> --previous
```
### Redis 连接失败
```bash
# 检查 Redis Service
kubectl get svc functional-scaffold-redis
# 测试 Redis 连接
kubectl run redis-test --rm -it --image=redis:7-alpine -- \
redis-cli -h functional-scaffold-redis ping
```
### Worker 不消费任务
```bash
# 检查 Worker 日志
kubectl logs -l component=worker -f
# 检查 Redis 队列
kubectl exec -it <redis-pod> -- redis-cli LLEN job:queue
```
## 相关文档
- [快速入门](getting-started.md)
- [监控指南](monitoring.md)
- [并发控制](concurrency-control.md)
- [日志集成](loki-quick-reference.md)

564
docs/loki-integration.md Normal file
View File

@@ -0,0 +1,564 @@
# Loki 日志收集系统集成文档
## 概述
本项目已集成 Grafana Loki 日志收集系统,支持两种日志收集模式:
1. **Docker stdio 收集**(推荐)- 从容器标准输出/错误收集日志
2. **Log 文件收集**(备用)- 从日志文件收集日志
## 架构
```
应用容器 (stdout/stderr)
Docker Engine
Promtail (日志采集器)
Loki (日志存储)
Grafana (可视化)
```
## 快速开始
### 1. 启动服务
```bash
cd deployment
docker-compose up -d
```
这将启动以下服务:
- **app**: 应用服务 (端口 8111)
- **loki**: 日志存储服务 (端口 3100)
- **promtail**: 日志采集服务 (端口 9080)
- **grafana**: 可视化服务 (端口 3000)
- **prometheus**: 指标收集服务 (端口 9090)
- **redis**: 缓存服务 (端口 6380)
### 2. 访问 Grafana
1. 打开浏览器访问 http://localhost:3000
2. 使用默认凭据登录:
- 用户名: `admin`
- 密码: `admin`
3. 首次登录后建议修改密码
### 3. 查看日志
#### 方式 1: 使用预配置的日志仪表板
1. 在 Grafana 左侧菜单点击 **Dashboards**
2. 选择 **日志监控** 仪表板
3. 查看以下面板:
- **日志流 (实时)**: 实时日志流
- **日志量趋势(按级别)**: 时间序列图表
- **日志级别分布**: 按级别统计
- **错误日志**: 只显示 ERROR 级别日志
#### 方式 2: 使用 Explore 功能
1. 在 Grafana 左侧菜单点击 **Explore** (指南针图标)
2. 选择 **Loki** 数据源
3. 输入 LogQL 查询语句(见下文)
## LogQL 查询示例
### 基础查询
```logql
# 查询所有应用日志
{job="functional-scaffold-app"}
# 查询特定级别的日志
{job="functional-scaffold-app", level="ERROR"}
{job="functional-scaffold-app", level="INFO"}
# 查询特定容器的日志
{container="functional-scaffold-app-1"}
```
### 文本过滤
```logql
# 包含特定文本
{job="functional-scaffold-app"} |= "request_id"
# 不包含特定文本
{job="functional-scaffold-app"} != "healthz"
# 正则表达式匹配
{job="functional-scaffold-app"} |~ "error|exception"
# 正则表达式不匹配
{job="functional-scaffold-app"} !~ "debug|trace"
```
### JSON 字段提取
```logql
# 提取 request_id 字段
{job="functional-scaffold-app"} | json | request_id != ""
# 提取并过滤特定 request_id
{job="functional-scaffold-app"} | json | request_id = "abc123"
# 提取 logger 字段
{job="functional-scaffold-app"} | json | logger = "functional_scaffold.api.routes"
```
### 聚合查询
```logql
# 统计日志数量
count_over_time({job="functional-scaffold-app"}[5m])
# 按级别统计
sum by (level) (count_over_time({job="functional-scaffold-app"}[5m]))
# 计算错误率
sum(rate({job="functional-scaffold-app", level="ERROR"}[5m]))
/
sum(rate({job="functional-scaffold-app"}[5m]))
```
## 日志收集模式
### 模式 1: Docker stdio 收集(默认,推荐)
**特点:**
- 无需修改应用代码
- 自动收集容器标准输出/错误
- 性能影响极小
- 配置简单
**工作原理:**
1. 应用将日志输出到 stdout/stderr
2. Docker Engine 捕获日志
3. Promtail 通过 Docker API 读取日志
4. 日志发送到 Loki 存储
**配置:**
- 应用容器需要添加标签:
```yaml
labels:
logging: "promtail"
logging_jobname: "functional-scaffold-app"
```
### 模式 2: Log 文件收集(备用)
**特点:**
- 日志持久化到文件
- 支持日志轮转
- 适合需要本地日志文件的场景
**启用方式:**
1. 修改 `deployment/docker-compose.yml`
```yaml
environment:
- LOG_FILE_ENABLED=true
- LOG_FILE_PATH=/var/log/app/app.log
```
2. 重启服务:
```bash
docker-compose up -d app
```
**日志文件配置:**
- 最大文件大小: 100MB
- 保留备份数: 5 个
- 总存储空间: 最多 500MB
## 配置说明
### Loki 配置 (monitoring/loki.yaml)
```yaml
limits_config:
retention_period: 168h # 日志保留 7 天
ingestion_rate_mb: 10 # 摄入速率限制 10MB/s
ingestion_burst_size_mb: 20 # 突发大小 20MB
```
**可调整参数:**
- `retention_period`: 日志保留时间(默认 7 天)
- `ingestion_rate_mb`: 每秒摄入速率限制
- `ingestion_burst_size_mb`: 突发流量大小
### Promtail 配置 (monitoring/promtail.yaml)
**Docker stdio 收集配置:**
```yaml
scrape_configs:
- job_name: docker
docker_sd_configs:
- host: unix:///var/run/docker.sock
filters:
- name: label
values: ["logging=promtail"]
```
**文件收集配置:**
```yaml
scrape_configs:
- job_name: app_files
static_configs:
- targets:
- localhost
labels:
job: functional-scaffold-app-files
__path__: /var/log/app/*.log
```
## 验证和测试
### 1. 检查服务状态
```bash
# 查看所有服务
docker-compose ps
# 检查 Loki 健康状态
curl http://localhost:3100/ready
# 检查 Promtail 健康状态
curl http://localhost:9080/ready
```
### 2. 生成测试日志
```bash
# 发送测试请求
curl -X POST http://localhost:8111/invoke \
-H "Content-Type: application/json" \
-d '{"algorithm": "PrimeChecker", "params": {"number": 17}}'
```
### 3. 查询日志
```bash
# 使用 Loki API 查询
curl -G -s "http://localhost:3100/loki/api/v1/query_range" \
--data-urlencode 'query={job="functional-scaffold-app"}' \
--data-urlencode 'limit=10' \
| jq '.data.result'
```
### 4. 在 Grafana 中验证
1. 访问 http://localhost:3000/explore
2. 选择 Loki 数据源
3. 输入查询: `{job="functional-scaffold-app"}`
4. 应该能看到应用日志
## 故障排查
### 问题 1: 看不到日志
**检查步骤:**
1. 确认 Promtail 正在运行:
```bash
docker-compose ps promtail
```
2. 检查 Promtail 日志:
```bash
docker-compose logs promtail
```
3. 确认应用容器有正确的标签:
```bash
docker inspect functional-scaffold-app-1 | grep -A 5 Labels
```
4. 检查 Loki 是否接收到日志:
```bash
curl -G -s "http://localhost:3100/loki/api/v1/label/job/values" | jq
```
### 问题 2: Promtail 无法访问 Docker socket
**错误信息:**
```
permission denied while trying to connect to the Docker daemon socket
```
**解决方案:**
在 macOS/Linux 上,确保 Docker socket 权限正确:
```bash
sudo chmod 666 /var/run/docker.sock
```
或者将 Promtail 容器添加到 docker 组Linux
```yaml
promtail:
user: root
group_add:
- docker
```
### 问题 3: 日志量过大
**症状:**
- Loki 响应缓慢
- 磁盘空间不足
**解决方案:**
1. 调整日志保留期:
```yaml
# monitoring/loki.yaml
limits_config:
retention_period: 72h # 改为 3 天
```
2. 增加摄入速率限制:
```yaml
limits_config:
ingestion_rate_mb: 5 # 降低到 5MB/s
```
3. 添加日志过滤:
```yaml
# monitoring/promtail.yaml
pipeline_stages:
- match:
selector: '{job="functional-scaffold-app"}'
stages:
- drop:
expression: ".*healthz.*" # 丢弃健康检查日志
```
### 问题 4: 文件模式下看不到日志
**检查步骤:**
1. 确认文件日志已启用:
```bash
docker-compose exec app env | grep LOG_FILE
```
2. 检查日志文件是否存在:
```bash
docker-compose exec app ls -lh /var/log/app/
```
3. 检查 Promtail 是否能访问日志文件:
```bash
docker-compose exec promtail ls -lh /var/log/app/
```
## 性能优化
### 1. 减少日志量
**在应用层面:**
- 调整日志级别为 WARNING 或 ERROR
- 过滤掉不必要的日志(如健康检查)
```yaml
# docker-compose.yml
environment:
- LOG_LEVEL=WARNING
```
**在 Promtail 层面:**
```yaml
# monitoring/promtail.yaml
pipeline_stages:
- drop:
expression: ".*healthz.*"
drop_counter_reason: "healthcheck"
```
### 2. 优化查询性能
**使用标签过滤:**
```logql
# 好:使用标签过滤(快)
{job="functional-scaffold-app", level="ERROR"}
# 差:使用文本过滤(慢)
{job="functional-scaffold-app"} |= "ERROR"
```
**限制时间范围:**
```logql
# 查询最近 5 分钟
{job="functional-scaffold-app"}[5m]
# 避免查询过长时间范围
{job="functional-scaffold-app"}[7d] # 慢
```
### 3. 存储优化
**定期清理旧数据:**
```bash
# Loki 会自动根据 retention_period 清理
# 也可以手动清理
docker-compose exec loki rm -rf /loki/chunks/*
```
**监控磁盘使用:**
```bash
docker-compose exec loki du -sh /loki/chunks
```
## 高级功能
### 1. 告警规则
在 Loki 中配置告警规则(需要 Loki Ruler
```yaml
# monitoring/loki-rules.yaml
groups:
- name: error_alerts
interval: 1m
rules:
- alert: HighErrorRate
expr: |
sum(rate({job="functional-scaffold-app", level="ERROR"}[5m]))
/
sum(rate({job="functional-scaffold-app"}[5m]))
> 0.05
for: 5m
labels:
severity: warning
annotations:
summary: "错误率过高"
description: "应用错误率超过 5%"
```
### 2. 日志导出
**导出为 JSON**
```bash
curl -G -s "http://localhost:3100/loki/api/v1/query_range" \
--data-urlencode 'query={job="functional-scaffold-app"}' \
--data-urlencode 'start=2024-01-01T00:00:00Z' \
--data-urlencode 'end=2024-01-02T00:00:00Z' \
| jq '.data.result' > logs.json
```
**导出为文本:**
```bash
curl -G -s "http://localhost:3100/loki/api/v1/query_range" \
--data-urlencode 'query={job="functional-scaffold-app"}' \
| jq -r '.data.result[].values[][1]' > logs.txt
```
### 3. 与 Prometheus 集成
在 Grafana 仪表板中同时显示日志和指标:
```json
{
"panels": [
{
"title": "错误率和错误日志",
"targets": [
{
"datasource": "Prometheus",
"expr": "rate(http_requests_total{status=\"error\"}[5m])"
},
{
"datasource": "Loki",
"expr": "{job=\"functional-scaffold-app\", level=\"ERROR\"}"
}
]
}
]
}
```
## 最佳实践
### 1. 日志格式
**使用结构化日志JSON**
```python
logger.info("处理请求", extra={
"request_id": "abc123",
"user_id": "user456",
"duration": 0.123
})
```
**输出:**
```json
{
"asctime": "2024-01-01 12:00:00,000",
"name": "functional_scaffold.api.routes",
"levelname": "INFO",
"message": "处理请求",
"request_id": "abc123",
"user_id": "user456",
"duration": 0.123
}
```
### 2. 标签策略
**好的标签:**
- 低基数(值的种类少)
- 用于过滤和分组
- 例如:`level`, `logger`, `container`
**不好的标签:**
- 高基数(值的种类多)
- 例如:`request_id`, `user_id`, `timestamp`
**正确做法:**
```logql
# 使用标签过滤
{job="functional-scaffold-app", level="ERROR"}
# 使用 JSON 提取高基数字段
{job="functional-scaffold-app"} | json | request_id = "abc123"
```
### 3. 查询优化
**使用时间范围:**
```logql
{job="functional-scaffold-app"}[5m] # 最近 5 分钟
```
**限制返回行数:**
```logql
{job="functional-scaffold-app"} | limit 100
```
**使用聚合减少数据量:**
```logql
sum by (level) (count_over_time({job="functional-scaffold-app"}[5m]))
```
## 参考资料
- [Loki 官方文档](https://grafana.com/docs/loki/latest/)
- [LogQL 查询语言](https://grafana.com/docs/loki/latest/logql/)
- [Promtail 配置](https://grafana.com/docs/loki/latest/clients/promtail/configuration/)
- [Grafana Explore](https://grafana.com/docs/grafana/latest/explore/)
## 总结
本项目的 Loki 集成提供了:
**开箱即用** - 无需额外配置即可收集日志
**双模式支持** - Docker stdio默认和文件收集
**自动化配置** - 数据源和仪表板自动加载
**结构化日志** - JSON 格式,支持字段提取
**高性能** - 低资源占用,快速查询
**易于扩展** - 支持自定义标签和过滤规则
如有问题,请参考故障排查章节或查阅官方文档。

View File

@@ -0,0 +1,237 @@
# Loki 快速参考
## 常用命令
### 服务管理
```bash
# 启动所有服务
cd deployment && docker-compose up -d
# 查看服务状态
docker-compose ps
# 查看日志
docker-compose logs -f loki
docker-compose logs -f promtail
# 重启服务
docker-compose restart loki promtail
# 停止服务
docker-compose down
```
### 健康检查
```bash
# Loki
curl http://localhost:3100/ready
# Promtail
curl http://localhost:9080/ready
# 验证脚本
./scripts/verify_loki.sh
```
## 常用 LogQL 查询
### 基础查询
```logql
# 所有日志
{job="functional-scaffold-app"}
# 错误日志
{job="functional-scaffold-app", level="ERROR"}
# 特定时间范围
{job="functional-scaffold-app"}[5m]
```
### 文本过滤
```logql
# 包含文本
{job="functional-scaffold-app"} |= "error"
# 不包含文本
{job="functional-scaffold-app"} != "healthz"
# 正则匹配
{job="functional-scaffold-app"} |~ "error|exception"
```
### JSON 提取
```logql
# 提取 request_id
{job="functional-scaffold-app"} | json | request_id != ""
# 按 request_id 过滤
{job="functional-scaffold-app"} | json | request_id = "abc123"
```
### 聚合统计
```logql
# 日志数量
count_over_time({job="functional-scaffold-app"}[5m])
# 按级别统计
sum by (level) (count_over_time({job="functional-scaffold-app"}[5m]))
# 错误率
sum(rate({job="functional-scaffold-app", level="ERROR"}[5m]))
/
sum(rate({job="functional-scaffold-app"}[5m]))
```
## API 查询
### 查询日志
```bash
# 查询最近的日志
curl -G -s "http://localhost:3100/loki/api/v1/query_range" \
--data-urlencode 'query={job="functional-scaffold-app"}' \
--data-urlencode 'limit=10' \
| jq '.data.result'
# 查询错误日志
curl -G -s "http://localhost:3100/loki/api/v1/query_range" \
--data-urlencode 'query={job="functional-scaffold-app", level="ERROR"}' \
| jq '.data.result'
```
### 查询标签
```bash
# 查询所有 job 标签值
curl -s "http://localhost:3100/loki/api/v1/label/job/values" | jq
# 查询所有 level 标签值
curl -s "http://localhost:3100/loki/api/v1/label/level/values" | jq
```
## 配置切换
### 启用文件日志
编辑 `deployment/docker-compose.yml`:
```yaml
environment:
- LOG_FILE_ENABLED=true
```
重启服务:
```bash
docker-compose up -d app
```
### 调整日志级别
编辑 `deployment/docker-compose.yml`:
```yaml
environment:
- LOG_LEVEL=WARNING # DEBUG, INFO, WARNING, ERROR, CRITICAL
```
### 修改保留期
编辑 `monitoring/loki.yaml`:
```yaml
limits_config:
retention_period: 72h # 改为 3 天
```
重启 Loki:
```bash
docker-compose restart loki
```
## 访问地址
| 服务 | 地址 | 凭据 |
|------|------|------|
| Grafana | http://localhost:3000 | admin/admin |
| Loki API | http://localhost:3100 | - |
| Promtail | http://localhost:9080 | - |
| Prometheus | http://localhost:9090 | - |
| App | http://localhost:8111 | - |
## 故障排查
### 看不到日志
```bash
# 1. 检查 Promtail 日志
docker-compose logs promtail | tail -50
# 2. 检查容器标签
docker inspect deployment-app-1 | grep -A 5 Labels
# 3. 查询 Loki
curl -s "http://localhost:3100/loki/api/v1/label/job/values" | jq
```
### Docker socket 权限
```bash
sudo chmod 666 /var/run/docker.sock
```
### 清理日志数据
```bash
# 停止 Loki
docker-compose stop loki
# 清理数据
docker-compose exec loki rm -rf /loki/chunks/*
# 重启 Loki
docker-compose start loki
```
## 性能优化
### 减少日志量
```yaml
# docker-compose.yml
environment:
- LOG_LEVEL=WARNING # 只记录警告和错误
```
### 过滤健康检查日志
编辑 `monitoring/promtail.yaml`:
```yaml
pipeline_stages:
- drop:
expression: ".*healthz.*"
```
### 限制查询范围
```logql
# 好:限制时间范围
{job="functional-scaffold-app"}[5m]
# 差:查询所有时间
{job="functional-scaffold-app"}
```
## 文档链接
- 完整文档: `docs/loki-integration.md`
- 实施总结: `docs/loki-implementation-summary.md`
- 验证脚本: `scripts/verify_loki.sh`

View File

@@ -0,0 +1,126 @@
# 指标过滤和路径规范化
## 变更说明
本次修改优化了 HTTP 请求指标的记录逻辑,主要包括两个方面:
### 1. 跳过健康检查端点
以下端点不再记录到 Prometheus 指标中:
- `/metrics` - 指标端点本身
- `/healthz` - 存活检查
- `/readyz` - 就绪检查
**原因**:这些端点通常被频繁调用(如 Kubernetes 健康检查、Prometheus 抓取),但对业务监控意义不大,会产生大量噪音数据。
### 2. 路径参数规范化
带有路径参数的端点会被规范化为模板形式:
| 原始路径 | 规范化后 |
|---------|---------|
| `GET /jobs/a1b2c3d4e5f6` | `GET /jobs/{job_id}` |
| `GET /jobs/xyz123456789` | `GET /jobs/{job_id}` |
**原因**避免因为不同的路径参数值产生过多的指标标签导致指标基数爆炸cardinality explosion影响 Prometheus 性能。
## 实现细节
### 代码修改
**文件:`src/functional_scaffold/main.py`**
1. 添加 `normalize_path()` 函数:
```python
def normalize_path(path: str) -> str:
"""规范化路径,将路径参数替换为模板形式"""
if path.startswith("/jobs/") and len(path) > 6:
return "/jobs/{job_id}"
return path
```
2. 修改 `track_metrics` 中间件:
```python
# 跳过不需要记录指标的端点
skip_paths = {"/metrics", "/readyz", "/healthz"}
if request.url.path in skip_paths:
return await call_next(request)
# 使用规范化后的路径记录指标
normalized_path = normalize_path(request.url.path)
incr("http_requests_total",
{"method": request.method, "endpoint": normalized_path, "status": status})
```
### 测试覆盖
**文件:`tests/test_middleware.py`**
新增 6 个测试用例:
- `test_normalize_jobs_path` - 测试任务路径规范化
- `test_normalize_other_paths` - 测试其他路径保持不变
- `test_normalize_jobs_root` - 测试 /jobs 根路径
- `test_skip_health_endpoints` - 测试跳过健康检查端点
- `test_record_normal_endpoints` - 测试记录普通端点
- `test_normalize_job_path` - 测试规范化任务路径的集成测试
所有测试通过:✅ 56/56 passed
## 验证方法
### 手动测试
使用提供的测试脚本:
```bash
./scripts/test_metrics_filtering.sh
```
### 预期结果
访问 `/metrics` 端点后,应该看到:
**应该出现的指标:**
```
http_requests_total{method="POST",endpoint="/invoke",status="success"} 1
http_requests_total{method="GET",endpoint="/jobs/{job_id}",status="error"} 2
```
**不应该出现的指标:**
```
http_requests_total{method="GET",endpoint="/healthz",...}
http_requests_total{method="GET",endpoint="/readyz",...}
http_requests_total{method="GET",endpoint="/metrics",...}
http_requests_total{method="GET",endpoint="/jobs/a1b2c3d4e5f6",...}
```
## 扩展性
如果需要添加更多路径规范化规则,只需修改 `normalize_path()` 函数:
```python
def normalize_path(path: str) -> str:
"""规范化路径,将路径参数替换为模板形式"""
# 任务路径
if path.startswith("/jobs/") and len(path) > 6:
return "/jobs/{job_id}"
# 用户路径(示例)
if path.startswith("/users/") and len(path) > 7:
return "/users/{user_id}"
# 其他路径保持不变
return path
```
## 影响范围
- ✅ 不影响现有功能
- ✅ 不影响 API 行为
- ✅ 仅影响指标记录逻辑
- ✅ 向后兼容
- ✅ 所有测试通过
## 相关文档
- [监控指南](../docs/monitoring.md) - 已更新指标说明
- [测试脚本](../scripts/test_metrics_filtering.sh) - 手动验证脚本

View File

@@ -61,6 +61,19 @@ docker-compose up -d redis prometheus grafana
| `http_request_duration_seconds` | Histogram | method, endpoint | HTTP 请求延迟分布 |
| `http_requests_in_progress` | Gauge | - | 当前进行中的请求数 |
**注意事项:**
1. **跳过的端点**:以下端点不会被记录到指标中,以减少噪音:
- `/metrics` - 指标端点本身
- `/healthz` - 存活检查
- `/readyz` - 就绪检查
2. **路径规范化**:带有路径参数的端点会被规范化为模板形式:
- `GET /jobs/a1b2c3d4e5f6``GET /jobs/{job_id}`
- `GET /jobs/xyz123456789``GET /jobs/{job_id}`
这样可以避免因为不同的路径参数值产生过多的指标标签,导致指标基数爆炸。
### 算法执行指标
| 指标 | 类型 | 标签 | 描述 |

16
main.py
View File

@@ -1,16 +0,0 @@
# 这是一个示例 Python 脚本。
# 按 ⌃R 执行或将其替换为您的代码。
# 按 双击 ⇧ 在所有地方搜索类、文件、工具窗口、操作和设置。
def print_hi(name):
# 在下面的代码行中使用断点来调试脚本。
print(f'Hi, {name}') # 按 ⌘F8 切换断点。
# 按装订区域中的绿色按钮以运行脚本。
if __name__ == '__main__':
print_hi('PyCharm')
# 访问 https://www.jetbrains.com/help/pycharm/ 获取 PyCharm 帮助

258
monitoring/README.md Normal file
View File

@@ -0,0 +1,258 @@
# Monitoring 目录说明
本目录包含所有监控和日志收集相关的配置文件。
## 目录结构
```
monitoring/
├── alerts/ # Prometheus 告警规则
│ └── rules.yaml # 告警规则配置
├── grafana/ # Grafana 配置
│ ├── datasources/ # 数据源自动配置
│ │ ├── prometheus.yaml # Prometheus 数据源
│ │ └── loki.yaml # Loki 数据源
│ └── dashboards/ # 仪表板自动加载
│ ├── provider.yaml # Dashboard provider 配置
│ ├── dashboard.json # 指标监控仪表板
│ └── logs-dashboard.json # 日志监控仪表板
├── loki.yaml # Loki 日志存储配置
├── promtail.yaml # Promtail 日志采集配置
└── prometheus.yml # Prometheus 指标收集配置
```
## 配置文件说明
### Prometheus 配置
**文件**: `prometheus.yml`
Prometheus 指标收集配置,包括:
- 抓取间隔: 5 秒
- 目标: app 服务的 `/metrics` 端点
- 告警规则: 从 `alerts/` 目录加载
### Loki 配置
**文件**: `loki.yaml`
Loki 日志存储配置,包括:
- 存储方式: 本地文件系统
- 日志保留期: 7 天
- 摄入速率限制: 10MB/s
- 自动压缩和清理
**关键配置**:
```yaml
limits_config:
retention_period: 168h # 7 天
ingestion_rate_mb: 10 # 10MB/s
```
### Promtail 配置
**文件**: `promtail.yaml`
Promtail 日志采集配置,支持两种模式:
**模式 1: Docker stdio 收集(默认)**
- 通过 Docker API 自动发现容器
- 过滤带有 `logging=promtail` 标签的容器
- 自动解析 JSON 日志
**模式 2: 文件收集(备用)**
-`/var/log/app/*.log` 读取日志文件
- 支持日志轮转
- 需要设置 `LOG_FILE_ENABLED=true`
### Grafana Provisioning
**数据源** (`grafana/datasources/`)
自动配置 Grafana 数据源:
- `prometheus.yaml`: Prometheus 数据源(默认)
- `loki.yaml`: Loki 数据源
**仪表板** (`grafana/dashboards/`)
自动加载 Grafana 仪表板:
- `provider.yaml`: Dashboard provider 配置
- `dashboard.json`: 指标监控仪表板HTTP 请求、算法执行等)
- `logs-dashboard.json`: 日志监控仪表板(日志流、错误日志等)
### 告警规则
**文件**: `alerts/rules.yaml`
Prometheus 告警规则,包括:
- 高错误率告警
- 高延迟告警
- 服务不可用告警
## 修改配置
### 调整日志保留期
编辑 `loki.yaml`:
```yaml
limits_config:
retention_period: 72h # 改为 3 天
```
重启 Loki:
```bash
cd deployment
docker-compose restart loki
```
### 调整指标抓取间隔
编辑 `prometheus.yml`:
```yaml
global:
scrape_interval: 10s # 改为 10 秒
```
重启 Prometheus:
```bash
cd deployment
docker-compose restart prometheus
```
### 添加新的告警规则
编辑 `alerts/rules.yaml`,添加新规则:
```yaml
groups:
- name: my_alerts
rules:
- alert: MyAlert
expr: my_metric > 100
for: 5m
labels:
severity: warning
annotations:
summary: "我的告警"
```
重启 Prometheus:
```bash
cd deployment
docker-compose restart prometheus
```
### 添加新的仪表板
1. 在 Grafana UI 中创建仪表板
2. 导出为 JSON
3. 保存到 `grafana/dashboards/my-dashboard.json`
4. 重启 Grafana或等待自动重载
```bash
cd deployment
docker-compose restart grafana
```
## 验证配置
### 检查 Prometheus 配置
```bash
# 访问 Prometheus UI
open http://localhost:9090
# 检查目标状态
open http://localhost:9090/targets
# 检查告警规则
open http://localhost:9090/alerts
```
### 检查 Loki 配置
```bash
# 检查 Loki 健康状态
curl http://localhost:3100/ready
# 查询标签
curl -s "http://localhost:3100/loki/api/v1/label/job/values" | jq
```
### 检查 Grafana 配置
```bash
# 访问 Grafana UI
open http://localhost:3000
# 检查数据源
curl -s -u admin:admin http://localhost:3000/api/datasources | jq
# 检查仪表板
curl -s -u admin:admin http://localhost:3000/api/search | jq
```
## 故障排查
### Prometheus 无法抓取指标
1. 检查 app 服务是否运行: `docker-compose ps app`
2. 检查 metrics 端点: `curl http://localhost:8111/metrics`
3. 查看 Prometheus 日志: `docker-compose logs prometheus`
### Loki 无法接收日志
1. 检查 Promtail 是否运行: `docker-compose ps promtail`
2. 查看 Promtail 日志: `docker-compose logs promtail`
3. 检查容器标签: `docker inspect <container> | grep Labels`
### Grafana 数据源未加载
1. 检查 provisioning 目录挂载: `docker-compose config | grep grafana -A 10`
2. 查看 Grafana 日志: `docker-compose logs grafana`
3. 手动重启 Grafana: `docker-compose restart grafana`
## 相关文档
- [Loki 集成文档](../docs/loki-integration.md) - 完整的 Loki 使用文档
- [Loki 快速参考](../docs/loki-quick-reference.md) - 常用命令和查询
- [Loki 实施总结](../docs/loki-implementation-summary.md) - 实施细节和架构说明
- [Prometheus 官方文档](https://prometheus.io/docs/)
- [Loki 官方文档](https://grafana.com/docs/loki/latest/)
- [Grafana 官方文档](https://grafana.com/docs/grafana/latest/)
## 性能建议
### 日志量控制
- 调整日志级别为 WARNING 或 ERROR
- 过滤掉不必要的日志(如健康检查)
- 减少日志保留期
### 指标优化
- 增加抓取间隔(如 15s 或 30s
- 减少指标基数(避免高基数标签)
- 定期清理旧数据
### 存储优化
- 监控磁盘使用: `docker-compose exec loki du -sh /loki`
- 定期备份重要数据
- 考虑使用对象存储S3/OSS作为后端
## 总结
本目录包含完整的监控和日志收集配置:
**Prometheus** - 指标收集和告警
**Loki** - 日志存储和查询
**Promtail** - 日志采集
**Grafana** - 可视化和仪表板
所有配置都支持自动加载,无需手动配置。

View File

@@ -1395,6 +1395,504 @@
],
"title": "Webhook 发送状态",
"type": "piechart"
},
{
"collapsed": false,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 53
},
"id": 200,
"panels": [],
"title": "队列监控",
"type": "row"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "任务数",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 20,
"gradientMode": "opacity",
"hideFrom": {
"tooltip": false,
"viz": false,
"legend": false
},
"lineInterpolation": "smooth",
"lineWidth": 2,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "never",
"spanNulls": true,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "yellow",
"value": 50
},
{
"color": "red",
"value": 100
}
]
},
"unit": "short"
},
"overrides": [
{
"matcher": {
"id": "byName",
"options": "pending"
},
"properties": [
{
"id": "color",
"value": {
"fixedColor": "blue",
"mode": "fixed"
}
}
]
},
{
"matcher": {
"id": "byName",
"options": "processing"
},
"properties": [
{
"id": "color",
"value": {
"fixedColor": "orange",
"mode": "fixed"
}
}
]
},
{
"matcher": {
"id": "byName",
"options": "dlq"
},
"properties": [
{
"id": "color",
"value": {
"fixedColor": "red",
"mode": "fixed"
}
}
]
}
]
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 54
},
"id": 19,
"options": {
"legend": {
"calcs": ["mean", "last", "max"],
"displayMode": "table",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"expr": "job_queue_length",
"legendFormat": "{{queue}}",
"refId": "A"
}
],
"title": "队列长度趋势",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "秒",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 20,
"gradientMode": "opacity",
"hideFrom": {
"tooltip": false,
"viz": false,
"legend": false
},
"lineInterpolation": "smooth",
"lineWidth": 2,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "never",
"spanNulls": true,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "line"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "yellow",
"value": 60
},
{
"color": "red",
"value": 300
}
]
},
"unit": "s"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 54
},
"id": 20,
"options": {
"legend": {
"calcs": ["mean", "last", "max"],
"displayMode": "table",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
},
"pluginVersion": "9.0.0",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"expr": "job_oldest_waiting_seconds",
"legendFormat": "最长等待时间",
"refId": "A"
}
],
"title": "最长任务等待时间",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "yellow",
"value": 10
},
{
"color": "red",
"value": 50
}
]
},
"unit": "short"
},
"overrides": []
},
"gridPos": {
"h": 4,
"w": 6,
"x": 0,
"y": 62
},
"id": 21,
"options": {
"colorMode": "value",
"graphMode": "area",
"justifyMode": "auto",
"orientation": "auto",
"reduceOptions": {
"values": false,
"calcs": ["last"],
"fields": ""
},
"textMode": "auto"
},
"pluginVersion": "9.0.0",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"expr": "job_queue_length{queue=\"pending\"}",
"refId": "A"
}
],
"title": "待处理队列",
"type": "stat"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "yellow",
"value": 5
},
{
"color": "red",
"value": 10
}
]
},
"unit": "short"
},
"overrides": []
},
"gridPos": {
"h": 4,
"w": 6,
"x": 6,
"y": 62
},
"id": 22,
"options": {
"colorMode": "value",
"graphMode": "area",
"justifyMode": "auto",
"orientation": "auto",
"reduceOptions": {
"values": false,
"calcs": ["last"],
"fields": ""
},
"textMode": "auto"
},
"pluginVersion": "9.0.0",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"expr": "job_queue_length{queue=\"processing\"}",
"refId": "A"
}
],
"title": "处理中队列",
"type": "stat"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 1
}
]
},
"unit": "short"
},
"overrides": []
},
"gridPos": {
"h": 4,
"w": 6,
"x": 12,
"y": 62
},
"id": 23,
"options": {
"colorMode": "value",
"graphMode": "area",
"justifyMode": "auto",
"orientation": "auto",
"reduceOptions": {
"values": false,
"calcs": ["last"],
"fields": ""
},
"textMode": "auto"
},
"pluginVersion": "9.0.0",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"expr": "job_queue_length{queue=\"dlq\"}",
"refId": "A"
}
],
"title": "死信队列",
"type": "stat"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
},
"unit": "short"
},
"overrides": []
},
"gridPos": {
"h": 4,
"w": 6,
"x": 18,
"y": 62
},
"id": 24,
"options": {
"colorMode": "value",
"graphMode": "area",
"justifyMode": "auto",
"orientation": "auto",
"reduceOptions": {
"values": false,
"calcs": ["last"],
"fields": ""
},
"textMode": "auto"
},
"pluginVersion": "9.0.0",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"expr": "sum(job_recovered_total) or vector(0)",
"refId": "A"
}
],
"title": "回收任务总数",
"type": "stat"
}
],
"refresh": "5s",

View File

@@ -0,0 +1,292 @@
{
"annotations": {
"list": [
{
"builtIn": 1,
"datasource": {
"type": "grafana",
"uid": "-- Grafana --"
},
"enable": true,
"hide": true,
"iconColor": "rgba(0, 211, 255, 1)",
"name": "Annotations & Alerts",
"type": "dashboard"
}
]
},
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 0,
"id": null,
"links": [],
"liveNow": false,
"panels": [
{
"datasource": {
"type": "loki",
"uid": "Loki"
},
"gridPos": {
"h": 10,
"w": 24,
"x": 0,
"y": 0
},
"id": 1,
"options": {
"dedupStrategy": "none",
"enableLogDetails": true,
"prettifyLogMessage": false,
"showCommonLabels": false,
"showLabels": false,
"showTime": true,
"sortOrder": "Descending",
"wrapLogMessage": false
},
"targets": [
{
"datasource": {
"type": "loki",
"uid": "Loki"
},
"editorMode": "code",
"expr": "{job=\"functional-scaffold-app\"} |= \"$request_id\"",
"queryType": "range",
"refId": "A"
}
],
"title": "日志流 (实时)",
"type": "logs"
},
{
"datasource": {
"type": "loki",
"uid": "Loki"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"tooltip": false,
"viz": false,
"legend": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "never",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
},
"unit": "short"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 10
},
"id": 2,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "single",
"sort": "none"
}
},
"targets": [
{
"datasource": {
"type": "loki",
"uid": "Loki"
},
"editorMode": "code",
"expr": "sum by (level) (count_over_time({job=\"functional-scaffold-app\"} |= \"$request_id\" [1m]))",
"queryType": "range",
"refId": "A"
}
],
"title": "日志量趋势(按级别)",
"type": "timeseries"
},
{
"datasource": {
"type": "loki",
"uid": "Loki"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "yellow",
"value": 10
},
{
"color": "red",
"value": 50
}
]
}
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 10
},
"id": 3,
"options": {
"orientation": "auto",
"reduceOptions": {
"values": false,
"calcs": [
"lastNotNull"
],
"fields": ""
},
"showThresholdLabels": false,
"showThresholdMarkers": true
},
"pluginVersion": "9.5.3",
"targets": [
{
"datasource": {
"type": "loki",
"uid": "Loki"
},
"editorMode": "code",
"expr": "sum by (level) (count_over_time({job=\"functional-scaffold-app\"} |= \"$request_id\" [$__range]))",
"queryType": "range",
"refId": "A"
}
],
"title": "日志级别分布",
"type": "gauge"
},
{
"datasource": {
"type": "loki",
"uid": "Loki"
},
"gridPos": {
"h": 10,
"w": 24,
"x": 0,
"y": 18
},
"id": 4,
"options": {
"dedupStrategy": "none",
"enableLogDetails": true,
"prettifyLogMessage": false,
"showCommonLabels": false,
"showLabels": false,
"showTime": true,
"sortOrder": "Descending",
"wrapLogMessage": false
},
"targets": [
{
"datasource": {
"type": "loki",
"uid": "Loki"
},
"editorMode": "code",
"expr": "{job=\"functional-scaffold-app\", level=\"ERROR\"} |= \"$request_id\"",
"queryType": "range",
"refId": "A"
}
],
"title": "错误日志",
"type": "logs"
}
],
"refresh": "5s",
"schemaVersion": 38,
"style": "dark",
"tags": ["logs", "loki"],
"templating": {
"list": [
{
"current": {
"selected": false,
"text": "",
"value": ""
},
"hide": 0,
"label": "Request ID",
"name": "request_id",
"options": [
{
"selected": true,
"text": "",
"value": ""
}
],
"query": "",
"skipUrlSync": false,
"type": "textbox"
}
]
},
"time": {
"from": "now-15m",
"to": "now"
},
"timepicker": {},
"timezone": "",
"title": "日志监控",
"uid": "logs-dashboard",
"version": 0,
"weekStart": ""
}

View File

@@ -0,0 +1,13 @@
apiVersion: 1
providers:
- name: 'default'
orgId: 1
folder: ''
type: file
disableDeletion: false
updateIntervalSeconds: 10
allowUiUpdates: true
options:
path: /etc/grafana/provisioning/dashboards
foldersFromFilesStructure: true

View File

@@ -0,0 +1,11 @@
apiVersion: 1
datasources:
- name: Loki
type: loki
access: proxy
url: http://loki:3100
isDefault: false
editable: false
jsonData:
maxLines: 1000

View File

@@ -0,0 +1,11 @@
apiVersion: 1
datasources:
- name: Prometheus
type: prometheus
access: proxy
url: http://prometheus:9090
isDefault: true
editable: false
jsonData:
timeInterval: "5s"

39
monitoring/loki.yaml Normal file
View File

@@ -0,0 +1,39 @@
auth_enabled: false
server:
http_listen_port: 3100
grpc_listen_port: 9096
common:
path_prefix: /loki
storage:
filesystem:
chunks_directory: /loki/chunks
rules_directory: /loki/rules
replication_factor: 1
ring:
instance_addr: 127.0.0.1
kvstore:
store: inmemory
schema_config:
configs:
- from: 2020-10-24
store: boltdb-shipper
object_store: filesystem
schema: v11
index:
prefix: index_
period: 24h
limits_config:
retention_period: 168h # 7 天
ingestion_rate_mb: 10
ingestion_burst_size_mb: 20
compactor:
working_directory: /loki/compactor
shared_store: filesystem
compaction_interval: 10m
retention_enabled: true
retention_delete_delay: 2h

71
monitoring/promtail.yaml Normal file
View File

@@ -0,0 +1,71 @@
server:
http_listen_port: 9080
grpc_listen_port: 0
positions:
filename: /tmp/positions.yaml
clients:
- url: http://loki:3100/loki/api/v1/push
scrape_configs:
# 场景 1: Docker stdio 收集(主要方式)
- job_name: docker
docker_sd_configs:
- host: unix:///var/run/docker.sock
refresh_interval: 5s
filters:
- name: label
values: ["logging=promtail"]
relabel_configs:
- source_labels: ['__meta_docker_container_name']
regex: '/(.*)'
target_label: 'container'
- source_labels: ['__meta_docker_container_label_logging_jobname']
target_label: 'job'
- source_labels: ['__meta_docker_container_id']
target_label: '__path__'
replacement: '/var/lib/docker/containers/$1/*.log'
pipeline_stages:
- json:
expressions:
log: log
stream: stream
time: time
- json:
source: log
expressions:
level: levelname
logger: name
message: message
request_id: request_id
- labels:
level:
logger:
- output:
source: log
# 场景 2: Log 文件收集(备用)
- job_name: app_files
static_configs:
- targets:
- localhost
labels:
job: functional-scaffold-app-files
__path__: /var/log/app/*.log
pipeline_stages:
- json:
expressions:
timestamp: asctime
level: levelname
logger: name
message: message
request_id: request_id
- timestamp:
source: timestamp
format: "2006-01-02 15:04:05,000"
- labels:
level:
logger:
- output:
source: message

View File

@@ -19,6 +19,14 @@ dependencies = [
"pydantic-settings>=2.0.0",
"prometheus-client>=0.19.0",
"python-json-logger>=2.0.7",
# Redis - 任务队列和指标存储
"redis>=5.0.0",
# YAML 配置解析
"pyyaml>=6.0.0",
# HTTP 客户端Webhook 回调)
"httpx>=0.27.0",
# 轻量级 HTTP 服务器Worker 健康检查)
"aiohttp>=3.9.0",
]
[project.optional-dependencies]
@@ -26,7 +34,6 @@ dev = [
"pytest>=7.4.0",
"pytest-asyncio>=0.21.0",
"pytest-cov>=4.1.0",
"httpx>=0.26.0",
"black>=23.12.0",
"ruff>=0.1.0",
]
@@ -48,3 +55,4 @@ python_files = ["test_*.py"]
python_classes = ["Test*"]
python_functions = ["test_*"]
addopts = "-v --strict-markers"
pythonpath = ["src"]

View File

@@ -1,16 +1,17 @@
# 核心依赖 - 与 pyproject.toml 保持同步
fastapi>=0.109.0
uvicorn[standard]>=0.27.0
pydantic>=2.5.0
pydantic-settings>=2.0.0
prometheus-client>=0.19.0
python-json-logger>=2.0.7
aiohttp>=3.9.0
# 指标存储方案(可选,根据选择的方案安装)
# 方案2Redis 方案需要
# Redis - 任务队列和指标存储
redis>=5.0.0
# YAML 配置解析
pyyaml>=6.0.0
# HTTP 客户端(用于 Webhook 回调)
# HTTP 客户端Webhook 回调)
httpx>=0.27.0

View File

@@ -21,4 +21,4 @@ pip install -e ".[dev]"
# 启动服务
echo "Starting server on http://localhost:8000"
echo "API docs available at http://localhost:8000/docs"
uvicorn src.functional_scaffold.main:app --reload --host 0.0.0.0 --port 8000
uvicorn functional_scaffold.main:app --reload --host 0.0.0.0 --port 8000

View File

@@ -1,114 +0,0 @@
#!/bin/bash
# 指标方案快速启动脚本
set -e
# 颜色定义
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m' # No Color
echo "=========================================="
echo "FunctionalScaffold 指标方案启动脚本"
echo "=========================================="
# 检查 docker-compose
if ! command -v docker-compose &> /dev/null; then
echo -e "${RED}错误: docker-compose 未安装${NC}"
exit 1
fi
# 选择方案
echo ""
echo "请选择指标方案:"
echo "1. Pushgateway推荐适合 Serverless"
echo "2. Redis + Exporter适合高并发"
echo "3. 两者都启动(用于对比测试)"
echo ""
read -p "输入选项 (1/2/3): " choice
cd "$(dirname "$0")/../deployment"
case $choice in
1)
echo -e "${GREEN}启动 Pushgateway 方案...${NC}"
docker-compose up -d redis pushgateway prometheus grafana
echo ""
echo -e "${GREEN}✓ Pushgateway 方案已启动${NC}"
echo ""
echo "服务地址:"
echo " - Pushgateway: http://localhost:9091"
echo " - Prometheus: http://localhost:9090"
echo " - Grafana: http://localhost:3000 (admin/admin)"
echo ""
echo "下一步:"
echo " 1. 修改代码导入: from functional_scaffold.core.metrics_pushgateway import ..."
echo " 2. 配置环境变量: PUSHGATEWAY_URL=localhost:9091"
echo " 3. 启动应用: ./scripts/run_dev.sh"
echo " 4. 运行测试: python scripts/test_metrics.py pushgateway"
;;
2)
echo -e "${GREEN}启动 Redis 方案...${NC}"
# 检查 redis 依赖
if ! python -c "import redis" 2>/dev/null; then
echo -e "${YELLOW}警告: redis 库未安装${NC}"
echo "正在安装 redis..."
pip install redis
fi
docker-compose up -d redis redis-exporter prometheus grafana
echo ""
echo -e "${GREEN}✓ Redis 方案已启动${NC}"
echo ""
echo "服务地址:"
echo " - Redis: localhost:6379"
echo " - Redis Exporter: http://localhost:8001/metrics"
echo " - Prometheus: http://localhost:9090"
echo " - Grafana: http://localhost:3000 (admin/admin)"
echo ""
echo "下一步:"
echo " 1. 修改代码导入: from functional_scaffold.core.metrics_redis import ..."
echo " 2. 配置环境变量: REDIS_HOST=localhost REDIS_PORT=6379"
echo " 3. 启动应用: ./scripts/run_dev.sh"
echo " 4. 运行测试: python scripts/test_metrics.py redis"
;;
3)
echo -e "${GREEN}启动所有服务...${NC}"
# 检查 redis 依赖
if ! python -c "import redis" 2>/dev/null; then
echo -e "${YELLOW}警告: redis 库未安装${NC}"
echo "正在安装 redis..."
pip install redis
fi
docker-compose up -d
echo ""
echo -e "${GREEN}✓ 所有服务已启动${NC}"
echo ""
echo "服务地址:"
echo " - 应用: http://localhost:8000"
echo " - Pushgateway: http://localhost:9091"
echo " - Redis: localhost:6379"
echo " - Redis Exporter: http://localhost:8001/metrics"
echo " - Prometheus: http://localhost:9090"
echo " - Grafana: http://localhost:3000 (admin/admin)"
echo ""
echo "下一步:"
echo " 1. 查看文档: cat docs/metrics-guide.md"
echo " 2. 运行测试: python scripts/test_metrics.py"
;;
*)
echo -e "${RED}无效的选项${NC}"
exit 1
;;
esac
echo ""
echo "=========================================="
echo "查看日志: docker-compose logs -f"
echo "停止服务: docker-compose down"
echo "查看文档: cat ../docs/metrics-guide.md"
echo "=========================================="

104
scripts/test_concurrency.sh Executable file
View File

@@ -0,0 +1,104 @@
#!/bin/bash
# 并发控制测试脚本
set -e
BASE_URL="http://localhost:8000"
echo "=== 异步任务并发控制测试 ==="
echo ""
# 1. 检查服务是否运行
echo "1. 检查服务状态..."
if ! curl -s "${BASE_URL}/healthz" > /dev/null; then
echo "❌ 服务未运行,请先启动服务"
exit 1
fi
echo "✅ 服务正常运行"
echo ""
# 2. 查询初始并发状态
echo "2. 查询初始并发状态..."
curl -s "${BASE_URL}/jobs/concurrency/status" | jq '.'
echo ""
# 3. 创建多个任务
echo "3. 创建 15 个任务(测试并发限制)..."
JOB_IDS=()
for i in {1..15}; do
# 使用较大的质数,让任务执行时间更长
NUMBER=$((10000 + i * 1000))
RESPONSE=$(curl -s -X POST "${BASE_URL}/jobs" \
-H "Content-Type: application/json" \
-d "{\"algorithm\": \"PrimeChecker\", \"params\": {\"number\": ${NUMBER}}}")
JOB_ID=$(echo "$RESPONSE" | jq -r '.job_id')
JOB_IDS+=("$JOB_ID")
echo " 创建任务 ${i}/15: job_id=${JOB_ID}"
# 短暂延迟,避免请求过快
sleep 0.1
done
echo ""
# 4. 立即查询并发状态(应该看到多个任务在运行)
echo "4. 查询并发状态(任务执行中)..."
for i in {1..5}; do
echo "${i} 次查询:"
STATUS=$(curl -s "${BASE_URL}/jobs/concurrency/status")
echo " $(echo "$STATUS" | jq -c '.')"
sleep 1
done
echo ""
# 5. 等待所有任务完成
echo "5. 等待任务完成..."
COMPLETED=0
TOTAL=${#JOB_IDS[@]}
while [ $COMPLETED -lt $TOTAL ]; do
COMPLETED=0
for JOB_ID in "${JOB_IDS[@]}"; do
STATUS=$(curl -s "${BASE_URL}/jobs/${JOB_ID}" | jq -r '.status')
if [ "$STATUS" = "completed" ] || [ "$STATUS" = "failed" ]; then
((COMPLETED++))
fi
done
echo " 进度: ${COMPLETED}/${TOTAL} 任务完成"
# 显示当前并发状态
CONCURRENCY=$(curl -s "${BASE_URL}/jobs/concurrency/status")
echo " 并发状态: $(echo "$CONCURRENCY" | jq -c '.')"
if [ $COMPLETED -lt $TOTAL ]; then
sleep 2
fi
done
echo ""
# 6. 查询最终并发状态
echo "6. 查询最终并发状态..."
curl -s "${BASE_URL}/jobs/concurrency/status" | jq '.'
echo ""
# 7. 显示任务结果统计
echo "7. 任务结果统计..."
COMPLETED_COUNT=0
FAILED_COUNT=0
for JOB_ID in "${JOB_IDS[@]}"; do
STATUS=$(curl -s "${BASE_URL}/jobs/${JOB_ID}" | jq -r '.status')
if [ "$STATUS" = "completed" ]; then
((COMPLETED_COUNT++))
elif [ "$STATUS" = "failed" ]; then
((FAILED_COUNT++))
fi
done
echo " 总任务数: ${TOTAL}"
echo " 成功: ${COMPLETED_COUNT}"
echo " 失败: ${FAILED_COUNT}"
echo ""
echo "=== 测试完成 ==="

View File

@@ -0,0 +1,39 @@
#!/bin/bash
# 测试指标过滤和路径规范化
echo "=== 测试指标过滤和路径规范化 ==="
echo ""
# 启动服务(假设已经在运行)
BASE_URL="http://localhost:8000"
echo "1. 访问健康检查端点(应该被跳过,不记录指标)"
curl -s "$BASE_URL/healthz" > /dev/null
curl -s "$BASE_URL/readyz" > /dev/null
echo " ✓ 已访问 /healthz 和 /readyz"
echo ""
echo "2. 访问普通端点(应该记录指标)"
curl -s -X POST "$BASE_URL/invoke" \
-H "Content-Type: application/json" \
-d '{"number": 17}' > /dev/null
echo " ✓ 已访问 POST /invoke"
echo ""
echo "3. 访问任务端点(应该规范化为 /jobs/{job_id}"
curl -s "$BASE_URL/jobs/a1b2c3d4e5f6" > /dev/null
curl -s "$BASE_URL/jobs/xyz123456789" > /dev/null
echo " ✓ 已访问 GET /jobs/a1b2c3d4e5f6 和 GET /jobs/xyz123456789"
echo ""
echo "4. 查看指标输出"
echo " 查找 http_requests_total 指标:"
curl -s "$BASE_URL/metrics" | grep 'http_requests_total{' | grep -v '#'
echo ""
echo " 预期结果:"
echo " - 应该看到 endpoint=\"/invoke\" 的记录"
echo " - 应该看到 endpoint=\"/jobs/{job_id}\" 的记录(而不是具体的 job_id"
echo " - 不应该看到 endpoint=\"/healthz\" 或 endpoint=\"/readyz\" 的记录"
echo " - 不应该看到 endpoint=\"/metrics\" 的记录"
echo ""
echo "=== 测试完成 ==="

View File

@@ -0,0 +1,69 @@
#!/bin/bash
# Grafana Request ID 过滤功能测试脚本
set -e
echo "========================================="
echo "Grafana Request ID 过滤功能测试"
echo "========================================="
echo ""
# 颜色定义
GREEN='\033[0;32m'
BLUE='\033[0;34m'
YELLOW='\033[1;33m'
NC='\033[0m' # No Color
echo "1. 生成测试请求..."
echo "-------------------"
RESPONSE=$(curl -X POST http://localhost:8111/invoke \
-H "Content-Type: application/json" \
-d '{"number": 43}' \
-s)
REQUEST_ID=$(echo "$RESPONSE" | jq -r '.request_id')
echo -e "${GREEN}✓ 请求成功${NC}"
echo -e "${BLUE}Request ID: $REQUEST_ID${NC}"
echo ""
echo "2. 等待日志收集 (5秒)..."
sleep 5
echo ""
echo "3. 测试 Loki 过滤..."
echo "-------------------"
# 测试过滤特定 request_id
LOG_COUNT=$(curl -G -s "http://localhost:3100/loki/api/v1/query_range" \
--data-urlencode "query={job=\"functional-scaffold-app\"} |= \"$REQUEST_ID\"" \
| jq '.data.result[0].values | length')
if [ "$LOG_COUNT" -gt 0 ]; then
echo -e "${GREEN}✓ 找到 $LOG_COUNT 条日志${NC}"
else
echo -e "${YELLOW}⚠ 没有找到日志,可能需要等待更长时间${NC}"
fi
echo ""
echo "4. 显示日志内容..."
echo "-------------------"
curl -G -s "http://localhost:3100/loki/api/v1/query_range" \
--data-urlencode "query={job=\"functional-scaffold-app\"} |= \"$REQUEST_ID\"" \
| jq -r '.data.result[0].values[].[-1]' \
| jq -r '.message' \
| nl
echo ""
echo "========================================="
echo "测试完成!"
echo "========================================="
echo ""
echo "在 Grafana 中测试:"
echo " 1. 访问: http://localhost:3000"
echo " 2. 进入 '日志监控' 仪表板"
echo " 3. 在顶部 'Request ID' 输入框中输入:"
echo -e " ${BLUE}$REQUEST_ID${NC}"
echo " 4. 按回车,查看过滤后的日志"
echo ""
echo "清空 Request ID 输入框可以查看所有日志"
echo ""

100
scripts/verify_loki.sh Executable file
View File

@@ -0,0 +1,100 @@
#!/bin/bash
# Loki 集成验证脚本
set -e
echo "========================================="
echo "Loki 日志收集系统验证"
echo "========================================="
echo ""
# 颜色定义
GREEN='\033[0;32m'
RED='\033[0;31m'
YELLOW='\033[1;33m'
NC='\033[0m' # No Color
# 检查服务状态
echo "1. 检查服务状态..."
echo "-------------------"
docker-compose ps
echo ""
echo "2. 检查 Loki 健康状态..."
echo "-------------------"
if curl -s http://localhost:3100/ready | grep -q "ready"; then
echo -e "${GREEN}✓ Loki 服务正常${NC}"
else
echo -e "${RED}✗ Loki 服务异常${NC}"
exit 1
fi
echo ""
echo "3. 检查 Promtail 健康状态..."
echo "-------------------"
if curl -s http://localhost:9080/ready | grep -q "ready"; then
echo -e "${GREEN}✓ Promtail 服务正常${NC}"
else
echo -e "${RED}✗ Promtail 服务异常${NC}"
exit 1
fi
echo ""
echo "4. 生成测试日志..."
echo "-------------------"
curl -X POST http://localhost:8111/invoke \
-H "Content-Type: application/json" \
-d '{"algorithm": "PrimeChecker", "params": {"number": 17}}' \
-s -o /dev/null -w "HTTP Status: %{http_code}\n"
echo ""
echo "5. 等待日志收集 (5秒)..."
sleep 5
echo ""
echo "6. 查询 Loki 日志..."
echo "-------------------"
LOGS=$(curl -G -s "http://localhost:3100/loki/api/v1/query_range" \
--data-urlencode 'query={job="functional-scaffold-app"}' \
--data-urlencode 'limit=5')
if echo "$LOGS" | jq -e '.data.result | length > 0' > /dev/null 2>&1; then
echo -e "${GREEN}✓ 成功查询到日志${NC}"
echo ""
echo "最近的日志条目:"
echo "$LOGS" | jq -r '.data.result[0].values[-1][1]' | head -3
else
echo -e "${YELLOW}⚠ 暂时没有查询到日志,可能需要等待更长时间${NC}"
fi
echo ""
echo "7. 检查 Grafana 数据源..."
echo "-------------------"
DATASOURCES=$(curl -s -u admin:admin http://localhost:3000/api/datasources)
if echo "$DATASOURCES" | jq -e '.[] | select(.name == "Loki")' > /dev/null 2>&1; then
echo -e "${GREEN}✓ Loki 数据源已配置${NC}"
else
echo -e "${RED}✗ Loki 数据源未配置${NC}"
fi
if echo "$DATASOURCES" | jq -e '.[] | select(.name == "Prometheus")' > /dev/null 2>&1; then
echo -e "${GREEN}✓ Prometheus 数据源已配置${NC}"
else
echo -e "${RED}✗ Prometheus 数据源未配置${NC}"
fi
echo ""
echo "========================================="
echo "验证完成!"
echo "========================================="
echo ""
echo "访问地址:"
echo " - Grafana: http://localhost:3000 (admin/admin)"
echo " - Loki: http://localhost:3100"
echo " - Promtail: http://localhost:9080"
echo ""
echo "查看日志:"
echo " 1. 访问 Grafana Explore: http://localhost:3000/explore"
echo " 2. 选择 Loki 数据源"
echo " 3. 输入查询: {job=\"functional-scaffold-app\"}"
echo ""

View File

@@ -32,7 +32,7 @@ class BaseAlgorithm(ABC):
Returns:
Dict[str, Any]: 包含结果和元数据的字典
"""
from ..core.metrics_unified import incr, observe
from ..core.metrics_unified import incr_sync, observe_sync
start_time = time.time()
status = "success"
@@ -71,5 +71,7 @@ class BaseAlgorithm(ABC):
finally:
# 记录算法执行指标
elapsed_time = time.time() - start_time
incr("algorithm_executions_total", {"algorithm": self.name, "status": status})
observe("algorithm_execution_duration_seconds", {"algorithm": self.name}, elapsed_time)
incr_sync("algorithm_executions_total", {"algorithm": self.name, "status": status})
observe_sync(
"algorithm_execution_duration_seconds", {"algorithm": self.name}, elapsed_time
)

View File

@@ -2,7 +2,7 @@
from typing import Dict, Any, List
from .base import BaseAlgorithm
from ..core.metrics_unified import incr
from ..core.metrics_unified import incr_sync
class PrimeChecker(BaseAlgorithm):
@@ -31,12 +31,12 @@ class PrimeChecker(BaseAlgorithm):
ValueError: 如果输入不是整数
"""
if not isinstance(number, int):
incr('prime_check',{"status":"invalid_input"})
incr_sync('prime_check', {"status": "invalid_input"})
raise ValueError(f"Input must be an integer, got {type(number).__name__}")
# 小于2的数不是质数
if number < 2:
incr('prime_check', {"status": "number_little_two"})
incr_sync('prime_check', {"status": "number_little_two"})
return {
"number": number,
"is_prime": False,
@@ -50,7 +50,7 @@ class PrimeChecker(BaseAlgorithm):
# 如果不是质数,计算因数
factors = [] if is_prime else self._get_factors(number)
incr('prime_check', {"status": "success"})
incr_sync('prime_check', {"status": "success"})
return {
"number": number,
"is_prime": is_prime,

View File

@@ -2,7 +2,7 @@
from fastapi import Header, HTTPException
from typing import Optional
from ..core.tracing import set_request_id, generate_request_id
from ..core.tracing import set_request_id, generate_request_id, get_request_id as get_current_request_id
async def get_request_id(x_request_id: Optional[str] = Header(None)) -> str:
@@ -15,6 +15,12 @@ async def get_request_id(x_request_id: Optional[str] = Header(None)) -> str:
Returns:
str: 请求ID
"""
# 先检查 ContextVar 中是否已经有 request_id由中间件设置
existing_request_id = get_current_request_id()
if existing_request_id:
return existing_request_id
# 如果没有,则从请求头获取或生成新的
request_id = x_request_id or generate_request_id()
set_request_id(request_id)
return request_id

View File

@@ -152,3 +152,21 @@ class JobStatusResponse(BaseModel):
result: Optional[Dict[str, Any]] = Field(None, description="执行结果(仅完成时返回)")
error: Optional[str] = Field(None, description="错误信息(仅失败时返回)")
metadata: Optional[Dict[str, Any]] = Field(None, description="元数据信息")
class ConcurrencyStatusResponse(BaseModel):
"""并发状态响应"""
model_config = ConfigDict(
json_schema_extra={
"example": {
"max_concurrent": 10,
"available_slots": 7,
"running_jobs": 3,
}
}
)
max_concurrent: int = Field(..., description="最大并发任务数")
available_slots: int = Field(..., description="当前可用槽位数")
running_jobs: int = Field(..., description="当前运行中的任务数")

View File

@@ -1,6 +1,5 @@
"""API 路由"""
import asyncio
from fastapi import APIRouter, HTTPException, Depends, status
import time
import logging
@@ -15,6 +14,7 @@ from .models import (
JobCreateResponse,
JobStatusResponse,
JobStatus,
ConcurrencyStatusResponse,
)
from .dependencies import get_request_id
from ..algorithms.prime_checker import PrimeChecker
@@ -199,10 +199,10 @@ async def create_job(
# 获取任务信息
job_data = await job_manager.get_job(job_id)
# 后台执行任务
asyncio.create_task(job_manager.execute_job(job_id))
# 任务入队,由 Worker 执行
await job_manager.enqueue_job(job_id)
logger.info(f"异步任务已创建: job_id={job_id}, request_id={request_id}")
logger.info(f"异步任务已创建并入队: job_id={job_id}, request_id={request_id}")
return JobCreateResponse(
job_id=job_id,
@@ -292,3 +292,57 @@ async def get_job_status(job_id: str):
"message": str(e),
},
)
@router.get(
"/jobs/concurrency/status",
response_model=ConcurrencyStatusResponse,
summary="查询并发状态",
description="查询任务管理器的并发执行状态",
responses={
200: {"description": "成功", "model": ConcurrencyStatusResponse},
503: {"description": "服务不可用", "model": ErrorResponse},
},
)
async def get_concurrency_status():
"""
查询并发状态
返回当前任务管理器的并发执行状态,包括:
- 最大并发任务数
- 当前可用槽位数
- 当前运行中的任务数
"""
try:
job_manager = await get_job_manager()
# 检查任务管理器是否可用
if not job_manager.is_available():
raise HTTPException(
status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
detail={
"error": "SERVICE_UNAVAILABLE",
"message": "任务管理器不可用",
},
)
concurrency_status = job_manager.get_concurrency_status()
return ConcurrencyStatusResponse(
max_concurrent=concurrency_status["max_concurrent"],
available_slots=concurrency_status["available_slots"],
running_jobs=concurrency_status["running_jobs"],
)
except HTTPException:
raise
except Exception as e:
logger.error(f"查询并发状态失败: {str(e)}", exc_info=True)
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail={
"error": "INTERNAL_ERROR",
"message": str(e),
},
)

View File

@@ -23,6 +23,8 @@ class Settings(BaseSettings):
# 日志配置
log_level: str = "INFO"
log_format: str = "json"
log_file_enabled: bool = False
log_file_path: str = "/var/log/app/app.log"
# 指标配置
metrics_enabled: bool = True
@@ -53,6 +55,27 @@ class Settings(BaseSettings):
job_result_ttl: int = 1800 # 结果缓存时间(秒),默认 30 分钟
webhook_max_retries: int = 3 # Webhook 最大重试次数
webhook_timeout: int = 10 # Webhook 超时时间(秒)
max_concurrent_jobs: int = 10 # 最大并发任务数
# Worker 配置
worker_poll_interval: float = 0.1 # Worker 轮询间隔(秒)
job_queue_key: str = "job:queue" # 任务队列 Redis Key
job_concurrency_key: str = "job:concurrency" # 全局并发计数器 Redis Key
job_lock_ttl: int = 300 # 任务锁 TTL
job_max_retries: int = 3 # 任务最大重试次数
job_execution_timeout: int = 300 # 任务执行超时(秒)
# 处理队列配置
job_processing_key: str = "job:processing" # 处理中队列
job_processing_ts_key: str = "job:processing:ts" # 处理时间戳 ZSET
job_dlq_key: str = "job:dlq" # 死信队列
# 锁配置扩展
job_lock_buffer: int = 60 # 锁 TTL 缓冲时间(秒)
# 回收器配置
job_sweeper_enabled: bool = True # 启用回收器
job_sweeper_interval: int = 60 # 回收扫描间隔(秒)
# 全局配置实例

View File

@@ -7,6 +7,7 @@ import asyncio
import json
import logging
import secrets
import time
from datetime import datetime, timezone
from typing import Any, Dict, List, Optional, Type
@@ -16,6 +17,7 @@ import redis.asyncio as aioredis
from ..algorithms.base import BaseAlgorithm
from ..config import settings
from ..core.metrics_unified import incr, observe
from ..core.tracing import set_request_id
logger = logging.getLogger(__name__)
@@ -23,10 +25,30 @@ logger = logging.getLogger(__name__)
class JobManager:
"""异步任务管理器"""
# Lua 脚本:安全释放锁(验证 token
RELEASE_LOCK_SCRIPT = """
local current = redis.call('GET', KEYS[1])
if current == ARGV[1] then
return redis.call('DEL', KEYS[1])
end
return 0
"""
# Lua 脚本:锁续租(验证 token 后延长 TTL
RENEW_LOCK_SCRIPT = """
local current = redis.call('GET', KEYS[1])
if current == ARGV[1] then
return redis.call('EXPIRE', KEYS[1], ARGV[2])
end
return 0
"""
def __init__(self):
self._redis_client: Optional[aioredis.Redis] = None
self._algorithm_registry: Dict[str, Type[BaseAlgorithm]] = {}
self._http_client: Optional[httpx.AsyncClient] = None
self._semaphore: Optional[asyncio.Semaphore] = None
self._max_concurrent_jobs: int = 0
async def initialize(self) -> None:
"""初始化 Redis 连接和 HTTP 客户端"""
@@ -51,6 +73,11 @@ class JobManager:
# 初始化 HTTP 客户端
self._http_client = httpx.AsyncClient(timeout=settings.webhook_timeout)
# 初始化并发控制信号量
self._max_concurrent_jobs = settings.max_concurrent_jobs
self._semaphore = asyncio.Semaphore(self._max_concurrent_jobs)
logger.info(f"任务并发限制已设置: {self._max_concurrent_jobs}")
# 注册算法
self._register_algorithms()
@@ -141,7 +168,7 @@ class JobManager:
await self._redis_client.hset(key, mapping=job_data)
# 记录指标
incr("jobs_created_total", {"algorithm": algorithm})
await incr("jobs_created_total", {"algorithm": algorithm})
logger.info(f"任务已创建: job_id={job_id}, algorithm={algorithm}")
return job_id
@@ -169,6 +196,7 @@ class JobManager:
"job_id": job_id,
"status": job_data.get("status", ""),
"algorithm": job_data.get("algorithm", ""),
"request_id": job_data.get("request_id") or None,
"created_at": job_data.get("created_at", ""),
"started_at": job_data.get("started_at") or None,
"completed_at": job_data.get("completed_at") or None,
@@ -203,6 +231,10 @@ class JobManager:
logger.error(f"Redis 不可用,无法执行任务: {job_id}")
return
if not self._semaphore:
logger.error(f"并发控制未初始化,无法执行任务: {job_id}")
return
key = f"job:{job_id}"
job_data = await self._redis_client.hgetall(key)
@@ -212,6 +244,11 @@ class JobManager:
algorithm_name = job_data.get("algorithm", "")
webhook_url = job_data.get("webhook", "")
request_id = job_data.get("request_id", "")
# 设置 request_id 上下文,确保日志中包含 request_id
if request_id:
set_request_id(request_id)
# 解析参数
try:
@@ -219,9 +256,13 @@ class JobManager:
except json.JSONDecodeError:
params = {}
# 使用信号量控制并发
async with self._semaphore:
# 更新状态为 running
started_at = self._get_timestamp()
await self._redis_client.hset(key, mapping={"status": "running", "started_at": started_at})
await self._redis_client.hset(
key, mapping={"status": "running", "started_at": started_at}
)
logger.info(f"开始执行任务: job_id={job_id}, algorithm={algorithm_name}")
@@ -279,10 +320,14 @@ class JobManager:
await self._redis_client.expire(key, settings.job_result_ttl)
# 记录指标
incr("jobs_completed_total", {"algorithm": algorithm_name, "status": status})
observe("job_execution_duration_seconds", {"algorithm": algorithm_name}, elapsed_time)
await incr("jobs_completed_total", {"algorithm": algorithm_name, "status": status})
await observe(
"job_execution_duration_seconds", {"algorithm": algorithm_name}, elapsed_time
)
logger.info(f"任务执行完成: job_id={job_id}, status={status}, elapsed={elapsed_time:.3f}s")
logger.info(
f"任务执行完成: job_id={job_id}, status={status}, elapsed={elapsed_time:.3f}s"
)
# 发送 Webhook 回调
if webhook_url:
@@ -329,7 +374,7 @@ class JobManager:
)
if response.status_code < 400:
incr("webhook_deliveries_total", {"status": "success"})
await incr("webhook_deliveries_total", {"status": "success"})
logger.info(
f"Webhook 发送成功: job_id={job_id}, url={webhook_url}, "
f"status_code={response.status_code}"
@@ -352,13 +397,445 @@ class JobManager:
await asyncio.sleep(delay)
# 所有重试都失败
incr("webhook_deliveries_total", {"status": "failed"})
await incr("webhook_deliveries_total", {"status": "failed"})
logger.error(f"Webhook 发送最终失败: job_id={job_id}, url={webhook_url}")
def is_available(self) -> bool:
"""检查任务管理器是否可用"""
return self._redis_client is not None
async def enqueue_job(self, job_id: str) -> bool:
"""将任务加入队列
Args:
job_id: 任务 ID
Returns:
bool: 是否成功入队
"""
if not self._redis_client:
logger.error(f"Redis 不可用,无法入队任务: {job_id}")
return False
try:
await self._redis_client.lpush(settings.job_queue_key, job_id)
logger.info(f"任务已入队: job_id={job_id}")
return True
except Exception as e:
logger.error(f"任务入队失败: job_id={job_id}, error={e}")
return False
async def dequeue_job(self, timeout: int = 5) -> Optional[str]:
"""从队列获取任务(阻塞式,转移式出队)
使用 BLMOVE 原子性地将任务从 job:queue 移动到 job:processing
防止 Worker 崩溃时任务丢失。
Args:
timeout: 阻塞超时时间(秒)
Returns:
Optional[str]: 任务 ID超时返回 None
"""
if not self._redis_client:
return None
try:
# 使用 BLMOVE 原子性转移任务
job_id = await self._redis_client.blmove(
settings.job_queue_key, # 源: job:queue
settings.job_processing_key, # 目标: job:processing
timeout,
"RIGHT",
"LEFT",
)
if job_id:
# 记录出队时间戳到 ZSET
await self._redis_client.zadd(settings.job_processing_ts_key, {job_id: time.time()})
logger.debug(f"任务已转移到处理队列: {job_id}")
return job_id
except Exception as e:
logger.error(f"任务出队失败: error={e}")
return None
async def acquire_job_lock(self, job_id: str) -> Optional[str]:
"""获取任务执行锁(分布式锁,带 Token
Args:
job_id: 任务 ID
Returns:
Optional[str]: 成功时返回锁 token失败返回 None
"""
if not self._redis_client:
return None
lock_key = f"job:lock:{job_id}"
lock_token = secrets.token_hex(16) # 随机 token
lock_ttl = settings.job_execution_timeout + settings.job_lock_buffer
try:
acquired = await self._redis_client.set(lock_key, lock_token, nx=True, ex=lock_ttl)
if acquired:
logger.debug(f"获取任务锁成功: job_id={job_id}")
return lock_token
return None
except Exception as e:
logger.error(f"获取任务锁失败: job_id={job_id}, error={e}")
return None
async def release_job_lock(self, job_id: str, lock_token: Optional[str] = None) -> bool:
"""释放任务执行锁(使用 Lua 脚本验证 token
Args:
job_id: 任务 ID
lock_token: 锁 token用于验证所有权
Returns:
bool: 是否成功释放锁
"""
if not self._redis_client:
return False
lock_key = f"job:lock:{job_id}"
try:
if lock_token:
# 使用 Lua 脚本安全释放锁
result = await self._redis_client.eval(
self.RELEASE_LOCK_SCRIPT, 1, lock_key, lock_token
)
if result == 1:
logger.debug(f"释放任务锁成功: job_id={job_id}")
return True
else:
logger.warning(f"释放任务锁失败token 不匹配): job_id={job_id}")
return False
else:
# 向后兼容:无 token 时直接删除
await self._redis_client.delete(lock_key)
logger.debug(f"释放任务锁成功(无 token 验证): job_id={job_id}")
return True
except Exception as e:
logger.error(f"释放任务锁失败: job_id={job_id}, error={e}")
return False
async def increment_concurrency(self) -> int:
"""增加全局并发计数
Returns:
int: 增加后的并发数
"""
if not self._redis_client:
return 0
try:
count = await self._redis_client.incr(settings.job_concurrency_key)
return count
except Exception as e:
logger.error(f"增加并发计数失败: error={e}")
return 0
async def decrement_concurrency(self) -> int:
"""减少全局并发计数
Returns:
int: 减少后的并发数
"""
if not self._redis_client:
return 0
try:
count = await self._redis_client.decr(settings.job_concurrency_key)
# 防止计数变为负数
if count < 0:
await self._redis_client.set(settings.job_concurrency_key, 0)
return 0
return count
except Exception as e:
logger.error(f"减少并发计数失败: error={e}")
return 0
async def get_global_concurrency(self) -> int:
"""获取当前全局并发数
Returns:
int: 当前并发数
"""
if not self._redis_client:
return 0
try:
count = await self._redis_client.get(settings.job_concurrency_key)
return int(count) if count else 0
except Exception as e:
logger.error(f"获取并发计数失败: error={e}")
return 0
async def can_execute(self) -> bool:
"""检查是否可以执行新任务(全局并发控制)
Returns:
bool: 是否可以执行
"""
current = await self.get_global_concurrency()
return current < settings.max_concurrent_jobs
async def get_job_retry_count(self, job_id: str) -> int:
"""获取任务重试次数
Args:
job_id: 任务 ID
Returns:
int: 重试次数
"""
if not self._redis_client:
return 0
key = f"job:{job_id}"
try:
retry_count = await self._redis_client.hget(key, "retry_count")
return int(retry_count) if retry_count else 0
except Exception:
return 0
async def increment_job_retry(self, job_id: str) -> int:
"""增加任务重试次数
Args:
job_id: 任务 ID
Returns:
int: 增加后的重试次数
"""
if not self._redis_client:
return 0
key = f"job:{job_id}"
try:
await self._redis_client.hincrby(key, "retry_count", 1)
retry_count = await self._redis_client.hget(key, "retry_count")
return int(retry_count) if retry_count else 1
except Exception as e:
logger.error(f"增加重试次数失败: job_id={job_id}, error={e}")
return 0
async def ack_job(self, job_id: str) -> bool:
"""确认任务完成(从处理队列移除)
Args:
job_id: 任务 ID
Returns:
bool: 是否成功确认
"""
if not self._redis_client:
return False
try:
async with self._redis_client.pipeline(transaction=True) as pipe:
pipe.lrem(settings.job_processing_key, 1, job_id)
pipe.zrem(settings.job_processing_ts_key, job_id)
await pipe.execute()
logger.debug(f"任务已确认完成: job_id={job_id}")
return True
except Exception as e:
logger.error(f"确认任务失败: job_id={job_id}, error={e}")
return False
async def nack_job(self, job_id: str, requeue: bool = True) -> bool:
"""拒绝任务(从处理队列移除,根据重试次数决定重新入队或进死信队列)
Args:
job_id: 任务 ID
requeue: 是否尝试重新入队
Returns:
bool: 是否成功处理
"""
if not self._redis_client:
return False
try:
retry_count = await self.get_job_retry_count(job_id)
async with self._redis_client.pipeline(transaction=True) as pipe:
pipe.lrem(settings.job_processing_key, 1, job_id)
pipe.zrem(settings.job_processing_ts_key, job_id)
if requeue and retry_count < settings.job_max_retries:
pipe.lpush(settings.job_queue_key, job_id)
logger.info(f"任务重新入队: job_id={job_id}, retry_count={retry_count}")
else:
pipe.lpush(settings.job_dlq_key, job_id)
logger.warning(f"任务进入死信队列: job_id={job_id}, retry_count={retry_count}")
await pipe.execute()
return True
except Exception as e:
logger.error(f"拒绝任务失败: job_id={job_id}, error={e}")
return False
async def renew_job_lock(self, job_id: str, lock_token: str) -> bool:
"""续租任务锁(延长 TTL
Args:
job_id: 任务 ID
lock_token: 锁 token
Returns:
bool: 是否成功续租
"""
if not self._redis_client:
return False
lock_key = f"job:lock:{job_id}"
lock_ttl = settings.job_execution_timeout + settings.job_lock_buffer
try:
result = await self._redis_client.eval(
self.RENEW_LOCK_SCRIPT, 1, lock_key, lock_token, lock_ttl
)
if result == 1:
logger.debug(f"锁续租成功: job_id={job_id}")
return True
else:
logger.warning(f"锁续租失败token 不匹配或锁已过期): job_id={job_id}")
return False
except Exception as e:
logger.error(f"锁续租失败: job_id={job_id}, error={e}")
return False
async def recover_stale_jobs(self) -> int:
"""回收超时任务
扫描 job:processing:ts ZSET找出超时的任务
根据重试次数决定重新入队或进死信队列。
Returns:
int: 回收的任务数量
"""
if not self._redis_client:
return 0
timeout = settings.job_execution_timeout + settings.job_lock_buffer
cutoff = time.time() - timeout
try:
# 获取超时任务列表
stale_jobs = await self._redis_client.zrangebyscore(
settings.job_processing_ts_key, "-inf", cutoff
)
recovered = 0
for job_id in stale_jobs:
# 增加重试次数
await self.increment_job_retry(job_id)
retry_count = await self.get_job_retry_count(job_id)
async with self._redis_client.pipeline(transaction=True) as pipe:
pipe.lrem(settings.job_processing_key, 1, job_id)
pipe.zrem(settings.job_processing_ts_key, job_id)
if retry_count < settings.job_max_retries:
pipe.lpush(settings.job_queue_key, job_id)
logger.info(f"超时任务重新入队: job_id={job_id}, retry_count={retry_count}")
else:
pipe.lpush(settings.job_dlq_key, job_id)
logger.warning(
f"超时任务进入死信队列: job_id={job_id}, retry_count={retry_count}"
)
await pipe.execute()
recovered += 1
if recovered > 0:
logger.info(f"回收超时任务完成: 共 {recovered}")
return recovered
except Exception as e:
logger.error(f"回收超时任务失败: error={e}")
return 0
def get_concurrency_status(self) -> Dict[str, int]:
"""获取并发状态
Returns:
Dict[str, int]: 包含以下键的字典
- max_concurrent: 最大并发数
- available_slots: 可用槽位数
- running_jobs: 当前运行中的任务数
"""
if not self._semaphore:
return {
"max_concurrent": 0,
"available_slots": 0,
"running_jobs": 0,
}
max_concurrent = self._max_concurrent_jobs
available_slots = self._semaphore._value
running_jobs = max_concurrent - available_slots
return {
"max_concurrent": max_concurrent,
"available_slots": available_slots,
"running_jobs": running_jobs,
}
async def collect_queue_metrics(self) -> Dict[str, Any]:
"""收集队列监控指标
Returns:
Dict[str, Any]: 包含以下键的字典
- queue_length: 待处理队列长度
- processing_length: 处理中队列长度
- dlq_length: 死信队列长度
- oldest_waiting_seconds: 最长等待时间(秒)
"""
if not self._redis_client:
return {
"queue_length": 0,
"processing_length": 0,
"dlq_length": 0,
"oldest_waiting_seconds": 0,
}
try:
# 使用 pipeline 批量获取队列长度
async with self._redis_client.pipeline(transaction=False) as pipe:
pipe.llen(settings.job_queue_key)
pipe.llen(settings.job_processing_key)
pipe.llen(settings.job_dlq_key)
pipe.zrange(settings.job_processing_ts_key, 0, 0, withscores=True)
results = await pipe.execute()
queue_length = results[0] or 0
processing_length = results[1] or 0
dlq_length = results[2] or 0
# 计算最长等待时间
oldest_waiting_seconds = 0
if results[3]:
# results[3] 是 [(job_id, timestamp), ...] 格式
oldest_ts = results[3][0][1]
oldest_waiting_seconds = time.time() - oldest_ts
# 更新指标
from .metrics_unified import set as metrics_set
await metrics_set("job_queue_length", {"queue": "pending"}, queue_length)
await metrics_set("job_queue_length", {"queue": "processing"}, processing_length)
await metrics_set("job_queue_length", {"queue": "dlq"}, dlq_length)
await metrics_set("job_oldest_waiting_seconds", None, oldest_waiting_seconds)
return {
"queue_length": queue_length,
"processing_length": processing_length,
"dlq_length": dlq_length,
"oldest_waiting_seconds": oldest_waiting_seconds,
}
except Exception as e:
logger.error(f"收集队列指标失败: error={e}")
return {
"queue_length": 0,
"processing_length": 0,
"dlq_length": 0,
"oldest_waiting_seconds": 0,
}
# 全局单例
_job_manager: Optional[JobManager] = None

View File

@@ -2,14 +2,39 @@
import logging
import sys
from pathlib import Path
from typing import Optional
from logging.handlers import RotatingFileHandler
from pythonjsonlogger.json import JsonFormatter
from .tracing import get_request_id
class RequestIdFilter(logging.Filter):
"""自动添加 request_id 到日志记录的过滤器"""
def filter(self, record: logging.LogRecord) -> bool:
"""
为日志记录添加 request_id 字段
Args:
record: 日志记录
Returns:
bool: 总是返回 True不过滤任何日志
"""
# 从 ContextVar 中获取 request_id
request_id = get_request_id()
# 添加到日志记录中,如果没有则设置为 None
record.request_id = request_id if request_id else "-"
return True
def setup_logging(
level: str = "INFO",
format_type: str = "json",
logger_name: Optional[str] = None,
file_path: Optional[str] = None,
) -> logging.Logger:
"""
配置日志系统
@@ -18,6 +43,7 @@ def setup_logging(
level: 日志级别 (DEBUG, INFO, WARNING, ERROR, CRITICAL)
format_type: 日志格式 ('json''text')
logger_name: 日志器名称None表示根日志器
file_path: 日志文件路径None表示不写入文件
Returns:
logging.Logger: 配置好的日志器
@@ -28,23 +54,45 @@ def setup_logging(
# 清除现有处理器
logger.handlers.clear()
# 创建控制台处理器
handler = logging.StreamHandler(sys.stdout)
handler.setLevel(getattr(logging, level.upper()))
# 设置格式
if format_type == "json":
formatter = JsonFormatter(
"%(asctime)s %(name)s %(levelname)s %(message)s",
"%(asctime)s %(name)s %(levelname)s %(message)s %(request_id)s",
timestamp=True,
)
else:
formatter = logging.Formatter(
"%(asctime)s - %(name)s - %(levelname)s - %(message)s",
"%(asctime)s - %(name)s - %(levelname)s - [%(request_id)s] - %(message)s",
datefmt="%Y-%m-%d %H:%M:%S",
)
handler.setFormatter(formatter)
logger.addHandler(handler)
# 创建 RequestIdFilter
request_id_filter = RequestIdFilter()
# 创建控制台处理器
console_handler = logging.StreamHandler(sys.stdout)
console_handler.setLevel(getattr(logging, level.upper()))
console_handler.setFormatter(formatter)
console_handler.addFilter(request_id_filter)
logger.addHandler(console_handler)
# 创建文件处理器(如果指定了文件路径)
if file_path:
# 确保日志目录存在
log_dir = Path(file_path).parent
log_dir.mkdir(parents=True, exist_ok=True)
# 创建 RotatingFileHandler
# 最大 100MB保留 5 个备份
file_handler = RotatingFileHandler(
file_path,
maxBytes=100 * 1024 * 1024, # 100MB
backupCount=5,
encoding="utf-8",
)
file_handler.setLevel(getattr(logging, level.upper()))
file_handler.setFormatter(formatter)
file_handler.addFilter(request_id_filter)
logger.addHandler(file_handler)
return logger

View File

@@ -1,19 +1,21 @@
"""统一指标管理模块
基于 Redis 的指标收集方案,支持多实例部署和 YAML 配置。
使用异步 Redis 客户端,避免在异步请求路径中阻塞事件循环。
"""
import os
import re
import socket
import logging
import asyncio
from pathlib import Path
from typing import Any, Dict, List, Optional
from functools import wraps
import time
import yaml
import redis
import redis.asyncio as aioredis
logger = logging.getLogger(__name__)
@@ -22,7 +24,7 @@ class MetricsManager:
"""统一指标管理器
支持从 YAML 配置文件加载指标定义,使用 Redis 存储指标数据,
并导出 Prometheus 格式的指标。
并导出 Prometheus 格式的指标。使用异步 Redis 客户端。
"""
def __init__(self, config_path: Optional[str] = None):
@@ -37,16 +39,22 @@ class MetricsManager:
self.instance_id = settings.metrics_instance_id or socket.gethostname()
self.config: Dict[str, Any] = {}
self.metrics_definitions: Dict[str, Dict[str, Any]] = {}
self._redis_client: Optional[redis.Redis] = None
self._redis_client: Optional[aioredis.Redis] = None
self._redis_available = False
self._initialized = False
# 加载配置
# 加载配置(同步操作)
self._load_config()
# 初始化 Redis 连接
self._init_redis()
# 注册指标定义
# 注册指标定义(同步操作)
self._register_metrics()
async def initialize(self) -> None:
"""异步初始化 Redis 连接"""
if self._initialized:
return
await self._init_redis()
self._initialized = True
def _load_config(self) -> None:
"""加载 YAML 配置文件"""
# 尝试多个路径
@@ -138,8 +146,8 @@ class MetricsManager:
"custom_metrics": {},
}
def _init_redis(self) -> None:
"""初始化 Redis 连接"""
async def _init_redis(self) -> None:
"""异步初始化 Redis 连接"""
from ..config import settings
redis_config = self.config.get("redis", {})
@@ -149,7 +157,7 @@ class MetricsManager:
password = redis_config.get("password") or settings.redis_password
try:
self._redis_client = redis.Redis(
self._redis_client = aioredis.Redis(
host=host,
port=port,
db=db,
@@ -159,10 +167,10 @@ class MetricsManager:
socket_timeout=5,
)
# 测试连接
self._redis_client.ping()
await self._redis_client.ping()
self._redis_available = True
logger.info(f"Redis 连接成功: {host}:{port}/{db}")
except redis.ConnectionError as e:
except aioredis.ConnectionError as e:
logger.warning(f"Redis 连接失败: {e},指标将不会被收集")
self._redis_available = False
except Exception as e:
@@ -235,7 +243,9 @@ class MetricsManager:
# === 简单 API业务代码使用===
def incr(self, name: str, labels: Optional[Dict[str, str]] = None, value: int = 1) -> None:
async def incr(
self, name: str, labels: Optional[Dict[str, str]] = None, value: int = 1
) -> None:
"""增加计数器
Args:
@@ -252,11 +262,13 @@ class MetricsManager:
try:
key = f"metrics:counter:{name}"
field = self._labels_to_key(labels) or "_default_"
self._redis_client.hincrbyfloat(key, field, value)
await self._redis_client.hincrbyfloat(key, field, value)
except Exception as e:
logger.error(f"增加计数器失败: {e}")
def set(self, name: str, labels: Optional[Dict[str, str]] = None, value: float = 0) -> None:
async def set(
self, name: str, labels: Optional[Dict[str, str]] = None, value: float = 0
) -> None:
"""设置仪表盘值
Args:
@@ -273,11 +285,11 @@ class MetricsManager:
try:
key = f"metrics:gauge:{name}"
field = self._labels_to_key(labels) or "_default_"
self._redis_client.hset(key, field, value)
await self._redis_client.hset(key, field, value)
except Exception as e:
logger.error(f"设置仪表盘失败: {e}")
def gauge_incr(
async def gauge_incr(
self, name: str, labels: Optional[Dict[str, str]] = None, value: float = 1
) -> None:
"""增加仪表盘值
@@ -296,11 +308,11 @@ class MetricsManager:
try:
key = f"metrics:gauge:{name}"
field = self._labels_to_key(labels) or "_default_"
self._redis_client.hincrbyfloat(key, field, value)
await self._redis_client.hincrbyfloat(key, field, value)
except Exception as e:
logger.error(f"增加仪表盘失败: {e}")
def gauge_decr(
async def gauge_decr(
self, name: str, labels: Optional[Dict[str, str]] = None, value: float = 1
) -> None:
"""减少仪表盘值
@@ -310,9 +322,11 @@ class MetricsManager:
labels: 标签字典
value: 减少的值
"""
self.gauge_incr(name, labels, -value)
await self.gauge_incr(name, labels, -value)
def observe(self, name: str, labels: Optional[Dict[str, str]] = None, value: float = 0) -> None:
async def observe(
self, name: str, labels: Optional[Dict[str, str]] = None, value: float = 0
) -> None:
"""记录直方图观测值
Args:
@@ -348,13 +362,13 @@ class MetricsManager:
# +Inf 桶总是增加
pipe.hincrbyfloat(f"metrics:histogram:{name}:bucket:+Inf", label_key, 1)
pipe.execute()
await pipe.execute()
except Exception as e:
logger.error(f"记录直方图失败: {e}")
# === 导出方法 ===
def export(self) -> str:
async def export(self) -> str:
"""导出 Prometheus 格式指标
Returns:
@@ -375,11 +389,11 @@ class MetricsManager:
lines.append(f"# TYPE {name} {metric_type}")
if metric_type == "counter":
lines.extend(self._export_counter(name))
lines.extend(await self._export_counter(name))
elif metric_type == "gauge":
lines.extend(self._export_gauge(name))
lines.extend(await self._export_gauge(name))
elif metric_type == "histogram":
lines.extend(self._export_histogram(name, definition))
lines.extend(await self._export_histogram(name, definition))
lines.append("") # 空行分隔
@@ -389,12 +403,12 @@ class MetricsManager:
return "\n".join(lines)
def _export_counter(self, name: str) -> List[str]:
async def _export_counter(self, name: str) -> List[str]:
"""导出计数器指标"""
lines = []
key = f"metrics:counter:{name}"
data = self._redis_client.hgetall(key)
data = await self._redis_client.hgetall(key)
for field, value in data.items():
if field == "_default_":
lines.append(f"{name} {value}")
@@ -404,12 +418,12 @@ class MetricsManager:
return lines
def _export_gauge(self, name: str) -> List[str]:
async def _export_gauge(self, name: str) -> List[str]:
"""导出仪表盘指标"""
lines = []
key = f"metrics:gauge:{name}"
data = self._redis_client.hgetall(key)
data = await self._redis_client.hgetall(key)
for field, value in data.items():
if field == "_default_":
lines.append(f"{name} {value}")
@@ -419,14 +433,14 @@ class MetricsManager:
return lines
def _export_histogram(self, name: str, definition: Dict[str, Any]) -> List[str]:
async def _export_histogram(self, name: str, definition: Dict[str, Any]) -> List[str]:
"""导出直方图指标"""
lines = []
buckets = definition.get("buckets", [])
# 获取所有标签组合
count_data = self._redis_client.hgetall(f"metrics:histogram:{name}:count")
sum_data = self._redis_client.hgetall(f"metrics:histogram:{name}:sum")
count_data = await self._redis_client.hgetall(f"metrics:histogram:{name}:count")
sum_data = await self._redis_client.hgetall(f"metrics:histogram:{name}:sum")
for label_key in count_data.keys():
prom_labels = self._key_to_prometheus_labels(label_key)
@@ -434,7 +448,7 @@ class MetricsManager:
# 导出各个桶
for bucket in buckets:
bucket_key = f"metrics:histogram:{name}:bucket:{bucket}"
bucket_value = self._redis_client.hget(bucket_key, label_key) or "0"
bucket_value = await self._redis_client.hget(bucket_key, label_key) or "0"
if label_key == "_default_":
lines.append(f'{name}_bucket{{le="{bucket}"}} {bucket_value}')
else:
@@ -442,7 +456,7 @@ class MetricsManager:
# +Inf 桶
inf_key = f"metrics:histogram:{name}:bucket:+Inf"
inf_value = self._redis_client.hget(inf_key, label_key) or "0"
inf_value = await self._redis_client.hget(inf_key, label_key) or "0"
if label_key == "_default_":
lines.append(f'{name}_bucket{{le="+Inf"}} {inf_value}')
else:
@@ -464,43 +478,79 @@ class MetricsManager:
"""检查 Redis 是否可用"""
return self._redis_available
def reset(self) -> None:
async def reset(self) -> None:
"""重置所有指标(主要用于测试)"""
if not self._redis_available:
return
try:
# 删除所有指标相关的 key
keys = self._redis_client.keys("metrics:*")
keys = await self._redis_client.keys("metrics:*")
if keys:
self._redis_client.delete(*keys)
await self._redis_client.delete(*keys)
logger.info("已重置所有指标")
except Exception as e:
logger.error(f"重置指标失败: {e}")
async def close(self) -> None:
"""关闭 Redis 连接"""
if self._redis_client:
await self._redis_client.close()
self._redis_client = None
self._redis_available = False
self._initialized = False
# 全局单例
_manager: Optional[MetricsManager] = None
_manager_lock = asyncio.Lock()
def get_metrics_manager() -> MetricsManager:
"""获取指标管理器单例"""
async def get_metrics_manager() -> MetricsManager:
"""获取指标管理器单例(异步)"""
global _manager
if _manager is None:
async with _manager_lock:
if _manager is None:
_manager = MetricsManager()
await _manager.initialize()
elif not _manager._initialized:
await _manager.initialize()
return _manager
def get_metrics_manager_sync() -> MetricsManager:
"""获取指标管理器单例(同步,仅用于非异步上下文)
注意:此方法不会初始化 Redis 连接,需要在异步上下文中调用 initialize()
"""
global _manager
if _manager is None:
_manager = MetricsManager()
return _manager
def reset_metrics_manager() -> None:
async def reset_metrics_manager() -> None:
"""重置指标管理器单例(主要用于测试)"""
global _manager
if _manager is not None:
await _manager.close()
_manager = None
def reset_metrics_manager_sync() -> None:
"""同步重置指标管理器单例(主要用于测试)
注意:此方法不会关闭 Redis 连接,仅重置单例引用
"""
global _manager
_manager = None
# === 便捷函数(业务代码直接调用)===
def incr(name: str, labels: Optional[Dict[str, str]] = None, value: int = 1) -> None:
async def incr(name: str, labels: Optional[Dict[str, str]] = None, value: int = 1) -> None:
"""增加计数器 - 便捷函数
Args:
@@ -508,10 +558,11 @@ def incr(name: str, labels: Optional[Dict[str, str]] = None, value: int = 1) ->
labels: 标签字典
value: 增加的值,默认为 1
"""
get_metrics_manager().incr(name, labels, value)
manager = await get_metrics_manager()
await manager.incr(name, labels, value)
def set(name: str, labels: Optional[Dict[str, str]] = None, value: float = 0) -> None:
async def set(name: str, labels: Optional[Dict[str, str]] = None, value: float = 0) -> None:
"""设置仪表盘 - 便捷函数
Args:
@@ -519,10 +570,13 @@ def set(name: str, labels: Optional[Dict[str, str]] = None, value: float = 0) ->
labels: 标签字典
value: 设置的值
"""
get_metrics_manager().set(name, labels, value)
manager = await get_metrics_manager()
await manager.set(name, labels, value)
def gauge_incr(name: str, labels: Optional[Dict[str, str]] = None, value: float = 1) -> None:
async def gauge_incr(
name: str, labels: Optional[Dict[str, str]] = None, value: float = 1
) -> None:
"""增加仪表盘 - 便捷函数
Args:
@@ -530,10 +584,13 @@ def gauge_incr(name: str, labels: Optional[Dict[str, str]] = None, value: float
labels: 标签字典
value: 增加的值
"""
get_metrics_manager().gauge_incr(name, labels, value)
manager = await get_metrics_manager()
await manager.gauge_incr(name, labels, value)
def gauge_decr(name: str, labels: Optional[Dict[str, str]] = None, value: float = 1) -> None:
async def gauge_decr(
name: str, labels: Optional[Dict[str, str]] = None, value: float = 1
) -> None:
"""减少仪表盘 - 便捷函数
Args:
@@ -541,10 +598,13 @@ def gauge_decr(name: str, labels: Optional[Dict[str, str]] = None, value: float
labels: 标签字典
value: 减少的值
"""
get_metrics_manager().gauge_decr(name, labels, value)
manager = await get_metrics_manager()
await manager.gauge_decr(name, labels, value)
def observe(name: str, labels: Optional[Dict[str, str]] = None, value: float = 0) -> None:
async def observe(
name: str, labels: Optional[Dict[str, str]] = None, value: float = 0
) -> None:
"""记录直方图 - 便捷函数
Args:
@@ -552,21 +612,105 @@ def observe(name: str, labels: Optional[Dict[str, str]] = None, value: float = 0
labels: 标签字典
value: 观测值
"""
get_metrics_manager().observe(name, labels, value)
manager = await get_metrics_manager()
await manager.observe(name, labels, value)
def export() -> str:
async def export() -> str:
"""导出指标 - 便捷函数
Returns:
Prometheus 文本格式的指标字符串
"""
return get_metrics_manager().export()
manager = await get_metrics_manager()
return await manager.export()
def is_available() -> bool:
async def is_available() -> bool:
"""检查 Redis 是否可用 - 便捷函数"""
return get_metrics_manager().is_available()
manager = await get_metrics_manager()
return manager.is_available()
# === 同步便捷函数(用于同步代码中的 fire-and-forget 模式)===
def _schedule_async(coro) -> None:
"""在后台调度异步协程fire-and-forget 模式)
如果当前没有运行的事件循环,则静默忽略。
"""
try:
loop = asyncio.get_running_loop()
loop.create_task(coro)
except RuntimeError:
# 没有运行的事件循环,静默忽略
pass
def incr_sync(
name: str, labels: Optional[Dict[str, str]] = None, value: int = 1
) -> None:
"""增加计数器 - 同步便捷函数fire-and-forget
Args:
name: 指标名称
labels: 标签字典
value: 增加的值,默认为 1
"""
_schedule_async(incr(name, labels, value))
def set_sync(
name: str, labels: Optional[Dict[str, str]] = None, value: float = 0
) -> None:
"""设置仪表盘 - 同步便捷函数fire-and-forget
Args:
name: 指标名称
labels: 标签字典
value: 设置的值
"""
_schedule_async(set(name, labels, value))
def gauge_incr_sync(
name: str, labels: Optional[Dict[str, str]] = None, value: float = 1
) -> None:
"""增加仪表盘 - 同步便捷函数fire-and-forget
Args:
name: 指标名称
labels: 标签字典
value: 增加的值
"""
_schedule_async(gauge_incr(name, labels, value))
def gauge_decr_sync(
name: str, labels: Optional[Dict[str, str]] = None, value: float = 1
) -> None:
"""减少仪表盘 - 同步便捷函数fire-and-forget
Args:
name: 指标名称
labels: 标签字典
value: 减少的值
"""
_schedule_async(gauge_decr(name, labels, value))
def observe_sync(
name: str, labels: Optional[Dict[str, str]] = None, value: float = 0
) -> None:
"""记录直方图 - 同步便捷函数fire-and-forget
Args:
name: 指标名称
labels: 标签字典
value: 观测值
"""
_schedule_async(observe(name, labels, value))
# === 装饰器(兼容旧 API===
@@ -593,8 +737,11 @@ def track_algorithm_execution(algorithm_name: str):
raise e
finally:
elapsed = time.time() - start_time
incr("algorithm_executions_total", {"algorithm": algorithm_name, "status": status})
observe(
incr_sync(
"algorithm_executions_total",
{"algorithm": algorithm_name, "status": status},
)
observe_sync(
"algorithm_execution_duration_seconds",
{"algorithm": algorithm_name},
elapsed,

View File

@@ -9,6 +9,7 @@ import time
from .api import router
from .config import settings
from .core.logging import setup_logging
from .core.tracing import generate_request_id, set_request_id, get_request_id
from .core.metrics_unified import (
get_metrics_manager,
incr,
@@ -20,7 +21,11 @@ from .core.metrics_unified import (
from .core.job_manager import get_job_manager, shutdown_job_manager
# 设置日志
setup_logging(level=settings.log_level, format_type=settings.log_format)
setup_logging(
level=settings.log_level,
format_type=settings.log_format,
file_path=settings.log_file_path if settings.log_file_enabled else None,
)
logger = logging.getLogger(__name__)
# 创建 FastAPI 应用
@@ -47,12 +52,37 @@ app.add_middleware(
@app.middleware("http")
async def log_requests(request: Request, call_next):
"""记录所有HTTP请求"""
# 从请求头获取或生成 request_id
request_id = request.headers.get("x-request-id") or generate_request_id()
set_request_id(request_id)
logger.info(f"Request: {request.method} {request.url.path}")
response = await call_next(request)
logger.info(f"Response: {response.status_code}")
return response
def normalize_path(path: str) -> str:
"""
规范化路径,将路径参数替换为模板形式
Args:
path: 原始路径
Returns:
规范化后的路径
Examples:
/jobs/a1b2c3d4e5f6 -> /jobs/{job_id}
/invoke -> /invoke
"""
# 匹配 /jobs/{任意字符串} 模式
if path.startswith("/jobs/") and len(path) > 6:
return "/jobs/{job_id}"
return path
# 指标跟踪中间件
@app.middleware("http")
async def track_metrics(request: Request, call_next):
@@ -60,11 +90,12 @@ async def track_metrics(request: Request, call_next):
if not settings.metrics_enabled:
return await call_next(request)
# 跳过 /metrics 端点本身,避免循环记录
if request.url.path == "/metrics":
# 跳过不需要记录指标的端点
skip_paths = {"/metrics", "/readyz", "/healthz"}
if request.url.path in skip_paths:
return await call_next(request)
gauge_incr("http_requests_in_progress")
await gauge_incr("http_requests_in_progress")
start_time = time.time()
status = "success"
@@ -79,16 +110,18 @@ async def track_metrics(request: Request, call_next):
raise e
finally:
elapsed = time.time() - start_time
incr(
# 使用规范化后的路径记录指标
normalized_path = normalize_path(request.url.path)
await incr(
"http_requests_total",
{"method": request.method, "endpoint": request.url.path, "status": status},
{"method": request.method, "endpoint": normalized_path, "status": status},
)
observe(
await observe(
"http_request_duration_seconds",
{"method": request.method, "endpoint": request.url.path},
{"method": request.method, "endpoint": normalized_path},
elapsed,
)
gauge_decr("http_requests_in_progress")
await gauge_decr("http_requests_in_progress")
# 注册路由
@@ -112,7 +145,7 @@ async def metrics():
return Response(content="Metrics disabled", status_code=404)
return Response(
content=export(),
content=await export(),
media_type="text/plain; version=0.0.4; charset=utf-8",
)
@@ -127,7 +160,7 @@ async def startup_event():
# 初始化指标管理器
if settings.metrics_enabled:
manager = get_metrics_manager()
manager = await get_metrics_manager()
if manager.is_available():
logger.info("Redis 指标收集已启用")
else:

View File

@@ -0,0 +1,373 @@
"""Worker 进程模块
基于 Redis 队列的任务 Worker支持分布式锁和全局并发控制。
"""
import asyncio
import logging
import signal
import sys
from typing import Optional
from aiohttp import web
from .config import settings
from .core.job_manager import JobManager
from .core.logging import setup_logging
from .core.tracing import set_request_id
logger = logging.getLogger(__name__)
class HealthCheckServer:
"""轻量级健康检查 HTTP 服务器
为 Worker 模式提供健康检查端点,满足 FC 3.0 容器健康检查要求。
"""
def __init__(self, host: str = "0.0.0.0", port: int = 8000):
self._host = host
self._port = port
self._app: Optional[web.Application] = None
self._runner: Optional[web.AppRunner] = None
self._site: Optional[web.TCPSite] = None
self._healthy = True
async def start(self) -> None:
"""启动健康检查服务器"""
self._app = web.Application()
self._app.router.add_get("/healthz", self._healthz_handler)
self._app.router.add_get("/readyz", self._readyz_handler)
self._runner = web.AppRunner(self._app)
await self._runner.setup()
self._site = web.TCPSite(self._runner, self._host, self._port)
await self._site.start()
logger.info(f"健康检查服务器已启动: http://{self._host}:{self._port}")
async def stop(self) -> None:
"""停止健康检查服务器"""
if self._runner:
await self._runner.cleanup()
logger.info("健康检查服务器已停止")
def set_healthy(self, healthy: bool) -> None:
"""设置健康状态"""
self._healthy = healthy
async def _healthz_handler(self, request: web.Request) -> web.Response:
"""存活检查端点"""
return web.json_response({"status": "healthy", "mode": "worker"})
async def _readyz_handler(self, request: web.Request) -> web.Response:
"""就绪检查端点"""
if self._healthy:
return web.json_response({"status": "ready", "mode": "worker"})
return web.json_response({"status": "not ready"}, status=503)
class JobWorker:
"""任务 Worker
从 Redis 队列获取任务并执行,支持:
- 分布式锁防止重复执行
- 全局并发控制
- 任务重试机制
- 锁续租机制
- 超时任务回收
- 优雅关闭
"""
def __init__(self):
self._job_manager: Optional[JobManager] = None
self._running: bool = False
self._current_job_id: Optional[str] = None
self._current_lock_token: Optional[str] = None
self._lock_renewal_task: Optional[asyncio.Task] = None
self._sweeper_task: Optional[asyncio.Task] = None
async def initialize(self) -> None:
"""初始化 Worker"""
self._job_manager = JobManager()
await self._job_manager.initialize()
logger.info("Worker 初始化完成")
async def shutdown(self) -> None:
"""关闭 Worker"""
logger.info("Worker 正在关闭...")
self._running = False
# 取消回收器任务
if self._sweeper_task and not self._sweeper_task.done():
self._sweeper_task.cancel()
try:
await self._sweeper_task
except asyncio.CancelledError:
pass
# 取消锁续租任务
if self._lock_renewal_task and not self._lock_renewal_task.done():
self._lock_renewal_task.cancel()
try:
await self._lock_renewal_task
except asyncio.CancelledError:
pass
# 等待当前任务完成
if self._current_job_id:
logger.info(f"等待当前任务完成: {self._current_job_id}")
if self._job_manager:
await self._job_manager.shutdown()
logger.info("Worker 已关闭")
async def run(self) -> None:
"""运行 Worker 主循环"""
self._running = True
logger.info(
f"Worker 启动,轮询间隔: {settings.worker_poll_interval}s"
f"最大并发: {settings.max_concurrent_jobs}"
)
# 启动超时任务回收器
if settings.job_sweeper_enabled:
self._sweeper_task = asyncio.create_task(self._sweeper_loop())
logger.info(f"超时任务回收器已启动,扫描间隔: {settings.job_sweeper_interval}s")
while self._running:
try:
await self._process_next_job()
except Exception as e:
logger.error(f"Worker 循环异常: {e}", exc_info=True)
await asyncio.sleep(settings.worker_poll_interval)
async def _process_next_job(self) -> None:
"""处理下一个任务"""
if not self._job_manager:
logger.error("JobManager 未初始化")
await asyncio.sleep(settings.worker_poll_interval)
return
# 从队列获取任务(转移式出队)
job_id = await self._job_manager.dequeue_job(timeout=int(settings.worker_poll_interval))
if not job_id:
return
# 获取任务信息以提取 request_id
job_data = await self._job_manager.get_job(job_id)
if job_data:
request_id = job_data.get("request_id") or job_id
set_request_id(request_id)
else:
set_request_id(job_id)
logger.info(f"从队列获取任务: {job_id}")
# 尝试获取分布式锁(返回 token
lock_token = await self._job_manager.acquire_job_lock(job_id)
if not lock_token:
logger.warning(f"无法获取任务锁,任务可能正在被其他 Worker 执行: {job_id}")
# 任务留在 processing 队列,等待回收器处理
return
self._current_lock_token = lock_token
# 启动锁续租协程
self._lock_renewal_task = asyncio.create_task(self._lock_renewal_loop(job_id, lock_token))
try:
# 检查全局并发限制
if not await self._job_manager.can_execute():
logger.info(f"达到并发限制,任务 NACK 重新入队: {job_id}")
await self._job_manager.nack_job(job_id, requeue=True)
return
# 增加并发计数
await self._job_manager.increment_concurrency()
self._current_job_id = job_id
try:
# 执行任务
success = await self._execute_with_retry(job_id)
if success:
await self._job_manager.ack_job(job_id)
else:
await self._job_manager.increment_job_retry(job_id)
await self._job_manager.nack_job(job_id, requeue=True)
finally:
# 减少并发计数
await self._job_manager.decrement_concurrency()
self._current_job_id = None
finally:
# 停止锁续租
if self._lock_renewal_task and not self._lock_renewal_task.done():
self._lock_renewal_task.cancel()
try:
await self._lock_renewal_task
except asyncio.CancelledError:
pass
self._lock_renewal_task = None
# 释放分布式锁
await self._job_manager.release_job_lock(job_id, lock_token)
self._current_lock_token = None
async def _execute_with_retry(self, job_id: str) -> bool:
"""执行任务(带重试机制)
Returns:
bool: 任务是否成功执行
"""
if not self._job_manager:
return False
try:
# 执行任务
await asyncio.wait_for(
self._job_manager.execute_job(job_id),
timeout=settings.job_execution_timeout,
)
return True
except asyncio.TimeoutError:
logger.error(f"任务执行超时: {job_id}")
await self._handle_job_failure(job_id, "任务执行超时")
return False
except Exception as e:
logger.error(f"任务执行异常: {job_id}, error={e}", exc_info=True)
await self._handle_job_failure(job_id, str(e))
return False
async def _handle_job_failure(self, job_id: str, error: str) -> None:
"""处理任务失败"""
if not self._job_manager:
return
retry_count = await self._job_manager.increment_job_retry(job_id)
if retry_count < settings.job_max_retries:
logger.info(f"任务将重试 ({retry_count}/{settings.job_max_retries}): {job_id}")
# 重新入队
await self._job_manager.enqueue_job(job_id)
else:
logger.error(f"任务达到最大重试次数,标记为失败: {job_id}")
# 更新任务状态为失败
if self._job_manager._redis_client:
key = f"job:{job_id}"
await self._job_manager._redis_client.hset(
key,
mapping={
"status": "failed",
"error": f"达到最大重试次数 ({settings.job_max_retries}): {error}",
},
)
async def _lock_renewal_loop(self, job_id: str, lock_token: str) -> None:
"""锁续租协程
定期续租任务锁,防止长任务执行时锁过期。
Args:
job_id: 任务 ID
lock_token: 锁 token
"""
# 续租间隔为锁 TTL 的一半
interval = (settings.job_execution_timeout + settings.job_lock_buffer) / 2
while True:
try:
await asyncio.sleep(interval)
if not self._job_manager:
break
if not await self._job_manager.renew_job_lock(job_id, lock_token):
logger.error(f"锁续租失败,可能已被其他进程获取: {job_id}")
break
logger.debug(f"锁续租成功: {job_id}")
except asyncio.CancelledError:
logger.debug(f"锁续租协程已取消: {job_id}")
break
except Exception as e:
logger.error(f"锁续租异常: {job_id}, error={e}")
break
async def _sweeper_loop(self) -> None:
"""超时任务回收协程
定期扫描处理中队列,回收超时任务,并收集队列监控指标。
"""
while self._running:
try:
await asyncio.sleep(settings.job_sweeper_interval)
if not self._job_manager:
continue
# 回收超时任务
recovered = await self._job_manager.recover_stale_jobs()
if recovered > 0:
logger.info(f"回收超时任务: {recovered}")
# 记录回收指标
from .core.metrics_unified import incr
await incr("job_recovered_total", None, recovered)
# 收集队列监控指标
await self._job_manager.collect_queue_metrics()
except asyncio.CancelledError:
logger.debug("超时任务回收协程已取消")
break
except Exception as e:
logger.error(f"超时任务回收异常: {e}")
def setup_signal_handlers(
worker: JobWorker,
health_server: HealthCheckServer,
loop: asyncio.AbstractEventLoop,
) -> None:
"""设置信号处理器"""
async def shutdown_all() -> None:
"""关闭所有服务"""
await worker.shutdown()
await health_server.stop()
def signal_handler(sig: signal.Signals) -> None:
logger.info(f"收到信号 {sig.name},准备关闭...")
loop.create_task(shutdown_all())
for sig in (signal.SIGTERM, signal.SIGINT):
loop.add_signal_handler(sig, signal_handler, sig)
async def main() -> None:
"""Worker 入口函数"""
# 设置日志
setup_logging(level=settings.log_level, format_type=settings.log_format)
# 创建健康检查服务器和 Worker
health_server = HealthCheckServer(port=8000)
worker = JobWorker()
# 设置信号处理
loop = asyncio.get_running_loop()
setup_signal_handlers(worker, health_server, loop)
try:
# 先启动健康检查服务器,确保 FC 健康检查能通过
await health_server.start()
# 初始化并运行 Worker
await worker.initialize()
await worker.run()
except Exception as e:
logger.error(f"Worker 异常退出: {e}", exc_info=True)
sys.exit(1)
finally:
await worker.shutdown()
await health_server.stop()
if __name__ == "__main__":
asyncio.run(main())

View File

@@ -2,7 +2,7 @@
import pytest
from fastapi.testclient import TestClient
from src.functional_scaffold.main import app
from functional_scaffold.main import app
@pytest.fixture

View File

@@ -1,7 +1,7 @@
"""算法单元测试"""
import pytest
from src.functional_scaffold.algorithms.prime_checker import PrimeChecker
from functional_scaffold.algorithms.prime_checker import PrimeChecker
class TestPrimeChecker:

View File

@@ -1,17 +1,13 @@
"""异步任务管理器测试"""
import asyncio
import json
import pytest
from unittest.mock import AsyncMock, MagicMock, patch
from fastapi import status
from src.functional_scaffold.core.job_manager import (
from functional_scaffold.core.job_manager import (
JobManager,
get_job_manager,
shutdown_job_manager,
)
from src.functional_scaffold.api.models import JobStatus
class TestJobManager:
@@ -186,6 +182,11 @@ class TestJobManagerWithMocks:
manager._redis_client = mock_redis
manager._register_algorithms()
# 初始化 semaphore
import asyncio
manager._semaphore = asyncio.Semaphore(10)
await manager.execute_job("test-job-id")
# 验证状态更新被调用
@@ -199,7 +200,7 @@ class TestJobsAPI:
def test_create_job_success(self, client):
"""测试成功创建任务"""
with patch(
"src.functional_scaffold.api.routes.get_job_manager", new_callable=AsyncMock
"functional_scaffold.api.routes.get_job_manager", new_callable=AsyncMock
) as mock_get_manager:
mock_manager = MagicMock()
mock_manager.is_available.return_value = True
@@ -213,7 +214,7 @@ class TestJobsAPI:
"created_at": "2026-02-02T10:00:00+00:00",
}
)
mock_manager.execute_job = AsyncMock()
mock_manager.enqueue_job = AsyncMock(return_value=True)
mock_get_manager.return_value = mock_manager
response = client.post(
@@ -233,7 +234,7 @@ class TestJobsAPI:
def test_create_job_algorithm_not_found(self, client):
"""测试创建任务时算法不存在"""
with patch(
"src.functional_scaffold.api.routes.get_job_manager", new_callable=AsyncMock
"functional_scaffold.api.routes.get_job_manager", new_callable=AsyncMock
) as mock_get_manager:
mock_manager = MagicMock()
mock_manager.is_available.return_value = True
@@ -255,7 +256,7 @@ class TestJobsAPI:
def test_create_job_service_unavailable(self, client):
"""测试服务不可用时创建任务"""
with patch(
"src.functional_scaffold.api.routes.get_job_manager", new_callable=AsyncMock
"functional_scaffold.api.routes.get_job_manager", new_callable=AsyncMock
) as mock_get_manager:
mock_manager = MagicMock()
mock_manager.is_available.return_value = False
@@ -274,7 +275,7 @@ class TestJobsAPI:
def test_get_job_status_success(self, client):
"""测试成功查询任务状态"""
with patch(
"src.functional_scaffold.api.routes.get_job_manager", new_callable=AsyncMock
"functional_scaffold.api.routes.get_job_manager", new_callable=AsyncMock
) as mock_get_manager:
mock_manager = MagicMock()
mock_manager.is_available.return_value = True
@@ -304,7 +305,7 @@ class TestJobsAPI:
def test_get_job_status_not_found(self, client):
"""测试查询不存在的任务"""
with patch(
"src.functional_scaffold.api.routes.get_job_manager", new_callable=AsyncMock
"functional_scaffold.api.routes.get_job_manager", new_callable=AsyncMock
) as mock_get_manager:
mock_manager = MagicMock()
mock_manager.is_available.return_value = True
@@ -320,7 +321,7 @@ class TestJobsAPI:
def test_get_job_status_service_unavailable(self, client):
"""测试服务不可用时查询任务"""
with patch(
"src.functional_scaffold.api.routes.get_job_manager", new_callable=AsyncMock
"functional_scaffold.api.routes.get_job_manager", new_callable=AsyncMock
) as mock_get_manager:
mock_manager = MagicMock()
mock_manager.is_available.return_value = False
@@ -391,7 +392,7 @@ class TestWebhook:
manager._http_client = mock_http
# 使用较短的重试间隔进行测试
with patch("src.functional_scaffold.core.job_manager.settings") as mock_settings:
with patch("functional_scaffold.core.job_manager.settings") as mock_settings:
mock_settings.webhook_max_retries = 2
mock_settings.webhook_timeout = 1
@@ -399,3 +400,771 @@ class TestWebhook:
# 验证重试次数
assert mock_http.post.call_count == 2
class TestConcurrencyControl:
"""测试并发控制功能"""
@pytest.mark.asyncio
async def test_get_concurrency_status(self):
"""测试获取并发状态"""
manager = JobManager()
# 初始化 semaphore
manager._max_concurrent_jobs = 10
manager._semaphore = asyncio.Semaphore(10)
status = manager.get_concurrency_status()
assert status["max_concurrent"] == 10
assert status["available_slots"] == 10
assert status["running_jobs"] == 0
@pytest.mark.asyncio
async def test_get_concurrency_status_without_semaphore(self):
"""测试未初始化 semaphore 时获取并发状态"""
manager = JobManager()
status = manager.get_concurrency_status()
assert status["max_concurrent"] == 0
assert status["available_slots"] == 0
assert status["running_jobs"] == 0
@pytest.mark.asyncio
async def test_concurrency_limit(self):
"""测试并发限制是否生效"""
manager = JobManager()
# 设置较小的并发限制
manager._max_concurrent_jobs = 2
manager._semaphore = asyncio.Semaphore(2)
# 模拟 Redis
mock_redis = AsyncMock()
mock_redis.hgetall = AsyncMock(
return_value={
"status": "pending",
"algorithm": "PrimeChecker",
"params": '{"number": 17}',
"webhook": "",
"request_id": "test-request-id",
"created_at": "2026-02-02T10:00:00+00:00",
}
)
mock_redis.hset = AsyncMock()
mock_redis.expire = AsyncMock()
manager._redis_client = mock_redis
manager._register_algorithms()
# 创建一个慢速任务
async def slow_execute():
async with manager._semaphore:
await asyncio.sleep(0.1)
# 启动 3 个任务
tasks = [asyncio.create_task(slow_execute()) for _ in range(3)]
# 等待一小段时间,让前两个任务获取 semaphore
await asyncio.sleep(0.01)
# 检查并发状态
status = manager.get_concurrency_status()
assert status["running_jobs"] == 2 # 只有 2 个任务在运行
assert status["available_slots"] == 0 # 没有可用槽位
# 等待所有任务完成
await asyncio.gather(*tasks)
# 检查最终状态
status = manager.get_concurrency_status()
assert status["running_jobs"] == 0
assert status["available_slots"] == 2
def test_concurrency_status_api(self, client):
"""测试并发状态 API 端点"""
with patch(
"functional_scaffold.api.routes.get_job_manager", new_callable=AsyncMock
) as mock_get_manager:
mock_manager = MagicMock()
mock_manager.is_available.return_value = True
mock_manager.get_concurrency_status.return_value = {
"max_concurrent": 10,
"available_slots": 8,
"running_jobs": 2,
}
mock_get_manager.return_value = mock_manager
response = client.get("/jobs/concurrency/status")
assert response.status_code == status.HTTP_200_OK
data = response.json()
assert "max_concurrent" in data
assert "available_slots" in data
assert "running_jobs" in data
assert isinstance(data["max_concurrent"], int)
assert isinstance(data["available_slots"], int)
assert isinstance(data["running_jobs"], int)
class TestJobQueue:
"""测试任务队列功能"""
@pytest.mark.asyncio
async def test_enqueue_job(self):
"""测试任务入队"""
manager = JobManager()
mock_redis = AsyncMock()
mock_redis.lpush = AsyncMock(return_value=1)
manager._redis_client = mock_redis
result = await manager.enqueue_job("test-job-id")
assert result is True
mock_redis.lpush.assert_called_once()
@pytest.mark.asyncio
async def test_enqueue_job_without_redis(self):
"""测试 Redis 不可用时入队"""
manager = JobManager()
result = await manager.enqueue_job("test-job-id")
assert result is False
@pytest.mark.asyncio
async def test_dequeue_job(self):
"""测试任务出队(使用 BLMOVE"""
manager = JobManager()
mock_redis = AsyncMock()
mock_redis.blmove = AsyncMock(return_value="test-job-id")
mock_redis.zadd = AsyncMock()
manager._redis_client = mock_redis
result = await manager.dequeue_job(timeout=5)
assert result == "test-job-id"
mock_redis.blmove.assert_called_once()
mock_redis.zadd.assert_called_once()
@pytest.mark.asyncio
async def test_dequeue_job_timeout(self):
"""测试任务出队超时"""
manager = JobManager()
mock_redis = AsyncMock()
mock_redis.blmove = AsyncMock(return_value=None)
manager._redis_client = mock_redis
result = await manager.dequeue_job(timeout=1)
assert result is None
@pytest.mark.asyncio
async def test_dequeue_job_without_redis(self):
"""测试 Redis 不可用时出队"""
manager = JobManager()
result = await manager.dequeue_job(timeout=1)
assert result is None
class TestDistributedLock:
"""测试分布式锁功能"""
@pytest.mark.asyncio
async def test_acquire_job_lock(self):
"""测试获取任务锁"""
manager = JobManager()
mock_redis = AsyncMock()
mock_redis.set = AsyncMock(return_value=True)
manager._redis_client = mock_redis
result = await manager.acquire_job_lock("test-job-id")
assert result is not None # 返回 token
assert len(result) == 32 # 16 字节的十六进制字符串
mock_redis.set.assert_called_once()
call_args = mock_redis.set.call_args
assert call_args[0][0] == "job:lock:test-job-id"
assert call_args[1]["nx"] is True
assert "ex" in call_args[1]
@pytest.mark.asyncio
async def test_acquire_job_lock_already_locked(self):
"""测试获取已被锁定的任务锁"""
manager = JobManager()
mock_redis = AsyncMock()
mock_redis.set = AsyncMock(return_value=None) # 锁已存在
manager._redis_client = mock_redis
result = await manager.acquire_job_lock("test-job-id")
assert result is None
@pytest.mark.asyncio
async def test_release_job_lock(self):
"""测试释放任务锁"""
manager = JobManager()
mock_redis = AsyncMock()
mock_redis.eval = AsyncMock(return_value=1)
manager._redis_client = mock_redis
result = await manager.release_job_lock("test-job-id", "valid-token")
assert result is True
mock_redis.eval.assert_called_once()
@pytest.mark.asyncio
async def test_release_job_lock_without_redis(self):
"""测试 Redis 不可用时释放锁"""
manager = JobManager()
result = await manager.release_job_lock("test-job-id", "token")
assert result is False
class TestGlobalConcurrency:
"""测试全局并发控制功能"""
@pytest.mark.asyncio
async def test_increment_concurrency(self):
"""测试增加并发计数"""
manager = JobManager()
mock_redis = AsyncMock()
mock_redis.incr = AsyncMock(return_value=5)
manager._redis_client = mock_redis
result = await manager.increment_concurrency()
assert result == 5
mock_redis.incr.assert_called_once()
@pytest.mark.asyncio
async def test_decrement_concurrency(self):
"""测试减少并发计数"""
manager = JobManager()
mock_redis = AsyncMock()
mock_redis.decr = AsyncMock(return_value=4)
manager._redis_client = mock_redis
result = await manager.decrement_concurrency()
assert result == 4
mock_redis.decr.assert_called_once()
@pytest.mark.asyncio
async def test_decrement_concurrency_prevent_negative(self):
"""测试防止并发计数变为负数"""
manager = JobManager()
mock_redis = AsyncMock()
mock_redis.decr = AsyncMock(return_value=-1)
mock_redis.set = AsyncMock()
manager._redis_client = mock_redis
result = await manager.decrement_concurrency()
assert result == 0
mock_redis.set.assert_called_once()
@pytest.mark.asyncio
async def test_get_global_concurrency(self):
"""测试获取全局并发数"""
manager = JobManager()
mock_redis = AsyncMock()
mock_redis.get = AsyncMock(return_value="7")
manager._redis_client = mock_redis
result = await manager.get_global_concurrency()
assert result == 7
@pytest.mark.asyncio
async def test_get_global_concurrency_empty(self):
"""测试获取空的全局并发数"""
manager = JobManager()
mock_redis = AsyncMock()
mock_redis.get = AsyncMock(return_value=None)
manager._redis_client = mock_redis
result = await manager.get_global_concurrency()
assert result == 0
@pytest.mark.asyncio
async def test_can_execute(self):
"""测试检查是否可执行"""
manager = JobManager()
mock_redis = AsyncMock()
mock_redis.get = AsyncMock(return_value="5")
manager._redis_client = mock_redis
with patch("functional_scaffold.core.job_manager.settings") as mock_settings:
mock_settings.max_concurrent_jobs = 10
result = await manager.can_execute()
assert result is True
@pytest.mark.asyncio
async def test_can_execute_at_limit(self):
"""测试达到并发限制时"""
manager = JobManager()
mock_redis = AsyncMock()
mock_redis.get = AsyncMock(return_value="10")
manager._redis_client = mock_redis
with patch("functional_scaffold.core.job_manager.settings") as mock_settings:
mock_settings.max_concurrent_jobs = 10
result = await manager.can_execute()
assert result is False
class TestJobRetry:
"""测试任务重试功能"""
@pytest.mark.asyncio
async def test_get_job_retry_count(self):
"""测试获取任务重试次数"""
manager = JobManager()
mock_redis = AsyncMock()
mock_redis.hget = AsyncMock(return_value="2")
manager._redis_client = mock_redis
result = await manager.get_job_retry_count("test-job-id")
assert result == 2
mock_redis.hget.assert_called_once_with("job:test-job-id", "retry_count")
@pytest.mark.asyncio
async def test_get_job_retry_count_empty(self):
"""测试获取空的重试次数"""
manager = JobManager()
mock_redis = AsyncMock()
mock_redis.hget = AsyncMock(return_value=None)
manager._redis_client = mock_redis
result = await manager.get_job_retry_count("test-job-id")
assert result == 0
@pytest.mark.asyncio
async def test_increment_job_retry(self):
"""测试增加任务重试次数"""
manager = JobManager()
mock_redis = AsyncMock()
mock_redis.hincrby = AsyncMock()
mock_redis.hget = AsyncMock(return_value="3")
manager._redis_client = mock_redis
result = await manager.increment_job_retry("test-job-id")
assert result == 3
mock_redis.hincrby.assert_called_once_with("job:test-job-id", "retry_count", 1)
class TestTransferDequeue:
"""测试转移式出队功能"""
@pytest.mark.asyncio
async def test_dequeue_job_with_blmove(self):
"""测试使用 BLMOVE 转移式出队"""
manager = JobManager()
mock_redis = AsyncMock()
mock_redis.blmove = AsyncMock(return_value="test-job-id")
mock_redis.zadd = AsyncMock()
manager._redis_client = mock_redis
result = await manager.dequeue_job(timeout=5)
assert result == "test-job-id"
mock_redis.blmove.assert_called_once()
mock_redis.zadd.assert_called_once()
@pytest.mark.asyncio
async def test_dequeue_job_timeout(self):
"""测试出队超时"""
manager = JobManager()
mock_redis = AsyncMock()
mock_redis.blmove = AsyncMock(return_value=None)
manager._redis_client = mock_redis
result = await manager.dequeue_job(timeout=1)
assert result is None
mock_redis.zadd.assert_not_called()
class TestTokenBasedLock:
"""测试带 Token 的安全锁"""
@pytest.mark.asyncio
async def test_acquire_job_lock_returns_token(self):
"""测试获取锁返回 token"""
manager = JobManager()
mock_redis = AsyncMock()
mock_redis.set = AsyncMock(return_value=True)
manager._redis_client = mock_redis
result = await manager.acquire_job_lock("test-job-id")
assert result is not None
assert len(result) == 32 # 16 字节的十六进制字符串
mock_redis.set.assert_called_once()
call_args = mock_redis.set.call_args
assert call_args[0][0] == "job:lock:test-job-id"
assert call_args[1]["nx"] is True
@pytest.mark.asyncio
async def test_acquire_job_lock_already_locked(self):
"""测试获取已被锁定的任务锁"""
manager = JobManager()
mock_redis = AsyncMock()
mock_redis.set = AsyncMock(return_value=None)
manager._redis_client = mock_redis
result = await manager.acquire_job_lock("test-job-id")
assert result is None
@pytest.mark.asyncio
async def test_release_job_lock_with_token(self):
"""测试使用 token 释放锁"""
manager = JobManager()
mock_redis = AsyncMock()
mock_redis.eval = AsyncMock(return_value=1)
manager._redis_client = mock_redis
result = await manager.release_job_lock("test-job-id", "valid-token")
assert result is True
mock_redis.eval.assert_called_once()
@pytest.mark.asyncio
async def test_release_job_lock_invalid_token(self):
"""测试使用无效 token 释放锁"""
manager = JobManager()
mock_redis = AsyncMock()
mock_redis.eval = AsyncMock(return_value=0)
manager._redis_client = mock_redis
result = await manager.release_job_lock("test-job-id", "invalid-token")
assert result is False
@pytest.mark.asyncio
async def test_release_job_lock_without_token(self):
"""测试不使用 token 释放锁(向后兼容)"""
manager = JobManager()
mock_redis = AsyncMock()
mock_redis.delete = AsyncMock()
manager._redis_client = mock_redis
result = await manager.release_job_lock("test-job-id")
assert result is True
mock_redis.delete.assert_called_once_with("job:lock:test-job-id")
class TestAckNack:
"""测试 ACK/NACK 机制"""
@pytest.mark.asyncio
async def test_ack_job(self):
"""测试确认任务完成"""
manager = JobManager()
mock_pipe = AsyncMock()
mock_pipe.lrem = MagicMock()
mock_pipe.zrem = MagicMock()
mock_pipe.execute = AsyncMock()
mock_pipe.__aenter__ = AsyncMock(return_value=mock_pipe)
mock_pipe.__aexit__ = AsyncMock()
mock_redis = AsyncMock()
mock_redis.pipeline = MagicMock(return_value=mock_pipe)
manager._redis_client = mock_redis
result = await manager.ack_job("test-job-id")
assert result is True
mock_pipe.lrem.assert_called_once()
mock_pipe.zrem.assert_called_once()
@pytest.mark.asyncio
async def test_nack_job_requeue(self):
"""测试拒绝任务并重新入队"""
manager = JobManager()
mock_pipe = AsyncMock()
mock_pipe.lrem = MagicMock()
mock_pipe.zrem = MagicMock()
mock_pipe.lpush = MagicMock()
mock_pipe.execute = AsyncMock()
mock_pipe.__aenter__ = AsyncMock(return_value=mock_pipe)
mock_pipe.__aexit__ = AsyncMock()
mock_redis = AsyncMock()
mock_redis.pipeline = MagicMock(return_value=mock_pipe)
mock_redis.hget = AsyncMock(return_value="0") # retry_count = 0
manager._redis_client = mock_redis
result = await manager.nack_job("test-job-id", requeue=True)
assert result is True
assert mock_pipe.lpush.call_count == 1
@pytest.mark.asyncio
async def test_nack_job_to_dlq(self):
"""测试拒绝任务进入死信队列"""
manager = JobManager()
mock_pipe = AsyncMock()
mock_pipe.lrem = MagicMock()
mock_pipe.zrem = MagicMock()
mock_pipe.lpush = MagicMock()
mock_pipe.execute = AsyncMock()
mock_pipe.__aenter__ = AsyncMock(return_value=mock_pipe)
mock_pipe.__aexit__ = AsyncMock()
mock_redis = AsyncMock()
mock_redis.pipeline = MagicMock(return_value=mock_pipe)
mock_redis.hget = AsyncMock(return_value="5") # retry_count > max_retries
manager._redis_client = mock_redis
with patch("functional_scaffold.core.job_manager.settings") as mock_settings:
mock_settings.job_max_retries = 3
mock_settings.job_processing_key = "job:processing"
mock_settings.job_processing_ts_key = "job:processing:ts"
mock_settings.job_dlq_key = "job:dlq"
mock_settings.job_queue_key = "job:queue"
result = await manager.nack_job("test-job-id", requeue=True)
assert result is True
class TestLockRenewal:
"""测试锁续租功能"""
@pytest.mark.asyncio
async def test_renew_job_lock_success(self):
"""测试锁续租成功"""
manager = JobManager()
mock_redis = AsyncMock()
mock_redis.eval = AsyncMock(return_value=1)
manager._redis_client = mock_redis
result = await manager.renew_job_lock("test-job-id", "valid-token")
assert result is True
mock_redis.eval.assert_called_once()
@pytest.mark.asyncio
async def test_renew_job_lock_invalid_token(self):
"""测试锁续租失败token 不匹配)"""
manager = JobManager()
mock_redis = AsyncMock()
mock_redis.eval = AsyncMock(return_value=0)
manager._redis_client = mock_redis
result = await manager.renew_job_lock("test-job-id", "invalid-token")
assert result is False
@pytest.mark.asyncio
async def test_renew_job_lock_without_redis(self):
"""测试 Redis 不可用时续租"""
manager = JobManager()
result = await manager.renew_job_lock("test-job-id", "token")
assert result is False
class TestStaleJobRecovery:
"""测试超时任务回收功能"""
@pytest.mark.asyncio
async def test_recover_stale_jobs_empty(self):
"""测试没有超时任务时的回收"""
manager = JobManager()
mock_redis = AsyncMock()
mock_redis.zrangebyscore = AsyncMock(return_value=[])
manager._redis_client = mock_redis
result = await manager.recover_stale_jobs()
assert result == 0
@pytest.mark.asyncio
async def test_recover_stale_jobs_requeue(self):
"""测试回收超时任务并重新入队"""
manager = JobManager()
mock_pipe = AsyncMock()
mock_pipe.lrem = MagicMock()
mock_pipe.zrem = MagicMock()
mock_pipe.lpush = MagicMock()
mock_pipe.execute = AsyncMock()
mock_pipe.__aenter__ = AsyncMock(return_value=mock_pipe)
mock_pipe.__aexit__ = AsyncMock()
mock_redis = AsyncMock()
mock_redis.zrangebyscore = AsyncMock(return_value=["stale-job-1", "stale-job-2"])
mock_redis.hincrby = AsyncMock()
mock_redis.hget = AsyncMock(return_value="1") # retry_count = 1
mock_redis.pipeline = MagicMock(return_value=mock_pipe)
manager._redis_client = mock_redis
with patch("functional_scaffold.core.job_manager.settings") as mock_settings:
mock_settings.job_execution_timeout = 300
mock_settings.job_lock_buffer = 60
mock_settings.job_max_retries = 3
mock_settings.job_processing_key = "job:processing"
mock_settings.job_processing_ts_key = "job:processing:ts"
mock_settings.job_dlq_key = "job:dlq"
mock_settings.job_queue_key = "job:queue"
result = await manager.recover_stale_jobs()
assert result == 2
@pytest.mark.asyncio
async def test_recover_stale_jobs_to_dlq(self):
"""测试回收超时任务进入死信队列"""
manager = JobManager()
mock_pipe = AsyncMock()
mock_pipe.lrem = MagicMock()
mock_pipe.zrem = MagicMock()
mock_pipe.lpush = MagicMock()
mock_pipe.execute = AsyncMock()
mock_pipe.__aenter__ = AsyncMock(return_value=mock_pipe)
mock_pipe.__aexit__ = AsyncMock()
mock_redis = AsyncMock()
mock_redis.zrangebyscore = AsyncMock(return_value=["stale-job-1"])
mock_redis.hincrby = AsyncMock()
mock_redis.hget = AsyncMock(return_value="5") # retry_count > max_retries
mock_redis.pipeline = MagicMock(return_value=mock_pipe)
manager._redis_client = mock_redis
with patch("functional_scaffold.core.job_manager.settings") as mock_settings:
mock_settings.job_execution_timeout = 300
mock_settings.job_lock_buffer = 60
mock_settings.job_max_retries = 3
mock_settings.job_processing_key = "job:processing"
mock_settings.job_processing_ts_key = "job:processing:ts"
mock_settings.job_dlq_key = "job:dlq"
mock_settings.job_queue_key = "job:queue"
result = await manager.recover_stale_jobs()
assert result == 1
@pytest.mark.asyncio
async def test_recover_stale_jobs_without_redis(self):
"""测试 Redis 不可用时回收"""
manager = JobManager()
result = await manager.recover_stale_jobs()
assert result == 0
class TestQueueMetrics:
"""测试队列监控指标收集"""
@pytest.mark.asyncio
async def test_collect_queue_metrics(self):
"""测试收集队列指标"""
manager = JobManager()
mock_pipe = AsyncMock()
mock_pipe.llen = MagicMock()
mock_pipe.zrange = MagicMock()
mock_pipe.execute = AsyncMock(return_value=[5, 2, 1, [("job-1", 1000.0)]])
mock_pipe.__aenter__ = AsyncMock(return_value=mock_pipe)
mock_pipe.__aexit__ = AsyncMock()
mock_redis = AsyncMock()
mock_redis.pipeline = MagicMock(return_value=mock_pipe)
manager._redis_client = mock_redis
with patch("functional_scaffold.core.job_manager.time") as mock_time:
mock_time.time.return_value = 1060.0 # 60 秒后
with patch("functional_scaffold.core.job_manager.set") as mock_set:
result = await manager.collect_queue_metrics()
assert result["queue_length"] == 5
assert result["processing_length"] == 2
assert result["dlq_length"] == 1
assert result["oldest_waiting_seconds"] == 60.0
@pytest.mark.asyncio
async def test_collect_queue_metrics_empty(self):
"""测试空队列时收集指标"""
manager = JobManager()
mock_pipe = AsyncMock()
mock_pipe.llen = MagicMock()
mock_pipe.zrange = MagicMock()
mock_pipe.execute = AsyncMock(return_value=[0, 0, 0, []])
mock_pipe.__aenter__ = AsyncMock(return_value=mock_pipe)
mock_pipe.__aexit__ = AsyncMock()
mock_redis = AsyncMock()
mock_redis.pipeline = MagicMock(return_value=mock_pipe)
manager._redis_client = mock_redis
with patch("functional_scaffold.core.job_manager.set"):
result = await manager.collect_queue_metrics()
assert result["queue_length"] == 0
assert result["processing_length"] == 0
assert result["dlq_length"] == 0
assert result["oldest_waiting_seconds"] == 0
@pytest.mark.asyncio
async def test_collect_queue_metrics_without_redis(self):
"""测试 Redis 不可用时收集指标"""
manager = JobManager()
result = await manager.collect_queue_metrics()
assert result["queue_length"] == 0
assert result["processing_length"] == 0
assert result["dlq_length"] == 0
assert result["oldest_waiting_seconds"] == 0

View File

@@ -1,158 +1,239 @@
"""metrics_unified 模块单元测试"""
import pytest
from unittest.mock import MagicMock, patch
from unittest.mock import AsyncMock, MagicMock, patch
@pytest.fixture(autouse=True)
def reset_manager():
"""每个测试前后重置管理器"""
from functional_scaffold.core.metrics_unified import reset_metrics_manager_sync
reset_metrics_manager_sync()
yield
reset_metrics_manager_sync()
class TestMetricsManager:
"""MetricsManager 类测试"""
@pytest.fixture
def mock_redis(self):
"""模拟 Redis 客户端"""
with patch("redis.Redis") as mock:
mock_instance = MagicMock()
mock_instance.ping.return_value = True
mock_instance.hincrbyfloat.return_value = 1.0
mock_instance.hset.return_value = True
mock_instance.hgetall.return_value = {}
mock_instance.hget.return_value = "0"
mock_instance.keys.return_value = []
mock_instance.pipeline.return_value = MagicMock()
mock.return_value = mock_instance
yield mock_instance
@pytest.fixture
def manager(self, mock_redis):
"""创建测试用的 MetricsManager"""
from src.functional_scaffold.core.metrics_unified import (
MetricsManager,
reset_metrics_manager,
)
reset_metrics_manager()
manager = MetricsManager()
return manager
def test_init_loads_default_config(self, manager):
def test_init_loads_default_config(self):
"""测试初始化加载默认配置"""
from functional_scaffold.core.metrics_unified import MetricsManager
manager = MetricsManager()
assert manager.config is not None
assert "builtin_metrics" in manager.config or len(manager.metrics_definitions) > 0
def test_metrics_definitions_registered(self, manager):
def test_metrics_definitions_registered(self):
"""测试指标定义已注册"""
from functional_scaffold.core.metrics_unified import MetricsManager
manager = MetricsManager()
assert "http_requests_total" in manager.metrics_definitions
assert "http_request_duration_seconds" in manager.metrics_definitions
assert "algorithm_executions_total" in manager.metrics_definitions
def test_incr_counter(self, manager, mock_redis):
@pytest.mark.asyncio
@patch("redis.asyncio.Redis")
async def test_incr_counter(self, mock_redis_class):
"""测试计数器增加"""
manager.incr("http_requests_total", {"method": "GET", "endpoint": "/", "status": "success"})
mock_redis.hincrbyfloat.assert_called()
mock_instance = AsyncMock()
mock_instance.ping = AsyncMock(return_value=True)
mock_instance.hincrbyfloat = AsyncMock(return_value=1.0)
mock_instance.close = AsyncMock()
mock_redis_class.return_value = mock_instance
def test_incr_with_invalid_metric_type(self, manager, mock_redis):
from functional_scaffold.core.metrics_unified import MetricsManager
manager = MetricsManager()
await manager.initialize()
await manager.incr(
"http_requests_total", {"method": "GET", "endpoint": "/", "status": "success"}
)
mock_instance.hincrbyfloat.assert_called()
def test_incr_with_invalid_metric_type(self):
"""测试对非计数器类型调用 incr"""
from functional_scaffold.core.metrics_unified import MetricsManager
manager = MetricsManager()
# http_request_duration_seconds 是 histogram 类型
manager.incr("http_request_duration_seconds", {})
# 不应该调用 Redis因为类型不匹配
# 验证没有调用 hincrbyfloat或者调用次数没有增加
# 验证不会抛出异常(因为 Redis 不可用)
def test_set_gauge(self, manager, mock_redis):
@pytest.mark.asyncio
@patch("redis.asyncio.Redis")
async def test_set_gauge(self, mock_redis_class):
"""测试设置仪表盘"""
manager.set("http_requests_in_progress", {}, 5)
mock_redis.hset.assert_called()
mock_instance = AsyncMock()
mock_instance.ping = AsyncMock(return_value=True)
mock_instance.hset = AsyncMock(return_value=True)
mock_instance.close = AsyncMock()
mock_redis_class.return_value = mock_instance
def test_gauge_incr(self, manager, mock_redis):
from functional_scaffold.core.metrics_unified import MetricsManager
manager = MetricsManager()
await manager.initialize()
await manager.set("http_requests_in_progress", {}, 5)
mock_instance.hset.assert_called()
@pytest.mark.asyncio
@patch("redis.asyncio.Redis")
async def test_gauge_incr(self, mock_redis_class):
"""测试增加仪表盘"""
manager.gauge_incr("http_requests_in_progress", {}, 1)
mock_redis.hincrbyfloat.assert_called()
mock_instance = AsyncMock()
mock_instance.ping = AsyncMock(return_value=True)
mock_instance.hincrbyfloat = AsyncMock(return_value=1.0)
mock_instance.close = AsyncMock()
mock_redis_class.return_value = mock_instance
def test_gauge_decr(self, manager, mock_redis):
from functional_scaffold.core.metrics_unified import MetricsManager
manager = MetricsManager()
await manager.initialize()
await manager.gauge_incr("http_requests_in_progress", {}, 1)
mock_instance.hincrbyfloat.assert_called()
@pytest.mark.asyncio
@patch("redis.asyncio.Redis")
async def test_gauge_decr(self, mock_redis_class):
"""测试减少仪表盘"""
manager.gauge_decr("http_requests_in_progress", {}, 1)
mock_redis.hincrbyfloat.assert_called()
mock_instance = AsyncMock()
mock_instance.ping = AsyncMock(return_value=True)
mock_instance.hincrbyfloat = AsyncMock(return_value=1.0)
mock_instance.close = AsyncMock()
mock_redis_class.return_value = mock_instance
def test_observe_histogram(self, manager, mock_redis):
from functional_scaffold.core.metrics_unified import MetricsManager
manager = MetricsManager()
await manager.initialize()
await manager.gauge_decr("http_requests_in_progress", {}, 1)
mock_instance.hincrbyfloat.assert_called()
@pytest.mark.asyncio
@patch("redis.asyncio.Redis")
async def test_observe_histogram(self, mock_redis_class):
"""测试直方图观测"""
mock_pipeline = MagicMock()
mock_redis.pipeline.return_value = mock_pipeline
mock_instance = AsyncMock()
mock_instance.ping = AsyncMock(return_value=True)
mock_instance.close = AsyncMock()
manager.observe("http_request_duration_seconds", {"method": "GET", "endpoint": "/"}, 0.05)
mock_pipeline = AsyncMock()
mock_pipeline.hincrbyfloat = MagicMock()
mock_pipeline.execute = AsyncMock(return_value=[])
mock_instance.pipeline = MagicMock(return_value=mock_pipeline)
mock_redis.pipeline.assert_called()
mock_pipeline.execute.assert_called()
mock_redis_class.return_value = mock_instance
def test_labels_to_key(self, manager):
from functional_scaffold.core.metrics_unified import MetricsManager
manager = MetricsManager()
await manager.initialize()
await manager.observe(
"http_request_duration_seconds", {"method": "GET", "endpoint": "/"}, 0.05
)
mock_instance.pipeline.assert_called()
def test_labels_to_key(self):
"""测试标签转换为 key"""
from functional_scaffold.core.metrics_unified import MetricsManager
manager = MetricsManager()
labels = {"method": "GET", "endpoint": "/api"}
key = manager._labels_to_key(labels)
assert "method=GET" in key
assert "endpoint=/api" in key
def test_labels_to_key_empty(self, manager):
def test_labels_to_key_empty(self):
"""测试空标签转换"""
from functional_scaffold.core.metrics_unified import MetricsManager
manager = MetricsManager()
key = manager._labels_to_key(None)
assert key == ""
key = manager._labels_to_key({})
assert key == ""
def test_is_available(self, manager):
@pytest.mark.asyncio
@patch("redis.asyncio.Redis")
async def test_is_available(self, mock_redis_class):
"""测试 Redis 可用性检查"""
mock_instance = AsyncMock()
mock_instance.ping = AsyncMock(return_value=True)
mock_instance.close = AsyncMock()
mock_redis_class.return_value = mock_instance
from functional_scaffold.core.metrics_unified import MetricsManager
manager = MetricsManager()
await manager.initialize()
assert manager.is_available() is True
class TestConvenienceFunctions:
"""便捷函数测试"""
@pytest.fixture(autouse=True)
def setup(self):
"""每个测试前重置管理器"""
from src.functional_scaffold.core.metrics_unified import reset_metrics_manager
reset_metrics_manager()
@patch("redis.Redis")
def test_incr_function(self, mock_redis_class):
@pytest.mark.asyncio
@patch("redis.asyncio.Redis")
async def test_incr_function(self, mock_redis_class):
"""测试 incr 便捷函数"""
mock_instance = MagicMock()
mock_instance.ping.return_value = True
mock_instance = AsyncMock()
mock_instance.ping = AsyncMock(return_value=True)
mock_instance.hincrbyfloat = AsyncMock(return_value=1.0)
mock_instance.close = AsyncMock()
mock_redis_class.return_value = mock_instance
from src.functional_scaffold.core.metrics_unified import incr, reset_metrics_manager
from functional_scaffold.core.metrics_unified import incr
reset_metrics_manager()
incr("http_requests_total", {"method": "GET", "endpoint": "/", "status": "success"})
await incr(
"http_requests_total", {"method": "GET", "endpoint": "/", "status": "success"}
)
mock_instance.hincrbyfloat.assert_called()
@patch("redis.Redis")
def test_set_function(self, mock_redis_class):
@pytest.mark.asyncio
@patch("redis.asyncio.Redis")
async def test_set_function(self, mock_redis_class):
"""测试 set 便捷函数"""
mock_instance = MagicMock()
mock_instance.ping.return_value = True
mock_instance = AsyncMock()
mock_instance.ping = AsyncMock(return_value=True)
mock_instance.hset = AsyncMock(return_value=True)
mock_instance.close = AsyncMock()
mock_redis_class.return_value = mock_instance
from src.functional_scaffold.core.metrics_unified import reset_metrics_manager, set
from functional_scaffold.core.metrics_unified import set
reset_metrics_manager()
set("http_requests_in_progress", {}, 10)
await set("http_requests_in_progress", {}, 10)
mock_instance.hset.assert_called()
@patch("redis.Redis")
def test_observe_function(self, mock_redis_class):
@pytest.mark.asyncio
@patch("redis.asyncio.Redis")
async def test_observe_function(self, mock_redis_class):
"""测试 observe 便捷函数"""
mock_instance = MagicMock()
mock_instance.ping.return_value = True
mock_pipeline = MagicMock()
mock_instance.pipeline.return_value = mock_pipeline
mock_instance = AsyncMock()
mock_instance.ping = AsyncMock(return_value=True)
mock_instance.close = AsyncMock()
mock_pipeline = AsyncMock()
mock_pipeline.hincrbyfloat = MagicMock()
mock_pipeline.execute = AsyncMock(return_value=[])
mock_instance.pipeline = MagicMock(return_value=mock_pipeline)
mock_redis_class.return_value = mock_instance
from src.functional_scaffold.core.metrics_unified import observe, reset_metrics_manager
from functional_scaffold.core.metrics_unified import observe
reset_metrics_manager()
observe("http_request_duration_seconds", {"method": "GET", "endpoint": "/"}, 0.1)
await observe("http_request_duration_seconds", {"method": "GET", "endpoint": "/"}, 0.1)
mock_instance.pipeline.assert_called()
@@ -160,42 +241,49 @@ class TestConvenienceFunctions:
class TestExport:
"""导出功能测试"""
@patch("redis.Redis")
def test_export_counter(self, mock_redis_class):
@pytest.mark.asyncio
@patch("redis.asyncio.Redis")
async def test_export_counter(self, mock_redis_class):
"""测试导出计数器"""
mock_instance = MagicMock()
mock_instance.ping.return_value = True
mock_instance.hgetall.return_value = {"method=GET,endpoint=/,status=success": "10"}
mock_instance = AsyncMock()
mock_instance.ping = AsyncMock(return_value=True)
mock_instance.hgetall = AsyncMock(
return_value={"method=GET,endpoint=/,status=success": "10"}
)
mock_instance.hget = AsyncMock(return_value="0")
mock_instance.close = AsyncMock()
mock_redis_class.return_value = mock_instance
from src.functional_scaffold.core.metrics_unified import export, reset_metrics_manager
from functional_scaffold.core.metrics_unified import export
reset_metrics_manager()
output = export()
output = await export()
assert "http_requests_total" in output
assert "HELP" in output
assert "TYPE" in output
@patch("redis.Redis")
def test_export_histogram(self, mock_redis_class):
@pytest.mark.asyncio
@patch("redis.asyncio.Redis")
async def test_export_histogram(self, mock_redis_class):
"""测试导出直方图"""
mock_instance = MagicMock()
mock_instance.ping.return_value = True
mock_instance.hgetall.side_effect = lambda key: (
{"method=GET,endpoint=/": "5"}
if "count" in key
else {"method=GET,endpoint=/": "0.5"}
if "sum" in key
else {}
)
mock_instance.hget.return_value = "3"
mock_instance = AsyncMock()
mock_instance.ping = AsyncMock(return_value=True)
async def mock_hgetall(key):
if "count" in key:
return {"method=GET,endpoint=/": "5"}
elif "sum" in key:
return {"method=GET,endpoint=/": "0.5"}
return {}
mock_instance.hgetall = mock_hgetall
mock_instance.hget = AsyncMock(return_value="3")
mock_instance.close = AsyncMock()
mock_redis_class.return_value = mock_instance
from src.functional_scaffold.core.metrics_unified import export, reset_metrics_manager
from functional_scaffold.core.metrics_unified import export
reset_metrics_manager()
output = export()
output = await export()
assert "http_request_duration_seconds" in output
@@ -206,7 +294,7 @@ class TestEnvVarSubstitution:
def test_substitute_env_vars(self):
"""测试环境变量替换"""
import os
from src.functional_scaffold.core.metrics_unified import MetricsManager
from functional_scaffold.core.metrics_unified import MetricsManager
# 设置测试环境变量
os.environ["TEST_VAR"] = "test_value"
@@ -226,21 +314,9 @@ class TestEnvVarSubstitution:
class TestTrackAlgorithmExecution:
"""track_algorithm_execution 装饰器测试"""
@patch("redis.Redis")
def test_decorator_success(self, mock_redis_class):
def test_decorator_success(self):
"""测试装饰器成功执行"""
mock_instance = MagicMock()
mock_instance.ping.return_value = True
mock_pipeline = MagicMock()
mock_instance.pipeline.return_value = mock_pipeline
mock_redis_class.return_value = mock_instance
from src.functional_scaffold.core.metrics_unified import (
reset_metrics_manager,
track_algorithm_execution,
)
reset_metrics_manager()
from functional_scaffold.core.metrics_unified import track_algorithm_execution
@track_algorithm_execution("test_algo")
def test_func():
@@ -249,21 +325,9 @@ class TestTrackAlgorithmExecution:
result = test_func()
assert result == "result"
@patch("redis.Redis")
def test_decorator_error(self, mock_redis_class):
def test_decorator_error(self):
"""测试装饰器错误处理"""
mock_instance = MagicMock()
mock_instance.ping.return_value = True
mock_pipeline = MagicMock()
mock_instance.pipeline.return_value = mock_pipeline
mock_redis_class.return_value = mock_instance
from src.functional_scaffold.core.metrics_unified import (
reset_metrics_manager,
track_algorithm_execution,
)
reset_metrics_manager()
from functional_scaffold.core.metrics_unified import track_algorithm_execution
@track_algorithm_execution("test_algo")
def test_func():

97
tests/test_middleware.py Normal file
View File

@@ -0,0 +1,97 @@
"""中间件测试"""
import pytest
from unittest.mock import patch, MagicMock
from fastapi.testclient import TestClient
from functional_scaffold.main import app, normalize_path
class TestNormalizePath:
"""测试路径规范化函数"""
def test_normalize_jobs_path(self):
"""测试 /jobs/{job_id} 路径规范化"""
assert normalize_path("/jobs/a1b2c3d4e5f6") == "/jobs/{job_id}"
assert normalize_path("/jobs/123456789012") == "/jobs/{job_id}"
assert normalize_path("/jobs/xyz") == "/jobs/{job_id}"
def test_normalize_other_paths(self):
"""测试其他路径保持不变"""
assert normalize_path("/invoke") == "/invoke"
assert normalize_path("/healthz") == "/healthz"
assert normalize_path("/readyz") == "/readyz"
assert normalize_path("/metrics") == "/metrics"
assert normalize_path("/docs") == "/docs"
def test_normalize_jobs_root(self):
"""测试 /jobs 根路径"""
assert normalize_path("/jobs") == "/jobs"
class TestMetricsMiddleware:
"""测试指标中间件"""
@patch("functional_scaffold.main.incr")
@patch("functional_scaffold.main.observe")
@patch("functional_scaffold.main.gauge_incr")
@patch("functional_scaffold.main.gauge_decr")
def test_skip_health_endpoints(self, mock_gauge_decr, mock_gauge_incr, mock_observe, mock_incr):
"""测试跳过健康检查端点"""
client = TestClient(app)
# 访问健康检查端点
client.get("/healthz")
client.get("/readyz")
client.get("/metrics")
# 验证没有记录指标
mock_incr.assert_not_called()
mock_observe.assert_not_called()
mock_gauge_incr.assert_not_called()
mock_gauge_decr.assert_not_called()
@patch("functional_scaffold.main.incr")
@patch("functional_scaffold.main.observe")
@patch("functional_scaffold.main.gauge_incr")
@patch("functional_scaffold.main.gauge_decr")
def test_record_normal_endpoints(self, mock_gauge_decr, mock_gauge_incr, mock_observe, mock_incr):
"""测试记录普通端点"""
client = TestClient(app)
# 访问普通端点
client.post("/invoke", json={"number": 17})
# 验证记录了指标
mock_gauge_incr.assert_called_once()
mock_gauge_decr.assert_called_once()
mock_incr.assert_called_once()
mock_observe.assert_called_once()
# 验证使用了正确的端点路径
incr_call_args = mock_incr.call_args
assert incr_call_args[0][1]["endpoint"] == "/invoke"
@patch("functional_scaffold.main.incr")
@patch("functional_scaffold.main.observe")
@patch("functional_scaffold.main.gauge_incr")
@patch("functional_scaffold.main.gauge_decr")
@patch("functional_scaffold.core.job_manager.get_job_manager")
def test_normalize_job_path(self, mock_get_manager, mock_gauge_decr, mock_gauge_incr, mock_observe, mock_incr):
"""测试规范化任务路径"""
# Mock job manager
mock_manager = MagicMock()
mock_manager.get_job.return_value = None
mock_get_manager.return_value = mock_manager
client = TestClient(app)
# 访问任务端点(会返回 404但中间件应该记录指标
client.get("/jobs/a1b2c3d4e5f6")
# 验证记录了指标
mock_incr.assert_called_once()
# 验证使用了规范化后的路径
incr_call_args = mock_incr.call_args
assert incr_call_args[0][1]["endpoint"] == "/jobs/{job_id}"