main:删除指标脚本并优化指标记录逻辑

变更内容:
- 删除 `start_metrics.sh` 脚本,精简项目结构,移除不再需要的启动逻辑。
- 优化 HTTP 请求指标记录,新增健康检查端点过滤和路径参数规范化功能。
- 更新文档,添加指标过滤及路径规范化的详细说明。
- 提高 Prometheus 指标的性能和可维护性,避免标签基数爆炸。
This commit is contained in:
2026-02-02 15:53:00 +08:00
parent b1077e78e9
commit 57b276d038
4 changed files with 167 additions and 118 deletions

View File

@@ -0,0 +1,126 @@
# 指标过滤和路径规范化
## 变更说明
本次修改优化了 HTTP 请求指标的记录逻辑,主要包括两个方面:
### 1. 跳过健康检查端点
以下端点不再记录到 Prometheus 指标中:
- `/metrics` - 指标端点本身
- `/healthz` - 存活检查
- `/readyz` - 就绪检查
**原因**:这些端点通常被频繁调用(如 Kubernetes 健康检查、Prometheus 抓取),但对业务监控意义不大,会产生大量噪音数据。
### 2. 路径参数规范化
带有路径参数的端点会被规范化为模板形式:
| 原始路径 | 规范化后 |
|---------|---------|
| `GET /jobs/a1b2c3d4e5f6` | `GET /jobs/{job_id}` |
| `GET /jobs/xyz123456789` | `GET /jobs/{job_id}` |
**原因**避免因为不同的路径参数值产生过多的指标标签导致指标基数爆炸cardinality explosion影响 Prometheus 性能。
## 实现细节
### 代码修改
**文件:`src/functional_scaffold/main.py`**
1. 添加 `normalize_path()` 函数:
```python
def normalize_path(path: str) -> str:
"""规范化路径,将路径参数替换为模板形式"""
if path.startswith("/jobs/") and len(path) > 6:
return "/jobs/{job_id}"
return path
```
2. 修改 `track_metrics` 中间件:
```python
# 跳过不需要记录指标的端点
skip_paths = {"/metrics", "/readyz", "/healthz"}
if request.url.path in skip_paths:
return await call_next(request)
# 使用规范化后的路径记录指标
normalized_path = normalize_path(request.url.path)
incr("http_requests_total",
{"method": request.method, "endpoint": normalized_path, "status": status})
```
### 测试覆盖
**文件:`tests/test_middleware.py`**
新增 6 个测试用例:
- `test_normalize_jobs_path` - 测试任务路径规范化
- `test_normalize_other_paths` - 测试其他路径保持不变
- `test_normalize_jobs_root` - 测试 /jobs 根路径
- `test_skip_health_endpoints` - 测试跳过健康检查端点
- `test_record_normal_endpoints` - 测试记录普通端点
- `test_normalize_job_path` - 测试规范化任务路径的集成测试
所有测试通过:✅ 56/56 passed
## 验证方法
### 手动测试
使用提供的测试脚本:
```bash
./scripts/test_metrics_filtering.sh
```
### 预期结果
访问 `/metrics` 端点后,应该看到:
**应该出现的指标:**
```
http_requests_total{method="POST",endpoint="/invoke",status="success"} 1
http_requests_total{method="GET",endpoint="/jobs/{job_id}",status="error"} 2
```
**不应该出现的指标:**
```
http_requests_total{method="GET",endpoint="/healthz",...}
http_requests_total{method="GET",endpoint="/readyz",...}
http_requests_total{method="GET",endpoint="/metrics",...}
http_requests_total{method="GET",endpoint="/jobs/a1b2c3d4e5f6",...}
```
## 扩展性
如果需要添加更多路径规范化规则,只需修改 `normalize_path()` 函数:
```python
def normalize_path(path: str) -> str:
"""规范化路径,将路径参数替换为模板形式"""
# 任务路径
if path.startswith("/jobs/") and len(path) > 6:
return "/jobs/{job_id}"
# 用户路径(示例)
if path.startswith("/users/") and len(path) > 7:
return "/users/{user_id}"
# 其他路径保持不变
return path
```
## 影响范围
- ✅ 不影响现有功能
- ✅ 不影响 API 行为
- ✅ 仅影响指标记录逻辑
- ✅ 向后兼容
- ✅ 所有测试通过
## 相关文档
- [监控指南](../docs/monitoring.md) - 已更新指标说明
- [测试脚本](../scripts/test_metrics_filtering.sh) - 手动验证脚本

View File

@@ -61,6 +61,19 @@ docker-compose up -d redis prometheus grafana
| `http_request_duration_seconds` | Histogram | method, endpoint | HTTP 请求延迟分布 | | `http_request_duration_seconds` | Histogram | method, endpoint | HTTP 请求延迟分布 |
| `http_requests_in_progress` | Gauge | - | 当前进行中的请求数 | | `http_requests_in_progress` | Gauge | - | 当前进行中的请求数 |
**注意事项:**
1. **跳过的端点**:以下端点不会被记录到指标中,以减少噪音:
- `/metrics` - 指标端点本身
- `/healthz` - 存活检查
- `/readyz` - 就绪检查
2. **路径规范化**:带有路径参数的端点会被规范化为模板形式:
- `GET /jobs/a1b2c3d4e5f6``GET /jobs/{job_id}`
- `GET /jobs/xyz123456789``GET /jobs/{job_id}`
这样可以避免因为不同的路径参数值产生过多的指标标签,导致指标基数爆炸。
### 算法执行指标 ### 算法执行指标
| 指标 | 类型 | 标签 | 描述 | | 指标 | 类型 | 标签 | 描述 |

View File

@@ -1,114 +0,0 @@
#!/bin/bash
# 指标方案快速启动脚本
set -e
# 颜色定义
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m' # No Color
echo "=========================================="
echo "FunctionalScaffold 指标方案启动脚本"
echo "=========================================="
# 检查 docker-compose
if ! command -v docker-compose &> /dev/null; then
echo -e "${RED}错误: docker-compose 未安装${NC}"
exit 1
fi
# 选择方案
echo ""
echo "请选择指标方案:"
echo "1. Pushgateway推荐适合 Serverless"
echo "2. Redis + Exporter适合高并发"
echo "3. 两者都启动(用于对比测试)"
echo ""
read -p "输入选项 (1/2/3): " choice
cd "$(dirname "$0")/../deployment"
case $choice in
1)
echo -e "${GREEN}启动 Pushgateway 方案...${NC}"
docker-compose up -d redis pushgateway prometheus grafana
echo ""
echo -e "${GREEN}✓ Pushgateway 方案已启动${NC}"
echo ""
echo "服务地址:"
echo " - Pushgateway: http://localhost:9091"
echo " - Prometheus: http://localhost:9090"
echo " - Grafana: http://localhost:3000 (admin/admin)"
echo ""
echo "下一步:"
echo " 1. 修改代码导入: from functional_scaffold.core.metrics_pushgateway import ..."
echo " 2. 配置环境变量: PUSHGATEWAY_URL=localhost:9091"
echo " 3. 启动应用: ./scripts/run_dev.sh"
echo " 4. 运行测试: python scripts/test_metrics.py pushgateway"
;;
2)
echo -e "${GREEN}启动 Redis 方案...${NC}"
# 检查 redis 依赖
if ! python -c "import redis" 2>/dev/null; then
echo -e "${YELLOW}警告: redis 库未安装${NC}"
echo "正在安装 redis..."
pip install redis
fi
docker-compose up -d redis redis-exporter prometheus grafana
echo ""
echo -e "${GREEN}✓ Redis 方案已启动${NC}"
echo ""
echo "服务地址:"
echo " - Redis: localhost:6379"
echo " - Redis Exporter: http://localhost:8001/metrics"
echo " - Prometheus: http://localhost:9090"
echo " - Grafana: http://localhost:3000 (admin/admin)"
echo ""
echo "下一步:"
echo " 1. 修改代码导入: from functional_scaffold.core.metrics_redis import ..."
echo " 2. 配置环境变量: REDIS_HOST=localhost REDIS_PORT=6379"
echo " 3. 启动应用: ./scripts/run_dev.sh"
echo " 4. 运行测试: python scripts/test_metrics.py redis"
;;
3)
echo -e "${GREEN}启动所有服务...${NC}"
# 检查 redis 依赖
if ! python -c "import redis" 2>/dev/null; then
echo -e "${YELLOW}警告: redis 库未安装${NC}"
echo "正在安装 redis..."
pip install redis
fi
docker-compose up -d
echo ""
echo -e "${GREEN}✓ 所有服务已启动${NC}"
echo ""
echo "服务地址:"
echo " - 应用: http://localhost:8000"
echo " - Pushgateway: http://localhost:9091"
echo " - Redis: localhost:6379"
echo " - Redis Exporter: http://localhost:8001/metrics"
echo " - Prometheus: http://localhost:9090"
echo " - Grafana: http://localhost:3000 (admin/admin)"
echo ""
echo "下一步:"
echo " 1. 查看文档: cat docs/metrics-guide.md"
echo " 2. 运行测试: python scripts/test_metrics.py"
;;
*)
echo -e "${RED}无效的选项${NC}"
exit 1
;;
esac
echo ""
echo "=========================================="
echo "查看日志: docker-compose logs -f"
echo "停止服务: docker-compose down"
echo "查看文档: cat ../docs/metrics-guide.md"
echo "=========================================="

View File

@@ -53,6 +53,27 @@ async def log_requests(request: Request, call_next):
return response return response
def normalize_path(path: str) -> str:
"""
规范化路径,将路径参数替换为模板形式
Args:
path: 原始路径
Returns:
规范化后的路径
Examples:
/jobs/a1b2c3d4e5f6 -> /jobs/{job_id}
/invoke -> /invoke
"""
# 匹配 /jobs/{任意字符串} 模式
if path.startswith("/jobs/") and len(path) > 6:
return "/jobs/{job_id}"
return path
# 指标跟踪中间件 # 指标跟踪中间件
@app.middleware("http") @app.middleware("http")
async def track_metrics(request: Request, call_next): async def track_metrics(request: Request, call_next):
@@ -60,8 +81,9 @@ async def track_metrics(request: Request, call_next):
if not settings.metrics_enabled: if not settings.metrics_enabled:
return await call_next(request) return await call_next(request)
# 跳过 /metrics 端点本身,避免循环记录 # 跳过不需要记录指标的端点
if request.url.path == "/metrics": skip_paths = {"/metrics", "/readyz", "/healthz"}
if request.url.path in skip_paths:
return await call_next(request) return await call_next(request)
gauge_incr("http_requests_in_progress") gauge_incr("http_requests_in_progress")
@@ -79,13 +101,15 @@ async def track_metrics(request: Request, call_next):
raise e raise e
finally: finally:
elapsed = time.time() - start_time elapsed = time.time() - start_time
# 使用规范化后的路径记录指标
normalized_path = normalize_path(request.url.path)
incr( incr(
"http_requests_total", "http_requests_total",
{"method": request.method, "endpoint": request.url.path, "status": status}, {"method": request.method, "endpoint": normalized_path, "status": status},
) )
observe( observe(
"http_request_duration_seconds", "http_request_duration_seconds",
{"method": request.method, "endpoint": request.url.path}, {"method": request.method, "endpoint": normalized_path},
elapsed, elapsed,
) )
gauge_decr("http_requests_in_progress") gauge_decr("http_requests_in_progress")