https://blog.ohtly.com/tags/prometheus/ Recent content in Prometheus on OHTLY Blog Hugo zh-cn Fri, 29 May 2026 20:21:07 +0800 https://blog.ohtly.com/posts/postgresql-observability/ Fri, 29 May 2026 20:21:07 +0800 https://blog.ohtly.com/posts/postgresql-observability/ <h2 id="前言"> 前言 <a class="heading-link" href="#%e5%89%8d%e8%a8%80"> <i class="fa-solid fa-link" aria-hidden="true" title="链接到标题"></i> <span class="sr-only">链接到标题</span> </a> </h2> <p>PostgreSQL 是生产环境最常用的关系型数据库之一。当服务挂了、连接爆了、死锁了，需要第一时间感知。</p> <p>单纯靠 Prometheus 指标可以告诉你「连接数超了」，但说不出原因；单纯靠日志可以告诉你「too many connections」，但没有量化趋势。</p> <p>本文的方案是 <strong>指标 + 日志协同监控</strong>，形成完整可观测性闭环：</p> <ul> <li>指标告警 → 感知异常（量变）</li> <li>日志告警 → 定位根因（质变）</li> <li>统一通知 → 飞书即时推送</li> </ul> <h2 id="架构"> 架构 <a class="heading-link" href="#%e6%9e%b6%e6%9e%84"> <i class="fa-solid fa-link" aria-hidden="true" title="链接到标题"></i> <span class="sr-only">链接到标题</span> </a> </h2> <p>下面是完整的数据流：</p> <div class="mermaid"> graph TD PG[PostgreSQL 17<br/>monkey:5432] --> PE[postgres-exporter<br/>monkey:9187] PE --> PM[Prometheus<br/>robin:9090] PM --> AM[Alertmanager<br/>robin:9093] PG -->|Docker logs| AL[Alloy<br/>monkey:12346] AL --> LK[Loki<br/>robin:3100] LK -->|Loki Ruler LogQL| AM AM --> AT[alert-transformer<br/>robin:9091] AT --> OC[OpenClaw<br/>rivo:18789] OC --> FS[飞书] subgraph 指标路径 PE PM end subgraph 日志路径 AL LK end subgraph 通知路径 AM AT OC FS end </div> <p>两条路径独立采集、独立告警，最终汇总到同一个通知链路。</p> https://blog.ohtly.com/posts/2026-05-29-minio-prometheus-monitoring/ Fri, 29 May 2026 19:29:59 +0800 https://blog.ohtly.com/posts/2026-05-29-minio-prometheus-monitoring/ <p>前四篇介绍了不同场景的监控方案：</p> <ul> <li><a href="https://blog.ohtly.com/posts/2026-05-29-couchdb-loki-monitoring/" >CouchDB 日志告警</a> → Alloy + Loki + LogQL</li> <li><a href="https://blog.ohtly.com/posts/2026-05-29-windmill-loki-monitoring/" >Windmill 日志告警</a> → Alloy + Loki + LogQL</li> <li><a href="https://blog.ohtly.com/posts/2026-05-29-alertmanager-feishu/" >告警通知到飞书</a> → Alertmanager → transformer → OpenClaw</li> <li>MinIO 指标监控（本篇）→ 一行配置 + Prometheus 指标告警</li> </ul> <p>这篇最短，因为 MinIO 自带 Prometheus 端点，不需要额外 exporter，也不需要日志采集。</p> https://blog.ohtly.com/posts/2026-05-29-alertmanager-feishu/ Fri, 29 May 2026 19:10:00 +0800 https://blog.ohtly.com/posts/2026-05-29-alertmanager-feishu/ <p>前两篇介绍了用 Alloy + Loki Ruler 实现日志告警：</p> <ul> <li><a href="https://blog.ohtly.com/posts/2026-05-29-couchdb-loki-monitoring/" >CouchDB 日志监控</a>：文本日志的 LogQL 告警</li> <li><a href="https://blog.ohtly.com/posts/2026-05-29-windmill-loki-monitoring/" >Windmill 日志监控</a>：JSON 日志的 LogQL 告警</li> </ul> <p>两条链路最终都汇入 Alertmanager，由它统一分派到飞书。但 Alertmanager 原生没有飞书 receiver——它的 webhook 输出的是原始 JSON，飞书看不懂。中间需要一层适配器来转换格式。这个适配器就是 <strong>alert-transformer</strong>。</p> https://blog.ohtly.com/posts/2026-05-16-docker-couchdb-prometheus-monitoring/ Sat, 16 May 2026 10:00:00 +0800 https://blog.ohtly.com/posts/2026-05-16-docker-couchdb-prometheus-monitoring/ <p>最近在项目中使用了 CouchDB 作为文档数据库，考虑到生产环境的需求，研究了一套基于 Prometheus 的监控方案。这套方案全部跑在 Docker 环境下，现在整理成文，供有需要的同学参考。</p> https://blog.ohtly.com/posts/2026-05-06-dcgm-exporter-gpu-alert/ Tue, 05 May 2026 21:53:58 +0800 https://blog.ohtly.com/posts/2026-05-06-dcgm-exporter-gpu-alert/ <p>GPU 是生产环境中重要的计算资源，温度过高、显存泄漏、硬件故障等问题如果不及时发现，可能影响线上服务。本文记录一套完整的 GPU 监控告警方案，从指标采集到飞书推送的全自动链路。</p> <h2 id="监控目标"> 监控目标 <a class="heading-link" href="#%e7%9b%91%e6%8e%a7%e7%9b%ae%e6%a0%87"> <i class="fa-solid fa-link" aria-hidden="true" title="链接到标题"></i> <span class="sr-only">链接到标题</span> </a> </h2> <table> <thead> <tr> <th>维度</th> <th>指标</th> <th>告警阈值</th> </tr> </thead> <tbody> <tr> <td>温度</td> <td>GPU 核心温度 / 显存温度</td> <td>&gt; 83°C / &gt; 100°C</td> </tr> <tr> <td>显存</td> <td>VRAM 使用率</td> <td>&gt; 90%</td> </tr> <tr> <td>功耗</td> <td>GPU 功率</td> <td>&gt; 150W</td> </tr> <tr> <td>硬件健康</td> <td>PCIe 重连、行重映射错误</td> <td>任何异常</td> </tr> <tr> <td>可用性</td> <td>DCGM-Exporter 进程是否存活</td> <td>宕机 1m</td> </tr> </tbody> </table> <h2 id="架构总览"> 架构总览 <a class="heading-link" href="#%e6%9e%b6%e6%9e%84%e6%80%bb%e8%a7%88"> <i class="fa-solid fa-link" aria-hidden="true" title="链接到标题"></i> <span class="sr-only">链接到标题</span> </a> </h2> <div class="mermaid"> flowchart LR A[DCGM-Exporter<br/>monkey:9400] -->|scrape| B[Prometheus<br/>robin:9090] B -->|alert rules| C[Alertmanager<br/>robin:9093] C -->|webhook| D[alert-transformer<br/>rivo:9091] D -->|hooks| E[OpenClaw] E -->|飞书机器人| F[飞书] </div> <h2 id="部署-dcgm-exporter"> 部署 DCGM-Exporter <a class="heading-link" href="#%e9%83%a8%e7%bd%b2-dcgm-exporter"> <i class="fa-solid fa-link" aria-hidden="true" title="链接到标题"></i> <span class="sr-only">链接到标题</span> </a> </h2> <p>在 GPU 节点上通过 Docker 部署 DCGM-Exporter。使用 DaoCloud 镜像加速国内拉取：</p> https://blog.ohtly.com/posts/2026-05-05-%E6%99%BA%E8%83%BD%E7%9B%91%E6%8E%A7%E6%8A%80%E6%9C%AF%E6%96%B9%E6%A1%88/ Tue, 05 May 2026 13:49:54 +0800 https://blog.ohtly.com/posts/2026-05-05-%E6%99%BA%E8%83%BD%E7%9B%91%E6%8E%A7%E6%8A%80%E6%9C%AF%E6%96%B9%E6%A1%88/ <h1 id="智能监控技术方案"> 智能监控技术方案 <a class="heading-link" href="#%e6%99%ba%e8%83%bd%e7%9b%91%e6%8e%a7%e6%8a%80%e6%9c%af%e6%96%b9%e6%a1%88"> <i class="fa-solid fa-link" aria-hidden="true" title="链接到标题"></i> <span class="sr-only">链接到标题</span> </a> </h1> <h2 id="概述"> 概述 <a class="heading-link" href="#%e6%a6%82%e8%bf%b0"> <i class="fa-solid fa-link" aria-hidden="true" title="链接到标题"></i> <span class="sr-only">链接到标题</span> </a> </h2> <p>本文档描述基于开源组件构建的智能监控告警体系，实现从指标采集到飞书通知的完整告警流程。</p> <h3 id="架构概览"> 架构概览 <a class="heading-link" href="#%e6%9e%b6%e6%9e%84%e6%a6%82%e8%a7%88"> <i class="fa-solid fa-link" aria-hidden="true" title="链接到标题"></i> <span class="sr-only">链接到标题</span> </a> </h3> <p><strong>告警处理流程</strong></p> <div class="mermaid"> flowchart TD A["node-exporter<br/>:9100"] B["Prometheus<br/>:9090"] C["Alertmanager<br/>:9093"] D["alert-transformer<br/>:9091"] E["OpenClaw<br/>:18789"] F["飞书 Webhook"] G["告警规则<br/>alerts.yml"] A -->|指标采集| B B -->|告警评估| C B -.->|加载| G C -->|Webhook| D D -->|格式转换| E E -->|飞书通知| F </div> <p><strong>指标采集流程</strong></p> <div class="mermaid"> flowchart LR subgraph 指标 H["node-exporter<br/>:9100"] I["Prometheus<br/>:9090"] H -->|"Prometheus 抓取 (15s)"| I end subgraph 日志 J["Docker 日志"] K["Alloy<br/>:12345"] L["Loki<br/>:3100"] J -->|采集| K K -->|发送| L end </div> <h3 id="组件职责"> 组件职责 <a class="heading-link" href="#%e7%bb%84%e4%bb%b6%e8%81%8c%e8%b4%a3"> <i class="fa-solid fa-link" aria-hidden="true" title="链接到标题"></i> <span class="sr-only">链接到标题</span> </a> </h3> <table> <thead> <tr> <th>组件</th> <th>职责</th> <th>端口</th> <th>部署位置</th> </tr> </thead> <tbody> <tr> <td>node-exporter</td> <td>系统指标采集 (CPU/内存/磁盘/网络)</td> <td>9100</td> <td>每台被监控主机</td> </tr> <tr> <td>Prometheus</td> <td>指标抓取、存储、告警规则评估</td> <td>9090</td> <td>监控服务器</td> </tr> <tr> <td>Alertmanager</td> <td>告警去重、分组、路由、抑制</td> <td>9093</td> <td>监控服务器</td> </tr> <tr> <td>alert-transformer</td> <td>告警格式转换为 OpenClaw API</td> <td>9091</td> <td>监控服务器</td> </tr> <tr> <td>OpenClaw</td> <td>AI 决策、自然语言处理、飞书通知</td> <td>18789</td> <td>OpenClaw 服务器</td> </tr> <tr> <td>Loki</td> <td>日志存储和查询</td> <td>3100</td> <td>监控服务器</td> </tr> <tr> <td>Alloy</td> <td>Docker 日志采集</td> <td>12345</td> <td>监控服务器</td> </tr> </tbody> </table> <h2 id="部署配置"> 部署配置 <a class="heading-link" href="#%e9%83%a8%e7%bd%b2%e9%85%8d%e7%bd%ae"> <i class="fa-solid fa-link" aria-hidden="true" title="链接到标题"></i> <span class="sr-only">链接到标题</span> </a> </h2> <h3 id="node-exporter-部署"> node-exporter 部署 <a class="heading-link" href="#node-exporter-%e9%83%a8%e7%bd%b2"> <i class="fa-solid fa-link" aria-hidden="true" title="链接到标题"></i> <span class="sr-only">链接到标题</span> </a> </h3> <p>每台被监控主机部署 node-exporter，建议使用 Docker Compose：</p>