• Welcome to the world's largest Chinese hacker forum

    Welcome to the world's largest Chinese hacker forum, our forum registration is open! You can now register for technical communication with us, this is a free and open to the world of the BBS, we founded the purpose for the study of network security, please don't release business of black/grey, or on the BBS posts, to seek help hacker if violations, we will permanently frozen your IP and account, thank you for your cooperation. Hacker attack and defense cracking or network Security

    business please click here: Creation Security  From CNHACKTEAM

Recommended Posts

目录开放跟踪规范为什么需要开放跟踪什么是跟踪?典型的Trace case Skywalking函数引入了总体架构跟踪、日志记录和度量。NET6对接天行部署天行环境对接。NET6程序添加依赖编辑skyapm.json手动编写skyapm.json自动生成Skyapm.json安装CLI(SkyAPM。DotNet.CLI)自动生成skya Pm.json文件在launchSettings.json文件中配置SK,在startup.cs文件中添加获取traceId自定义调用链接的信息,访问微服务网关,访问后台微服务网关,添加依赖副本配置文件并简单修改,在launchsettings.json中添加环境变量,修改网关配置文件,添加OrderServiceInstance微服务的路由,启动网关,订购微服务访问,添加依赖副本配置文件并简单修改,在launchsettings.json中添加环境变量,启动订购微服务用户微

OpenTracing规范

OpenTracing是一种分布式系统链路跟踪的设计原则、规范、标准。's JDBC式的规范主要是提供一套标准的JDBC API。OpenTracing也是如此,它是一种规范,旨在为链接跟踪提供一组标准API。OpenTracing提供了平台无关和厂商无关的API,使得开发者可以很容易地添加(或替换)跟踪系统的实现。

为什么需要OpenTracing

OpenTracing提供了平台无关和厂商无关的API,使得开发者可以很容易地添加(或替换)跟踪系统的实现。OpenTracing为运营支撑系统和特定平台提供辅助库。

什么是一个Trace

广义地说,跟踪表示(分布式)系统中事务或流程的执行过程。在OpenTracing标准中,trace是由若干个区间组成的有向无环图(DAG ),每个区间代表trace中一个命名的、定时的连续执行段。

什么是一个Trace

分布式跟踪中的每个组件都包含自己的一个或多个区间。比如在一个常规的RPC调用过程中,OpenTracing建议在RPC的客户端和服务器端至少有一个span,用来记录RPC调用的客户端和服务器的信息。

什么是一个Trace2

父跨度将显示多个子跨度以并行或串行方式开始。在OpenTracing标准中,即使是一个子跨度也可以有多个父跨度(例如,并行写入的缓存,可以通过刷新操作写入)。

一个典型的Trace案例

m/blogs/619627/galleries/1815096/o_220501025757_Snipaste_2022-05-01_10-51-14.png"/>

在一个分布式系统中,追踪一个事务或者调用流一般如上图所示。虽然这种图对于看清各组件的组合关系是很有用的,但是,它不能很好显示组件的调用时间,是串行调用还是并行调用,如果展现更复杂的调用关系,会更加复杂,甚至无法画出这样的图。另外,这种图也无法显示调用间的时间间隔以及是否通过定时调用来启动调用。一种更有效的展现一个典型的trace过程,如下图所示:

一个典型的Trace案例2

这种展现方式增加显示了执行时间的上下文,相关服务间的层次关系,进程或者任务的串行或并行调用关系。这样的视图有助于发现系统调用的关键路径。通过关注关键路径的执行过程,项目团队可能专注于优化路径中的关键位置,最大幅度的提升系统性能。例如:可以通过追踪一个资源定位的调用情况,明确底层的调用情况,发现哪些操作有阻塞的情况。

Skywalking

Skywalking是一款APM(Application Performance Management & Monitoring)系统。Skywalking是分布式系统应用程序性能监视工具,专为微服务、云原生架构和基于容器(Docker、K8s、Mesos)架构而设计。提供分布式追踪、服务网格遥测分析、度量聚合和可视化一体化解决方案。

功能介绍

  • 多种监控手段。可以通过语言探针和 service mesh 获得监控是数据。
  • 多个语言自动探针。包括 Java,.NET Core 和 Node.JS。
  • 轻量高效。无需大数据平台,和大量的服务器资源。
  • 模块化。UI、存储、集群管理都有多种机制可选。
  • 支持告警。
  • 优秀的可视化解决方案。

整体架构

Skywalking

整个架构,分成上、下、左、右四部分:

  • 探针基于不同的来源可能是不一样的, 但作用都是收集数据, 将数据格式化为 SkyWalking 适用的格式.
  • 平台后端是一个支持集群模式运行的后台, 用于数据聚合, 数据分析以及驱动数据流从探针到用户界面的流程. 平台后端还提供了各种可插拔的能力, 如不同来源数据(如来自 Zipkin)格式化, 不同存储系统以及集群管理. 你甚至还可以使用观测分析语言来进行自定义聚合分析.
  • 存储是开放式的. 你可以选择一个既有的存储系统, 如 ElasticSearch, H2 或 MySQL 集群(Sharding-Sphere 管理), 也可以选择自己实现一个存储系统. 当然, 我们非常欢迎你贡献新的存储系统实现.
  • 用户界面对于 SkyWalking 的最终用户来说非常炫酷且强大. 同样它也是可定制以匹配你已存在的后端的

Tracing、Logging和Metrics

在微服务领域,很早以来就形成了Tracing、Logging和Metrics相辅相成,合力支撑多维度、多形态的监控体系,三类监控各有侧重:

Tracing:它在单次请求的范围内,处理信息。 任何的数据、元数据信息都被绑定到系统中的单个事务上。例如:一次调用远程服务的RPC执行过程;一次实际的SQL查询语句;一次HTTP请求的业务性ID;

Logging:日志,不知道大家有没有想过它的定义或者边界。Logging即是记录处理的离散事件,比如我们应用的调试信息或者错误信息等发送到ES;审计跟踪时间信息通过Kafka处理送到BigTable等数据仓储等等,大多数情况下记录的数据很分散,并且相互独立,也许是错误信息,也许仅仅只是记录当前的事件状态,或者是警告信息等等。

Metrics:当我们想知道我们服务的请求QPS是多少,或者当天的用户登录次数等等,这时我们可能需要将一部分事件进行聚合或计数,也就是我们说的Metrics。可聚合性即是Metrics的特征,它们是一段时间内某个度量(计数器或者直方图)的原子或者是元数据。例如接收的HTTP数量可以被建模为计数器,每次的HTTP请求即是我们的度量元数据,可以进行简单的加法聚合,当持续了一段时间我们又可以建模为直方图。

关系图

.NET6 对接 Skywalking

部署Skywalking环境

version: '3.3'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.5.0
    container_name: elasticsearch
    restart: always
    ports:
      - 9200:9200
    environment:
      - discovery.type=single-node
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms256m -Xmx256m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
  oap:
    image: apache/skywalking-oap-server:8.8.0
    container_name: oap
    depends_on:
      - elasticsearch
    links:
      - elasticsearch
    restart: always
    ports:
      - 11800:11800
      - 12800:12800
    environment:
      SW_STORAGE: elasticsearch
      SW_STORAGE_ES_CLUSTER_NODES: elasticsearch:9200
  ui:
    image: apache/skywalking-ui
    container_name: ui
    depends_on:
      - oap
    links:
      - oap
    restart: always
    ports:
      - 8080:8080
    environment:
      SW_OAP_ADDRESS: http://oap:12800

对接.NET6 程序

添加依赖

<ItemGroup>
    <PackageReference Include="SkyAPM.Agent.AspNetCore" Version="1.3.0" />
</ItemGroup>

编辑Skywalking配置文件skyapm.json

手动编写skyapm.json

{
  "SkyWalking": {
    "ServiceName": "MySkyWalkingDemoTest",
    "Namespace": "",
    "HeaderVersions": [
      "sw8"
    ],
    "Sampling": {
      "SamplePer3Secs": -1,
      "Percentage": -1.0
    },
    "Logging": {
      "Level": "Information",
      "FilePath": "logs\\skyapm-{Date}.log"
    },
    "Transport": {
      "Interval": 3000,
      "ProtocolVersion": "v8",
      "QueueSize": 30000,
      "BatchSize": 3000,
      "gRPC": {
        "Servers": "192.168.3.245:11800",
        "Timeout": 10000,
        "ConnectTimeout": 10000,
        "ReportTimeout": 600000,
        "Authentication": ""
      }
    }
  }
}

自动生成Skyapm.json

安装CLI(SkyAPM.DotNet.CLI)
 dotnet tool install -g SkyAPM.DotNet.CLI
自动生成skyapm.json文件

server name指的就是您刚才配置的SKYWALKING__SERVICENAME,server指的是您Skywalking的ip地址。执行命令后,会自动生成一个skywalking.json 。

 dotnet skyapm config [service name] [server]:11800
 eg: dotnet skyapm config MySkyWalking_OrderService 192.168.3.245:11800

SkyAPM Config 配置说明

ServiceName
服务名称

Sampling
采样配置节点

  • SamplePer3Secs 每3秒采样数
  • Percentage 采样百分比,例如10%采样则配置为10

Logging
日志配置节点

  • Level 日志级别
  • FilePath 日志保存路径

Transport

传输配置节点

  • Interval 每多少毫秒刷新
  • gRPC
    gRPC配置节点
  • Servers gRPC地址,多个用逗号“,”
  • Timeout 创建gRPC链接的超时时间,毫秒
  • ConnectTimeout gRPC最长链接时间,毫秒

launchSettings.json文件配置SK

"profiles": { // 项目
    "IIS Express": { // IIS部署项
      "commandName": "IISExpress",
      "launchBrowser": true,
      "launchUrl": "weatherforecast",
      "environmentVariables": {
        "ASPNETCORE_ENVIRONMENT": "Development",
        "ASPNETCORE_HOSTINGSTARTUPASSEMBLIES": "SkyAPM.Agent.AspNetCore",
        "SKYWALKING__SERVICENAME": "MySkyWalkingDemoTest"
      }
    },
    "SkyWalkingDemo": { // castrol部署项
      "commandName": "Project",
      "launchBrowser": true,
      "launchUrl": "weatherforecast",
      "applicationUrl": "http://localhost:5000",
      "environmentVariables": {
        "ASPNETCORE_ENVIRONMENT": "Development",
        "ASPNETCORE_HOSTINGSTARTUPASSEMBLIES": "SkyAPM.Agent.AspNetCore", // 必须配置
        "SKYWALKING__SERVICENAME": "MySkyWalkingDemoTest" // 必须配置,在skywalking做标识
      }
    }
  }

startup.cs文件中添加

public void ConfigureServices(IServiceCollection services)
{
    services.AddSkyApmExtensions(); // 添加Skywalking相关配置
    services.AddControllers();
    services.AddHttpClient();
}

获取traceId

private readonly IEntrySegmentContextAccessor segContext;
public SkywalkingController(IEntrySegmentContextAccessor segContext)
{
    this.segContext = segContext;
}
/// <summary>
/// 获取链接追踪ID
/// </summary>
/// <returns></returns>
[HttpGet("traceId")]
public string GetSkywalkingTraceId()
{
    return segContext.Context.TraceId;
}

自定义调用链路的信息

[HttpGet]
public async Task<IActionResult> SkywalkingTest()
{
    //获取全局的skywalking的TracId
    var TraceId = _segContext.Context.TraceId;
    Console.WriteLine($"TraceId={TraceId}");
    _segContext.Context.Span.AddLog(LogEvent.Message($"SkywalkingTest---Worker running at: {DateTime.Now}"));
    System.Threading.Thread.Sleep(1000);
    _segContext.Context.Span.AddLog(LogEvent.Message($"SkywalkingTest---Worker running at--end: {DateTime.Now}"));
    return Ok($"Ok,SkywalkingTest-TraceId={TraceId} ");
}

接入微服务网关+后台微服务

网关接入

添加依赖

<ItemGroup>
    <PackageReference Include="SkyAPM.Agent.AspNetCore" Version="1.3.0" />
</ItemGroup>

拷贝配置文件并简单修改

{
  "SkyWalking": {
    "ServiceName": "MySkyWalking_Gateway", #修改名称就OK
    "Namespace": "",
    "HeaderVersions": [
      "sw8"
    ],
    "Sampling": {
      "SamplePer3Secs": -1,
      "Percentage": -1.0
    },
    "Logging": {
      "Level": "Debug",
      "FilePath": "logs\\skyapm-{Date}.log"
    },
    "Transport": {
      "Interval": 3000,
      "ProtocolVersion": "v8",
      "QueueSize": 30000,
      "BatchSize": 3000,
      "gRPC": {
        "Servers": "192.168.3.245:11800",
        "Timeout": 10000,
        "ConnectTimeout": 10000,
        "ReportTimeout": 600000,
        "Authentication": ""
      }
    }
  }
}

launchsettings.json添加环境变量

"profiles": {
    "Zhaoxi.MicroService.GatewayCenter": {
      "commandName": "Project",
      "dotnetRunMessages": true,
      "launchBrowser": true,
      "launchUrl": "swagger",
      "applicationUrl": "https://localhost:7141;http://localhost:5141",
      "environmentVariables": {
        "ASPNETCORE_ENVIRONMENT": "Development",
        "ASPNETCORE_HOSTINGSTARTUPASSEMBLIES": "SkyAPM.Agent.AspNetCore", #添加HOST变量
        "SKYWALKING__SERVICENAME": "MySkyWalking_Gateway" #添加服务名称
      }
    },
    "IIS Express": {
      "commandName": "IISExpress",
      "launchBrowser": true,
      "launchUrl": "swagger",
      "environmentVariables": {
        "ASPNETCORE_ENVIRONMENT": "Development",
        "ASPNETCORE_HOSTINGSTARTUPASSEMBLIES": "SkyAPM.Agent.AspNetCore",
        "SKYWALKING__SERVICENAME": "MySkyWalking_Gateway"
      }
    }
  }

修改网关配置文件,添加OrderServiceInstance微服务的路由

{
    "DownstreamPathTemplate": "/api/{url}", //服务地址--url变量
    "DownstreamScheme": "http",
    "UpstreamPathTemplate": "/microservice/{url}", //网关地址--url变量
    "UpstreamHttpMethod": [ "Get", "Post" ],
    "UseServiceDiscovery": true,
    "ServiceName": "OrderService", //consul服务名称
    "LoadBalancerOptions": {
        "Type": "RoundRobin" //轮询
}

启动网关

dotnet run --urls=http://*:6299

订单微服务接入

添加依赖

<ItemGroup>
    <PackageReference Include="SkyAPM.Agent.AspNetCore" Version="1.3.0" />
</ItemGroup>

拷贝配置文件并简单修改

{
  "SkyWalking": {
    "ServiceName": "MySkyWalking_OrderService",
    "Namespace": "",
    "HeaderVersions": [
      "sw8"
    ],
    "Sampling": {
      "SamplePer3Secs": -1,
      "Percentage": -1.0
    },
    "Logging": {
      "Level": "Debug",
      "FilePath": "logs\\skyapm-{Date}.log"
    },
    "Transport": {
      "Interval": 3000,
      "ProtocolVersion": "v8",
      "QueueSize": 30000,
      "BatchSize": 3000,
      "gRPC": {
        "Servers": "192.168.3.245:11800",
        "Timeout": 10000,
        "ConnectTimeout": 10000,
        "ReportTimeout": 600000,
        "Authentication": ""
      }
    }
  }
}

launchsettings.json添加环境变量

"profiles": {
    "Zhaoxi.MicroService.OrderServiceInstance": {
      "commandName": "Project",
      "dotnetRunMessages": true,
      "launchBrowser": true,
      "launchUrl": "swagger",
      "applicationUrl": "http://192.168.3.105:7900",
      "environmentVariables": {
        "ASPNETCORE_ENVIRONMENT": "Development",
        "ASPNETCORE_HOSTINGSTARTUPASSEMBLIES": "SkyAPM.Agent.AspNetCore",
        "SKYWALKING__SERVICENAME": "MySkyWalking_OrderService"
      }
    },
    "IIS Express": {
      "commandName": "IISExpress",
      "launchBrowser": true,
      "launchUrl": "swagger",
      "environmentVariables": {
        "ASPNETCORE_ENVIRONMENT": "Development"
      }
    }
  }

启动订单微服务

dotnet run

用户微服务接入

步骤和订单微服务一样

配置Skywalking告警

配置告警规则

docker exec -it 12f053748e85 /bin/sh
ls -l

Skywalking告警

查阅配置规则文件及配置规则解读

通过cat alarm-settings.yml可以查阅文件内容,如下:

docker cp 12f053748e85:/skywalking/config/alarm-settings.yml .
# Sample alarm rules.
rules:
  # Rule unique name, must be ended with `_rule`.
  service_resp_time_rule:
    metrics-name: service_resp_time
    op: ">"
    threshold: 1000
    period: 10
    count: 3
    silence-period: 5
    message: Response time of service {name} is more than 1000ms in 3 minutes of last 10 minutes.
  service_sla_rule:
    # Metrics value need to be long, double or int
    metrics-name: service_sla
    op: "<"
    threshold: 8000
    # The length of time to evaluate the metrics
    period: 10
    # How many times after the metrics match the condition, will trigger alarm
    count: 2
    # How many times of checks, the alarm keeps silence after alarm triggered, default as same as period.
    silence-period: 3
    message: Successful rate of service {name} is lower than 80% in 2 minutes of last 10 minutes
  service_resp_time_percentile_rule:
    # Metrics value need to be long, double or int
    metrics-name: service_percentile
    op: ">"
    threshold: 1000,1000,1000,1000,1000
    period: 10
    count: 3
    silence-period: 5
    message: Percentile response time of service {name} alarm in 3 minutes of last 10 minutes, due to more than one condition of p50 > 1000, p75 > 1000, p90 > 1000, p95 > 1000, p99 > 1000
  service_instance_resp_time_rule:
    metrics-name: service_instance_resp_time
    op: ">"
    threshold: 1000
    period: 10
    count: 2
    silence-period: 5
    message: Response time of service instance {name} is more than 1000ms in 2 minutes of last 10 minutes
  database_access_resp_time_rule:
    metrics-name: database_access_resp_time
    threshold: 1000
    op: ">"
    period: 10
    count: 2
    message: Response time of database access {name} is more than 1000ms in 2 minutes of last 10 minutes
  endpoint_relation_resp_time_rule:
    metrics-name: endpoint_relation_resp_time
    threshold: 1000
    op: ">"
    period: 10
    count: 2
    message: Response time of endpoint relation {name} is more than 1000ms in 2 minutes of last 10 minutes
#  Active endpoint related metrics alarm will cost more memory than service and service instance metrics alarm.
#  Because the number of endpoint is much more than service and instance.
#
#  endpoint_avg_rule:
#    metrics-name: endpoint_avg
#    op: ">"
#    threshold: 1000
#    period: 10
#    count: 2
#    silence-period: 5
#    message: Response time of endpoint {name} is more than 1000ms in 2 minutes of last 10 minutes
webhooks:
#  - http://127.0.0.1/notify/
#  - http://127.0.0.1/go-wechat/

规则常用指标解读
rule name: 规则名称,必须唯一,必须以 _rule结尾;
metrics name: oal(Observability Analysis Language)脚本中的度量名;名称在SkyWalking后端服务中已经定义,进入容器skywalking-oap之后,进入如下目录就可以找到。

include names: 本规则告警生效的实体名称,如服务名,终端名;
exclude-names:将此规则作用于不匹配的实体名称上,如服务名,终端名;
threshold: 阈值,可以是一个数组,即可以配置多个值;
op: 操作符, 可以设定 >, <, =;
period: 多久检查一次当前的指标数据是否符合告警规则;以分钟为单位
count: 超过阈值条件,达到count次数,触发告警;
silence period:在同一个周期,指定的silence period时间内,忽略相同的告警消息;
更多告警规则详情,请参照这个地址:https://github.com/apache/skywalking/blob/master/docs/en/setup/backend/backend-alarm.md

修改告警规则

rules:
	service_test_sal_rule:
		# 指定指标名称
    	metrics-name: service_test_sal
    	# 小于
    	op: "<"
    	# 指定阈值
    	threshold: 8000
    	# 每2分钟检测告警该规则
    	period: 2
    	# 触发1次规则就告警
    	count: 1
    	# 设置三分钟内容相同告警,不重复告警
    	silence-period: 3
    	# 配置告警信息
    	message: Successful rate of service {name} is lower than 80% in 2 minutes of last 10 minutes

概要:服务成功率在过去2分钟内低于80%

告警API编写

本质还是SkyWalking根据规则进行检查,如果符合规则条件,就通过WebHook、gRPCHook、WeChat Hook、Dingtalk Hook等方式进行消息通知;接收到告警数据信息之后,可以自行处理消息。这里为了方便,就采用WebHook的方式进行演示,即触发告警条件之后,SkyWalking会调用配置的WebHook 接口,并传递对应的告警信息;

定义数据模型

public class AlarmMsg
{
    public int scopeId { get; set; }
    public string? scope { get; set; }
    public string? name { get; set; }
    public string? id0 { get; set; }
    public string? id1 { get; set; }
    public string? ruleName { get; set; }
    public string? alarmMessage { get; set; }
}

定义WebHook调用API

/// <summary>
/// 告警API
/// </summary>
/// <param name="msgs"></param>
/// <returns></returns>
[HttpPost("AlarmMsg")]
public void AlarmMsg(List<AlarmMsg> msgs)
{
    string msg = "触发告警:";
    msg += msgs.FirstOrDefault()?.alarmMessage;
    Console.WriteLine(msg);
    SendMail(msg);
}

配置webHook

http://192.168.3.105:7900/api/Skywalking/AlarmMsg
# Sample alarm rules.
rules:
  # Rule unique name, must be ended with `_rule`.
  service_resp_time_rule:
    metrics-name: service_resp_time
    op: ">"
    threshold: 1000
    period: 10
    count: 3
    silence-period: 5
    message: Response time of service {name} is more than 1000ms in 3 minutes of last 10 minutes.
  service_sla_rule:
    # Metrics value need to be long, double or int
    metrics-name: service_sla
    op: "<"
    threshold: 8000
    # The length of time to evaluate the metrics
    period: 10
    # How many times after the metrics match the condition, will trigger alarm
    count: 2
    # How many times of checks, the alarm keeps silence after alarm triggered, default as same as period.
    silence-period: 3
    message: Successful rate of service {name} is lower than 80% in 2 minutes of last 10 minutes
  service_resp_time_percentile_rule:
    # Metrics value need to be long, double or int
    metrics-name: service_percentile
    op: ">"
    threshold: 1000,1000,1000,1000,1000
    period: 10
    count: 3
    silence-period: 5
    message: Percentile response time of service {name} alarm in 3 minutes of last 10 minutes, due to more than one condition of p50 > 1000, p75 > 1000, p90 > 1000, p95 > 1000, p99 > 1000
  service_instance_resp_time_rule:
    metrics-name: service_instance_resp_time
    op: ">"
    threshold: 1000
    period: 10
    count: 2
    silence-period: 5
    message: Response time of service instance {name} is more than 1000ms in 2 minutes of last 10 minutes
  database_access_resp_time_rule:
    metrics-name: database_access_resp_time
    threshold: 1000
    op: ">"
    period: 10
    count: 2
    message: Response time of database access {name} is more than 1000ms in 2 minutes of last 10 minutes
  endpoint_relation_resp_time_rule:
    metrics-name: endpoint_relation_resp_time
    threshold: 1000
    op: ">"
    period: 10
    count: 2
    message: Response time of endpoint relation {name} is more than 1000ms in 2 minutes of last 10 minutes
#  Active endpoint related metrics alarm will cost more memory than service and service instance metrics alarm.
#  Because the number of endpoint is much more than service and instance.
#
#  endpoint_avg_rule:
#    metrics-name: endpoint_avg
#    op: ">"
#    threshold: 1000
#    period: 10
#    count: 2
#    silence-period: 5
#    message: Response time of endpoint {name} is more than 1000ms in 2 minutes of last 10 minutes
webhooks:
  - http://192.168.3.105:7900/api/Skywalking/AlarmMsg
#  - http://127.0.0.1/go-wechat/
rules:
  # 告警规则名称,必须唯一,以_rule结尾
  service_sla_rule:
     # 指定metrics-name
     metrics-name: service_sla
     # 小于
     op: "<" 
     # 指定阈值
     threshold: 8000                                                                           
     # 10分钟检测一次告警规则                                                 
     period: 10                                                                                 
     # 触发2次告警规则就告警                   
     count: 2                                                                                   
     # 设置的3分钟时间段有相同的告警,不重复告警.
     silence-period: 3 
      # 配置告警消息
     message: Successful rate of service {name} is lower than 80% in 2 minutes of last 10 minutes
webhooks:
- http://192.168.3.105:7900/api/Skywalking/AlarmMsg
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now