1172 lines
29 KiB
Markdown
1172 lines
29 KiB
Markdown
# 【E21现场】IGW站点security policy block www.target.com 失败
|
||
|
||
| ID | Creation Date | Assignee | Status |
|
||
|----|----------------|----------|--------|
|
||
| OMPUB-466 | 2022-04-26T22:02:07.000+0800 | 刘学利 | 已关闭 |
|
||
|
||
|
||
---
|
||
|
||
业主自己尝试配置security policy block 网站：[https://www.target.com/] ，
|
||
|
||
策略配置如下：
|
||
|
||
client ip=196.188.136.150（办公室公网ip）
|
||
|
||
application：ssl
|
||
|
||
sni-FQDN:*target.com
|
||
|
||
*targetimg1.com
|
||
|
||
$target.scene7.com
|
||
|
||
$target.com
|
||
|
||
Sub Action:reset
|
||
|
||
Effective Devices：{color:#0747a6}*所有IGW站点*{color}
|
||
|
||
|
||
|
||
策略测试效果：
|
||
|
||
浏览器一直访问[https://www.target.com/ |https://www.target.com/]，最初无法访问成功，大概一分钟左右网站可以正常访问。
|
||
|
||
|
||
|
||
并将以下内容上传到附件中：
|
||
|
||
策略配置内容截图
|
||
|
||
处理机计算板上drop情况
|
||
|
||
该策略安全策略命中日志
|
||
|
||
会话日志里搜索sni =%target% client ip=196.188.136.150导出session records
|
||
|
||
|
||
|
||
查询安全策略命中日志和会话日志时stream direction 只看到double 和c2s,没有s2c日志记录。
|
||
|
||
|
||
|
||
**liuxueli** commented on *2022-04-27T10:01:47.937+0800*:
|
||
|
||
* [~liuju] 这种问题需要现场捕包，在客户端捕包发我，我分析一下
|
||
* 另外我在京版尝试能不能复现
|
||
|
||
|
||
|
||
---
|
||
|
||
**dongxiaoyan** commented on *2022-04-27T10:44:19.639+0800*:
|
||
|
||
[~liuxueli]信息港复测：
|
||
1、浏览器正常访问，多访问几次打开一次；
|
||
2、无痕模式访问，一直未打开；
|
||
3、再次清理缓存后，浏览器正常访问，隔几分钟刷新一次，一直未打开；
|
||
|
||
|
||
|
||
---
|
||
|
||
**liuxueli** commented on *2022-04-27T11:45:50.356+0800*:
|
||
|
||
* 京版复现情况(京版环境不能稳定复现)：
|
||
|
||
*
|
||
** 京版唯一复现的一次，同时开启了捕包，数据包中存在18个ssl链接的SNI包含(target.com)的域名，
|
||
*** 16个链接在client hello收到功能端发出的RESET后，链接关闭
|
||
*** 2个链接未收到RESET,，链接正常建立
|
||
**** 在SessionRecord中未查到该2个链接的记录，客户端端口为17375/17376
|
||
**** SSL解析层测试18个链接均能解析出SNI
|
||
*** [^18-target.com.pcap]
|
||
|
||
|
||
|
||
---
|
||
|
||
**gitlab** commented on *2022-04-27T11:49:29.384+0800*:
|
||
|
||
[刘学利|https://git.mesalab.cn/liuxueli] mentioned this issue in [a commit|https://git.mesalab.cn/MESA_Platform/ssl/-/commit/8e49d1f437fe377a6dabae5bcbc623c2bc34b5d3] of [MESA Platform / ssl|https://git.mesalab.cn/MESA_Platform/ssl] on branch [bugfix-add-test-case|https://git.mesalab.cn/MESA_Platform/ssl/-/tree/bugfix-add-test-case]:{quote}OMPUB-466: 增加相应的测试用例{quote}
|
||
|
||
|
||
|
||
---
|
||
|
||
**liuxueli** commented on *2022-04-27T11:51:39.774+0800*:
|
||
|
||
* [bugfix-add-test-case|https://git.mesalab.cn/MESA_Platform/ssl/-/tree/bugfix-add-test-case]: 分支未修复任何BUG，仅增加测试用例，需继续定位该问题。
|
||
|
||
|
||
|
||
---
|
||
|
||
**liuxueli** commented on *2022-04-27T15:20:25.195+0800*:
|
||
|
||
* E现场复测，客户端未收到RESET包，查看日志发现RESET包发送失败报错
|
||
** 参见：TSG-10508
|
||
|
||
|
||
|
||
---
|
||
|
||
**liuxueli** commented on *2022-04-27T17:29:09.456+0800*:
|
||
|
||
* 分析E现场反馈回来数据包，有3个链接未阻断成功，经分析原因：在Bole IGW站点的CPU使用过载触发SAPP overload protection机制导致链接被Bypass(sapp未建立对应的流表)
|
||
** 当单核CPU使用率超95%时，会触发SAPP的overload protection机制，SAPP配置文件如下：
|
||
|
||
*
|
||
**
|
||
*** !image-2022-04-27-17-53-42-859.png!
|
||
** 分析在NPB上捕获的数据包，找到3个未阻断成功链接对应的client ISN，查询SessionRecord日志，仅在Old Airport站点查到日志
|
||
*** NPB捕获的数据包
|
||
**** [^target.com-196.188.136.150-151.101.2.187.443.pcap]
|
||
*** Session日志记录
|
||
**** [^sessionRecords-deny-target.com-failed.xlsx]
|
||
** 查看NPB DDOS Bypass监控，TCP Bypass 3~5K/S个链接，UDP Bypass 1.55K/S个链接：
|
||
*** !image-2022-04-27-17-12-00-094.png|width=1090,height=508!
|
||
** 查看NPB原始流量
|
||
*** !image-2022-04-27-17-14-25-901.png|width=1093,height=506!
|
||
** 查看NPB 整体CPU使用率
|
||
*** !image-2022-04-27-17-15-02-774.png|width=1079,height=492!
|
||
** 查看NPB 单核CPU使用率
|
||
*** !image-2022-04-27-17-17-32-639.png!
|
||
** 查看perf top -C cpuid的结果
|
||
*** !image-2022-04-27-17-19-09-881.png!
|
||
*** !image-2022-04-27-17-18-43-458.png!
|
||
** 查看火焰图
|
||
*** [^perf.svg]
|
||
*** [^perf33.svg]
|
||
*** [^perf37.svg]
|
||
*** [^perf40.svg]
|
||
|
||
|
||
|
||
---
|
||
|
||
**liuxueli** commented on *2022-04-27T17:36:10.537+0800*:
|
||
|
||
* 火焰图中_int_malloc调用较多的地方
|
||
** tsg_master
|
||
*** !image-2022-04-27-17-36-03-049.png|width=1139,height=433!
|
||
** MAAT
|
||
*** !image-2022-04-27-17-35-20-050.png|width=1152,height=213!
|
||
**
|
||
|
||
|
||
|
||
---
|
||
|
||
**liuxueli** commented on *2022-04-28T10:27:29.680+0800*:
|
||
|
||
* E现场反馈回来的扫描状态显示，FQDN命中率较高：
|
||
** APP SKETCH扫描状态[^app_sketch_maat.txt]
|
||
*** FQDN识别特征命中率为80%，
|
||
*** APP_SIG_SESSION_ATTRIBUTE_STRING表命中率: 35%
|
||
*** APP_SIG_SESSION_ATTRIBUTE_INTEGER表命中率: 10%
|
||
** TSG功能端扫描状态[^tsg_static_maat.txt]
|
||
*** FQDN命中率为: 58%
|
||
*** KEYWORDS命中率: 3.6%
|
||
*** APP_ID命中率: 96%(无性能影响)
|
||
*** FQDN_CAT命中率: 100%(无性能影响)
|
||
|
||
|
||
|
||
---
|
||
|
||
**liuxueli** commented on *2022-04-28T11:03:48.107+0800*:
|
||
|
||
* [~liuju] 到现场后备份一下Bifang的MariaDB，我需要验证现场的哪个FQDN的对象命中率比较高
|
||
** 京版环境可能跟现场有差别
|
||
* E现场备份回来的21.11版本，存在重复的FQDN Object现象，具体列表：
|
||
** [^e21-version-21.11-dup-fqdn-object.txt]
|
||
** ^经分析确认为APP内置特征(VPN)，一个FQDN在多个Object中出现^
|
||
** ^对应BUG: TSG-10517^
|
||
|
||
|
||
|
||
---
|
||
|
||
**liuxueli** commented on *2022-04-28T14:34:33.918+0800*:
|
||
|
||
* E现场备份回来的Bifang MariaDB v21.11版本发现，APP内置特征中包含三个极易命中的特征
|
||
** APP_SIG_SESSION_ATTRIBUTE_STRING
|
||
***
|
||
{code:java}
|
||
12482085 113748 http.user_agent chrome 0 0 0 1 1639035425000000 0
|
||
9021592 112096 http.user_agent Chrome 0 0 0 1 1637987142000000 0
|
||
9021593 112097 http.content_type text/html 0 0 0 1 1637987142000000 0 {code}
|
||
|
||
|
||
|
||
---
|
||
|
||
**zhengchao** commented on *2022-04-28T14:41:38.582+0800*:
|
||
|
||
现场http的比例多大？是触发overload protection的原因吗？
|
||
|
||
|
||
|
||
---
|
||
|
||
**liuxueli** commented on *2022-04-28T15:43:28.368+0800*:
|
||
|
||
* Bole IGW NPB5的协议统计，HTTP的比例8%，可能是APP_SIG_SESSION_ATTRIBUTE_STRING表命中率较高的原因
|
||
** 导出PDF结果中QUIC拼写错误，参见：TSG-10523
|
||
** [^L7protocal_sessions-Bole IGW NPB2.csv]
|
||
** [^L7protocal_sessions-Bole IGW NPB5.csv]
|
||
** [^L7protocal_sessions-all-NPB.csv]
|
||
** [^L7protocal_sessions-all-NPB-new.csv]
|
||
** !image-2022-04-28-15-43-09-227.png!
|
||
|
||
|
||
|
||
---
|
||
|
||
**liuxueli** commented on *2022-04-28T16:27:44.265+0800*:
|
||
|
||
* FQDN扫描命中率高的原因应该是：APP_SKETCH扫描了TOPN SNI，识别Psiphon3和Freegate非的条件。
|
||
** [^Bole IGW NPB5-http_domain_session.csv]
|
||
** [^Bole IGW NPB5-QUIC.SNI_session.csv]
|
||
** [^Bole IGW NPB5-SSL.SNI_session.csv]
|
||
|
||
|
||
|
||
---
|
||
|
||
**liuxueli** commented on *2022-04-28T16:54:53.829+0800*:
|
||
|
||
* APP_SKETCH扫描了TOPN SNI高命中率导致的CPU跑满，从而引发Bole IGW站点的SAPP overload protection机制导致链接被Bypass。[~luqiuwen]
|
||
|
||
|
||
|
||
---
|
||
|
||
**zhengchao** commented on *2022-04-28T17:01:59.390+0800*:
|
||
|
||
IGW能不能把TFE的CPU让出来？[~luqiuwen]
|
||
|
||
|
||
|
||
---
|
||
|
||
**luqiuwen** commented on *2022-04-28T17:02:57.016+0800*:
|
||
|
||
可以在IGW的计算板上关闭Proxy功能，将Proxy使用的CPU资源调整为Firewall使用。Provision中已预留相关选项，操作步骤：
|
||
* 修改/data/tsg-os-provision/provision.yml文件，将proxy->enable选项由1调整为0：
|
||
|
||
{code:java}
|
||
proxy:
|
||
enable: 1 {code}
|
||
修改为：
|
||
{code:java}
|
||
proxy:
|
||
enable: 0 {code}
|
||
* 令上述配置生效，运行：
|
||
|
||
{code:java}
|
||
$ sudo provision-config-apply{code}
|
||
* 检查tfe是否在运行
|
||
|
||
{code:java}
|
||
$ systemctl status tfe
|
||
● tfe.service - Tango Frontend Engine
|
||
Loaded: loaded (/usr/lib/systemd/system/tfe.service; disabled; vendor preset: disabled)
|
||
Drop-In: /usr/lib/systemd/system/tfe.service.d
|
||
└─require-mrzcpd.conf, service_add_ConditionPathExists.conf, service_override_env.conf, service_override_slice.conf
|
||
Active: inactive (dead) since Thu 2022-04-28 11:46:17 EAT; 15min ago
|
||
Process: 18732 ExecStart=/opt/tsg/tfe/bin/tfe (code=killed, signal=TERM)
|
||
Main PID: 18732 (code=killed, signal=TERM){code}
|
||
其中，Active项应为inactive。
|
||
|
||
需要考虑的问题是：
|
||
* 在关闭Proxy功能后，所有拦截策略和代理策略将无法在IGW生效，如何向业主解释。
|
||
|
||
|
||
|
||
---
|
||
|
||
**zhengchao** commented on *2022-04-28T17:06:42.366+0800*:
|
||
|
||
对业主解释：由于IGW存在单向流，Proxy功能不在IGW生效。
|
||
|
||
|
||
|
||
---
|
||
|
||
**gitlab** commented on *2022-04-28T18:18:44.475+0800*:
|
||
|
||
[刘学利|https://git.mesalab.cn/liuxueli] mentioned this issue in [a merge request|https://git.mesalab.cn/MESA_Platform/ssl/-/merge_requests/34] of [MESA Platform / ssl|https://git.mesalab.cn/MESA_Platform/ssl] on branch [feature-add-test-case|https://git.mesalab.cn/MESA_Platform/ssl/-/tree/feature-add-test-case]:{quote}OMPUB-466: 增加相应的测试用例{quote}
|
||
|
||
|
||
|
||
---
|
||
|
||
**liuxueli** commented on *2022-04-29T09:44:47.117+0800*:
|
||
|
||
* Bole IGW NPB5 TFE的CPU让出来后，被SAPP主动Bypass的链接有减少，但是没有完全消除
|
||
** !image-2022-04-29-09-44-38-025.png|width=1108,height=549!
|
||
|
||
|
||
|
||
---
|
||
|
||
**liuxueli** commented on *2022-04-29T10:19:39.687+0800*:
|
||
|
||
* [~liuju] 到现场后把/opt/tsg/sapp/sysinfo.log文件发回来一份。
|
||
** [^Bole IGW NPB5 sysinfo.log]
|
||
|
||
|
||
|
||
---
|
||
|
||
**liuxueli** commented on *2022-05-05T14:23:24.256+0800*:
|
||
|
||
* 新疆环境统计，xx.xxx.192.175：
|
||
** 实时流量约6Gbps，monit_device统计：
|
||
***
|
||
*** !XJ-192.175-monit_device.png!
|
||
** 整体的CPU使用率，单核使用率均在60~65%左右：
|
||
*** !XJ-192.175-top-all-cpu.png|thumbnail!
|
||
** SAPP的TCP/UDP链接流标大小为100000，sysinfo统计： [^XJ-192.175-sysinfo.txt]
|
||
*** !XJ-192.175-sysinfo.png!
|
||
** TSG MAAT句柄的扫描状态统计：[^XJ-192.175-tsg_static_maat.txt]
|
||
*** FQDN字符串扫描命中率为：53%
|
||
*** IP归属地字符串扫描命中率为：51%
|
||
*** !XJ-192.175-tsg-maat.png!
|
||
** APP MAAT句柄扫描统计：[^XJ-192.175-app_sketch_maat.txt]
|
||
*** FQDN字符串扫描命中率为：66%
|
||
*** !XJ-192.175-app-maat.png!
|
||
** 单核CPU使用，perf top -C 6：
|
||
*** !XJ-192.175-perf-top-cpu6.png!
|
||
** CPU 6的火焰图[^XJ-192.175-perf-cpu-6.svg]
|
||
*** ^__clock_gettime 是SAPP统计包处理延时造成的，E现场未开启该功能^
|
||
|
||
|
||
|
||
---
|
||
|
||
**zhengchao** commented on *2022-05-05T14:45:02.941+0800*:
|
||
|
||
FQDN对比，命中率、扫描性能({_}PROC_Tps{_})相近，E21的扫描次数{_}(IN_Tps=5.30e+04){_}是新疆(1.62e+04)的3倍
|
||
{code:java}
|
||
TSG_OBJ_FQDN (XJ) 160158 0 0 2.86e+05 5.66e+06 1.62e+04 3.20e+05 5.72e-01
|
||
TSG_OBJ_FQDN (E21) 4532 0 0 1.15e+06 7.26e+06 5.30e+04 3.36e+05 5.80e-01 {code}
|
||
|
||
|
||
|
||
|
||
---
|
||
|
||
**yangwei** commented on *2022-05-05T14:53:03.447+0800*:
|
||
|
||
补充说明：
|
||
# 新疆单核perf排名第一的函数为clock_get_time，原因是sapp开启了包处理延迟统计，E现场没有开启该功能
|
||
# 新疆CPU型号为Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz，9140的CPU型号为Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
|
||
|
||
|
||
|
||
---
|
||
|
||
**liuxueli** commented on *2022-05-06T11:00:08.046+0800*:
|
||
|
||
* 使用京版E现场升级验证环境测试，仪表构造流量，加载E现场的配置：
|
||
** {color:#de350b}测试结果：不加载E现场的APP特征配置，新建性能可以提升一倍{color}
|
||
*** {color:#de350b}新建由3.3万/s提升到7.3wan/s{color}
|
||
*** {color:#de350b}其中加载E现场APP特征，损耗CPU较多的函数为“msort_with_tmp”{color}
|
||
** {color:#172b4d}测试分为两组对比测试，{color}
|
||
|
||
*
|
||
**
|
||
*** {color:#172b4d}加载E现场APP特征配置{color}
|
||
**** {color:#172b4d}整体CPU使用率：{color}
|
||
***** {color:#172b4d}!XXG-40.82-load-app-maat-all-cpu.png!{color}
|
||
**** {color:#172b4d}单核CPU使用率，perf top -C 14{color}
|
||
***** {color:#172b4d}!XXG-40.82-load-app-maat-perf-top-cpu14.png!{color}
|
||
**** SAPP处理新建连接数({color:#de350b}3.3万/s{color})：[^XXG-40.82-load-app-maat-sysinfo.txt]
|
||
***** !XXG-40.82-load-app-maat-new-link.png!
|
||
*** {color:#172b4d}不加载E现场APP特征配置{color}
|
||
**** {color:#172b4d}整体CPU使用率：{color}
|
||
***** !XXG-40.82-no-app-maat--all-cpu.png!
|
||
**** {color:#172b4d}单核CPU使用率，perf top -C 14{color}
|
||
***** {color:#172b4d}!XXG-40.82-no-app-maat-perf-top-cpu14.png!{color}
|
||
**** {color:#172b4d}SAPP处理新建连接数(7.3万/s{color})，[^XXG-40.82-no-app-maat-sysinfo.txt]：
|
||
***** !XXG-40.82-no-app-maat-new-link.png!
|
||
|
||
|
||
|
||
---
|
||
|
||
**liuxueli** commented on *2022-05-06T11:01:15.064+0800*:
|
||
|
||
* 等待[~zhangwei] 优化APP特征组织结构后进行复测。
|
||
** 去除APP特征中APP ID的引用
|
||
|
||
|
||
|
||
---
|
||
|
||
**yangwei** commented on *2022-05-06T18:21:12.898+0800*:
|
||
|
||
分析E现场使用的App特征，在APP_SIG_SESSION_ATTRIBUTE_INTEGER表中，general.session.analysis.app_id存在较多重复
|
||
* 如下图所示，第一列为重复次数，第二列为AppID，其中67为HTTP，199为SSL，按前述统计，这俩种协议分别占E现场流量的65%和8%
|
||
|
||
可以部分解释前述在新建连接较高的情况下，单核出现CPU使用较高（perf采样占用第一的为msort_xxx函数调用）导致触发sapp自我保护的现象
|
||
|
||
!image-2022-05-06-18-15-08-649.png!
|
||
|
||
|
||
|
||
---
|
||
|
||
**liuxueli** commented on *2022-05-07T16:16:36.389+0800*:
|
||
|
||
* 优化APP特征组织结构后进行复测（去除APP特征中APP ID的引用），使用同一个数据包进行测试，对比APP的识别结果
|
||
** {color:#de350b}优化前单核新建处理能力为: 1000/s，单核CPU使用率75%左右{color}
|
||
** {color:#de350b}优化后单核新建处理能力为: 2500/s，单核CPU使用率80%左右 {color}
|
||
** {color:#de350b}有3个APP识别结果不一致{color}
|
||
*** {color:#de350b}!image-2022-05-07-16-12-05-474.png!{color}
|
||
|
||
|
||
|
||
---
|
||
|
||
**zhengchao** commented on *2022-05-07T17:00:00.363+0800*:
|
||
|
||
确认优化后的AppSketchDB可以在22.02版本正确加载，之后提供给E21现场。
|
||
|
||
|
||
|
||
---
|
||
|
||
**liuxueli** commented on *2022-05-09T10:51:10.555+0800*:
|
||
|
||
* App_GooseVPN_20211208识别结果存在差别，分析App_GooseVPN_20211208特征，特征条件变化，可能导致一定的误识别
|
||
** {color:#de350b}优化前特证：IP+PROTOCOl(IKE){color}
|
||
** {color:#de350b}优化后特征：仅IP{color}
|
||
|
||
|
||
|
||
---
|
||
|
||
**liuxueli** commented on *2022-05-09T17:24:52.318+0800*:
|
||
|
||
* 调整App_GooseVPN_20211208的特征，引用app_id=IKE的特征保留，
|
||
* 手动删除其余对app_id特征的引用，优化前后的APP识别结果一致（Psiphon3、unknown除外）。
|
||
** {color:#de350b}E现场升级验证环境，Psiphon3的识别特征有变化，导致识别结果存在差异，识别结果变少{color}
|
||
** {color:#de350b}Psiphon3识别结果减少，unknown相应会增加{color}
|
||
* {color:#172b4d}已提供至[~liuju] E21现场更新{color}
|
||
|
||
|
||
|
||
---
|
||
|
||
**liuju** commented on *2022-05-09T22:01:03.929+0800*:
|
||
|
||
1、已根据张东旭提供的《E现场修正app特征操作说明》文档，完成对E现场列表里提供的所有app自定义特征进行修改更新，将general.session.analysis.app_id的Condition删除。
|
||
|
||
2、并根据提供的sql语句对更新结果进行验证，验证更新无误。[~liuxueli]
|
||
|
||
|
||
|
||
---
|
||
|
||
**liuxueli** commented on *2022-05-10T09:37:05.586+0800*:
|
||
|
||
* [~liuju] 观察Nezha监控上的DDOS Bypass的统计是否还存在？
|
||
* 统计自定义APP的特征修正前后识别链接数是否较大的差异（统计12小时）？
|
||
** 统计Session Record日志中的common_app_label字段
|
||
* 登录Bole IGW NPB5查看一下整体CPU，使用perf top -C cpuid看一下单核的CPU
|
||
|
||
|
||
|
||
---
|
||
|
||
**liuxueli** commented on *2022-05-10T17:47:36.817+0800*:
|
||
|
||
* 建议[~liuju] 申请在IGW站点复测业主提出的网站阻断效果不佳的问题
|
||
* 优化APP自定义特征后，观察Bole IGW的统计，{color:#de350b}其中由于SAPP overload protection机制导致链接被Bypass的统计(DDOS Bypass)降为0{color}，{color:#de350b}偶尔会存在几个链接被Byapss{color}
|
||
** Nezha统计新建及DDOS Bypass
|
||
*** !Bole IGW NPB1 new connect.png!
|
||
*** !Bole IGW NPB1 DDOS Bypass connect.png!
|
||
*** !Bole IGW NPB2 new connect.png!
|
||
*** !Bole IGW NPB2 DDOS Bypass connect.png!
|
||
*** !Bole IGW NPB3 new connect.png!
|
||
*** !Bole IGW NPB3 DDOS Bypass connect.png!
|
||
|
||
|
||
|
||
---
|
||
|
||
**liuxueli** commented on *2022-05-10T17:50:27.077+0800*:
|
||
|
||
* [~liuju] Bole IGW NPB5 Nezha采集出现异常，请另外提一个BUG。
|
||
|
||
|
||
|
||
---
|
||
|
||
**liuju** commented on *2022-05-10T21:19:49.593+0800*:
|
||
|
||
[~liuxueli] 好的 采集问题已提交BUG 另外在IGW站点业主提出的网站阻断效果不佳的问题，业主已在进行复测，后续有结果会继续反馈
|
||
|
||
|
||
|
||
---
|
||
|
||
**liuxueli** commented on *2022-05-13T17:39:22.275+0800*:
|
||
|
||
* 从现场Nezha监控看，{color:#de350b}目前Bole IGW NPB5还存在由于SAPP overload protection机制导致链接被Bypass，流量峰值(10Gbps左右)时Bypass 2K/s个链接，CPU使用率在60%左右{color}
|
||
** {color:#172b4d}OMPUB-481影响，20220511~20220512统计出现异常，在20220512北京时间晚上重启sapp恢复统计{color}
|
||
** {color:#172b4d}DDOS Bypass统计{color}
|
||
*** {color:#172b4d}!Bole IGW NPB5 DDOS bypass 20220513.png|width=1743,height=810!{color}
|
||
** {color:#172b4d}新建链接统计{color}
|
||
*** {color:#172b4d}!Bole IGW NPB5 new connections 20220513.png|width=1777,height=827!{color}
|
||
** {color:#172b4d}流量统计{color}
|
||
*** {color:#172b4d}!Bole IGW NPB5 throughtput 20220513.png|width=1581,height=743!{color}
|
||
** {color:#172b4d}CPU使用率{color}
|
||
*** !Bole IGW NPB5 CPU 20220513.png!
|
||
** {color:#172b4d}perf top -C 32/44{color}
|
||
*** {color:#172b4d}!Bole IGW NPB5 perf top cpu32 20220513.png!{color}
|
||
*** {color:#172b4d}!Bole IGW NPB5 perf top cpu44 20220513.png!{color}
|
||
|
||
|
||
|
||
---
|
||
|
||
**liuxueli** commented on *2022-05-16T15:33:32.233+0800*:
|
||
|
||
* [~liuju] 建议关闭Bole IGW NPB SAPP的bypass功能，修改配置项：
|
||
** 修改/data/tsg-os-provision/provision.yml文件，将feature->enable_stream_bypass_under_ddos选项由1调整为0
|
||
*** !image-2022-05-16-15-33-27-227.png!
|
||
** 执行命令使上述配置生效，运行：
|
||
*** sudo provision-config-apply
|
||
** 执行完成后检查sapp配置文件etc/sapp.toml
|
||
*** stream_bypass_enabled=0
|
||
*** !image-2022-05-16-15-32-36-510.png!
|
||
|
||
|
||
|
||
---
|
||
|
||
**liuju** commented on *2022-05-16T23:59:04.497+0800*:
|
||
|
||
BOLE-IGW 10.225.11.1~5 已经按你的更新要求完成更新现在，均已检查完更新之后配置stream_bypass_enabled=0。[~liuxueli] 待明天申请配置策略效果验证完，告诉更新之后的效果。
|
||
|
||
|
||
|
||
---
|
||
|
||
**liuxueli** commented on *2022-05-17T08:50:35.226+0800*:
|
||
|
||
* [~liuju] 观察Bole IGW 是否存在丢包现象。
|
||
|
||
|
||
|
||
---
|
||
|
||
**liuju** commented on *2022-05-17T20:50:49.691+0800*:
|
||
|
||
关闭Bole IGW NPB SAPP的bypass功能后，今天业主重新进行复测IGW站点 security policy deny功能，复测结果都deny成功。[~liuxueli] 墨处询问该问题的具体原因，我回复说我需要告诉你们复测结果之后，待家里你们再告知我具体问题原因。墨处要求明天知道问题原因。
|
||
|
||
|
||
|
||
---
|
||
|
||
**zhengchao** commented on *2022-05-17T21:09:19.754+0800*:
|
||
|
||
回复业主：问题原因北京还在定位。
|
||
|
||
[~liuju] 观察Bole IGW 是否存在丢包现象。
|
||
|
||
|
||
|
||
---
|
||
|
||
**liuju** commented on *2022-05-17T21:14:30.862+0800*:
|
||
|
||
收到，好的[~zhengchao] BOLE-IGW 早上和现场观察结果还存在丢包。具体每个NPB丢包情况已微信提供给学利。
|
||
|
||
|
||
|
||
---
|
||
|
||
**liuxueli** commented on *2022-05-18T10:04:24.965+0800*:
|
||
|
||
* Bole IGW 存在偶尔丢包的情况，范围在200~1300pps，[~liuju] 需要同步丢包时刻的流量、CPU的监控
|
||
** !Bole IGW NPB1 application drop pkts.png!
|
||
** !Bole IGW NPB2 application drop pkts.png!
|
||
** !Bole IGW NPB3 application drop pkts.png!
|
||
** !Bole IGW NPB4 application drop pkts.png!
|
||
** !Bole IGW NPB5 application drop pkts.png!
|
||
|
||
|
||
|
||
---
|
||
|
||
**liuju** commented on *2022-05-18T15:18:25.026+0800*:
|
||
|
||
好的[~liuxueli] BOLE-IGW 最近24小时丢包及流量、CPU情况 截图 因本地网络上传jira困难，已将数据图片微信发送给你。
|
||
|
||
|
||
|
||
---
|
||
|
||
**liuxueli** commented on *2022-05-24T17:07:53.129+0800*:
|
||
|
||
* [~liuju] 观察Nezha监控，找一个由于CPU使用过载触发SAPP overload protection机制导致链接被Bypass较多得NPB，调整SAPP的配置文件，观察Bypass的情况
|
||
** 调整SAPP参数，位于sapp.toml，{color:#de350b}bypass_trigger_cpu_usage参数值由90改为99{color}，重启SAPP，观察bypass的情况
|
||
*** !image-2022-05-24-17-06-33-618.png!
|
||
|
||
|
||
|
||
---
|
||
|
||
**liuju** commented on *2022-05-25T15:15:02.793+0800*:
|
||
|
||
[~zhengchao] 超哥，业主处长这近期一直追问进展，故障原因。
|
||
|
||
|
||
|
||
---
|
||
|
||
**liuju** commented on *2022-05-25T15:31:32.750+0800*:
|
||
|
||
[~liuxueli] 已对比近7天bypass数据之后 挑选了MWV-IGW 10.227.11.9 修改了/opt/tsg/sapp/etc/sapp.toml配置内容bypass_trigger_cpu_usage=90改完bypass_trigger_cpu_usage=99,已重启sapp，待观察更新后效果。
|
||
|
||
|
||
|
||
---
|
||
|
||
**zhengchao** commented on *2022-05-25T16:56:30.458+0800*:
|
||
|
||
[~liuxueli] 提供新的TSG OS文件，22.02继续升级。
|
||
|
||
IGW站点的overload bypass的阈值调整为99。[~yangwei]
|
||
|
||
请 [~liuju] 提供现场perf，研发进一步优化性能。
|
||
|
||
|
||
|
||
---
|
||
|
||
**zhengchao** commented on *2022-05-25T16:58:01.728+0800*:
|
||
|
||
回复业主：由于之前的App特征过于消耗计算资源，影响了阻断功能，目前已优化了一轮，正在线上观察效果。北京方面也在开展更进一步的优化。
|
||
{quote}超哥，业主处长这近期一直追问进展，故障原因。
|
||
{quote}
|
||
|
||
|
||
|
||
---
|
||
|
||
**liuju** commented on *2022-05-25T22:09:07.905+0800*:
|
||
|
||
嗯嗯，好的~收到！
|
||
|
||
|
||
|
||
---
|
||
|
||
**liuxueli** commented on *2022-05-26T15:24:58.130+0800*:
|
||
|
||
* 20220526查看Bole IGW NPB5的CPU使用情况，查看perf top cpu48的使用分布
|
||
** 使用top查看整体CPU使用
|
||
*** !Bole IGW NPB5 all cpu 20220526.png!
|
||
** perf top cpu48结果
|
||
*** !Bole IGW NPB5 perf top cpu48 20220526.png!
|
||
|
||
|
||
|
||
---
|
||
|
||
**liuxueli** commented on *2022-05-31T09:46:06.827+0800*:
|
||
|
||
* [~liuju] 鉴于IGW站点的overload bypass的阈值调整为99后还存在Bypass的现象，找两个Bypass较多的NPB
|
||
** 一个NPB使用cpuages采集CPU的使用率，采集24小时
|
||
** 一个NPB调整sapp配置文件(/opt/tsg/sapp/etc/sapp.toml)参数，smooth_avg_window有2调整为20
|
||
*** !image-2022-05-31-09-44-22-020.png!
|
||
** !image-2022-05-31-09-45-23-952.png!
|
||
** !image-2022-05-31-09-46-02-588.png!
|
||
|
||
|
||
|
||
---
|
||
|
||
**liuju** commented on *2022-05-31T14:28:57.759+0800*:
|
||
|
||
收到 好的[~liuxueli]
|
||
|
||
|
||
|
||
---
|
||
|
||
**liuxueli** commented on *2022-06-10T15:38:27.387+0800*:
|
||
|
||
* NPB调整sapp配置文件(/opt/tsg/sapp/etc/sapp.toml)参数，smooth_avg_window有2调整为20，还是存在被SAPP Bypass的链接
|
||
** !image-2022-06-10-15-36-23-768.png!
|
||
** !image-2022-06-10-15-37-33-666.png!
|
||
** !image-2022-06-10-15-38-10-175.png!
|
||
**
|
||
|
||
|
||
|
||
---
|
||
|
||
**liuxueli** commented on *2022-06-10T16:49:20.613+0800*:
|
||
|
||
* [~liuju] smooth_avg_window参数由2调整为20 的NPB，使用cpusage命令采集一下CPU的使用率
|
||
|
||
|
||
|
||
---
|
||
|
||
**liuxueli** commented on *2022-06-15T10:14:53.829+0800*:
|
||
|
||
* smooth_avg_window参数由2调整为20 的NPB，还是存在Bypass的情况，查看cpusage采集的CPU使用率，Bypass前后时刻的CPU单核使用率均未超过95%，[~yangwei]
|
||
** 采集日志及截图存在于: 40.146:/home/E21/CPU20220610~20220611.zip
|
||
|
||
|
||
|
||
---
|
||
|
||
**liuxueli** commented on *2022-06-17T16:45:53.990+0800*:
|
||
|
||
* 京版9140环境也存在链接被SAPP Bypass的现象。
|
||
** !XXG-9140.sapp.bypass.png!
|
||
|
||
|
||
|
||
---
|
||
|
||
**gitlab** commented on *2022-07-29T10:24:05.585+0800*:
|
||
|
||
[刘学利|https://git.mesalab.cn/liuxueli] mentioned this issue in [a merge request|https://git.mesalab.cn/MESA_Platform/ssl/-/merge_requests/35] of [MESA Platform / ssl|https://git.mesalab.cn/MESA_Platform/ssl] on branch [feature-add-test-case|https://git.mesalab.cn/MESA_Platform/ssl/-/tree/feature-add-test-case]:{quote}OMPUB-466: 增加SSL相应的测试用例{quote}
|
||
|
||
|
||
|
||
---
|
||
|
||
**gitlab** commented on *2022-07-29T10:24:09.209+0800*:
|
||
|
||
[刘学利|https://git.mesalab.cn/liuxueli] mentioned this issue in [a commit|https://git.mesalab.cn/MESA_Platform/ssl/-/commit/1325788848f31f58d173c57767a2d1d47a0c2598] of [MESA Platform / ssl|https://git.mesalab.cn/MESA_Platform/ssl] on branch [master|https://git.mesalab.cn/MESA_Platform/ssl/-/tree/master]:{quote}OMPUB-466: 增加SSL相应的测试用例{quote}
|
||
|
||
|
||
|
||
---
|
||
|
||
|
||
|
||
## Attachments
|
||
|
||
**27574/18-target.com.pcap**
|
||
|
||
---
|
||
|
||
**27641/app_sketch_maat.txt**
|
||
|
||
---
|
||
|
||
**28062/Bole+IGW++NPB1+application+drop+pkts.png**
|
||
|
||
---
|
||
|
||
**28067/Bole+IGW++NPB1+application+drop+pkts-1.png**
|
||
|
||
---
|
||
|
||
**27863/Bole+IGW++NPB1+DDOS+Bypass++connect.png**
|
||
|
||
---
|
||
|
||
**27862/Bole+IGW++NPB1+new+connect.png**
|
||
|
||
---
|
||
|
||
**28063/Bole+IGW++NPB2+application+drop+pkts.png**
|
||
|
||
---
|
||
|
||
**28068/Bole+IGW++NPB2+application+drop+pkts-1.png**
|
||
|
||
---
|
||
|
||
**27865/Bole+IGW++NPB2+DDOS+Bypass++connect.png**
|
||
|
||
---
|
||
|
||
**27864/Bole+IGW++NPB2+new+connect.png**
|
||
|
||
---
|
||
|
||
**28064/Bole+IGW++NPB3+application+drop+pkts.png**
|
||
|
||
---
|
||
|
||
**28069/Bole+IGW++NPB3+application+drop+pkts-1.png**
|
||
|
||
---
|
||
|
||
**27867/Bole+IGW++NPB3+DDOS+Bypass++connect.png**
|
||
|
||
---
|
||
|
||
**27866/Bole+IGW++NPB3+new+connect.png**
|
||
|
||
---
|
||
|
||
**28065/Bole+IGW++NPB4+application+drop+pkts.png**
|
||
|
||
---
|
||
|
||
**28070/Bole+IGW++NPB4+application+drop+pkts-1.png**
|
||
|
||
---
|
||
|
||
**28268/Bole+IGW+NPB5+all+cpu+20220526.png**
|
||
|
||
---
|
||
|
||
**28066/Bole+IGW++NPB5+application+drop+pkts.png**
|
||
|
||
---
|
||
|
||
**28071/Bole+IGW++NPB5+application+drop+pkts-1.png**
|
||
|
||
---
|
||
|
||
**27980/Bole+IGW+NPB5+CPU+20220513.png**
|
||
|
||
---
|
||
|
||
**27978/Bole+IGW+NPB5+DDOS+bypass+20220513.png**
|
||
|
||
---
|
||
|
||
**27668/Bole+IGW+NPB5-http_domain_session.csv**
|
||
|
||
---
|
||
|
||
**27979/Bole+IGW+NPB5+new+connections+20220513.png**
|
||
|
||
---
|
||
|
||
**27976/Bole+IGW+NPB5+perf+top+cpu32+20220513.png**
|
||
|
||
---
|
||
|
||
**27975/Bole+IGW+NPB5+perf+top+cpu44+20220513.png**
|
||
|
||
---
|
||
|
||
**27977/Bole+IGW+NPB5+perf+top+cpu44+20220513-1.png**
|
||
|
||
---
|
||
|
||
**28269/Bole+IGW+NPB5+perf+top+cpu48+20220526.png**
|
||
|
||
---
|
||
|
||
**27669/Bole+IGW+NPB5-QUIC.SNI_session.csv**
|
||
|
||
---
|
||
|
||
**27670/Bole+IGW+NPB5-SSL.SNI_session.csv**
|
||
|
||
---
|
||
|
||
**27696/Bole+IGW+NPB5+sysinfo.log**
|
||
|
||
---
|
||
|
||
**27972/Bole+IGW+NPB5+throughtput+20220513.png**
|
||
|
||
---
|
||
|
||
**27750/E21-sapp-overload-protection-bypass-20220427~20220503.png**
|
||
|
||
---
|
||
|
||
**27643/e21-version-21.11-dup-fqdn-object.txt**
|
||
|
||
---
|
||
|
||
**27599/image-2022-04-27-17-12-00-094.png**
|
||
|
||
---
|
||
|
||
**27600/image-2022-04-27-17-14-25-901.png**
|
||
|
||
---
|
||
|
||
**27601/image-2022-04-27-17-15-02-774.png**
|
||
|
||
---
|
||
|
||
**27602/image-2022-04-27-17-17-32-639.png**
|
||
|
||
---
|
||
|
||
**27604/image-2022-04-27-17-18-43-458.png**
|
||
|
||
---
|
||
|
||
**27605/image-2022-04-27-17-19-09-881.png**
|
||
|
||
---
|
||
|
||
**27615/image-2022-04-27-17-35-20-050.png**
|
||
|
||
---
|
||
|
||
**27616/image-2022-04-27-17-36-03-049.png**
|
||
|
||
---
|
||
|
||
**27617/image-2022-04-27-17-53-42-859.png**
|
||
|
||
---
|
||
|
||
**27656/image-2022-04-28-15-43-09-227.png**
|
||
|
||
---
|
||
|
||
**27682/image-2022-04-29-09-44-38-025.png**
|
||
|
||
---
|
||
|
||
**27779/image-2022-05-06-18-15-08-649.png**
|
||
|
||
---
|
||
|
||
**27789/image-2022-05-07-16-12-05-474.png**
|
||
|
||
---
|
||
|
||
**28003/image-2022-05-16-15-32-36-510.png**
|
||
|
||
---
|
||
|
||
**28006/image-2022-05-16-15-33-27-227.png**
|
||
|
||
---
|
||
|
||
**28229/image-2022-05-24-17-06-33-618.png**
|
||
|
||
---
|
||
|
||
**28335/image-2022-05-31-09-44-22-020.png**
|
||
|
||
---
|
||
|
||
**28336/image-2022-05-31-09-45-23-952.png**
|
||
|
||
---
|
||
|
||
**28337/image-2022-05-31-09-46-02-588.png**
|
||
|
||
---
|
||
|
||
**28628/image-2022-06-10-15-36-23-768.png**
|
||
|
||
---
|
||
|
||
**28629/image-2022-06-10-15-37-33-666.png**
|
||
|
||
---
|
||
|
||
**28630/image-2022-06-10-15-38-10-175.png**
|
||
|
||
---
|
||
|
||
**27660/L7protocal_sessions-all-NPB.csv**
|
||
|
||
---
|
||
|
||
**27661/L7protocal_sessions-all-NPB-new.csv**
|
||
|
||
---
|
||
|
||
**27662/L7protocal_sessions-Bole+IGW+NPB2.csv**
|
||
|
||
---
|
||
|
||
**27663/L7protocal_sessions-Bole+IGW+NPB5.csv**
|
||
|
||
---
|
||
|
||
**27664/L7protocol_sessions-Bole+IGW+NPB5.pdf**
|
||
|
||
---
|
||
|
||
**27665/L7protocol_ssesion-all-NPB.pdf**
|
||
|
||
---
|
||
|
||
**27666/L7protocol_ssesion-Bole+IGW+NPB2.pdf**
|
||
|
||
---
|
||
|
||
**27606/perf.svg**
|
||
|
||
---
|
||
|
||
**27607/perf33.svg**
|
||
|
||
---
|
||
|
||
**27610/perf33-1.svg**
|
||
|
||
---
|
||
|
||
**27608/perf37.svg**
|
||
|
||
---
|
||
|
||
**27609/perf40.svg**
|
||
|
||
---
|
||
|
||
**27544/securityEvents-target.xlsx**
|
||
|
||
---
|
||
|
||
**27613/sessionRecords-deny-target.com-failed.xlsx**
|
||
|
||
---
|
||
|
||
**27543/sessionRecords+-target-clientip.xlsx**
|
||
|
||
---
|
||
|
||
**27614/target.com-196.188.136.150-151.101.2.187.443.pcap**
|
||
|
||
---
|
||
|
||
**27642/tsg_static_maat.txt**
|
||
|
||
---
|
||
|
||
**27720/XJ-192.175-app_sketch_maat.txt**
|
||
|
||
---
|
||
|
||
**27716/XJ-192.175-app-maat.png**
|
||
|
||
---
|
||
|
||
**27712/XJ-192.175-monit_device.png**
|
||
|
||
---
|
||
|
||
**27723/XJ-192.175-perf-cpu-6.svg**
|
||
|
||
---
|
||
|
||
**27717/XJ-192.175-perf-top-cpu6.png**
|
||
|
||
---
|
||
|
||
**27714/XJ-192.175-sysinfo.png**
|
||
|
||
---
|
||
|
||
**27718/XJ-192.175-sysinfo.txt**
|
||
|
||
---
|
||
|
||
**27713/XJ-192.175-top-all-cpu.png**
|
||
|
||
---
|
||
|
||
**27719/XJ-192.175-tsg_static_maat.txt**
|
||
|
||
---
|
||
|
||
**27715/XJ-192.175-tsg-maat.png**
|
||
|
||
---
|
||
|
||
**27742/XXG-40.82-load-app-maat-all-cpu.png**
|
||
|
||
---
|
||
|
||
**27745/XXG-40.82-load-app-maat-new-link.png**
|
||
|
||
---
|
||
|
||
**27743/XXG-40.82-load-app-maat-perf-top-cpu14.png**
|
||
|
||
---
|
||
|
||
**27744/XXG-40.82-load-app-maat-sysinfo.txt**
|
||
|
||
---
|
||
|
||
**27746/XXG-40.82-no-app-maat--all-cpu.png**
|
||
|
||
---
|
||
|
||
**27748/XXG-40.82-no-app-maat-new-link.png**
|
||
|
||
---
|
||
|
||
**27747/XXG-40.82-no-app-maat-perf-top-cpu14.png**
|
||
|
||
---
|
||
|
||
**27749/XXG-40.82-no-app-maat-sysinfo.txt**
|
||
|
||
---
|
||
|
||
**28933/XXG-9140.sapp.bypass.png**
|
||
|
||
---
|
||
|
||
**27547/微信图片_20220426170327.png**
|
||
|
||
---
|
||
|
||
**27546/微信图片_20220426170336.png**
|
||
|
||
---
|
||
|
||
**27545/微信图片_20220426170342.png**
|
||
|
||
---
|
||
|
||
**27549/微信图片_20220426170353.png**
|
||
|
||
---
|
||
|
||
**27548/微信图片_20220426170400-1.png**
|
||
|
||
---
|
||
|
||
**27551/微信图片_20220426170415.png**
|
||
|
||
---
|
||
|
||
**27550/微信图片_20220426170429.png**
|
||
|
||
---
|
||
|
||
**27555/微信图片_20220426170433.png**
|
||
|
||
---
|
||
|
||
**27554/微信图片_20220426170438.png**
|
||
|
||
---
|
||
|
||
**27553/微信图片_20220426170442.png**
|
||
|
||
---
|
||
|
||
**27552/微信图片_20220426170446.png**
|
||
|
||
---
|
||
|