136 lines
4.5 KiB
Markdown
136 lines
4.5 KiB
Markdown
|
# 福建项目：泉州联通局点升级至TSG v24.07后，测试出现穿透现象，流量到TSG-OS但不是每次都能命中策略
|
|||
|
|
|
|||
|
|
| ID | Creation Date | Assignee | Status |
|
|||
|
|
|----|----------------|----------|--------|
|
|||
|
|
| OMPUB-1498 | 2024-10-10T09:18:12.000+0800 | 杨威 | 已关闭 |
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
2024-10-09凌晨，福建泉州联通机房的TSG-OS升级至v24.07版本，测试封堵效果时发现穿透率很高，通过抓包发现流量能到firewall，但不是每次都能命中策略，穿透的时候firewall能捕到完整的c2s侧流量，但没有发rst包，且界面无命中日志。
|
|||
|
|
|
|||
|
|
TSG-OS ：
|
|||
|
|
192.168.21.1，CPU Intel 4216 16c *2，128G内存
|
|||
|
|
192.168.21.2，CPU Intel 4216 16c *2，128G内存
|
|||
|
|
192.168.23.1，CPU Intel 5218 16c *2，160G内存
|
|||
|
|
192.168.23.2，CPU Intel 5218 16c *2，256G内存
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
Deny策略：Destination IP + Protocol（http、ssl）
|
|||
|
|
!image-2024-10-10-09-07-06-208.png|thumbnail!
|
|||
|
|
|
|||
|
|
firewall容器内捕包（tcpdump_mesa）：
|
|||
|
|
!image-2024-10-10-09-08-02-438.png|thumbnail!
|
|||
|
|
|
|||
|
|
**yangwei** commented on *2024-10-10T10:05:03.517+0800*:
|
|||
|
|
|
|||
|
|
穿透原因
|
|||
|
|
* 触发overload protection，所以流量能被sapp捕获，但是未被firewall处理，导致穿透
|
|||
|
|
|
|||
|
|
处理
|
|||
|
|
* 关闭overload protection后，CPU使用较高，出现丢包，处理中
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**zhangzhihan** commented on *2024-10-10T14:34:07.684+0800*:
|
|||
|
|
|
|||
|
|
目前泉州联通4台已调整完毕，尖峰时间段（12:40 - 13:20）CPU占用满，丢包严重，测试穿透明显。其他时间段CPU占用降低后，无穿透。
|
|||
|
|
|
|||
|
|
调整参数如下：
|
|||
|
|
1、关policy sketch
|
|||
|
|
set template name tsg_traffic_engine_default policy_sketch enable no
|
|||
|
|
|
|||
|
|
2、关闭traffic sketch
|
|||
|
|
set template name tsg_traffic_engine_default traffic_sketch enable no
|
|||
|
|
|
|||
|
|
3、关闭overload_protection
|
|||
|
|
set template name tsg_traffic_engine_default overload_protection enable no
|
|||
|
|
|
|||
|
|
4、关重复报文识别
|
|||
|
|
set template name tsg_traffic_engine_default session_manager tcp_duplicated_packet_filter no
|
|||
|
|
set template name tsg_traffic_engine_default session_manager udp_duplicated_packet_filter no
|
|||
|
|
set template name tsg_traffic_engine_default session_manager inject_duplicated_packet_filter no
|
|||
|
|
|
|||
|
|
5、调整firewall资源占用
|
|||
|
|
set service_function name tsg-traffic-engine-vsys-1 quotas firewall 90
|
|||
|
|
|
|||
|
|
|
|||
|
|
6、TCP流表改到20000
|
|||
|
|
set template name tsg_traffic_engine_default session_manager tcp_session_max 20000
|
|||
|
|
|
|||
|
|
7、UDP流表改到5000
|
|||
|
|
set template name tsg_traffic_engine_default session_manager udp_session_max 5000
|
|||
|
|
|
|||
|
|
8、调整CPU绑定
|
|||
|
|
set system resources packet_io_engine_cores 1,16
|
|||
|
|
set system resources workload_cores 2,3,4,5,6,7,8,9,10,11,12,13,14,15,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**yangwei** commented on *2024-10-10T17:01:18.295+0800*:
|
|||
|
|
|
|||
|
|
建议在firewall的firewall_prestart_script.sh中添加如下行，暂时关闭firewall输出的metric节省CPU（预期下降5%-10%）
|
|||
|
|
{code:java}
|
|||
|
|
crudini --inplace --set /opt/tsg/sapp/tsgconf/main.conf FIREWALL_LOCAL_STAT STAT_INTERVAL_TIME_S 0{code}
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**zhangzhihan** commented on *2024-10-11T10:15:36.716+0800*:
|
|||
|
|
|
|||
|
|
2024.10.10晚20点开始，泉州联通的压力增大，穿透严重，192.168.21.1-2两台CPU基本处于爆满状态，23.1-2的CPU压力适中，traffic_sketch功能均处于关闭状态。
|
|||
|
|
用户为了保证业务稳定，减少了约一半流量。流量降低后CPU使用率降低到80%左右，此时测试不穿透。尝试开启traffic_sketch功能后，CPU攀升至95%，测试开始穿透，然后就关闭了traffic_sketch功能。
|
|||
|
|
用户希望继续排查CPU压力的问题。
|
|||
|
|
|
|||
|
|
补充：泉州联通的TSG升级前CPU压力并不是很大（约60%-70%），升级后CPU压力明显增大，用户反馈流量、策略基本没有太大变化。
|
|||
|
|
!screenshot-1.png|thumbnail!
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**yangwei** commented on *2024-10-12T11:38:10.640+0800*:
|
|||
|
|
|
|||
|
|
参照升级前的功能设置，可以考虑关闭ssl decoder解析证书，http decoder 解压gzip，和dns decoder，tsg-os-cli命令如下：
|
|||
|
|
{code:java}
|
|||
|
|
tsg-os-cli -1 set template name tsg_traffic_engine_default decoders SSL_CERT no
|
|||
|
|
tsg-os-cli -1 set template name tsg_traffic_engine_default decoders SSL_JA3 no
|
|||
|
|
tsg-os-cli -1 set template name tsg_traffic_engine_default decoders HTTP_GZIP no
|
|||
|
|
set template name tsg_traffic_engine_default decoders DNS no {code}
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**zhangzhihan** commented on *2024-10-17T11:45:03.735+0800*:
|
|||
|
|
|
|||
|
|
近期观察，CPU使用率已经降低很多，高峰时期不再丢包，穿透解决
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
## Attachments
|
|||
|
|
|
|||
|
|
**63454/image-2024-10-10-09-07-06-208.png**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**63453/image-2024-10-10-09-08-02-438.png**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**63474/screenshot-1.png**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|