202 lines
11 KiB
Markdown
202 lines
11 KiB
Markdown
# 【E21现场】会话日志部分字段不写入Kafka
|
||
|
||
| ID | Creation Date | Assignee | Status |
|
||
|----|----------------|----------|--------|
|
||
| OMPUB-684 | 2022-11-02T11:28:36.000+0800 | 刘学利 | 完成 |
|
||
|
||
|
||
---
|
||
|
||
从现场监控和系统日志分析，Bole-IGW 和Microwave-IGW 在流量高峰时，部分会话日志无法写入kafka（从Network zodiac 发现大约10%～15%）。目前现状如下：
|
||
* 原始日志单条大小较升级前增加了25%左右；单条平均大小2KB～2.7KB；其中SSL 2.3KB ～3KB，DNS 2.4KB～2.7KB
|
||
* 每个 Device Group服务器理论最大写速率450MB/s ；Bole-IGW 峰值日志量16w/s，kafka IO 320MB/s ～ 432MB/s 。
|
||
|
||
考虑E现场，每个Device Group 通过KVM形式混部各个组件，为避免IO抢占，同时为安全事件日志预留IO，建议SESSION-RECORD 和 INTERIM-SESSION-RECORD 对如下字段不写入kafka。
|
||
* common_link_info_s2c : 1.3TB (7%)
|
||
* common_link_info_c2s : 9.6TB (5%)
|
||
* common_app_id ：8TB (4.5%)
|
||
* dns_rr: 4.1TB (2.3%)
|
||
|
||
**doufenghu** commented on *2022-11-02T11:32:42.214+0800*:
|
||
|
||
通过Confluence页面补充下日志丢失排查详细过程。[~qidaijie]
|
||
|
||
|
||
|
||
---
|
||
|
||
**yangwei** commented on *2022-11-02T14:43:25.329+0800*:
|
||
|
||
* common_link_info_c2s和common_link_info_s2c
|
||
** 含义：用于WANNAT收集链路信息
|
||
** 如何关闭：在功能端关闭wire_graft插件（注释plug/conflist.inf中对应行）实现不发送该字段
|
||
** 副作用：关闭后，对应功能端的WANNAT功能失效
|
||
* common_app_id
|
||
** 含义：记录第三方DPI、内置特征和用户自定义特征识别出的app_id，内部调试用，界面不显示
|
||
** 如何关闭：注释tsgconf/tsg_log_field.conf中包含common_app_id的行
|
||
** 副作用：app识别结果不符合预期时，缺乏调试信息
|
||
* dns_rr
|
||
** 含义：记录dns response records， 内容为json数组
|
||
** 如何关闭：需要更新插件，增加开关
|
||
|
||
|
||
|
||
---
|
||
|
||
**liuxueli** commented on *2022-11-14T11:02:57.680+0800*:
|
||
|
||
* 2022-11-14例会：provision增加开关控制是否发送上述日志字段(common_link_info_c2s/common_link_info_s2c/common_app_id/dns_rr)。TSG-12613
|
||
|
||
|
||
|
||
---
|
||
|
||
**gitlab** commented on *2022-11-14T17:27:01.670+0800*:
|
||
|
||
[刘学利|https://git.mesalab.cn/liuxueli] mentioned this issue in [a commit|https://git.mesalab.cn/tango/tsg_master/-/commit/d8e1578b936c95d04a16639660782f5d9c04004b] of [TSG Appliance / tsg_master|https://git.mesalab.cn/tango/tsg_master] on branch [Branch_v5.7.6|https://git.mesalab.cn/tango/tsg_master/-/tree/Branch_v5.7.6]:{quote}OMPUB-684: 增加开关控制是否发送日志字段(common_link_info_c2s/common_link_info_s2c/common_app_id){quote}
|
||
|
||
|
||
|
||
---
|
||
|
||
**gitlab** commented on *2022-11-15T10:52:22.607+0800*:
|
||
|
||
[彭宣正|https://git.mesalab.cn/pxz] mentioned this issue in [a commit|https://git.mesalab.cn/tango/tsg_conn_record/-/commit/ccc751d7659b14a2e89e052663aeffad94fb9edc] of [TSG Appliance / tsg_conn_record|https://git.mesalab.cn/tango/tsg_conn_record] on branch [ompub-684|https://git.mesalab.cn/tango/tsg_conn_record/-/tree/ompub-684]:{quote}✨ feat(OMPUB-684): 增加send_dns_rr_switch控制是否发送dns rr，默认不发送{quote}
|
||
|
||
|
||
|
||
---
|
||
|
||
**gitlab** commented on *2022-11-18T14:33:48.776+0800*:
|
||
|
||
[彭宣正|https://git.mesalab.cn/pxz] mentioned this issue in [a merge request|https://git.mesalab.cn/tango/tsg_conn_record/-/merge_requests/206] of [TSG Appliance / tsg_conn_record|https://git.mesalab.cn/tango/tsg_conn_record] on branch [ompub-684|https://git.mesalab.cn/tango/tsg_conn_record/-/tree/ompub-684]:{quote}✨ feat(OMPUB-684): 增加send_dns_rr_switch控制是否发送dns rr，默认不发送{quote}
|
||
|
||
|
||
|
||
---
|
||
|
||
**gitlab** commented on *2022-11-18T14:35:30.410+0800*:
|
||
|
||
[彭宣正|https://git.mesalab.cn/pxz] mentioned this issue in [a commit|https://git.mesalab.cn/tango/tsg_conn_record/-/commit/d43ebcde8fd4929291e6e3024c4da0f28659c7a6] of [TSG Appliance / tsg_conn_record|https://git.mesalab.cn/tango/tsg_conn_record] on branch [ompub-684|https://git.mesalab.cn/tango/tsg_conn_record/-/tree/ompub-684]:{quote}✨ feat(OMPUB-684): 增加send_dns_rr_switch控制是否发送dns rr，默认不发送{quote}
|
||
|
||
|
||
|
||
---
|
||
|
||
**gitlab** commented on *2022-11-18T18:34:24.833+0800*:
|
||
|
||
[彭宣正|https://git.mesalab.cn/pxz] mentioned this issue in [a merge request|https://git.mesalab.cn/tango/tsg_conn_record/-/merge_requests/207] of [TSG Appliance / tsg_conn_record|https://git.mesalab.cn/tango/tsg_conn_record] on branch [ompub-684|https://git.mesalab.cn/tango/tsg_conn_record/-/tree/ompub-684]:{quote}✨ feat(OMPUB-684): 增加send_dns_rr_switch控制是否发送dns rr，默认不发送{quote}
|
||
|
||
|
||
|
||
---
|
||
|
||
**gitlab** commented on *2022-11-18T18:34:37.963+0800*:
|
||
|
||
[彭宣正|https://git.mesalab.cn/pxz] mentioned this issue in [a commit|https://git.mesalab.cn/tango/tsg_conn_record/-/commit/6d61fc5d38607e5689d5e1f4606e9b38b2e02d77] of [TSG Appliance / tsg_conn_record|https://git.mesalab.cn/tango/tsg_conn_record] on branch [ompub-684|https://git.mesalab.cn/tango/tsg_conn_record/-/tree/ompub-684]:{quote}✨ feat(OMPUB-684): 增加send_dns_rr_switch控制是否发送dns rr，默认不发送{quote}
|
||
|
||
|
||
|
||
---
|
||
|
||
**gitlab** commented on *2022-11-18T18:42:53.535+0800*:
|
||
|
||
[彭宣正|https://git.mesalab.cn/pxz] mentioned this issue in [a merge request|https://git.mesalab.cn/tango/tsg_conn_record/-/merge_requests/208] of [TSG Appliance / tsg_conn_record|https://git.mesalab.cn/tango/tsg_conn_record] on branch [ompub-684|https://git.mesalab.cn/tango/tsg_conn_record/-/tree/ompub-684]:{quote}✨ feat(OMPUB-684): 增加send_dns_rr_switch控制是否发送dns rr，默认不发送{quote}
|
||
|
||
|
||
|
||
---
|
||
|
||
**gitlab** commented on *2022-11-18T19:00:31.534+0800*:
|
||
|
||
[彭宣正|https://git.mesalab.cn/pxz] mentioned this issue in [a commit|https://git.mesalab.cn/tango/tsg_conn_record/-/commit/23bd4efc0ae62469bb5f0b61b9606e280a7ba3a0] of [TSG Appliance / tsg_conn_record|https://git.mesalab.cn/tango/tsg_conn_record] on branch [ompub-684|https://git.mesalab.cn/tango/tsg_conn_record/-/tree/ompub-684]:{quote}✨ feat(OMPUB-684): 增加send_dns_rr_switch控制是否发送dns rr，默认不发送{quote}
|
||
|
||
|
||
|
||
---
|
||
|
||
**gitlab** commented on *2022-11-18T19:00:56.596+0800*:
|
||
|
||
[彭宣正|https://git.mesalab.cn/pxz] mentioned this issue in [a merge request|https://git.mesalab.cn/tango/tsg_conn_record/-/merge_requests/210] of [TSG Appliance / tsg_conn_record|https://git.mesalab.cn/tango/tsg_conn_record] on branch [ompub-684|https://git.mesalab.cn/tango/tsg_conn_record/-/tree/ompub-684]:{quote}✨ feat(OMPUB-684): 增加send_dns_rr_switch控制是否发送dns rr，默认不发送{quote}
|
||
|
||
|
||
|
||
---
|
||
|
||
**gitlab** commented on *2022-11-18T20:11:37.958+0800*:
|
||
|
||
[彭宣正|https://git.mesalab.cn/pxz] mentioned this issue in [a commit|https://git.mesalab.cn/tsg/tsg-os-buildimage/-/commit/c60d16e447d16f2e2c3c544a9fac6c9073d315aa] of [TSG / tsg-os-buildimage|https://git.mesalab.cn/tsg/tsg-os-buildimage] on branch [update-tsg_conn_sketch|https://git.mesalab.cn/tsg/tsg-os-buildimage/-/tree/update-tsg_conn_sketch]:{quote}🐞 fix(TSG-12733,OMPUB-684): update tsg_conn_sketch-3.3.13 to tsg_conn_sketch-3.3.14{quote}
|
||
|
||
|
||
|
||
---
|
||
|
||
**gitlab** commented on *2022-11-18T20:12:10.228+0800*:
|
||
|
||
[彭宣正|https://git.mesalab.cn/pxz] mentioned this issue in [a merge request|https://git.mesalab.cn/tsg/tsg-os-buildimage/-/merge_requests/984] of [TSG / tsg-os-buildimage|https://git.mesalab.cn/tsg/tsg-os-buildimage] on branch [update-tsg_conn_sketch|https://git.mesalab.cn/tsg/tsg-os-buildimage/-/tree/update-tsg_conn_sketch]:{quote}🐞 fix(TSG-12733,OMPUB-684): update tsg_conn_sketch-3.3.13 to tsg_conn_sketch-3.3.14{quote}
|
||
|
||
|
||
|
||
---
|
||
|
||
**gitlab** commented on *2022-11-18T20:14:23.157+0800*:
|
||
|
||
[彭宣正|https://git.mesalab.cn/pxz] mentioned this issue in [a commit|https://git.mesalab.cn/tsg/tsg-os-buildimage/-/commit/611c1e6e648e7c2060246a8c6153e2216353bc5d] of [TSG / tsg-os-buildimage|https://git.mesalab.cn/tsg/tsg-os-buildimage] on branch [update-tsg_conn_sketch|https://git.mesalab.cn/tsg/tsg-os-buildimage/-/tree/update-tsg_conn_sketch]:{quote}🐞 fix(TSG-12733,OMPUB-684): update tsg_conn_sketch-3.3.13 to tsg_conn_sketch-3.3.14{quote}
|
||
|
||
|
||
|
||
---
|
||
|
||
**gitlab** commented on *2022-12-30T13:41:17.939+0800*:
|
||
|
||
[刘学利|https://git.mesalab.cn/liuxueli] mentioned this issue in [a commit|https://git.mesalab.cn/tango/tsg_master/-/commit/ceb492bcb3a8892fbc55110f20f2129a0449c2ef] of [TSG Appliance / tsg_master|https://git.mesalab.cn/tango/tsg_master] on branch [feature-OMPUB-684|https://git.mesalab.cn/tango/tsg_master/-/tree/feature-OMPUB-684]:{quote}OMPUB-684: 增加开关控制是否发送日志字段(common_link_info_c2s/common_link_info_s2c/common_app_id){quote}
|
||
|
||
|
||
|
||
---
|
||
|
||
**gitlab** commented on *2022-12-30T13:42:57.823+0800*:
|
||
|
||
[刘学利|https://git.mesalab.cn/liuxueli] mentioned this issue in [a commit|https://git.mesalab.cn/tango/tsg_master/-/commit/3bd6a5642f32b8d26739e6d34532e56cd60a408f] of [TSG Appliance / tsg_master|https://git.mesalab.cn/tango/tsg_master] on branch [feature-OMPUB-684|https://git.mesalab.cn/tango/tsg_master/-/tree/feature-OMPUB-684]:{quote}OMPUB-684: 增加开关控制是否发送日志字段(common_link_info_c2s/common_link_info_s2c/common_app_id){quote}
|
||
|
||
|
||
|
||
---
|
||
|
||
**gitlab** commented on *2022-12-30T13:57:22.745+0800*:
|
||
|
||
[刘学利|https://git.mesalab.cn/liuxueli] mentioned this issue in [a merge request|https://git.mesalab.cn/tango/tsg_master/-/merge_requests/264] of [TSG Appliance / tsg_master|https://git.mesalab.cn/tango/tsg_master] on branch [feature-OMPUB-684|https://git.mesalab.cn/tango/tsg_master/-/tree/feature-OMPUB-684]:{quote}OMPUB-684: 增加开关控制是否发送日志字段(common_link_info_c2s/common_link_info_s2c/common_app_id){quote}
|
||
|
||
|
||
|
||
---
|
||
|
||
**gitlab** commented on *2022-12-30T14:18:48.577+0800*:
|
||
|
||
[刘学利|https://git.mesalab.cn/liuxueli] mentioned this issue in [a commit|https://git.mesalab.cn/tango/tsg_master/-/commit/eeae469a3f74d85fa8ccddf1426243c68f148f60] of [TSG Appliance / tsg_master|https://git.mesalab.cn/tango/tsg_master] on branch [TSG-OS.v22.11|https://git.mesalab.cn/tango/tsg_master/-/tree/TSG-OS.v22.11]:{quote}OMPUB-684: 增加开关控制是否发送日志字段(common_link_info_c2s/common_link_info_s2c/common_app_id){quote}
|
||
|
||
|
||
|
||
---
|
||
|
||
**gitlab** commented on *2023-01-13T10:19:11.044+0800*:
|
||
|
||
[刘学利|https://git.mesalab.cn/liuxueli] mentioned this issue in [a commit|https://git.mesalab.cn/tsg/tsg-os-buildimage/-/commit/a0bc07c27637ad97286041483217c2c72e27b6c6] of [TSG / tsg-os-buildimage|https://git.mesalab.cn/tsg/tsg-os-buildimage] on branch [update-v22.11-firewall-v6|https://git.mesalab.cn/tsg/tsg-os-buildimage/-/tree/update-v22.11-firewall-v6]:{quote}更新sapp、tsg_master、fw_dns_plug，修复或新增：{quote}
|
||
|
||
|
||
|
||
---
|
||
|
||
**gitlab** commented on *2023-01-13T10:20:04.994+0800*:
|
||
|
||
[刘学利|https://git.mesalab.cn/liuxueli] mentioned this issue in [a merge request|https://git.mesalab.cn/tsg/tsg-os-buildimage/-/merge_requests/1067] of [TSG / tsg-os-buildimage|https://git.mesalab.cn/tsg/tsg-os-buildimage] on branch [update-v22.11-firewall-v6|https://git.mesalab.cn/tsg/tsg-os-buildimage/-/tree/update-v22.11-firewall-v6]:{quote}更新sapp、tsg_master、fw_dns_plug，修复或新增：{quote}
|
||
|
||
|
||
|
||
---
|
||
|
||
|
||
|
||
## Attachments
|
||
|