11 KiB
【E21现场】会话日志部分字段不写入Kafka
| ID | Creation Date | Assignee | Status |
|---|---|---|---|
| OMPUB-684 | 2022-11-02T11:28:36.000+0800 | 刘学利 | 完成 |
从现场监控和系统日志分析，Bole-IGW 和Microwave-IGW 在流量高峰时，部分会话日志无法写入kafka（从Network zodiac 发现大约10%～15%）。目前现状如下：
- 原始日志单条大小较升级前增加了25%左右；单条平均大小2KB～2.7KB；其中SSL 2.3KB ～3KB，DNS 2.4KB～2.7KB
- 每个 Device Group服务器理论最大写速率450MB/s ；Bole-IGW 峰值日志量16w/s，kafka IO 320MB/s ～ 432MB/s 。
考虑E现场，每个Device Group 通过KVM形式混部各个组件，为避免IO抢占，同时为安全事件日志预留IO，建议SESSION-RECORD 和 INTERIM-SESSION-RECORD 对如下字段不写入kafka。
- common_link_info_s2c : 1.3TB (7%)
- common_link_info_c2s : 9.6TB (5%)
- common_app_id ：8TB (4.5%)
- dns_rr: 4.1TB (2.3%)
doufenghu commented on 2022-11-02T11:32:42.214+0800:
通过Confluence页面补充下日志丢失排查详细过程。[~qidaijie]
yangwei commented on 2022-11-02T14:43:25.329+0800:
- common_link_info_c2s和common_link_info_s2c ** 含义：用于WANNAT收集链路信息 ** 如何关闭：在功能端关闭wire_graft插件（注释plug/conflist.inf中对应行）实现不发送该字段 ** 副作用：关闭后，对应功能端的WANNAT功能失效
- common_app_id ** 含义：记录第三方DPI、内置特征和用户自定义特征识别出的app_id，内部调试用，界面不显示 ** 如何关闭：注释tsgconf/tsg_log_field.conf中包含common_app_id的行 ** 副作用：app识别结果不符合预期时，缺乏调试信息
- dns_rr ** 含义：记录dns response records， 内容为json数组 ** 如何关闭：需要更新插件，增加开关
liuxueli commented on 2022-11-14T11:02:57.680+0800:
- 2022-11-14例会：provision增加开关控制是否发送上述日志字段(common_link_info_c2s/common_link_info_s2c/common_app_id/dns_rr)。TSG-12613
gitlab commented on 2022-11-14T17:27:01.670+0800:
[刘学利|https://git.mesalab.cn/liuxueli] mentioned this issue in [a commit|d8e1578b93] of [TSG Appliance / tsg_master|https://git.mesalab.cn/tango/tsg_master] on branch [Branch_v5.7.6|https://git.mesalab.cn/tango/tsg_master/-/tree/Branch_v5.7.6]:{quote}OMPUB-684: 增加开关控制是否发送日志字段(common_link_info_c2s/common_link_info_s2c/common_app_id){quote}
gitlab commented on 2022-11-15T10:52:22.607+0800:
[彭宣正|https://git.mesalab.cn/pxz] mentioned this issue in [a commit|ccc751d765] of [TSG Appliance / tsg_conn_record|https://git.mesalab.cn/tango/tsg_conn_record] on branch [ompub-684|https://git.mesalab.cn/tango/tsg_conn_record/-/tree/ompub-684]:{quote}✨ feat(OMPUB-684): 增加send_dns_rr_switch控制是否发送dns rr，默认不发送{quote}
gitlab commented on 2022-11-18T14:33:48.776+0800:
[彭宣正|https://git.mesalab.cn/pxz] mentioned this issue in [a merge request|https://git.mesalab.cn/tango/tsg_conn_record/-/merge_requests/206] of [TSG Appliance / tsg_conn_record|https://git.mesalab.cn/tango/tsg_conn_record] on branch [ompub-684|https://git.mesalab.cn/tango/tsg_conn_record/-/tree/ompub-684]:{quote}✨ feat(OMPUB-684): 增加send_dns_rr_switch控制是否发送dns rr，默认不发送{quote}
gitlab commented on 2022-11-18T14:35:30.410+0800:
[彭宣正|https://git.mesalab.cn/pxz] mentioned this issue in [a commit|d43ebcde8f] of [TSG Appliance / tsg_conn_record|https://git.mesalab.cn/tango/tsg_conn_record] on branch [ompub-684|https://git.mesalab.cn/tango/tsg_conn_record/-/tree/ompub-684]:{quote}✨ feat(OMPUB-684): 增加send_dns_rr_switch控制是否发送dns rr，默认不发送{quote}
gitlab commented on 2022-11-18T18:34:24.833+0800:
[彭宣正|https://git.mesalab.cn/pxz] mentioned this issue in [a merge request|https://git.mesalab.cn/tango/tsg_conn_record/-/merge_requests/207] of [TSG Appliance / tsg_conn_record|https://git.mesalab.cn/tango/tsg_conn_record] on branch [ompub-684|https://git.mesalab.cn/tango/tsg_conn_record/-/tree/ompub-684]:{quote}✨ feat(OMPUB-684): 增加send_dns_rr_switch控制是否发送dns rr，默认不发送{quote}
gitlab commented on 2022-11-18T18:34:37.963+0800:
[彭宣正|https://git.mesalab.cn/pxz] mentioned this issue in [a commit|6d61fc5d38] of [TSG Appliance / tsg_conn_record|https://git.mesalab.cn/tango/tsg_conn_record] on branch [ompub-684|https://git.mesalab.cn/tango/tsg_conn_record/-/tree/ompub-684]:{quote}✨ feat(OMPUB-684): 增加send_dns_rr_switch控制是否发送dns rr，默认不发送{quote}
gitlab commented on 2022-11-18T18:42:53.535+0800:
[彭宣正|https://git.mesalab.cn/pxz] mentioned this issue in [a merge request|https://git.mesalab.cn/tango/tsg_conn_record/-/merge_requests/208] of [TSG Appliance / tsg_conn_record|https://git.mesalab.cn/tango/tsg_conn_record] on branch [ompub-684|https://git.mesalab.cn/tango/tsg_conn_record/-/tree/ompub-684]:{quote}✨ feat(OMPUB-684): 增加send_dns_rr_switch控制是否发送dns rr，默认不发送{quote}
gitlab commented on 2022-11-18T19:00:31.534+0800:
[彭宣正|https://git.mesalab.cn/pxz] mentioned this issue in [a commit|23bd4efc0a] of [TSG Appliance / tsg_conn_record|https://git.mesalab.cn/tango/tsg_conn_record] on branch [ompub-684|https://git.mesalab.cn/tango/tsg_conn_record/-/tree/ompub-684]:{quote}✨ feat(OMPUB-684): 增加send_dns_rr_switch控制是否发送dns rr，默认不发送{quote}
gitlab commented on 2022-11-18T19:00:56.596+0800:
[彭宣正|https://git.mesalab.cn/pxz] mentioned this issue in [a merge request|https://git.mesalab.cn/tango/tsg_conn_record/-/merge_requests/210] of [TSG Appliance / tsg_conn_record|https://git.mesalab.cn/tango/tsg_conn_record] on branch [ompub-684|https://git.mesalab.cn/tango/tsg_conn_record/-/tree/ompub-684]:{quote}✨ feat(OMPUB-684): 增加send_dns_rr_switch控制是否发送dns rr，默认不发送{quote}
gitlab commented on 2022-11-18T20:11:37.958+0800:
[彭宣正|https://git.mesalab.cn/pxz] mentioned this issue in [a commit|c60d16e447] of [TSG / tsg-os-buildimage|https://git.mesalab.cn/tsg/tsg-os-buildimage] on branch [update-tsg_conn_sketch|https://git.mesalab.cn/tsg/tsg-os-buildimage/-/tree/update-tsg_conn_sketch]:{quote}🐞 fix(TSG-12733,OMPUB-684): update tsg_conn_sketch-3.3.13 to tsg_conn_sketch-3.3.14{quote}
gitlab commented on 2022-11-18T20:12:10.228+0800:
[彭宣正|https://git.mesalab.cn/pxz] mentioned this issue in [a merge request|https://git.mesalab.cn/tsg/tsg-os-buildimage/-/merge_requests/984] of [TSG / tsg-os-buildimage|https://git.mesalab.cn/tsg/tsg-os-buildimage] on branch [update-tsg_conn_sketch|https://git.mesalab.cn/tsg/tsg-os-buildimage/-/tree/update-tsg_conn_sketch]:{quote}🐞 fix(TSG-12733,OMPUB-684): update tsg_conn_sketch-3.3.13 to tsg_conn_sketch-3.3.14{quote}
gitlab commented on 2022-11-18T20:14:23.157+0800:
[彭宣正|https://git.mesalab.cn/pxz] mentioned this issue in [a commit|611c1e6e64] of [TSG / tsg-os-buildimage|https://git.mesalab.cn/tsg/tsg-os-buildimage] on branch [update-tsg_conn_sketch|https://git.mesalab.cn/tsg/tsg-os-buildimage/-/tree/update-tsg_conn_sketch]:{quote}🐞 fix(TSG-12733,OMPUB-684): update tsg_conn_sketch-3.3.13 to tsg_conn_sketch-3.3.14{quote}
gitlab commented on 2022-12-30T13:41:17.939+0800:
[刘学利|https://git.mesalab.cn/liuxueli] mentioned this issue in [a commit|ceb492bcb3] of [TSG Appliance / tsg_master|https://git.mesalab.cn/tango/tsg_master] on branch [feature-OMPUB-684|https://git.mesalab.cn/tango/tsg_master/-/tree/feature-OMPUB-684]:{quote}OMPUB-684: 增加开关控制是否发送日志字段(common_link_info_c2s/common_link_info_s2c/common_app_id){quote}
gitlab commented on 2022-12-30T13:42:57.823+0800:
[刘学利|https://git.mesalab.cn/liuxueli] mentioned this issue in [a commit|3bd6a5642f] of [TSG Appliance / tsg_master|https://git.mesalab.cn/tango/tsg_master] on branch [feature-OMPUB-684|https://git.mesalab.cn/tango/tsg_master/-/tree/feature-OMPUB-684]:{quote}OMPUB-684: 增加开关控制是否发送日志字段(common_link_info_c2s/common_link_info_s2c/common_app_id){quote}
gitlab commented on 2022-12-30T13:57:22.745+0800:
[刘学利|https://git.mesalab.cn/liuxueli] mentioned this issue in [a merge request|https://git.mesalab.cn/tango/tsg_master/-/merge_requests/264] of [TSG Appliance / tsg_master|https://git.mesalab.cn/tango/tsg_master] on branch [feature-OMPUB-684|https://git.mesalab.cn/tango/tsg_master/-/tree/feature-OMPUB-684]:{quote}OMPUB-684: 增加开关控制是否发送日志字段(common_link_info_c2s/common_link_info_s2c/common_app_id){quote}
gitlab commented on 2022-12-30T14:18:48.577+0800:
[刘学利|https://git.mesalab.cn/liuxueli] mentioned this issue in [a commit|eeae469a3f] of [TSG Appliance / tsg_master|https://git.mesalab.cn/tango/tsg_master] on branch [TSG-OS.v22.11|https://git.mesalab.cn/tango/tsg_master/-/tree/TSG-OS.v22.11]:{quote}OMPUB-684: 增加开关控制是否发送日志字段(common_link_info_c2s/common_link_info_s2c/common_app_id){quote}
gitlab commented on 2023-01-13T10:19:11.044+0800:
[刘学利|https://git.mesalab.cn/liuxueli] mentioned this issue in [a commit|a0bc07c276] of [TSG / tsg-os-buildimage|https://git.mesalab.cn/tsg/tsg-os-buildimage] on branch [update-v22.11-firewall-v6|https://git.mesalab.cn/tsg/tsg-os-buildimage/-/tree/update-v22.11-firewall-v6]:{quote}更新sapp、tsg_master、fw_dns_plug，修复或新增：{quote}
gitlab commented on 2023-01-13T10:20:04.994+0800:
[刘学利|https://git.mesalab.cn/liuxueli] mentioned this issue in [a merge request|https://git.mesalab.cn/tsg/tsg-os-buildimage/-/merge_requests/1067] of [TSG / tsg-os-buildimage|https://git.mesalab.cn/tsg/tsg-os-buildimage] on branch [update-v22.11-firewall-v6|https://git.mesalab.cn/tsg/tsg-os-buildimage/-/tree/update-v22.11-firewall-v6]:{quote}更新sapp、tsg_master、fw_dns_plug，修复或新增：{quote}