admin/geedge-jira
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
6a8bfef5778fc7ddd06fa05776bf1ecd0f61d4c4
geedge-jira/md/OMPUB-1274.md
hello 46daec8ae4 first
2025-09-14 21:52:36 +00:00

7.5 KiB
Raw Blame History

【M22项目】成功导入50w FQDN(150秒后删除)，MAAT加载50w FQDN使用94分54秒(11:57:35->13:32:29)

ID Creation Date Assignee Status
OMPUB-1274 2024-05-07T14:17:00.000+0800 杨威 已关闭

复现步骤：

  • 导入50w FQDN （已发给功能端同事）
  • 导入成功后等待150秒，删除该FQDN
  • 下发Monitor Rule： !image-2024-05-07-12-44-16-054.png|thumbnail!

当前问题：Monitor Rule下发已经超过半小时，当前新产生的流量仍然无法命中该Monitor Rule

  • 创建时间： !image-2024-05-07-12-48-55-523.png|thumbnail!
  • 新日志中的Monitor Rule List为空： !image-2024-05-07-12-49-45-295.png|thumbnail!

问题排查：

  • 联系CM同事排查，该Rule已可以在CM的redis中成功查询到 !image-2024-05-07-12-46-53-494.png|thumbnail!liuxueli commented on 2024-05-07T14:24:43.187+0800:

  • 查看MAAT日志：发现Start INC update: 5593 -> 5594 (500000 entries)1小时后还未完成，导致所有新下发策略未生效。 ** !image-2024-05-07-12-53-29-342.png!

  • 栈信息： ** [^20240507.YGN-MYTEL-TSGX001.stack.pid.104.txt]  ** [^20240507.YGN-MYTEL-TSGX001.stack.pid.104.2.txt] ** {code:java} #0  0x00007f2e4e0d87d1 in ue2::RoseInstrBaseOneTarget<(RoseInstructionCode)29, ROSE_STRUCT_DEDUPE_SOM, ue2::RoseInstrDedupeSom>::update_target(ue2::RoseInstruction const*, ue2::RoseInstruction const*) () from /opt/tsg/framework/lib/libmaatframe.so.4 #1  0x00007f2e4e0a69c3 in ue2::RoseProgram::add_block(ue2::RoseProgram&&) () from /opt/tsg/framework/lib/libmaatframe.so.4 #2  0x00007f2e4e0a835e in ue2::assembleProgramBlocks(std::vector<ue2::RoseProgram, std::allocatorue2::RoseProgram >&&) () from /opt/tsg/framework/lib/libmaatframe.so.4 #3  0x00007f2e4e06ee9b in ue2::buildLiteralPrograms(ue2::RoseBuildImpl const&, std::vector<ue2::LitFragment, std::allocatorue2::LitFragment >&, ue2::(anonymous namespace)::build_context&, ue2::ProgramBuild&, ue2::LitProto*, ue2::LitProto*, ue2::LitProto*, ue2::LitProto*) () from /opt/tsg/framework/lib/libmaatframe.so.4 #4  0x00007f2e4e07719e in ue2::RoseBuildImpl::buildFinalEngine(unsigned int) () from /opt/tsg/framework/lib/libmaatframe.so.4 #5  0x00007f2e4ddcda4e in ue2::RoseBuildImpl::buildRose(unsigned int) () from /opt/tsg/framework/lib/libmaatframe.so.4 #6  0x00007f2e4da9f314 in ue2::build(ue2::NG&, unsigned int*, unsigned char) () from /opt/tsg/framework/lib/libmaatframe.so.4 #7  0x00007f2e4da9c6da in ue2::hs_compile_lit_multi_int(char const* const*, unsigned int const*, unsigned int const*, hs_expr_ext const* const*, unsigned long const*, unsigned int, unsigned int, hs_platform_info const*, hs_database**, hs_compile_error**, ue2::Grey const&) () from /opt/tsg/framework/lib/libmaatframe.so.4 #8  0x00007f2e4da9d101 in hs_compile_lit_multi () from /opt/tsg/framework/lib/libmaatframe.so.4 #9  0x00007f2e4da94a2c in hs_build_lit_db () from /opt/tsg/framework/lib/libmaatframe.so.4 #10 0x00007f2e4da92c94 in expr_matcher_new () from /opt/tsg/framework/lib/libmaatframe.so.4 #11 0x00007f2e4d9ee937 in expr_runtime_commit () from /opt/tsg/framework/lib/libmaatframe.so.4 #12 0x00007f2e4d9d3a45 in maat_runtime_commit.isra () from /opt/tsg/framework/lib/libmaatframe.so.4 #13 0x00007f2e4d9d3b5f in maat_finish_cb () from /opt/tsg/framework/lib/libmaatframe.so.4 #14 0x00007f2e4d9db587 in redis_monitor_traverse () from /opt/tsg/framework/lib/libmaatframe.so.4 #15 0x00007f2e4d9d466b in rule_monitor_loop () from /opt/tsg/framework/lib/libmaatframe.so.4 #16 0x00007f300ed071ca in start_thread () from /lib64/libpthread.so.0 #17 0x00007f300dbf5e73 in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f2e4bc40700 (LWP 116)): {code}

liuxueli commented on 2024-05-07T15:07:30.832+0800:

  • 50w FQDN加载需使用94分54秒（11:57:35 -> 13:32:29） ** {code:java} Tue May 7 11:57:20 2024, INFO, maat.rule(118), rule_monitor_loop thread still alive......... Tue May 7 11:57:30 2024, INFO, maat.rule(118), rule_monitor_loop thread still alive......... Tue May 7 11:57:32 2024, INFO, maat.redis_monitor(118), Inc Update from instance_version 5593 to 5594 (500000 entries) Tue May 7 11:57:33 2024, INFO, maat.redis_monitor(118), Start INC update: 5593 -> 5594 (500000 entries) Tue May 7 11:57:35 2024, INFO, maat.expr_matcher(118), expr_matcher module: build bool matcher of 545441 expressions with 51685516 bytes memory Tue May 7 13:32:29 2024, INFO, maat.expr(118), table[TSG_OBJ_FQDN] has 545441 rules, commit 545441 expr rules(literal_rules:545441 regex_rules:0) and rebuild expr_matcher(hyperscan) completed, version:5594, consume:5694107ms Tue May 7 13:32:29 2024, INFO, maat.rule(118), table:<TSG_COMPILE> rule_count:96 Tue May 7 13:32:29 2024, INFO, maat.rule(118), table:<SECURITY_COMPILE_PLUGIN> rule_count:16 Tue May 7 13:32:29 2024, INFO, maat.rule(118), table:<STATISTICS_COMPILE_PLUGIN> rule_count:18 Tue May 7 13:32:29 2024, INFO, maat.rule(118), table:<APP_SIG_COMPILE_PLUGIN> rule_count:58 Tue May 7 13:32:29 2024, INFO, maat.rule(118), table:<TSG_GROUP_COMPILE_RELATION> rule_count:188 Tue May 7 13:32:29 2024, INFO, maat.rule(118), table:<TSG_GROUP_GROUP_RELATION> rule_count:9 Tue May 7 13:32:29 2024, INFO, maat.rule(118), table:<TSG_IP_ADDR> rule_count:404141 Tue May 7 13:32:29 2024, INFO, maat.rule(118), table:<TSG_OBJ_ACCOUNT> rule_count:1 Tue May 7 13:32:29 2024, INFO, maat.rule(118), table:<TSG_OBJ_URL> rule_count:14 Tue May 7 13:32:29 2024, INFO, maat.rule(118), table:<TSG_OBJ_FQDN> rule_count:545441 Tue May 7 13:32:29 2024, INFO, maat.rule(118), table:<TSG_OBJ_KEYWORDS> rule_count:23 {code}

yangwei commented on 2024-05-07T17:30:01.642+0800:

  • FQDN表类型为expr，maat中对于该类型使用Hyperscan引擎执行扫描
  • 导入的50万随机规则格式如下，{}末位均为.abcdef.com{}，直接使用Hyperscan加载，耗时同样较长

!image-2024-05-07-17-26-38-928.png!

  • 使用测试组提供的另一组50万随机性更高的规则，Hyperscan加载速度为26s

!image-2024-05-07-17-28-44-692.png|width=413,height=212!

  • 对第一组50万随机规则进行修剪，将末位的.abcdef.com调整为.com，直接使用Hyperscan加载速率也在30s左右

!image-2024-05-07-17-30-33-029.png!

 

初步结论：Hyperscan对于后缀重复度较高的文本规则，加载速度较慢，考虑使用Rulescan进行加载速度验证

yangwei commented on 2024-05-08T09:45:34.560+0800:

maat内部对expr规则使用双引擎（Hyperscan+Rulescan），当规则条目大于5万时，设计上自动启用Rulescan。

但是上述特性仅在全量时有效，加载增量时存在bug（https://jira.geedge.net/browse/TSG-21089），未进行有效判断，默认仍使用Hyperscan，造成了加载issue中50万规则集慢的问题。

待[~liuchang]修复上述增量加载的bug后，在京版环境复测验证修复后的加载时间。 

yangwei commented on 2024-08-30T13:34:32.860+0800:

参见TSG-21089，现场已更新，关闭此issue

 

Attachments

56796/20240507.YGN-MYTEL-TSGX001.stack.pid.104.2.txt

56797/20240507.YGN-MYTEL-TSGX001.stack.pid.104.txt

56791/image-2024-05-07-12-44-16-054.png

56790/image-2024-05-07-12-46-53-494.png

56792/image-2024-05-07-12-48-55-523.png

56793/image-2024-05-07-12-49-45-295.png

56794/image-2024-05-07-12-53-29-342.png

56823/image-2024-05-07-17-26-38-928.png

56825/image-2024-05-07-17-28-44-692.png

56826/image-2024-05-07-17-30-33-029.png