WMS-UTR现场: MAAT加载SD的动态配置大量报错

ID	Creation Date	Assignee	Status
OMPUB-1221	2024-04-09T13:33:56.000+0800	刘学利	已关闭

MAAT加载SD的动态配置大量报错 ** [^firewall.sd.maat.status.txt] ** {code:java} Tue Apr 9 00:00:00 2024, INFO, maat.ipport_plugin(148), table[TSG_DYN_IPPORT_SUBSCRIBER_MAPPING] commit 7945740 ipport_plugin rules and rebuild ipport_matcher completed, version:136774, consume:1138ms Tue Apr 9 00:00:01 2024, INFO, maat.rule(148), table:<TSG_DYN_IPPORT_SUBSCRIBER_MAPPING> rule_count:7945740 Tue Apr 9 00:00:01 2024, INFO, maat.rule(148), Inc config version 136774 load 7945740 entries complete Tue Apr 9 00:00:01 2024, INFO, maat.rule(148), Actual update config version 136774, 7945740 entries load to maat runtime. Tue Apr 9 00:00:02 2024, INFO, maat.redis_monitor(148), Inc Update from instance_version 136774 to 136818 (203726 entries) Tue Apr 9 00:00:02 2024, INFO, maat.redis_monitor(148), Start INC update: 136774 -> 136818 (203726 entries) Tue Apr 9 00:00:07 2024, FATAL, maat.ipport_plugin(148), [ipport_plugin_runtime_update:462]ipport_plugin table:<TSG_DYN_IPPORT_SUBSCRIBER_MAPPING> update one line failed, line:1090201366 4 154.80.12.233 45472 45503 923020068292 {"imsi":"410010097282951","phone_number":"923020068292"} 1 1712599689434 f645d22f-2782-4233-8924-d9b87070ab65 Tue Apr 9 00:00:07 2024, FATAL, maat.rule(148), [maat_update_cb:166] table<TSG_DYN_IPPORT_SUBSCRIBER_MAPPING> update runtime error for rules:1090201366 4 154.80.12.233 45472 45503 923020068292 {"imsi":"410010097282951","phone_number":"923020068292"} 1 1712599689434 f645d22f-2782-4233-8924-d9b87070ab65 Tue Apr 9 00:00:07 2024, FATAL, maat.ipport_plugin(148), [ipport_plugin_runtime_update:462]ipport_plugin table:<TSG_DYN_IPPORT_SUBSCRIBER_MAPPING> update one line failed, line:1090209204 4 154.80.97.13 46304 46335 923464806173 {"imsi":"410010131200122","phone_number":"923464806173"} 1 1712599689429 457de963-5fa4-4e17-bf59-48d3394b2813 Tue Apr 9 00:00:07 2024, FATAL, maat.rule(148), [maat_update_cb:166] table<TSG_DYN_IPPORT_SUBSCRIBER_MAPPING> update runtime error for rules:1090209204 4 154.80.97.13 46304 46335 923464806173 {"imsi":"410010131200122","phone_number":"923464806173"} 1 1712599689429 457de963-5fa4-4e17-bf59-48d3394b2813 Tue Apr 9 00:00:07 2024, FATAL, maat.ipport_plugin(148), [ipport_plugin_runtime_update:462]ipport_plugin table:<TSG_DYN_IPPORT_SUBSCRIBER_MAPPING> update one line failed, line:1358585105 4 154.80.25.7 61888 61919 923285682672 {"imsi":"410010207709822","phone_number":"923285682672"} 1 1712599689453 32904e0a-7239-474b-855a-89e5cc59b44a Tue Apr 9 00:00:07 2024, FATAL, maat.rule(148), [maat_update_cb:166] table<TSG_DYN_IPPORT_SUBSCRIBER_MAPPING> update runtime error for rules:1358585105 4 154.80.25.7 61888 61919 923285682672 {"imsi":"410010207709822","phone_number":"923285682672"} 1 1712599689453 32904e0a-7239-474b-855a-89e5cc59b44a Tue Apr 9 00:00:07 2024, FATAL, maat.ipport_plugin(148), [ipport_plugin_runtime_update:462]ipport_plugin table:<TSG_DYN_IPPORT_SUBSCRIBER_MAPPING> update one line failed, line:1358694849 4 154.80.42.81 39968 39999 923097725939 {"imsi":"410018148732171","phone_number":"923097725939"} 1 1712599689414 a462d5f2-7b6b-4c01-9401-1563f80979ea Tue Apr 9 00:00:07 2024, FATAL, maat.rule(148), [maat_update_cb:166] table<TSG_DYN_IPPORT_SUBSCRIBER_MAPPING> update runtime error for rules:1358694849 4 154.80.42.81 39968 39999 923097725939 {"imsi":"410018148732171","phone_number":"923097725939"} 1 1712599689414 a462d5f2-7b6b-4c01-9401-1563f80979ea Tue Apr 9 00:00:07 2024, FATAL, maat.ipport_plugin(148), [ipport_plugin_runtime_update:462]ipport_plugin table:<TSG_DYN_IPPORT_SUBSCRIBER_MAPPING> update one line failed, line:16145380 4 154.80.34.116 61504 61535 923287165971 {"imsi":"410010181073151","phone_number":"923287165971"} 1 1712599689354 4ecc5c70-ff17-441c-afc9-552fd163c7b3 Tue Apr 9 00:00:07 2024, FATAL, maat.rule(148), [maat_update_cb:166] table<TSG_DYN_IPPORT_SUBSCRIBER_MAPPING> update runtime error for rules:16145380 4 154.80.34.116 61504 61535 923287165971 {"imsi":"410010181073151","phone_number":"923287165971"} 1 1712599689354 4ecc5c70-ff17-441c-afc9-552fd163c7b3 Tue Apr 9 00:00:07 2024, FATAL, maat.ipport_plugin(148), [ipport_plugin_runtime_update:462]ipport_plugin table:<TSG_DYN_IPPORT_SUBSCRIBER_MAPPING> update one line failed, line:16191199 4 154.80.34.5 42304 42335 923164100042 {"imsi":"410010017449622","phone_number":"923164100042"} 1 1712599689356 23de7430-57d2-467c-b671-cbf4e5ce03a8 Tue Apr 9 00:00:07 2024, FATAL, maat.rule(148), [maat_update_cb:166] table<TSG_DYN_IPPORT_SUBSCRIBER_MAPPING> update runtime error for rules:16191199 4 154.80.34.5 42304 42335 923164100042 {"imsi":"410010017449622","phone_number":"923164100042"} 1 1712599689356 23de7430-57d2-467c-b671-cbf4e5ce03a8 Tue Apr 9 00:00:07 2024, FATAL, maat.ipport_plugin(148), [ipport_plugin_runtime_update:462]ipport_plugin table:<TSG_DYN_IPPORT_SUBSCRIBER_MAPPING> update one line failed, line:1626660397 4 154.80.113.152 41408 41439 923056403868 {"imsi":"410018067860169","phone_number":"923056403868"} 1 1712599689419 2b2ffec7-1af5-4daf-b50c-ed92df6b06f5 Tue Apr 9 00:00:07 2024, FATAL, maat.rule(148), [maat_update_cb:166] table<TSG_DYN_IPPORT_SUBSCRIBER_MAPPING> update runtime error for rules:1626660397 4 154.80.113.152 41408 41439 923056403868 {"imsi":"410018067860169","phone_number":"923056403868"} 1 1712599689419 2b2ffec7-1af5-4daf-b50c-ed92df6b06f5 Tue Apr 9 00:00:07 2024, FATAL, maat.ipport_plugin(148), [ipport_plugin_runtime_update:462]ipport_plugin table:<TSG_DYN_IPPORT_SUBSCRIBER_MAPPING> update one line failed, line:1627032536 4 154.80.41.145 44864 44895 923024151564 {"imsi":"410010026176839","phone_number":"923024151564"} 1 1712599689356 62074b70-98b7-474a-986c-46fab867f78f Tue Apr 9 00:00:07 2024, FATAL, maat.rule(148), [maat_update_cb:166] table<TSG_DYN_IPPORT_SUBSCRIBER_MAPPING> update runtime error for rules:1627032536 4 154.80.41.145 44864 44895 923024151564 {"imsi":"410010026176839","phone_number":"923024151564"} 1 1712599689356 62074b70-98b7-474a-986c-46fab867f78f Tue Apr 9 00:00:07 2024, FATAL, maat.ipport_plugin(148), [ipport_plugin_runtime_update:462]ipport_plugin table:<TSG_DYN_IPPORT_SUBSCRIBER_MAPPING> update one line failed, line:1895094699 4 154.80.42.112 44512 44543 923237173765 {"imsi":"410010077938140","phone_number":"923237173765"} 1 1712599689343 ac505e2f-59ad-4846-92ed-a92d6a91f2c9 Tue Apr 9 00:00:07 2024, FATAL, maat.rule(148), [maat_update_cb:166] table<TSG_DYN_IPPORT_SUBSCRIBER_MAPPING> update runtime error for rules:1895094699 4 154.80.42.112 44512 44543 923237173765 {"imsi":"410010077938140","phone_number":"923237173765"} 1 1712599689343 ac505e2f-59ad-4846-92ed-a92d6a91f2c9 Tue Apr 9 00:00:07 2024, FATAL, maat.ipport_plugin(148), [ipport_plugin_runtime_update:462]ipport_plugin table:<TSG_DYN_IPPORT_SUBSCRIBER_MAPPING> update one line failed, line:1895127712 4 154.80.4.124 42400 42431 923008084142 {"imsi":"410010006449996","phone_number":"923008084142"} 1 1712599689456 e612d0f8-df45-4ba8-9989-8f635155f109 Tue Apr 9 00:00:07 2024, FATAL, maat.rule(148), [maat_update_cb:166] table<TSG_DYN_IPPORT_SUBSCRIBER_MAPPING> update runtime error for rules:1895127712 4 154.80.4.124 42400 42431 923008084142 {"imsi":"410010006449996","phone_number":"923008084142"} 1 1712599689456 e612d0f8-df45-4ba8-9989-8f635155f109 {code}

** !image-2024-04-09-13-33-46-540.png!duandongmei commented on 2024-04-17T16:21:34.089+0800:

问题现状： ** TSG在跨版本的配置扫描中，同一个条配置只能获取两个MAAT_UPDATE_STATUS记录(实际跨版本的范围内同一条ID有多次MAAT_UPDATE_STATUS的更新)，此现象导致SAPP配置加载时会出现与本地ID重复的问题。
问题复现：我与与刘畅在信息港复现该问题（测试过程中的monitor的日志见附件[^monitor.20240417.log]） ** Provisioner ZADD的记录： *** !image-2024-04-17-16-09-13-961.png|width=539,height=178! ** TSG跨版本获取MAAT_UPDATE_STATUS的记录 *** 初始从404版本进行了一次全量配置扫描，此时12配置状态为1（有效） *** 增量开始，TSG ZRANGEBYSCORE日志 **** 1713333715.891403 [0 192.168.40.137:43086] "ZRANGEBYSCORE" "MAAT_UPDATE_STATUS" "(404" "412" "WITHSCORES"

{code:java} Wed Apr 17 14:01:55 2024, INFO, maat.redis_monitor(302693), ZRANGEBYSCORE MAAT_UPDATE_STATUS (404 412 WITHSCORES: element_str: ADD,TSG_DYN_IPPORT_SUBSCRIBER_MAPPING,12 Wed Apr 17 14:01:55 2024, INFO, maat.redis_monitor(302693), ZRANGEBYSCORE MAAT_UPDATE_STATUS (404 412 WITHSCORES: element_score: 406

Wed Apr 17 14:01:55 2024, INFO, maat.redis_monitor(302693), ZRANGEBYSCORE MAAT_UPDATE_STATUS (404 412 WITHSCORES: element_str: DEL,TSG_DYN_IPPORT_SUBSCRIBER_MAPPING,12 Wed Apr 17 14:01:55 2024, INFO, maat.redis_monitor(302693), ZRANGEBYSCORE MAAT_UPDATE_STATUS (404 412 WITHSCORES: element_score: 407{code}

** 问题复现情况说明 *** 当MAAT_VERSION为404时，TSG功能端全量配置中12配置为生效 *** 接着功能端增量扫描(404,412]更新状态时，12配置扫描结果为406 ADD,407DEL *** 此时功能端进行12配置的ADD操作，发现与本地的12生效配置重复 *** 到此复现反馈的ID重复问题

* 问题分析： * ** 通过分析Redis monitor未发现ZADD命令遗漏更新SCORE的情况，进一步分析Redis的ZADD和ZRANGEBYSCORE执行逻辑 ** MAAT_UPDATE_STATUS操作为ZADD命令，Redis ZADD命令逻辑为不存在成员SCORE执行新增，已存在的成员会更新SCORE。 ** Redis ZADD命令的执行逻辑可以保证当前版本内的配置状态正确执行，可不做调整。

* 建议： * ** 建议功能端调整此问题日志级别

liuxueli commented on 2024-05-10T16:10:42.370+0800:

2024/04/28 已Hoxfix修复，参见： https://docs.geedge.net/pages/viewpage.action?pageId=129092431

Attachments

54646/firewall.sd.maat.status.txt

54642/image-2024-04-09-13-33-46-540.png

55104/image-2024-04-12-16-57-10-244.png

55446/image-2024-04-17-16-09-13-961.png

55451/monitor.20240417.log

11 KiB Raw Blame History Unescape Escape

WMS-UTR现场: MAAT加载SD的动态配置大量报错

Attachments

11 KiB

Raw Blame History