113 lines
11 KiB
Markdown
113 lines
11 KiB
Markdown
# WMS-UTR现场: MAAT加载SD的动态配置大量报错
|
||
|
||
| ID | Creation Date | Assignee | Status |
|
||
|----|----------------|----------|--------|
|
||
| OMPUB-1221 | 2024-04-09T13:33:56.000+0800 | 刘学利 | 已关闭 |
|
||
|
||
|
||
---
|
||
|
||
* MAAT加载SD的动态配置大量报错
|
||
** [^firewall.sd.maat.status.txt]
|
||
**
|
||
{code:java}
|
||
Tue Apr 9 00:00:00 2024, INFO, maat.ipport_plugin(148), table[TSG_DYN_IPPORT_SUBSCRIBER_MAPPING] commit 7945740 ipport_plugin rules and rebuild ipport_matcher completed, version:136774, consume:1138ms
|
||
Tue Apr 9 00:00:01 2024, INFO, maat.rule(148), table:<TSG_DYN_IPPORT_SUBSCRIBER_MAPPING> rule_count:7945740
|
||
Tue Apr 9 00:00:01 2024, INFO, maat.rule(148), Inc config version 136774 load 7945740 entries complete
|
||
Tue Apr 9 00:00:01 2024, INFO, maat.rule(148), Actual update config version 136774, 7945740 entries load to maat runtime.
|
||
Tue Apr 9 00:00:02 2024, INFO, maat.redis_monitor(148), Inc Update from instance_version 136774 to 136818 (203726 entries)
|
||
Tue Apr 9 00:00:02 2024, INFO, maat.redis_monitor(148), Start INC update: 136774 -> 136818 (203726 entries)
|
||
Tue Apr 9 00:00:07 2024, FATAL, maat.ipport_plugin(148), [ipport_plugin_runtime_update:462]ipport_plugin table:<TSG_DYN_IPPORT_SUBSCRIBER_MAPPING> update one line failed, line:1090201366 4 154.80.12.233 45472 45503 923020068292 {"imsi":"410010097282951","phone_number":"923020068292"} 1 1712599689434 f645d22f-2782-4233-8924-d9b87070ab65
|
||
Tue Apr 9 00:00:07 2024, FATAL, maat.rule(148), [maat_update_cb:166] table<TSG_DYN_IPPORT_SUBSCRIBER_MAPPING> update runtime error for rules:1090201366 4 154.80.12.233 45472 45503 923020068292 {"imsi":"410010097282951","phone_number":"923020068292"} 1 1712599689434 f645d22f-2782-4233-8924-d9b87070ab65
|
||
Tue Apr 9 00:00:07 2024, FATAL, maat.ipport_plugin(148), [ipport_plugin_runtime_update:462]ipport_plugin table:<TSG_DYN_IPPORT_SUBSCRIBER_MAPPING> update one line failed, line:1090209204 4 154.80.97.13 46304 46335 923464806173 {"imsi":"410010131200122","phone_number":"923464806173"} 1 1712599689429 457de963-5fa4-4e17-bf59-48d3394b2813
|
||
Tue Apr 9 00:00:07 2024, FATAL, maat.rule(148), [maat_update_cb:166] table<TSG_DYN_IPPORT_SUBSCRIBER_MAPPING> update runtime error for rules:1090209204 4 154.80.97.13 46304 46335 923464806173 {"imsi":"410010131200122","phone_number":"923464806173"} 1 1712599689429 457de963-5fa4-4e17-bf59-48d3394b2813
|
||
Tue Apr 9 00:00:07 2024, FATAL, maat.ipport_plugin(148), [ipport_plugin_runtime_update:462]ipport_plugin table:<TSG_DYN_IPPORT_SUBSCRIBER_MAPPING> update one line failed, line:1358585105 4 154.80.25.7 61888 61919 923285682672 {"imsi":"410010207709822","phone_number":"923285682672"} 1 1712599689453 32904e0a-7239-474b-855a-89e5cc59b44a
|
||
Tue Apr 9 00:00:07 2024, FATAL, maat.rule(148), [maat_update_cb:166] table<TSG_DYN_IPPORT_SUBSCRIBER_MAPPING> update runtime error for rules:1358585105 4 154.80.25.7 61888 61919 923285682672 {"imsi":"410010207709822","phone_number":"923285682672"} 1 1712599689453 32904e0a-7239-474b-855a-89e5cc59b44a
|
||
Tue Apr 9 00:00:07 2024, FATAL, maat.ipport_plugin(148), [ipport_plugin_runtime_update:462]ipport_plugin table:<TSG_DYN_IPPORT_SUBSCRIBER_MAPPING> update one line failed, line:1358694849 4 154.80.42.81 39968 39999 923097725939 {"imsi":"410018148732171","phone_number":"923097725939"} 1 1712599689414 a462d5f2-7b6b-4c01-9401-1563f80979ea
|
||
Tue Apr 9 00:00:07 2024, FATAL, maat.rule(148), [maat_update_cb:166] table<TSG_DYN_IPPORT_SUBSCRIBER_MAPPING> update runtime error for rules:1358694849 4 154.80.42.81 39968 39999 923097725939 {"imsi":"410018148732171","phone_number":"923097725939"} 1 1712599689414 a462d5f2-7b6b-4c01-9401-1563f80979ea
|
||
Tue Apr 9 00:00:07 2024, FATAL, maat.ipport_plugin(148), [ipport_plugin_runtime_update:462]ipport_plugin table:<TSG_DYN_IPPORT_SUBSCRIBER_MAPPING> update one line failed, line:16145380 4 154.80.34.116 61504 61535 923287165971 {"imsi":"410010181073151","phone_number":"923287165971"} 1 1712599689354 4ecc5c70-ff17-441c-afc9-552fd163c7b3
|
||
Tue Apr 9 00:00:07 2024, FATAL, maat.rule(148), [maat_update_cb:166] table<TSG_DYN_IPPORT_SUBSCRIBER_MAPPING> update runtime error for rules:16145380 4 154.80.34.116 61504 61535 923287165971 {"imsi":"410010181073151","phone_number":"923287165971"} 1 1712599689354 4ecc5c70-ff17-441c-afc9-552fd163c7b3
|
||
Tue Apr 9 00:00:07 2024, FATAL, maat.ipport_plugin(148), [ipport_plugin_runtime_update:462]ipport_plugin table:<TSG_DYN_IPPORT_SUBSCRIBER_MAPPING> update one line failed, line:16191199 4 154.80.34.5 42304 42335 923164100042 {"imsi":"410010017449622","phone_number":"923164100042"} 1 1712599689356 23de7430-57d2-467c-b671-cbf4e5ce03a8
|
||
Tue Apr 9 00:00:07 2024, FATAL, maat.rule(148), [maat_update_cb:166] table<TSG_DYN_IPPORT_SUBSCRIBER_MAPPING> update runtime error for rules:16191199 4 154.80.34.5 42304 42335 923164100042 {"imsi":"410010017449622","phone_number":"923164100042"} 1 1712599689356 23de7430-57d2-467c-b671-cbf4e5ce03a8
|
||
Tue Apr 9 00:00:07 2024, FATAL, maat.ipport_plugin(148), [ipport_plugin_runtime_update:462]ipport_plugin table:<TSG_DYN_IPPORT_SUBSCRIBER_MAPPING> update one line failed, line:1626660397 4 154.80.113.152 41408 41439 923056403868 {"imsi":"410018067860169","phone_number":"923056403868"} 1 1712599689419 2b2ffec7-1af5-4daf-b50c-ed92df6b06f5
|
||
Tue Apr 9 00:00:07 2024, FATAL, maat.rule(148), [maat_update_cb:166] table<TSG_DYN_IPPORT_SUBSCRIBER_MAPPING> update runtime error for rules:1626660397 4 154.80.113.152 41408 41439 923056403868 {"imsi":"410018067860169","phone_number":"923056403868"} 1 1712599689419 2b2ffec7-1af5-4daf-b50c-ed92df6b06f5
|
||
Tue Apr 9 00:00:07 2024, FATAL, maat.ipport_plugin(148), [ipport_plugin_runtime_update:462]ipport_plugin table:<TSG_DYN_IPPORT_SUBSCRIBER_MAPPING> update one line failed, line:1627032536 4 154.80.41.145 44864 44895 923024151564 {"imsi":"410010026176839","phone_number":"923024151564"} 1 1712599689356 62074b70-98b7-474a-986c-46fab867f78f
|
||
Tue Apr 9 00:00:07 2024, FATAL, maat.rule(148), [maat_update_cb:166] table<TSG_DYN_IPPORT_SUBSCRIBER_MAPPING> update runtime error for rules:1627032536 4 154.80.41.145 44864 44895 923024151564 {"imsi":"410010026176839","phone_number":"923024151564"} 1 1712599689356 62074b70-98b7-474a-986c-46fab867f78f
|
||
Tue Apr 9 00:00:07 2024, FATAL, maat.ipport_plugin(148), [ipport_plugin_runtime_update:462]ipport_plugin table:<TSG_DYN_IPPORT_SUBSCRIBER_MAPPING> update one line failed, line:1895094699 4 154.80.42.112 44512 44543 923237173765 {"imsi":"410010077938140","phone_number":"923237173765"} 1 1712599689343 ac505e2f-59ad-4846-92ed-a92d6a91f2c9
|
||
Tue Apr 9 00:00:07 2024, FATAL, maat.rule(148), [maat_update_cb:166] table<TSG_DYN_IPPORT_SUBSCRIBER_MAPPING> update runtime error for rules:1895094699 4 154.80.42.112 44512 44543 923237173765 {"imsi":"410010077938140","phone_number":"923237173765"} 1 1712599689343 ac505e2f-59ad-4846-92ed-a92d6a91f2c9
|
||
Tue Apr 9 00:00:07 2024, FATAL, maat.ipport_plugin(148), [ipport_plugin_runtime_update:462]ipport_plugin table:<TSG_DYN_IPPORT_SUBSCRIBER_MAPPING> update one line failed, line:1895127712 4 154.80.4.124 42400 42431 923008084142 {"imsi":"410010006449996","phone_number":"923008084142"} 1 1712599689456 e612d0f8-df45-4ba8-9989-8f635155f109
|
||
Tue Apr 9 00:00:07 2024, FATAL, maat.rule(148), [maat_update_cb:166] table<TSG_DYN_IPPORT_SUBSCRIBER_MAPPING> update runtime error for rules:1895127712 4 154.80.4.124 42400 42431 923008084142 {"imsi":"410010006449996","phone_number":"923008084142"} 1 1712599689456 e612d0f8-df45-4ba8-9989-8f635155f109 {code}
|
||
|
||
|
||
** !image-2024-04-09-13-33-46-540.png!**duandongmei** commented on *2024-04-17T16:21:34.089+0800*:
|
||
|
||
* 问题现状：
|
||
** TSG在跨版本的配置扫描中，同一个条配置只能获取两个MAAT_UPDATE_STATUS记录(实际跨版本的范围内同一条ID有多次MAAT_UPDATE_STATUS的更新)，此现象导致SAPP配置加载时会出现与本地ID重复的问题。
|
||
* 问题复现：我与与刘畅在信息港复现该问题（测试过程中的monitor的日志见附件[^monitor.20240417.log]）
|
||
** Provisioner ZADD的记录：
|
||
*** !image-2024-04-17-16-09-13-961.png|width=539,height=178!
|
||
** TSG跨版本获取MAAT_UPDATE_STATUS的记录
|
||
*** 初始从404版本进行了一次全量配置扫描，此时12配置状态为1（有效）
|
||
*** 增量开始，TSG ZRANGEBYSCORE日志
|
||
**** 1713333715.891403 [0 192.168.40.137:43086] "ZRANGEBYSCORE" "MAAT_UPDATE_STATUS" "(404" "412" "WITHSCORES"
|
||
***
|
||
{code:java}
|
||
Wed Apr 17 14:01:55 2024, INFO, maat.redis_monitor(302693), ZRANGEBYSCORE MAAT_UPDATE_STATUS (404 412 WITHSCORES: element_str: ADD,TSG_DYN_IPPORT_SUBSCRIBER_MAPPING,12
|
||
Wed Apr 17 14:01:55 2024, INFO, maat.redis_monitor(302693), ZRANGEBYSCORE MAAT_UPDATE_STATUS (404 412 WITHSCORES: element_score: 406
|
||
|
||
Wed Apr 17 14:01:55 2024, INFO, maat.redis_monitor(302693), ZRANGEBYSCORE MAAT_UPDATE_STATUS (404 412 WITHSCORES: element_str: DEL,TSG_DYN_IPPORT_SUBSCRIBER_MAPPING,12
|
||
Wed Apr 17 14:01:55 2024, INFO, maat.redis_monitor(302693), ZRANGEBYSCORE MAAT_UPDATE_STATUS (404 412 WITHSCORES: element_score: 407{code}
|
||
|
||
** 问题复现情况说明
|
||
*** 当MAAT_VERSION为404时，TSG功能端全量配置中12配置为生效
|
||
*** 接着功能端增量扫描(404,412]更新状态时，12配置扫描结果为406 ADD,407DEL
|
||
*** 此时功能端进行12配置的ADD操作，发现与本地的12生效配置重复
|
||
*** 到此复现反馈的ID重复问题
|
||
|
||
* 问题分析：
|
||
*
|
||
** 通过分析Redis monitor未发现ZADD命令遗漏更新SCORE的情况，进一步分析Redis的ZADD和ZRANGEBYSCORE执行逻辑
|
||
** MAAT_UPDATE_STATUS操作为ZADD命令，Redis ZADD命令逻辑为不存在成员SCORE执行新增，已存在的成员会更新SCORE。
|
||
** Redis ZADD命令的执行逻辑可以保证当前版本内的配置状态正确执行，可不做调整。
|
||
|
||
* 建议：
|
||
*
|
||
** 建议功能端调整此问题日志级别
|
||
|
||
|
||
|
||
---
|
||
|
||
**liuxueli** commented on *2024-05-10T16:10:42.370+0800*:
|
||
|
||
* 2024/04/28 已Hoxfix修复，参见： https://docs.geedge.net/pages/viewpage.action?pageId=129092431
|
||
|
||
|
||
|
||
---
|
||
|
||
|
||
|
||
## Attachments
|
||
|
||
**54646/firewall.sd.maat.status.txt**
|
||
|
||
---
|
||
|
||
**54642/image-2024-04-09-13-33-46-540.png**
|
||
|
||
---
|
||
|
||
**55104/image-2024-04-12-16-57-10-244.png**
|
||
|
||
---
|
||
|
||
**55446/image-2024-04-17-16-09-13-961.png**
|
||
|
||
---
|
||
|
||
**55451/monitor.20240417.log**
|
||
|
||
---
|
||
|