183 lines
13 KiB
Markdown
183 lines
13 KiB
Markdown
# 江苏项目：功能端开启gtp_signaling_plug插件后，sapp异常重启
|
||
|
||
| ID | Creation Date | Assignee | Status |
|
||
|----|----------------|----------|--------|
|
||
| OMPUB-573 | 2022-08-03T10:19:46.000+0800 | 刘学利 | 已关闭 |
|
||
|
||
|
||
---
|
||
|
||
开启gtp_signaling_plug插件后，sapp出现大量丢包，16s后就会重启
|
||
**liuxueli** commented on *2022-08-03T11:37:03.311+0800*:
|
||
|
||
* GTP-C日志量：1.5万条/s
|
||
**
|
||
{code:java}
|
||
[admin@localhost sapp]$ cat tsg_stat.log
|
||
============================================================Wed Aug 3 11:35:22 2022============================================================
|
||
tcp_links udp_links bypass hit_addr hit_share intercept exclusion D_result
|
||
sum 0 0 0 0 0 0 0 0
|
||
speed/s 454 39626 0 8060 0 0 0 0
|
||
Q_result U_result B_result inject_succuess inject_failed mirror_pkt_suc mirror_byte_suc mirror_pkt_fai
|
||
sum 0 0 0 0 0 0 0 0
|
||
speed/s 0 0 0 0 0 0 0 0
|
||
mirror_byte_fai set_timeout_suc set_timeout_fai tamper_sucess tamper_nopload tamper_noswap abort_allow abort_deny
|
||
sum 0 0 0 0 0 0 0 0
|
||
speed/s 0 204 0 0 0 0 0 0
|
||
abort_monitor abort_intercept abort_unknown create_log_cnt dup_log_cnt append_log_cnt free_log_cnt free_rapid_size
|
||
sum 0 0 0 0 0 0 0 0
|
||
speed/s 0 0 0 46819 0 1452522 38116 177538908
|
||
free_rapid_capacity
|
||
sum 0
|
||
speed/s 309056704
|
||
________________________________________________________________________________________________________________________________________________
|
||
T_success_log T_fail_log T_drop_log success_log/s fail_log/s drop_log/s
|
||
SUM 188497 0 0 3.77e+04 0.00e+00 0.00e+00
|
||
SECURITY-EVENT 53205 0 0 1.07e+04 0.00e+00 0.00e+00
|
||
SESSION-RECORD 479 0 0 9.60e+01 0.00e+00 0.00e+00
|
||
INTERNAL-RTP-RECORD 8060 0 0 1.62e+03 0.00e+00 0.00e+00
|
||
VOIP-RECORD 4 0 0 8.00e-01 0.00e+00 0.00e+00
|
||
INTERIM-SESSION-RECORD 0 0 0 0.00e+00 0.00e+00 0.00e+00
|
||
TRANSACTION-RECORD 54041 0 0 1.08e+04 0.00e+00 0.00e+00
|
||
GTPC-RECORD 75307 0 0 1.51e+04 0.00e+00 0.00e+00
|
||
________________________________________________________________________________________________________________________________________________ {code}
|
||
|
||
|
||
|
||
---
|
||
|
||
**liuxueli** commented on *2022-08-03T11:42:45.299+0800*:
|
||
|
||
* 根据栈现场：MESA_htable锁冲突严重，引发sapp watchdog timeout导致重启
|
||
**
|
||
{code:java}
|
||
71 LWP 84656 0x00007ffff5783658 in nanosleep () from /lib64/libc.so.6
|
||
72 LWP 84657 0x00007ffff79a8170 in nanosleep () from /lib64/libpthread.so.0
|
||
73 LWP 84658 0x00007ffff79a8170 in nanosleep () from /lib64/libpthread.so.0
|
||
74 LWP 84659 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
|
||
75 LWP 84660 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
|
||
76 LWP 84661 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
|
||
77 LWP 84662 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
|
||
78 LWP 84663 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
|
||
79 LWP 84664 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
|
||
80 LWP 84665 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
|
||
81 LWP 84666 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
|
||
82 LWP 84667 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
|
||
83 LWP 84668 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
|
||
84 LWP 84669 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
|
||
--Type <RET> for more, q to quit, c to continue without paging--
|
||
85 LWP 84670 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
|
||
86 LWP 84671 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
|
||
87 LWP 84672 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
|
||
88 LWP 84673 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
|
||
89 LWP 84674 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
|
||
90 LWP 84675 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
|
||
91 LWP 84676 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
|
||
92 LWP 84677 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
|
||
93 LWP 84678 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
|
||
94 LWP 84679 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
|
||
95 LWP 84680 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
|
||
96 LWP 84681 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
|
||
97 LWP 84682 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
|
||
98 LWP 84683 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
|
||
99 LWP 84684 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
|
||
100 LWP 84685 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
|
||
101 LWP 84686 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
|
||
102 LWP 84687 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
|
||
103 LWP 84688 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
|
||
104 LWP 84689 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
|
||
105 LWP 84690 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
|
||
106 LWP 84691 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
|
||
107 LWP 84692 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
|
||
108 LWP 84693 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
|
||
109 LWP 84694 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
|
||
110 LWP 84696 0x00007ffff5783658 in nanosleep () from /lib64/libc.so.6
|
||
(gdb)
|
||
(gdb)
|
||
(gdb)
|
||
(gdb)
|
||
(gdb) thr 90
|
||
[Switching to thread 90 (LWP 84675)]
|
||
#0 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
|
||
(gdb) bt
|
||
#0 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
|
||
#1 0x00007ffff79a0b94 in pthread_mutex_lock () from /lib64/libpthread.so.0
|
||
#2 0x00007ffff6d00dc6 in MESA_htable_search_cb () from /opt/tsg/framework/lib/libMESA_htable.so
|
||
#3 0x00007ff970ff57f6 in GTP_SIGNALING_PLUG_ENTRY () from ./plug/business/gtp_signaling_plug/gtp_signaling_plug.so
|
||
#4 0x00000000004478d6 in plugin_call_appentry ()
|
||
#5 0x0000000000447d81 in plugin_process_data ()
|
||
#6 0x0000000000447f36 in PROT_PROCESS ()
|
||
#7 0x00007ffbe819ffa0 in gtp_callback_business_plug(streaminfo*, gtp_c_session*, gtp_c_header*, IE_unit*, int, int, void*) () from ./plug/protocol/gtp/gtp.so
|
||
#8 0x00007ffbe81a1710 in dissect_gtpv2_control_packet(streaminfo*, void**, int, void*) () from ./plug/protocol/gtp/gtp.so
|
||
#9 0x00007ffbe81a1cb1 in GTP_C_UDP_ENTRY () from ./plug/protocol/gtp/gtp.so
|
||
#10 0x000000000042f2d7 in plugin_call_streamentry ()
|
||
#11 0x000000000042f950 in stream_process ()
|
||
#12 0x000000000042fd31 in stream_process_udp ()
|
||
#13 0x000000000043c245 in dealipv4udppkt ()
|
||
#14 0x0000000000430bb5 in ipv4_entry ()
|
||
#15 0x000000000043b383 in eth_entry ()
|
||
#16 0x0000000000428207 in mesa_default_pkt_cb ()
|
||
#17 0x00000000004a51e1 in marsio4_process_packet ()
|
||
#18 0x00000000004a5bd0 in marsio4_worker ()
|
||
#19 0x00007ffff799e1cf in start_thread () from /lib64/libpthread.so.0
|
||
#20 0x00007ffff56c2dd3 in clone () from /lib64/libc.so.6
|
||
(gdb) thr 89
|
||
[Switching to thread 89 (LWP 84674)]
|
||
#0 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
|
||
(gdb) bt
|
||
#0 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
|
||
#1 0x00007ffff79a0b94 in pthread_mutex_lock () from /lib64/libpthread.so.0
|
||
#2 0x00007ffff6d00dc6 in MESA_htable_search_cb () from /opt/tsg/framework/lib/libMESA_htable.so
|
||
#3 0x00007ff970ff57f6 in GTP_SIGNALING_PLUG_ENTRY () from ./plug/business/gtp_signaling_plug/gtp_signaling_plug.so
|
||
#4 0x00000000004478d6 in plugin_call_appentry ()
|
||
#5 0x0000000000447ba9 in plugin_process_pending ()
|
||
#6 0x0000000000447fdc in PROT_PROCESS ()
|
||
#7 0x00007ffbe819ffa0 in gtp_callback_business_plug(streaminfo*, gtp_c_session*, gtp_c_header*, IE_unit*, int, int, void*) () from ./plug/protocol/gtp/gtp.so
|
||
#8 0x00007ffbe81a1710 in dissect_gtpv2_control_packet(streaminfo*, void**, int, void*) () from ./plug/protocol/gtp/gtp.so
|
||
#9 0x00007ffbe81a1cb1 in GTP_C_UDP_ENTRY () from ./plug/protocol/gtp/gtp.so
|
||
#10 0x000000000042f2d7 in plugin_call_streamentry ()
|
||
#11 0x000000000042f70e in stream_process ()
|
||
#12 0x000000000042fd31 in stream_process_udp ()
|
||
#13 0x000000000043c245 in dealipv4udppkt ()
|
||
#14 0x0000000000430bb5 in ipv4_entry ()
|
||
#15 0x000000000043b383 in eth_entry ()
|
||
#16 0x0000000000428207 in mesa_default_pkt_cb ()
|
||
#17 0x00000000004a51e1 in marsio4_process_packet ()
|
||
#18 0x00000000004a5bd0 in marsio4_worker ()
|
||
#19 0x00007ffff799e1cf in start_thread () from /lib64/libpthread.so.0
|
||
#20 0x00007ffff56c2dd3 in clone () from /lib64/libc.so.6
|
||
(gdb) {code}
|
||
|
||
|
||
|
||
---
|
||
|
||
**gitlab** commented on *2022-08-04T16:41:22.300+0800*:
|
||
|
||
[刘学利|https://git.mesalab.cn/liuxueli] mentioned this issue in [a commit|https://git.mesalab.cn/tango/tsg_master/-/commit/cb93a78dcdf3809313a27823cd6532960fba0a02] of [TSG Appliance / tsg_master|https://git.mesalab.cn/tango/tsg_master] on branch [bugifx-OMPUB-573|https://git.mesalab.cn/tango/tsg_master/-/tree/bugifx-OMPUB-573]:{quote}OMPUB-573: 使用gtp_c_key2index函数替换TEID的HASH算法{quote}
|
||
|
||
|
||
|
||
---
|
||
|
||
**gitlab** commented on *2022-08-30T18:26:32.855+0800*:
|
||
|
||
[刘学利|https://git.mesalab.cn/liuxueli] mentioned this issue in [a commit|https://git.mesalab.cn/tango/tsg_master/-/commit/0fcf2b3dff4582ef6e2c47e048dc64edc3aee2de] of [TSG Appliance / tsg_master|https://git.mesalab.cn/tango/tsg_master] on branch [bugifx-OMPUB-573|https://git.mesalab.cn/tango/tsg_master/-/tree/bugifx-OMPUB-573]:{quote}OMPUB-573: 使用gtp_c_key2index函数替换TEID的HASH算法{quote}
|
||
|
||
|
||
|
||
---
|
||
|
||
**gitlab** commented on *2022-08-30T18:27:36.204+0800*:
|
||
|
||
[刘学利|https://git.mesalab.cn/liuxueli] mentioned this issue in [a merge request|https://git.mesalab.cn/tango/tsg_master/-/merge_requests/250] of [TSG Appliance / tsg_master|https://git.mesalab.cn/tango/tsg_master] on branch [bugifx-OMPUB-573|https://git.mesalab.cn/tango/tsg_master/-/tree/bugifx-OMPUB-573]:{quote}OMPUB-573: 使用gtp_c_key2index函数替换TEID的HASH算法{quote}
|
||
|
||
|
||
|
||
---
|
||
|
||
|
||
|
||
## Attachments
|
||
|