13 KiB
江苏项目：功能端开启gtp_signaling_plug插件后，sapp异常重启
| ID | Creation Date | Assignee | Status |
|---|---|---|---|
| OMPUB-573 | 2022-08-03T10:19:46.000+0800 | 刘学利 | 已关闭 |
开启gtp_signaling_plug插件后，sapp出现大量丢包，16s后就会重启 liuxueli commented on 2022-08-03T11:37:03.311+0800:
- GTP-C日志量：1.5万条/s ** {code:java} [admin@localhost sapp]$ cat tsg_stat.log ============================================================Wed Aug 3 11:35:22 2022============================================================ tcp_links udp_links bypass hit_addr hit_share intercept exclusion D_result sum 0 0 0 0 0 0 0 0 speed/s 454 39626 0 8060 0 0 0 0 Q_result U_result B_result inject_succuess inject_failed mirror_pkt_suc mirror_byte_suc mirror_pkt_fai sum 0 0 0 0 0 0 0 0 speed/s 0 0 0 0 0 0 0 0 mirror_byte_fai set_timeout_suc set_timeout_fai tamper_sucess tamper_nopload tamper_noswap abort_allow abort_deny sum 0 0 0 0 0 0 0 0 speed/s 0 204 0 0 0 0 0 0 abort_monitor abort_intercept abort_unknown create_log_cnt dup_log_cnt append_log_cnt free_log_cnt free_rapid_size sum 0 0 0 0 0 0 0 0 speed/s 0 0 0 46819 0 1452522 38116 177538908 free_rapid_capacity sum 0 speed/s 309056704
T_success_log T_fail_log T_drop_log success_log/s fail_log/s drop_log/s SUM 188497 0 0 3.77e+04 0.00e+00 0.00e+00 SECURITY-EVENT 53205 0 0 1.07e+04 0.00e+00 0.00e+00 SESSION-RECORD 479 0 0 9.60e+01 0.00e+00 0.00e+00 INTERNAL-RTP-RECORD 8060 0 0 1.62e+03 0.00e+00 0.00e+00 VOIP-RECORD 4 0 0 8.00e-01 0.00e+00 0.00e+00 INTERIM-SESSION-RECORD 0 0 0 0.00e+00 0.00e+00 0.00e+00 TRANSACTION-RECORD 54041 0 0 1.08e+04 0.00e+00 0.00e+00 GTPC-RECORD 75307 0 0 1.51e+04 0.00e+00 0.00e+00 ________________________________________________________________________________________________________________________________________________ {code}
liuxueli commented on 2022-08-03T11:42:45.299+0800:
- 根据栈现场：MESA_htable锁冲突严重，引发sapp watchdog timeout导致重启 ** {code:java} 71 LWP 84656 0x00007ffff5783658 in nanosleep () from /lib64/libc.so.6 72 LWP 84657 0x00007ffff79a8170 in nanosleep () from /lib64/libpthread.so.0 73 LWP 84658 0x00007ffff79a8170 in nanosleep () from /lib64/libpthread.so.0 74 LWP 84659 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0 75 LWP 84660 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0 76 LWP 84661 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0 77 LWP 84662 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0 78 LWP 84663 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0 79 LWP 84664 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0 80 LWP 84665 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0 81 LWP 84666 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0 82 LWP 84667 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0 83 LWP 84668 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0 84 LWP 84669 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0 --Type for more, q to quit, c to continue without paging-- 85 LWP 84670 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0 86 LWP 84671 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0 87 LWP 84672 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0 88 LWP 84673 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0 89 LWP 84674 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0 90 LWP 84675 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0 91 LWP 84676 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0 92 LWP 84677 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0 93 LWP 84678 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0 94 LWP 84679 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0 95 LWP 84680 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0 96 LWP 84681 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0 97 LWP 84682 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0 98 LWP 84683 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0 99 LWP 84684 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0 100 LWP 84685 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0 101 LWP 84686 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0 102 LWP 84687 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0 103 LWP 84688 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0 104 LWP 84689 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0 105 LWP 84690 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0 106 LWP 84691 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0 107 LWP 84692 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0 108 LWP 84693 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0 109 LWP 84694 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0 110 LWP 84696 0x00007ffff5783658 in nanosleep () from /lib64/libc.so.6 (gdb) (gdb) (gdb) (gdb) (gdb) thr 90 [Switching to thread 90 (LWP 84675)] #0 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0 (gdb) bt #0 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007ffff79a0b94 in pthread_mutex_lock () from /lib64/libpthread.so.0 #2 0x00007ffff6d00dc6 in MESA_htable_search_cb () from /opt/tsg/framework/lib/libMESA_htable.so #3 0x00007ff970ff57f6 in GTP_SIGNALING_PLUG_ENTRY () from ./plug/business/gtp_signaling_plug/gtp_signaling_plug.so #4 0x00000000004478d6 in plugin_call_appentry () #5 0x0000000000447d81 in plugin_process_data () #6 0x0000000000447f36 in PROT_PROCESS () #7 0x00007ffbe819ffa0 in gtp_callback_business_plug(streaminfo*, gtp_c_session*, gtp_c_header*, IE_unit*, int, int, void*) () from ./plug/protocol/gtp/gtp.so #8 0x00007ffbe81a1710 in dissect_gtpv2_control_packet(streaminfo*, void**, int, void*) () from ./plug/protocol/gtp/gtp.so #9 0x00007ffbe81a1cb1 in GTP_C_UDP_ENTRY () from ./plug/protocol/gtp/gtp.so #10 0x000000000042f2d7 in plugin_call_streamentry () #11 0x000000000042f950 in stream_process () #12 0x000000000042fd31 in stream_process_udp () #13 0x000000000043c245 in dealipv4udppkt () #14 0x0000000000430bb5 in ipv4_entry () #15 0x000000000043b383 in eth_entry () #16 0x0000000000428207 in mesa_default_pkt_cb () #17 0x00000000004a51e1 in marsio4_process_packet () #18 0x00000000004a5bd0 in marsio4_worker () #19 0x00007ffff799e1cf in start_thread () from /lib64/libpthread.so.0 #20 0x00007ffff56c2dd3 in clone () from /lib64/libc.so.6 (gdb) thr 89 [Switching to thread 89 (LWP 84674)] #0 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0 (gdb) bt #0 0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007ffff79a0b94 in pthread_mutex_lock () from /lib64/libpthread.so.0 #2 0x00007ffff6d00dc6 in MESA_htable_search_cb () from /opt/tsg/framework/lib/libMESA_htable.so #3 0x00007ff970ff57f6 in GTP_SIGNALING_PLUG_ENTRY () from ./plug/business/gtp_signaling_plug/gtp_signaling_plug.so #4 0x00000000004478d6 in plugin_call_appentry () #5 0x0000000000447ba9 in plugin_process_pending () #6 0x0000000000447fdc in PROT_PROCESS () #7 0x00007ffbe819ffa0 in gtp_callback_business_plug(streaminfo*, gtp_c_session*, gtp_c_header*, IE_unit*, int, int, void*) () from ./plug/protocol/gtp/gtp.so #8 0x00007ffbe81a1710 in dissect_gtpv2_control_packet(streaminfo*, void**, int, void*) () from ./plug/protocol/gtp/gtp.so #9 0x00007ffbe81a1cb1 in GTP_C_UDP_ENTRY () from ./plug/protocol/gtp/gtp.so #10 0x000000000042f2d7 in plugin_call_streamentry () #11 0x000000000042f70e in stream_process () #12 0x000000000042fd31 in stream_process_udp () #13 0x000000000043c245 in dealipv4udppkt () #14 0x0000000000430bb5 in ipv4_entry () #15 0x000000000043b383 in eth_entry () #16 0x0000000000428207 in mesa_default_pkt_cb () #17 0x00000000004a51e1 in marsio4_process_packet () #18 0x00000000004a5bd0 in marsio4_worker () #19 0x00007ffff799e1cf in start_thread () from /lib64/libpthread.so.0 #20 0x00007ffff56c2dd3 in clone () from /lib64/libc.so.6 (gdb) {code}
gitlab commented on 2022-08-04T16:41:22.300+0800:
[刘学利|https://git.mesalab.cn/liuxueli] mentioned this issue in [a commit|cb93a78dcd] of [TSG Appliance / tsg_master|https://git.mesalab.cn/tango/tsg_master] on branch [bugifx-OMPUB-573|https://git.mesalab.cn/tango/tsg_master/-/tree/bugifx-OMPUB-573]:{quote}OMPUB-573: 使用gtp_c_key2index函数替换TEID的HASH算法{quote}
gitlab commented on 2022-08-30T18:26:32.855+0800:
[刘学利|https://git.mesalab.cn/liuxueli] mentioned this issue in [a commit|0fcf2b3dff] of [TSG Appliance / tsg_master|https://git.mesalab.cn/tango/tsg_master] on branch [bugifx-OMPUB-573|https://git.mesalab.cn/tango/tsg_master/-/tree/bugifx-OMPUB-573]:{quote}OMPUB-573: 使用gtp_c_key2index函数替换TEID的HASH算法{quote}
gitlab commented on 2022-08-30T18:27:36.204+0800:
[刘学利|https://git.mesalab.cn/liuxueli] mentioned this issue in [a merge request|https://git.mesalab.cn/tango/tsg_master/-/merge_requests/250] of [TSG Appliance / tsg_master|https://git.mesalab.cn/tango/tsg_master] on branch [bugifx-OMPUB-573|https://git.mesalab.cn/tango/tsg_master/-/tree/bugifx-OMPUB-573]:{quote}OMPUB-573: 使用gtp_c_key2index函数替换TEID的HASH算法{quote}