admin/geedge-jira
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
6a8bfef5778fc7ddd06fa05776bf1ecd0f61d4c4
geedge-jira/md/OMPUB-573.md

183 lines
13 KiB
Markdown
Raw Normal View History

first
2025-09-14 21:52:36 +00:00
# 江苏项目：功能端开启gtp_signaling_plug插件后，sapp异常重启
| ID | Creation Date | Assignee | Status |
|----|----------------|----------|--------|
| OMPUB-573 | 2022-08-03T10:19:46.000+0800 | 刘学利 | 已关闭 |
---
开启gtp_signaling_plug插件后，sapp出现大量丢包，16s后就会重启
**liuxueli** commented on *2022-08-03T11:37:03.311+0800*:
* GTP-C日志量：1.5万条/s
**
{code:java}
[admin@localhost sapp]$ cat tsg_stat.log 
============================================================Wed Aug  3 11:35:22 2022============================================================
     tcp_links     udp_links        bypass      hit_addr     hit_share     intercept     exclusion      D_result    
sum             0             0             0             0             0             0             0             0    
speed/s           454         39626             0          8060             0             0             0             0    
      Q_result      U_result      B_result    inject_succuess    inject_failed    mirror_pkt_suc    mirror_byte_suc    mirror_pkt_fai    
sum             0             0             0             0             0             0             0             0    
speed/s             0             0             0             0             0             0             0             0    
    mirror_byte_fai    set_timeout_suc    set_timeout_fai    tamper_sucess    tamper_nopload    tamper_noswap    abort_allow    abort_deny    
sum             0             0             0             0             0             0             0             0    
speed/s             0           204             0             0             0             0             0             0    
    abort_monitor    abort_intercept    abort_unknown    create_log_cnt    dup_log_cnt    append_log_cnt    free_log_cnt    free_rapid_size    
sum             0             0             0             0             0             0             0             0    
speed/s             0             0             0         46819             0       1452522         38116     177538908    
    free_rapid_capacity    
sum             0    
speed/s     309056704    
________________________________________________________________________________________________________________________________________________
            T_success_log    T_fail_log    T_drop_log    success_log/s    fail_log/s    drop_log/s
SUM                         188497             0             0      3.77e+04      0.00e+00      0.00e+00    
SECURITY-EVENT               53205             0             0      1.07e+04      0.00e+00      0.00e+00    
SESSION-RECORD                 479             0             0      9.60e+01      0.00e+00      0.00e+00    
INTERNAL-RTP-RECORD           8060             0             0      1.62e+03      0.00e+00      0.00e+00    
VOIP-RECORD                      4             0             0      8.00e-01      0.00e+00      0.00e+00    
INTERIM-SESSION-RECORD             0             0             0      0.00e+00      0.00e+00      0.00e+00    
TRANSACTION-RECORD           54041             0             0      1.08e+04      0.00e+00      0.00e+00    
GTPC-RECORD                  75307             0             0      1.51e+04      0.00e+00      0.00e+00    
________________________________________________________________________________________________________________________________________________ {code}
---
**liuxueli** commented on *2022-08-03T11:42:45.299+0800*:
* 根据栈现场：MESA_htable锁冲突严重，引发sapp watchdog timeout导致重启
**
{code:java}
  71   LWP 84656         0x00007ffff5783658 in nanosleep () from /lib64/libc.so.6
  72   LWP 84657         0x00007ffff79a8170 in nanosleep () from /lib64/libpthread.so.0
  73   LWP 84658         0x00007ffff79a8170 in nanosleep () from /lib64/libpthread.so.0
  74   LWP 84659         0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
  75   LWP 84660         0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
  76   LWP 84661         0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
  77   LWP 84662         0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
  78   LWP 84663         0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
  79   LWP 84664         0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
  80   LWP 84665         0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
  81   LWP 84666         0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
  82   LWP 84667         0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
  83   LWP 84668         0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
  84   LWP 84669         0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
--Type <RET> for more, q to quit, c to continue without paging--
  85   LWP 84670         0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
  86   LWP 84671         0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
  87   LWP 84672         0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
  88   LWP 84673         0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
  89   LWP 84674         0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
  90   LWP 84675         0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
  91   LWP 84676         0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
  92   LWP 84677         0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
  93   LWP 84678         0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
  94   LWP 84679         0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
  95   LWP 84680         0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
  96   LWP 84681         0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
  97   LWP 84682         0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
  98   LWP 84683         0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
  99   LWP 84684         0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
  100  LWP 84685         0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
  101  LWP 84686         0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
  102  LWP 84687         0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
  103  LWP 84688         0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
  104  LWP 84689         0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
  105  LWP 84690         0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
  106  LWP 84691         0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
  107  LWP 84692         0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
  108  LWP 84693         0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
  109  LWP 84694         0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
  110  LWP 84696         0x00007ffff5783658 in nanosleep () from /lib64/libc.so.6
(gdb) 
(gdb) 
(gdb) 
(gdb) 
(gdb) thr 90
[Switching to thread 90 (LWP 84675)]
#0  0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
(gdb) bt
#0  0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007ffff79a0b94 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2  0x00007ffff6d00dc6 in MESA_htable_search_cb () from /opt/tsg/framework/lib/libMESA_htable.so
#3  0x00007ff970ff57f6 in GTP_SIGNALING_PLUG_ENTRY () from ./plug/business/gtp_signaling_plug/gtp_signaling_plug.so
#4  0x00000000004478d6 in plugin_call_appentry ()
#5  0x0000000000447d81 in plugin_process_data ()
#6  0x0000000000447f36 in PROT_PROCESS ()
#7  0x00007ffbe819ffa0 in gtp_callback_business_plug(streaminfo*, gtp_c_session*, gtp_c_header*, IE_unit*, int, int, void*) () from ./plug/protocol/gtp/gtp.so
#8  0x00007ffbe81a1710 in dissect_gtpv2_control_packet(streaminfo*, void**, int, void*) () from ./plug/protocol/gtp/gtp.so
#9  0x00007ffbe81a1cb1 in GTP_C_UDP_ENTRY () from ./plug/protocol/gtp/gtp.so
#10 0x000000000042f2d7 in plugin_call_streamentry ()
#11 0x000000000042f950 in stream_process ()
#12 0x000000000042fd31 in stream_process_udp ()
#13 0x000000000043c245 in dealipv4udppkt ()
#14 0x0000000000430bb5 in ipv4_entry ()
#15 0x000000000043b383 in eth_entry ()
#16 0x0000000000428207 in mesa_default_pkt_cb ()
#17 0x00000000004a51e1 in marsio4_process_packet ()
#18 0x00000000004a5bd0 in marsio4_worker ()
#19 0x00007ffff799e1cf in start_thread () from /lib64/libpthread.so.0
#20 0x00007ffff56c2dd3 in clone () from /lib64/libc.so.6
(gdb) thr 89
[Switching to thread 89 (LWP 84674)]
#0  0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
(gdb) bt
#0  0x00007ffff79a781d in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007ffff79a0b94 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2  0x00007ffff6d00dc6 in MESA_htable_search_cb () from /opt/tsg/framework/lib/libMESA_htable.so
#3  0x00007ff970ff57f6 in GTP_SIGNALING_PLUG_ENTRY () from ./plug/business/gtp_signaling_plug/gtp_signaling_plug.so
#4  0x00000000004478d6 in plugin_call_appentry ()
#5  0x0000000000447ba9 in plugin_process_pending ()
#6  0x0000000000447fdc in PROT_PROCESS ()
#7  0x00007ffbe819ffa0 in gtp_callback_business_plug(streaminfo*, gtp_c_session*, gtp_c_header*, IE_unit*, int, int, void*) () from ./plug/protocol/gtp/gtp.so
#8  0x00007ffbe81a1710 in dissect_gtpv2_control_packet(streaminfo*, void**, int, void*) () from ./plug/protocol/gtp/gtp.so
#9  0x00007ffbe81a1cb1 in GTP_C_UDP_ENTRY () from ./plug/protocol/gtp/gtp.so
#10 0x000000000042f2d7 in plugin_call_streamentry ()
#11 0x000000000042f70e in stream_process ()
#12 0x000000000042fd31 in stream_process_udp ()
#13 0x000000000043c245 in dealipv4udppkt ()
#14 0x0000000000430bb5 in ipv4_entry ()
#15 0x000000000043b383 in eth_entry ()
#16 0x0000000000428207 in mesa_default_pkt_cb ()
#17 0x00000000004a51e1 in marsio4_process_packet ()
#18 0x00000000004a5bd0 in marsio4_worker ()
#19 0x00007ffff799e1cf in start_thread () from /lib64/libpthread.so.0
#20 0x00007ffff56c2dd3 in clone () from /lib64/libc.so.6
(gdb) {code}
---
**gitlab** commented on *2022-08-04T16:41:22.300+0800*:
[刘学利|https://git.mesalab.cn/liuxueli] mentioned this issue in [a commit|https://git.mesalab.cn/tango/tsg_master/-/commit/cb93a78dcdf3809313a27823cd6532960fba0a02] of [TSG Appliance / tsg_master|https://git.mesalab.cn/tango/tsg_master] on branch [bugifx-OMPUB-573|https://git.mesalab.cn/tango/tsg_master/-/tree/bugifx-OMPUB-573]:{quote}OMPUB-573: 使用gtp_c_key2index函数替换TEID的HASH算法{quote}
---
**gitlab** commented on *2022-08-30T18:26:32.855+0800*:
[刘学利|https://git.mesalab.cn/liuxueli] mentioned this issue in [a commit|https://git.mesalab.cn/tango/tsg_master/-/commit/0fcf2b3dff4582ef6e2c47e048dc64edc3aee2de] of [TSG Appliance / tsg_master|https://git.mesalab.cn/tango/tsg_master] on branch [bugifx-OMPUB-573|https://git.mesalab.cn/tango/tsg_master/-/tree/bugifx-OMPUB-573]:{quote}OMPUB-573: 使用gtp_c_key2index函数替换TEID的HASH算法{quote}
---
**gitlab** commented on *2022-08-30T18:27:36.204+0800*:
[刘学利|https://git.mesalab.cn/liuxueli] mentioned this issue in [a merge request|https://git.mesalab.cn/tango/tsg_master/-/merge_requests/250] of [TSG Appliance / tsg_master|https://git.mesalab.cn/tango/tsg_master] on branch [bugifx-OMPUB-573|https://git.mesalab.cn/tango/tsg_master/-/tree/bugifx-OMPUB-573]:{quote}OMPUB-573: 使用gtp_c_key2index函数替换TEID的HASH算法{quote}
---
## Attachments
Reference in New Issue Copy Permalink