120 lines
12 KiB
Markdown
120 lines
12 KiB
Markdown
# P-POC现场APP第三方识别引擎库(libqmbundle)出现segmentfault导致SAPP重启
|
||
|
||
| ID | Creation Date | Assignee | Status |
|
||
|----|----------------|----------|--------|
|
||
| OMPUB-1038 | 2023-10-17T18:59:36.000+0800 | 冯伟浩 | 已解决 |
|
||
|
||
|
||
---
|
||
|
||
* P-POC现场第三方引擎库(libqmbundle)出现segmentfault导致SAPP重启
|
||
** coredump-tool ls
|
||
***
|
||
{code:java}
|
||
| 76863 | /opt/tsg/mrzcpd/icelake-server/bin/mrzcpd | NULL | 2023-10-11 22:35:03 |
|
||
| 940872 | /opt/tsg/mrzcpd/icelake-server/bin/mrzcpd | NULL | 2023-10-12 19:44:05 |
|
||
| 390967 | /opt/tsg/sapp/sapp | registry.gdnt-cloud.website/tsg-firewall:v23.07.18-591aed7 | 2023-10-17 15:01:29 |
|
||
+---------+--------------------------------------------+------------------------------------------------------------+---------------------+ {code}
|
||
|
||
*
|
||
** dmesg结果显示为segmentfault
|
||
***
|
||
{code:java}
|
||
[258580.919257] x86/split lock detection: #AC: pkt:worker-14/13954 took a split_lock trap at address: 0x7f6cb4a3dd2b
|
||
[412312.924760] sapp_marsio_32[395924]: segfault at 0 ip 00007fa06c0f2f42 sp 00007f98fd3fc2a8 error 4 in libc-2.28.so[7fa06c0bc000+1bb000]
|
||
[412312.924776] Code: 4c 8d 0c 16 4c 39 cf 0f 82 63 01 00 00 48 89 d1 f3 a4 c3 80 fa 08 73 12 80 fa 04 73 1e 80 fa 01 77 26 72 05 0f b6 0e 88 0f c3 <48> 8b 4c 16 f8 48 8b 36 48 89 4c 17 f8 48 89 37 c3 8b 4c 16 fc 8b
|
||
[414158.213746] cnibr0: port 21(veth7b4bfe57) entered blocking state
|
||
[414158.213751] cnibr0: port 21(veth7b4bfe57) entered disabled state {code}
|
||
|
||
*
|
||
** gdb coredump 显示LWP=239
|
||
***
|
||
{code:java}
|
||
[New LWP 191]
|
||
[New LWP 175]
|
||
Failed to read a valid object file image from memory.
|
||
Core was generated by `/opt/tsg/sapp/sapp'.
|
||
#0 0x00007fa06c0f2f42 in ?? ()
|
||
[Current thread is 1 (LWP 239)]
|
||
(gdb) q
|
||
[root@node-debug-43b48915-9699-4647-96d3-189287b1af40 /]# exit
|
||
{code}
|
||
|
||
*
|
||
** sapp_process_latency文件记录现场([^sapp_process_latency.log.2023-10-17.txt])，TID=239
|
||
***
|
||
{code:java}
|
||
[root@nfs-tsgx02 ~]# cat /var/log/traffic-engine/traffic-engine-tsg-traffic-engine-vsys-1/sapp/sapp_process_latency.log.2023-10-17
|
||
Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 86, TID:239 recv signal SIGUSR2, backtrace start----------------Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 89, TID:239 backtrace:/opt/tsg/sapp/sapp() [0x4687d7]
|
||
Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 89, TID:239 backtrace:/lib64/libc.so.6(+0x4eb50) [0x7fa06c10ab50]
|
||
Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 89, TID:239 backtrace:/opt/tsg/framework/lib/libbreakpad_mini.so(+0x3f8d) [0x7fa06d76ef8d]
|
||
Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 89, TID:239 backtrace:/opt/tsg/framework/lib/libbreakpad_mini.so(+0x4368) [0x7fa06d76f368]
|
||
Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 89, TID:239 backtrace:/lib64/libpthread.so.0(+0x12cf0) [0x7fa06d211cf0]
|
||
Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 89, TID:239 backtrace:/lib64/libc.so.6(+0x36f42) [0x7fa06c0f2f42]
|
||
Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 89, TID:239 backtrace:./plug/platform/app_proto_engine/libqmbundle.so.tmp.ukKqUX(+0x1c92e8) [0x7f9ab9fdd2e8]
|
||
Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 89, TID:239 backtrace:./plug/platform/app_proto_engine/libqmbundle.so.tmp.ukKqUX(+0x6b0f11) [0x7f9aba4c4f11]
|
||
Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 89, TID:239 backtrace:./plug/platform/app_proto_engine/libqmbundle.so.tmp.ukKqUX(+0x6b13f1) [0x7f9aba4c53f1]
|
||
Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 89, TID:239 backtrace:./plug/platform/app_proto_engine/libqmbundle.so.tmp.ukKqUX(+0x1ab376) [0x7f9ab9fbf376]
|
||
Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 89, TID:239 backtrace:./plug/platform/app_proto_engine/libqmbundle.so.tmp.ukKqUX(+0x1b1618) [0x7f9ab9fc5618]
|
||
Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 89, TID:239 backtrace:./plug/platform/app_proto_engine/libqmbundle.so.tmp.ukKqUX(+0x1b3a4d) [0x7f9ab9fc7a4d]
|
||
Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 89, TID:239 backtrace:./plug/platform/app_proto_engine/libqmbundle.so.tmp.ukKqUX(+0x1b5ff8) [0x7f9ab9fc9ff8]
|
||
Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 89, TID:239 backtrace:./plug/platform/app_proto_engine/libqmbundle.so.tmp.ukKqUX(+0x1a8b02) [0x7f9ab9fbcb02]
|
||
Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 89, TID:239 backtrace:./plug/platform/app_proto_engine/libqmbundle.so.tmp.ukKqUX(+0x18f76c) [0x7f9ab9fa376c]
|
||
Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 89, TID:239 backtrace:./plug/platform/app_proto_engine/libqmengine.so(+0x7e7ae) [0x7f9b5b66c7ae]
|
||
Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 89, TID:239 backtrace:./plug/platform/app_proto_engine/libqmengine.so(qmdpi_worker_process+0xa9) [0x7f9b5b66d0b9]
|
||
Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 89, TID:239 backtrace:./plug/platform/app_proto_engine/app_proto_engine.so(_Z18app_worker_processP10streaminfoP13app_proto_ctxPciij+0x146) [0x7f9b5c1f29d6]
|
||
Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 89, TID:239 backtrace:./plug/platform/app_proto_engine/app_proto_engine.so(+0x4511) [0x7f9b5c1f0511]
|
||
Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 89, TID:239 backtrace:./plug/platform/app_proto_engine/app_proto_engine.so(APP_PROTO_ENGINE_MAIN_UDP_ENTRY+0x7b) [0x7f9b5c1f0f7b]
|
||
Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 89, TID:239 backtrace:/opt/tsg/sapp/sapp(plugin_call_streamentry+0x67) [0x4354e7]
|
||
Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 89, TID:239 backtrace:/opt/tsg/sapp/sapp(call_streamentry+0x55) [0x435915]
|
||
Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 89, TID:239 backtrace:/opt/tsg/sapp/sapp(stream_process+0x53a) [0x4360da]
|
||
Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 89, TID:239 backtrace:/opt/tsg/sapp/sapp(stream_process_udp+0x57) [0x436567]
|
||
Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 89, TID:239 backtrace:/opt/tsg/sapp/sapp(dealipv4udppkt+0x377) [0x444547]
|
||
Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 89, TID:239 backtrace:/opt/tsg/sapp/sapp(ipv4_entry+0x2ab) [0x4374cb]
|
||
Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 89, TID:239 backtrace:/opt/tsg/sapp/sapp(eth_entry+0x344) [0x443194]
|
||
Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 89, TID:239 backtrace:/opt/tsg/sapp/sapp() [0x42e9f2]
|
||
Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 89, TID:239 backtrace:/opt/tsg/sapp/sapp() [0x507a27]
|
||
Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 89, TID:239 backtrace:/opt/tsg/sapp/sapp() [0x508132]
|
||
Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 89, TID:239 backtrace:/lib64/libpthread.so.0(+0x81ca) [0x7fa06d2071ca]
|
||
Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 89, TID:239 backtrace:/lib64/libc.so.6(clone+0x43) [0x7fa06c0f5e73]
|
||
Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 91, TID:239 recv signal SIGUSR2, backtrace end----------------
|
||
{code}**fengweihao** commented on *2023-11-01T19:00:42.787+0800*:
|
||
|
||
初步怀疑：
|
||
* APP_sketch_db的升级包中的PB版本存在问题
|
||
|
||
更新记录：
|
||
* 怀疑此次更新
|
||
|
||
!image-2023-11-01-18-55-44-864.png|width=307,height=105!
|
||
|
||
App_sketch_db_23.08更新的PB为1.670-0-21，App_sketch_db_23.09版的PB没有更新
|
||
|
||
解决方案：
|
||
* 请使用最新App_sketch_db_23.10升级包进行更细，PB版本为1.680.0.20
|
||
|
||
|
||
|
||
---
|
||
|
||
**fengweihao** commented on *2023-11-16T15:26:52.919+0800*:
|
||
|
||
* 升级APP Sketch DB 23.10.680.2， 已运行一星期
|
||
|
||
|
||
|
||
---
|
||
|
||
|
||
|
||
## Attachments
|
||
|
||
**46652/image-2023-11-01-18-55-44-864.png**
|
||
|
||
---
|
||
|
||
**46047/sapp_process_latency.log.2023-10-17.txt**
|
||
|
||
---
|
||
|