12 KiB
P-POC现场APP第三方识别引擎库(libqmbundle)出现segmentfault导致SAPP重启
| ID | Creation Date | Assignee | Status |
|---|---|---|---|
| OMPUB-1038 | 2023-10-17T18:59:36.000+0800 | 冯伟浩 | 已解决 |
- P-POC现场第三方引擎库(libqmbundle)出现segmentfault导致SAPP重启 ** coredump-tool ls
{code:java} | 76863 | /opt/tsg/mrzcpd/icelake-server/bin/mrzcpd | NULL | 2023-10-11 22:35:03 | | 940872 | /opt/tsg/mrzcpd/icelake-server/bin/mrzcpd | NULL | 2023-10-12 19:44:05 | | 390967 | /opt/tsg/sapp/sapp | registry.gdnt-cloud.website/tsg-firewall:v23.07.18-591aed7 | 2023-10-17 15:01:29 | +---------+--------------------------------------------+------------------------------------------------------------+---------------------+ {code}
** dmesg结果显示为segmentfault
{code:java} [258580.919257] x86/split lock detection: #AC: pkt:worker-14/13954 took a split_lock trap at address: 0x7f6cb4a3dd2b [412312.924760] sapp_marsio_32[395924]: segfault at 0 ip 00007fa06c0f2f42 sp 00007f98fd3fc2a8 error 4 in libc-2.28.so[7fa06c0bc000+1bb000] [412312.924776] Code: 4c 8d 0c 16 4c 39 cf 0f 82 63 01 00 00 48 89 d1 f3 a4 c3 80 fa 08 73 12 80 fa 04 73 1e 80 fa 01 77 26 72 05 0f b6 0e 88 0f c3 <48> 8b 4c 16 f8 48 8b 36 48 89 4c 17 f8 48 89 37 c3 8b 4c 16 fc 8b [414158.213746] cnibr0: port 21(veth7b4bfe57) entered blocking state [414158.213751] cnibr0: port 21(veth7b4bfe57) entered disabled state {code}
** gdb coredump 显示LWP=239
{code:java} [New LWP 191] [New LWP 175] Failed to read a valid object file image from memory. Core was generated by `/opt/tsg/sapp/sapp'. #0 0x00007fa06c0f2f42 in ?? () [Current thread is 1 (LWP 239)] (gdb) q [root@node-debug-43b48915-9699-4647-96d3-189287b1af40 /]# exit {code}
** sapp_process_latency文件记录现场([^sapp_process_latency.log.2023-10-17.txt])，TID=239
{code:java} [root@nfs-tsgx02 ~]# cat /var/log/traffic-engine/traffic-engine-tsg-traffic-engine-vsys-1/sapp/sapp_process_latency.log.2023-10-17 Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 86, TID:239 recv signal SIGUSR2, backtrace start----------------Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 89, TID:239 backtrace:/opt/tsg/sapp/sapp() [0x4687d7] Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 89, TID:239 backtrace:/lib64/libc.so.6(+0x4eb50) [0x7fa06c10ab50] Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 89, TID:239 backtrace:/opt/tsg/framework/lib/libbreakpad_mini.so(+0x3f8d) [0x7fa06d76ef8d] Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 89, TID:239 backtrace:/opt/tsg/framework/lib/libbreakpad_mini.so(+0x4368) [0x7fa06d76f368] Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 89, TID:239 backtrace:/lib64/libpthread.so.0(+0x12cf0) [0x7fa06d211cf0] Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 89, TID:239 backtrace:/lib64/libc.so.6(+0x36f42) [0x7fa06c0f2f42] Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 89, TID:239 backtrace:./plug/platform/app_proto_engine/libqmbundle.so.tmp.ukKqUX(+0x1c92e8) [0x7f9ab9fdd2e8] Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 89, TID:239 backtrace:./plug/platform/app_proto_engine/libqmbundle.so.tmp.ukKqUX(+0x6b0f11) [0x7f9aba4c4f11] Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 89, TID:239 backtrace:./plug/platform/app_proto_engine/libqmbundle.so.tmp.ukKqUX(+0x6b13f1) [0x7f9aba4c53f1] Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 89, TID:239 backtrace:./plug/platform/app_proto_engine/libqmbundle.so.tmp.ukKqUX(+0x1ab376) [0x7f9ab9fbf376] Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 89, TID:239 backtrace:./plug/platform/app_proto_engine/libqmbundle.so.tmp.ukKqUX(+0x1b1618) [0x7f9ab9fc5618] Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 89, TID:239 backtrace:./plug/platform/app_proto_engine/libqmbundle.so.tmp.ukKqUX(+0x1b3a4d) [0x7f9ab9fc7a4d] Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 89, TID:239 backtrace:./plug/platform/app_proto_engine/libqmbundle.so.tmp.ukKqUX(+0x1b5ff8) [0x7f9ab9fc9ff8] Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 89, TID:239 backtrace:./plug/platform/app_proto_engine/libqmbundle.so.tmp.ukKqUX(+0x1a8b02) [0x7f9ab9fbcb02] Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 89, TID:239 backtrace:./plug/platform/app_proto_engine/libqmbundle.so.tmp.ukKqUX(+0x18f76c) [0x7f9ab9fa376c] Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 89, TID:239 backtrace:./plug/platform/app_proto_engine/libqmengine.so(+0x7e7ae) [0x7f9b5b66c7ae] Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 89, TID:239 backtrace:./plug/platform/app_proto_engine/libqmengine.so(qmdpi_worker_process+0xa9) [0x7f9b5b66d0b9] Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 89, TID:239 backtrace:./plug/platform/app_proto_engine/app_proto_engine.so(_Z18app_worker_processP10streaminfoP13app_proto_ctxPciij+0x146) [0x7f9b5c1f29d6] Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 89, TID:239 backtrace:./plug/platform/app_proto_engine/app_proto_engine.so(+0x4511) [0x7f9b5c1f0511] Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 89, TID:239 backtrace:./plug/platform/app_proto_engine/app_proto_engine.so(APP_PROTO_ENGINE_MAIN_UDP_ENTRY+0x7b) [0x7f9b5c1f0f7b] Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 89, TID:239 backtrace:/opt/tsg/sapp/sapp(plugin_call_streamentry+0x67) [0x4354e7] Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 89, TID:239 backtrace:/opt/tsg/sapp/sapp(call_streamentry+0x55) [0x435915] Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 89, TID:239 backtrace:/opt/tsg/sapp/sapp(stream_process+0x53a) [0x4360da] Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 89, TID:239 backtrace:/opt/tsg/sapp/sapp(stream_process_udp+0x57) [0x436567] Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 89, TID:239 backtrace:/opt/tsg/sapp/sapp(dealipv4udppkt+0x377) [0x444547] Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 89, TID:239 backtrace:/opt/tsg/sapp/sapp(ipv4_entry+0x2ab) [0x4374cb] Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 89, TID:239 backtrace:/opt/tsg/sapp/sapp(eth_entry+0x344) [0x443194] Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 89, TID:239 backtrace:/opt/tsg/sapp/sapp() [0x42e9f2] Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 89, TID:239 backtrace:/opt/tsg/sapp/sapp() [0x507a27] Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 89, TID:239 backtrace:/opt/tsg/sapp/sapp() [0x508132] Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 89, TID:239 backtrace:/lib64/libpthread.so.0(+0x81ca) [0x7fa06d2071ca] Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 89, TID:239 backtrace:/lib64/libc.so.6(clone+0x43) [0x7fa06c0f5e73] Tue Oct 17 15:01:29 2023, FATAL, process_latency , file /tmp/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX_PREFIX/MESA_Platform/sapp/src/sapp_dev/sapp_plug.c, line 91, TID:239 recv signal SIGUSR2, backtrace end---------------- {code}fengweihao commented on 2023-11-01T19:00:42.787+0800:
初步怀疑：
- APP_sketch_db的升级包中的PB版本存在问题
更新记录：
- 怀疑此次更新
!image-2023-11-01-18-55-44-864.png|width=307,height=105!
App_sketch_db_23.08更新的PB为1.670-0-21，App_sketch_db_23.09版的PB没有更新
解决方案：
- 请使用最新App_sketch_db_23.10升级包进行更细，PB版本为1.680.0.20
fengweihao commented on 2023-11-16T15:26:52.919+0800:
- 升级APP Sketch DB 23.10.680.2， 已运行一星期
Attachments
46652/image-2023-11-01-18-55-44-864.png
46047/sapp_process_latency.log.2023-10-17.txt