284 lines
14 KiB
Markdown
284 lines
14 KiB
Markdown
# 福建：安全日志中的http_request_body下载失败
|
||
|
||
| ID | Creation Date | Assignee | Status |
|
||
|----|----------------|----------|--------|
|
||
| OMPUB-411 | 2022-03-01T09:19:29.000+0800 | 彭宣正 | 已关闭 |
|
||
|
||
|
||
---
|
||
|
||
No description
|
||
|
||
---
|
||
|
||
**liuxueli** commented on *2022-03-01T09:56:41.796+0800*:
|
||
|
||
* [~pengxuanzheng] 描述一下排查过程及结论
|
||
|
||
|
||
|
||
---
|
||
|
||
**pengxuanzheng** commented on *2022-03-01T18:49:25.052+0800*:
|
||
|
||
hos请求任务缓存大于了每个线程允许的最大缓存数，丢弃这条请求，没有发送hos请求。
|
||
|
||
!image-2022-03-01-18-25-30-689.png!
|
||
|
||
但是，在安全日志里面却出现了这条这条URL
|
||
|
||
!image-2022-03-01-18-27-45-315.png!
|
||
|
||
|
||
|
||
{color:#172b4d}*原因*{color}：检查代码发现，在不满足发送hos请求条件的情况下（http content内容不满足hos最小的发送条件），会将需要上传的内容缓存起来，并返回成功（此时并没有发送hos请求，而是等待缓存内容足够大时，发送hos请求，旨在减少发送hos频率提高性能）。此时fw_http_plug会认为上传成功，并将hos 的url写入日志中。当hos真正准备发送put请求时，发现此时的任务请求队列已经满了，遂丢弃该请求，但此时日志已经发送出去了，导致界面上出现了有url却无法下载的情况。
|
||
|
||
|
||
|
||
{color:#FF0000}临时解决方法{color}：暂时关闭hos的缓存功能，来不及发送的请求，不再缓存。
|
||
|
||
{color:#FF0000}具体操作{color}：在/opt/tsg/sapp/tsgconf/main.conf中， [HOS_CONF]模块的hos_cache_size设置为0，hos_request_num设置为0
|
||
|
||
{color:#FF0000}风险{color}：虽然可以避免出现有url却无法下载的情况，但在高并发的情况下，多数hos请求将会由于性能限制被丢弃。也就是说界面上的安全日志的http_request_body和http_response_body的可下载文件将会
|
||
|
||
比设置之前少
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
---
|
||
|
||
**zhangzhihan** commented on *2022-03-02T15:12:26.886+0800*:
|
||
|
||
修改[HOS_CONF]的hos_cache_size和hos_request_num配置后，sapp内存占用会非常快，大概20min就会占用130+GB的内存，然后服务器没有内存了，sapp报错“sapp.service watchdog timeout(limit 10s)!”，然后sapp重启
|
||
|
||
!image-2022-03-02-15-11-23-873.png!
|
||
|
||
|
||
|
||
---
|
||
|
||
**pengxuanzheng** commented on *2022-03-02T15:23:19.814+0800*:
|
||
|
||
查看代码发现cache_size为0的情况下，没有对cache_size进行检查就进行了减操作，导致产生了一个特别大的数，在hos缓存了大量数据，在快速消耗内存
|
||
|
||
|
||
|
||
---
|
||
|
||
**gitlab** commented on *2022-03-02T18:35:58.072+0800*:
|
||
|
||
[彭宣正|https://git.mesalab.cn/pxz] mentioned this issue in [a commit|https://git.mesalab.cn/pxz/hos_client_cpp_module/-/commit/d4e8b149c85b7310d3073b3eea2b0d5f90c0c351] of [彭宣正 / hos_client_cpp_module|https://git.mesalab.cn/pxz/hos_client_cpp_module] on branch [bugfix-9807|https://git.mesalab.cn/pxz/hos_client_cpp_module/-/tree/bugfix-9807]:{quote}🐞 fix(TSG-9807): 规范错误信息输出，并增加返回的错误码{quote}
|
||
|
||
|
||
|
||
---
|
||
|
||
**gitlab** commented on *2022-03-04T10:59:23.110+0800*:
|
||
|
||
[彭宣正|https://git.mesalab.cn/pxz] mentioned this issue in [a commit|https://git.mesalab.cn/pxz/hos_client_cpp_module/-/commit/7bc08418adce8ed6cdde85c158654bf4f9eab5e6] of [彭宣正 / hos_client_cpp_module|https://git.mesalab.cn/pxz/hos_client_cpp_module] on branch [bugfix-9807|https://git.mesalab.cn/pxz/hos_client_cpp_module/-/tree/bugfix-9807]:{quote}🐞 fix(TSG-9807): 修复cache_size设置为0导致的内存快速消耗{quote}
|
||
|
||
|
||
|
||
---
|
||
|
||
**gitlab** commented on *2022-03-04T11:31:41.259+0800*:
|
||
|
||
[彭宣正|https://git.mesalab.cn/pxz] mentioned this issue in [a commit|https://git.mesalab.cn/pxz/hos_client_cpp_module/-/commit/9f8d0a7614fa7beb46f8fdfd67b37bff7ef28a96] of [彭宣正 / hos_client_cpp_module|https://git.mesalab.cn/pxz/hos_client_cpp_module] on branch [bugfix-9807|https://git.mesalab.cn/pxz/hos_client_cpp_module/-/tree/bugfix-9807]:{quote}🐞 fix(TSG-9807): 修复cache_size设置为0导致的内存快速消耗{quote}
|
||
|
||
|
||
|
||
---
|
||
|
||
**gitlab** commented on *2022-03-04T11:41:52.113+0800*:
|
||
|
||
[彭宣正|https://git.mesalab.cn/pxz] mentioned this issue in [a commit|https://git.mesalab.cn/pxz/hos_client_cpp_module/-/commit/e2fc84674a45730ffd9a60b5040fb4ec6171fe3c] of [彭宣正 / hos_client_cpp_module|https://git.mesalab.cn/pxz/hos_client_cpp_module] on branch [bugfix-9807|https://git.mesalab.cn/pxz/hos_client_cpp_module/-/tree/bugfix-9807]:{quote}🐞 fix(TSG-9807): 修复cache_size设置为0导致的内存快速消耗{quote}
|
||
|
||
|
||
|
||
---
|
||
|
||
**gitlab** commented on *2022-03-04T11:52:14.525+0800*:
|
||
|
||
[彭宣正|https://git.mesalab.cn/pxz] mentioned this issue in [a commit|https://git.mesalab.cn/pxz/hos_client_cpp_module/-/commit/c042dc30d3c66dc7168411a15569c84e13db85bf] of [彭宣正 / hos_client_cpp_module|https://git.mesalab.cn/pxz/hos_client_cpp_module] on branch [bugfix-9807|https://git.mesalab.cn/pxz/hos_client_cpp_module/-/tree/bugfix-9807]:{quote}🐞 fix(TSG-9807): 修复cache_size设置为0导致的内存快速消耗{quote}
|
||
|
||
|
||
|
||
---
|
||
|
||
**gitlab** commented on *2022-03-04T12:21:29.048+0800*:
|
||
|
||
[彭宣正|https://git.mesalab.cn/pxz] mentioned this issue in [a commit|https://git.mesalab.cn/pxz/hos_client_cpp_module/-/commit/b2364885f5f5955f390797fcb31e14cc3f11c978] of [彭宣正 / hos_client_cpp_module|https://git.mesalab.cn/pxz/hos_client_cpp_module] on branch [bugfix-9807|https://git.mesalab.cn/pxz/hos_client_cpp_module/-/tree/bugfix-9807]:{quote}🐞 fix(TSG-9807): 修复cache_size设置为0导致的内存快速消耗{quote}
|
||
|
||
|
||
|
||
---
|
||
|
||
**gitlab** commented on *2022-03-04T12:36:05.478+0800*:
|
||
|
||
[彭宣正|https://git.mesalab.cn/pxz] mentioned this issue in [a commit|https://git.mesalab.cn/pxz/hos_client_cpp_module/-/commit/16d71d2fe65c457308512dfadeb7d15c17d5eeae] of [彭宣正 / hos_client_cpp_module|https://git.mesalab.cn/pxz/hos_client_cpp_module] on branch [bugfix-9807|https://git.mesalab.cn/pxz/hos_client_cpp_module/-/tree/bugfix-9807]:{quote}🐞 fix(TSG-9807): 修复cache_size设置为0导致的内存快速消耗{quote}
|
||
|
||
|
||
|
||
---
|
||
|
||
**gitlab** commented on *2022-03-04T13:00:26.486+0800*:
|
||
|
||
[彭宣正|https://git.mesalab.cn/pxz] mentioned this issue in [a commit|https://git.mesalab.cn/pxz/hos_client_cpp_module/-/commit/977ace35c69ed75634d11632220747d09288197b] of [彭宣正 / hos_client_cpp_module|https://git.mesalab.cn/pxz/hos_client_cpp_module] on branch [bugfix-9807|https://git.mesalab.cn/pxz/hos_client_cpp_module/-/tree/bugfix-9807]:{quote}🐞 fix(TSG-9807): 修复cache_size设置为0导致的内存快速消耗{quote}
|
||
|
||
|
||
|
||
---
|
||
|
||
**gitlab** commented on *2022-03-04T13:08:53.937+0800*:
|
||
|
||
[彭宣正|https://git.mesalab.cn/pxz] mentioned this issue in [a merge request|https://git.mesalab.cn/pxz/hos_client_cpp_module/-/merge_requests/47] of [彭宣正 / hos_client_cpp_module|https://git.mesalab.cn/pxz/hos_client_cpp_module] on branch [bugfix-9807|https://git.mesalab.cn/pxz/hos_client_cpp_module/-/tree/bugfix-9807]:{quote}🐞 fix(TSG-9807): 修复cache_size设置为0导致的内存快速消耗{quote}
|
||
|
||
|
||
|
||
---
|
||
|
||
**gitlab** commented on *2022-03-04T13:09:01.240+0800*:
|
||
|
||
[彭宣正|https://git.mesalab.cn/pxz] mentioned this issue in [a commit|https://git.mesalab.cn/pxz/hos_client_cpp_module/-/commit/fce8d9749583465aa0e1ddc0cce3b299725cfb9f] of [彭宣正 / hos_client_cpp_module|https://git.mesalab.cn/pxz/hos_client_cpp_module] on branch [bugfix-9807|https://git.mesalab.cn/pxz/hos_client_cpp_module/-/tree/bugfix-9807]:{quote}🐞 fix(TSG-9807): 修复cache_size设置为0导致的内存快速消耗{quote}
|
||
|
||
|
||
|
||
---
|
||
|
||
**gitlab** commented on *2022-03-04T13:32:20.157+0800*:
|
||
|
||
[彭宣正|https://git.mesalab.cn/pxz] mentioned this issue in [a commit|https://git.mesalab.cn/tsg/tsg-os-buildimage/-/commit/4299c07a02e3427c622bd1f2559b270a2d5d8f64] of [TSG / tsg-os-buildimage|https://git.mesalab.cn/tsg/tsg-os-buildimage] on branch [update-hos-client-cpp|https://git.mesalab.cn/tsg/tsg-os-buildimage/-/tree/update-hos-client-cpp]:{quote}🐞 fix(TSG-9807): update hos-client-cpp-3.0.2 to hos-client-cpp-3.0.3{quote}
|
||
|
||
|
||
|
||
---
|
||
|
||
**gitlab** commented on *2022-03-04T13:33:20.190+0800*:
|
||
|
||
[彭宣正|https://git.mesalab.cn/pxz] mentioned this issue in [a merge request|https://git.mesalab.cn/tsg/tsg-os-buildimage/-/merge_requests/645] of [TSG / tsg-os-buildimage|https://git.mesalab.cn/tsg/tsg-os-buildimage] on branch [update-hos-client-cpp|https://git.mesalab.cn/tsg/tsg-os-buildimage/-/tree/update-hos-client-cpp]:{quote}🐞 fix(TSG-9807): update hos-client-cpp-3.0.2 to hos-client-cpp-3.0.3{quote}
|
||
|
||
|
||
|
||
---
|
||
|
||
**gitlab** commented on *2022-03-07T16:26:59.232+0800*:
|
||
|
||
[彭宣正|https://git.mesalab.cn/pxz] mentioned this issue in [a commit|https://git.mesalab.cn/pxz/hos_client_cpp_module/-/commit/dbdf757aac2a12674d93182334279be6bb3037e6] of [彭宣正 / hos_client_cpp_module|https://git.mesalab.cn/pxz/hos_client_cpp_module] on branch [bugfix-tsg-9807|https://git.mesalab.cn/pxz/hos_client_cpp_module/-/tree/bugfix-tsg-9807]:{quote}🐞 fix(TSG-9807): 保持fd在整个上传过程中一致{quote}
|
||
|
||
|
||
|
||
---
|
||
|
||
**gitlab** commented on *2022-03-07T16:27:29.368+0800*:
|
||
|
||
[彭宣正|https://git.mesalab.cn/pxz] mentioned this issue in [a merge request|https://git.mesalab.cn/pxz/hos_client_cpp_module/-/merge_requests/48] of [彭宣正 / hos_client_cpp_module|https://git.mesalab.cn/pxz/hos_client_cpp_module] on branch [bugfix-tsg-9807|https://git.mesalab.cn/pxz/hos_client_cpp_module/-/tree/bugfix-tsg-9807]:{quote}🐞 fix(TSG-9807): 保持fd在整个上传过程中一致{quote}
|
||
|
||
|
||
|
||
---
|
||
|
||
**gitlab** commented on *2022-03-08T14:32:27.521+0800*:
|
||
|
||
[彭宣正|https://git.mesalab.cn/pxz] mentioned this issue in [a commit|https://git.mesalab.cn/pxz/hos_client_cpp_module/-/commit/7de1970f87d37377e640387b0dcfde18890eb6de] of [彭宣正 / hos_client_cpp_module|https://git.mesalab.cn/pxz/hos_client_cpp_module] on branch [develop-tsg-9807|https://git.mesalab.cn/pxz/hos_client_cpp_module/-/tree/develop-tsg-9807]:{quote}🎈 perf(TSG-9807): 修改打印信息{quote}
|
||
|
||
|
||
|
||
---
|
||
|
||
**zhangzhihan** commented on *2022-03-13T21:47:37.363+0800*:
|
||
|
||
最新进展：
|
||
在福州部署完TSG 22.01集群后，当天更新了2台功能端连接新集群，下载日志文件正常，3.12日发现无法下载短时间的日志文件，只能下载超过1小时的日志文件
|
||
|
||
目前hos版本：
|
||
libhos-client-cpp-3.0.2.31c56ed-1.x86_64
|
||
hos_cache_size=102400
|
||
hos_request_num=1000
|
||
|
||
fw_http_plug版本：
|
||
fw_http_plug-4.0.12.010ebc5-1.x86_64
|
||
|
||
|
||
|
||
|
||
---
|
||
|
||
**pengxuanzheng** commented on *2022-03-17T15:03:02.375+0800*:
|
||
|
||
目前的情况是，hos采用异步上传的方式，为了保证日志可以及时的发出去，并不会等待hos上传的结果。在日志中填写hos url下载路径的时机是在hos客户端将hos上传任务添加至任务队列时，所以在发送日志的时候，hos上传请求不一定已经发送出去。而发送hos上传请求的线程需要等待hos服务端的响应才能结束一次完整的上传。如果响应时间过长将会导致任务队列的任务积压，导致后续的任务无法及时执行。
|
||
|
||
在公司环境和hos 服务端联调测试，hos客户端最多一秒可以上传2800左右的上传请求，每次上传内容5k左右，hos服务端均可以及时响应。现场经计算，大概每秒会产生5500个左右的请求（30台机器共）。
|
||
|
||
后续希望可以监测到每条hos请求入队出队以及执行完成所花费的时间。由于aws s3不支持这种检测行为，并且aws s3 多数功能并不在生产环境中使用，拟使用轻量级的curl加无锁线程池来代替aws s3的功能，对hos的每个过程进行完整的性能监测
|
||
|
||
|
||
|
||
---
|
||
|
||
**gitlab** commented on *2022-03-22T14:53:33.502+0800*:
|
||
|
||
[彭宣正|https://git.mesalab.cn/pxz] mentioned this issue in [a merge request|https://git.mesalab.cn/pxz/hos_client_cpp_module/-/merge_requests/50] of [彭宣正 / hos_client_cpp_module|https://git.mesalab.cn/pxz/hos_client_cpp_module] on branch [develop-tsg-9807|https://git.mesalab.cn/pxz/hos_client_cpp_module/-/tree/develop-tsg-9807]:{quote}🎈 perf(TSG-9807): 修改打印信息{quote}
|
||
|
||
|
||
|
||
---
|
||
|
||
**gitlab** commented on *2022-06-07T19:07:05.067+0800*:
|
||
|
||
[彭宣正|https://git.mesalab.cn/pxz] mentioned this issue in [a commit|https://git.mesalab.cn/tsg/tsg-os-buildimage/-/commit/d4c4941079bf49f80c78564e28428bceb6db054c] of [TSG / tsg-os-buildimage|https://git.mesalab.cn/tsg/tsg-os-buildimage]:{quote}🐞 fix(OMPUB-411): 使用curl上传取代aws s3，增加性能监测{quote}
|
||
|
||
|
||
|
||
---
|
||
|
||
**gitlab** commented on *2022-06-07T19:07:05.144+0800*:
|
||
|
||
[彭宣正|https://git.mesalab.cn/pxz] mentioned this issue in [a commit|https://git.mesalab.cn/tsg/tsg-os-buildimage/-/commit/743914361e5d502a81b839c1d42392900c7fdf05] of [TSG / tsg-os-buildimage|https://git.mesalab.cn/tsg/tsg-os-buildimage] on branch [update-tsg_conn_sketch|https://git.mesalab.cn/tsg/tsg-os-buildimage/-/tree/update-tsg_conn_sketch]:{quote}✨ feat(OMPUB-411): 更新HOS 配置{quote}
|
||
|
||
|
||
|
||
---
|
||
|
||
**gitlab** commented on *2022-06-07T19:07:49.348+0800*:
|
||
|
||
[彭宣正|https://git.mesalab.cn/pxz] mentioned this issue in [a commit|https://git.mesalab.cn/tsg/tsg-os-buildimage/-/commit/8e99c1034697a951d2c8ebe40648f47096d0a3e1] of [TSG / tsg-os-buildimage|https://git.mesalab.cn/tsg/tsg-os-buildimage] on branch [update-tsg_conn_sketch|https://git.mesalab.cn/tsg/tsg-os-buildimage/-/tree/update-tsg_conn_sketch]:{quote}✨ feat(OMPUB-411): 更新HOS 配置{quote}
|
||
|
||
|
||
|
||
---
|
||
|
||
**gitlab** commented on *2022-06-07T19:07:49.556+0800*:
|
||
|
||
[彭宣正|https://git.mesalab.cn/pxz] mentioned this issue in [a commit|https://git.mesalab.cn/tsg/tsg-os-buildimage/-/commit/3b3747b4ac238e2a59ce8c5484c0d00277d0cf05] of [TSG / tsg-os-buildimage|https://git.mesalab.cn/tsg/tsg-os-buildimage]:{quote}🐞 fix(OMPUB-411): 使用curl上传取代aws s3，增加性能监测{quote}
|
||
|
||
|
||
|
||
---
|
||
|
||
|
||
|
||
## Attachments
|
||
|
||
**25767/image-2022-03-01-18-25-30-689.png**
|
||
|
||
---
|
||
|
||
**25768/image-2022-03-01-18-27-45-315.png**
|
||
|
||
---
|
||
|
||
**25769/image-2022-03-01-18-30-28-642.png**
|
||
|
||
---
|
||
|
||
**25785/image-2022-03-02-15-11-23-873.png**
|
||
|
||
---
|
||
|