211 lines
7.7 KiB
Markdown
211 lines
7.7 KiB
Markdown
|
# 【XJ-NPM现场】APP sketch中Unknown Application情况
|
|||
|
|
|
|||
|
|
| ID | Creation Date | Assignee | Status |
|
|||
|
|
|----|----------------|----------|--------|
|
|||
|
|
| OMPUB-472 | 2022-04-20T12:02:19.000+0800 | 贾依蒙 | 已关闭 |
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
新疆IDC环境中，CN生成的服务质量报告显示，Unknow Application排名第二、流量占比为14%。该APP流量情况及描述信息如下：
|
|||
|
|
{code:java}
|
|||
|
|
description:
|
|||
|
|
Unknown is a virtual protocol created for DPI that represents flows that are not recognized by the system. Most of the time, its presence is due to the existence of a business application whose specifications are not made available to the public. Such flows can be defined in the system by means of a User Defined Application
|
|||
|
|
{code}
|
|||
|
|
|
|||
|
|
|
|||
|
|
!image-2022-04-20-11-49-03-101.png|width=579,height=247!
|
|||
|
|
|
|||
|
|
查看APP详情页发现，标记为Unknow APP的流量中，主要为快手、抖音等提供的服务。同时，识别为kuaishou application的流量仅占总流量的1.3%，不符合预期。 !image-2022-04-20-12-01-34-024.png|width=583,height=268!
|
|||
|
|
|
|||
|
|
**lizhao** commented on *2022-04-21T16:32:10.967+0800*:
|
|||
|
|
|
|||
|
|
该问题已移至ompub，需现场运维同事协助解决。
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**jiayimeng** commented on *2022-04-29T12:56:37.231+0800*:
|
|||
|
|
|
|||
|
|
1、通过CN界面中unknown --> 流量 中相关域名排名 TOP10的域名是kwimgs.com，vivo.com.cn，lxdns.net，douyinvod.com，heytapimage.com，douyincdn.com，badambiz.com，douyinstatic.com，pangolin-sdk-toutiao.com，toutiaoapi.com，
|
|||
|
|
|
|||
|
|
2、识别为unknown的 TOP10 域名中，douyin开头的有三个，分别是douyinvod.com，douyincdn.com，douyinstatic.com，通过CN界面中搜索douyin --> 流量 中相关域名排名 TOP10的域名douyinvod.com，amemv.com，douyinpic.com，dbankcdn.com，weathercn.com，yximgs.com，heytapdownload.com，snssdk.com，kwimgs.com，bytegecko.com
|
|||
|
|
|
|||
|
|
识别为douyin APP的 TOP10域名 douyinvod.com，amemv.com，douyinpic.com，dbankcdn.com，weathercn.com，yximgs.com，heytapdownload.com，snssdk.com，kwimgs.com，bytegecko.com
|
|||
|
|
|
|||
|
|
识别为unknown APP的 TOP10域名 kwimgs.com，vivo.com.cn，lxdns.net，douyinvod.com，heytapimage.com，douyincdn.com，badambiz.com，douyinstatic.com，pangolin-sdk-toutiao.com，toutiaoapi.com
|
|||
|
|
|
|||
|
|
其中 kwimgs.com和douyincdn.com同时出现在douyin和unknown 中，unknown中的douyincdn.com和douyinstatic.com明显是属于douyin app的域名。
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**jiayimeng** commented on *2022-04-29T17:33:35.323+0800*:
|
|||
|
|
|
|||
|
|
初步将unknown中TOP100的域名根据备案公司名称 配置到新的自定义APP中，经与[~lizhao] 讨论，对于CN来说会有问题，暂时不进行配置。
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**jiayimeng** commented on *2022-04-29T19:29:06.415+0800*:
|
|||
|
|
|
|||
|
|
使用联通手机号拨测douyin的流量，一部分去了新疆联通IDC，一部分去了新疆联通省口；
|
|||
|
|
|
|||
|
|
4次抓包，同一域名v26-web.douyinvod.com每次分别与新疆联通IDC的1个IP建立TLS连接，查看日志，common_app_label两次被识别为douyin，两次被识别为300hu。
|
|||
|
|
|
|||
|
|
PCAP与导出的会话日志：https://files.geedge.net/d/084c3b79b96648b6b889/
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**lizhao** commented on *2022-04-29T20:12:53.695+0800*:
|
|||
|
|
|
|||
|
|
# app识别不会完全准确，这个可以接受。
|
|||
|
|
# 目前存在一些未识别的APP流量，如果能够通过增加已有APP特征，把其中用户特别关注的流量修正一下最好。
|
|||
|
|
# 目前只支持增加新APP修正这部分流量，建议暂时先不修正，影响不大。
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**fengweihao** commented on *2022-05-05T14:46:37.249+0800*:
|
|||
|
|
|
|||
|
|
|
|||
|
|
||抓包次数||数据包||数据包中会话数||APP识别连接数||APP识别结果||
|
|||
|
|
|第一次|[第一次抓包.pcap|https://files.geedge.net/d/084c3b79b96648b6b889/files/?p=%2F%E7%AC%AC%E4%B8%80%E6%AC%A1%2F%E7%AC%AC%E4%B8%80%E6%AC%A1%E6%8A%93%E5%8C%85.pcap]|1|1|douyin|
|
|||
|
|
|第二次|[cucc-idc-2.pcap|https://files.geedge.net/d/084c3b79b96648b6b889/files/?p=%2F%E7%AC%AC%E4%BA%8C%E6%AC%A1%2Fcucc-idc-2.pcap]|19|19|douyin|
|
|||
|
|
|第三次|[第三次抓包.pcap|https://files.geedge.net/d/084c3b79b96648b6b889/files/?p=%2F%E7%AC%AC%E4%B8%89%E6%AC%A1%2F%E7%AC%AC%E4%B8%89%E6%AC%A1%E6%8A%93%E5%8C%85.pcap]|25|24|douyin|
|
|||
|
|
|第四次|[第四次抓包.pcap|https://files.geedge.net/d/084c3b79b96648b6b889/files/?p=%2F%E7%AC%AC%E5%9B%9B%E6%AC%A1%2F%E7%AC%AC%E5%9B%9B%E6%AC%A1%E6%8A%93%E5%8C%85.pcap]|30|30|douyin|
|
|||
|
|
|
|||
|
|
结论：
|
|||
|
|
|
|||
|
|
以上数据包，第三方识别引擎能正常识别，不存在误识别现象
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**yangwei** commented on *2022-05-05T15:49:41.555+0800*:
|
|||
|
|
|
|||
|
|
[~jiayimeng] 看前述描述，我理解现在存在的问题如下：
|
|||
|
|
|
|||
|
|
背景：
|
|||
|
|
|
|||
|
|
联通CN探针目前仅启用第三方DPI进行App识别
|
|||
|
|
|
|||
|
|
现象：
|
|||
|
|
# 观察整体App识别结果，标识为unknow的会话中，存在包含“douyin”的FQDN
|
|||
|
|
# 通过四次拨测，发现存在拨测客户端访问抖音的行为，在CN的会话记录日志中，被误识别为300hu，经伟浩读包测试，发现都被正确识别为douyin
|
|||
|
|
|
|||
|
|
可能的问题：
|
|||
|
|
|
|||
|
|
现象1：第三方DPI的识别逻辑可能是IP，也可能是FQDN，在没有捕获到识别为unknow的pcap包并且读包复现前，不好下定论，也没法跟第三方DPI交流误识别的问题
|
|||
|
|
|
|||
|
|
现象2：对于拨测行为有识别为300hu的现象，结合伟浩前述本地读包没有误识别的结果，怀疑为AppID->AppLabel的映射有误
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
后续操作：
|
|||
|
|
|
|||
|
|
现象1：继续拨测，或者在CN探针(TSG功能端)尝试捕获识别为unknow的流量，尝试找到能够复现识别为unknow的包
|
|||
|
|
|
|||
|
|
现象2：提供下原始的会话记录日志（SESSION_RECORD）信息，协助确认是否由于AppID->AppLabel映射错误导致的误识别，common_app_id字段中包含原始的AppID，此外不排除现象1与现象2原因相同的可能性
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**fengweihao** commented on *2022-05-05T18:04:17.653+0800*:
|
|||
|
|
|
|||
|
|
现象一：
|
|||
|
|
|
|||
|
|
目前从现场提供的日志，标记为unknow的会话分为三种情况：
|
|||
|
|
# 在抖音直播过程中产生的HTTP请求，且Host为 pull-flv-xxx. douyincdn.com 这一类HTTP请求第三方无法识别
|
|||
|
|
# ipv6地址的抖音，第三方无法识别
|
|||
|
|
# HTTP请求，且Host为lf3-webcastcdn-tos.douyinstatic.com第三方无法识别
|
|||
|
|
|
|||
|
|
需要和第三方沟通后，进一步定位
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**jiayimeng** commented on *2022-05-05T18:34:31.406+0800*:
|
|||
|
|
|
|||
|
|
现象2：
|
|||
|
|
|
|||
|
|
今日拨测5次，均识别正确，未出现识别为300hu的情况，app_id也显示的是douyin的id，原始日志与pcap包：[https://files.geedge.net/d/bf7637d24ecf4744b517/]
|
|||
|
|
|
|||
|
|
待明日继续拨测
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**lizhao** commented on *2022-05-06T14:17:03.027+0800*:
|
|||
|
|
|
|||
|
|
后续操作建议：
|
|||
|
|
# TSG增加功能，支持用户对已有APP添加新的特征。[~liuyang]
|
|||
|
|
# XJ现场，自建名为douyin_customized和kuaishou_customized的app，将unknown中douyin相关与kuaishou相关域名作为特征配置进去，评估对app误识别修正的效果。[~jiayimeng]
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**jiayimeng** commented on *2022-05-06T17:40:02.855+0800*:
|
|||
|
|
|
|||
|
|
XJ现场，自建名为douyin_customized的app，将unknown中的douyincdn.com，douyinvod.com，douyinstatic.com，idouyinvod.com，huoshanvod.com 共计5条douyin相关的域名作为SNI特征已配置；自建名为kuaishou_customized的app，将unknown中kwimgs.com，etoote.com，eckwai.com，yximgs.com，共计4条kuaishou相关域名作为SNI特征已配置。
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**jiayimeng** commented on *2022-05-18T10:47:07.189+0800*:
|
|||
|
|
|
|||
|
|
XJ现场，目前unknown仍在top 10以内，主要为vivo.com.cn占比很高
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
## Attachments
|
|||
|
|
|
|||
|
|
**27259/image-2022-04-20-11-49-03-101.png**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**27257/image-2022-04-20-12-01-34-024.png**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**27692/image-20220429122249102.png**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**27691/image-2022-04-29-12-55-35-415.png**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**27721/image-2022-05-05-14-41-06-885.png**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**27722/image-2022-05-05-14-43-20-912.png**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**27724/image-2022-05-05-14-44-43-675.png**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|