gfwleak/tango-tsg-master
Archived
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Squashed commit of the following:

Browse Source 
commit 91574444c3
Author: liuxueli <liuxueli@iie.ac.cn>
Date:   Tue Dec 3 11:44:16 2019 +0800

    公共字段中的通信市时长单位是：ms；转换单位
    KNI发日志需等待TFE结束，需提前构造streaminfo日志字段，调整包数字节数统计位置
    细化调整日志

commit ff053e3e2c
Author: liuxueli <liuxueli@iie.ac.cn>
Date:   Wed Nov 27 11:38:35 2019 +0800

    增加调试日志信息

commit 2d1a530165
Author: liuxueli <liuxueli@iie.ac.cn>
Date:   Wed Nov 27 11:27:06 2019 +0800

    使用前置声明结构体，修改接口函数原型

commit 27f0cfc91a
Author: liuxueli <liuxueli@iie.ac.cn>
Date:   Wed Nov 27 10:33:48 2019 +0800

    提供TLD_append_streaminfo函数接口供KNI使用
    修改配置文件

commit ed0b17a49d
Merge: 132d807 04963fe
Author: 崔一鸣 <cuiyiming@iie.ac.cn>
Date:   Tue Nov 26 17:44:28 2019 +0800

    Merge branch 'feature-kni' into 'develop'

    增加http解析部分

    See merge request tango/tsg_master!3

commit 04963fe1bf
Author: 崔一鸣 <cuiyiming@iie.ac.cn>
Date:   Tue Nov 26 17:41:43 2019 +0800

    增加http解析部分

commit 132d807af1
Author: liuxueli <liuxueli@iie.ac.cn>
Date:   Tue Nov 26 16:58:54 2019 +0800

    调整调试日志内容

commit 8cfc6b32fd
Author: liuxueli <liuxueli@iie.ac.cn>
Date:   Mon Nov 25 19:30:08 2019 +0800

    增加调试日志信息

commit 4aef0628e5
Author: liuxueli <liuxueli@iie.ac.cn>
Date:   Mon Nov 25 18:36:48 2019 +0800

    增加调试信息

commit 7c9dabdab5
Author: liuxueli <liuxueli@iie.ac.cn>
Date:   Mon Nov 25 10:51:37 2019 +0800

    矫正tableinfo配置项

commit 34ec2059ea
Merge: 27cb2ad 2b96c05
Author: liuxueli <liuxueli@iie.ac.cn>
Date:   Mon Nov 25 10:00:18 2019 +0800

    Merge branch 'develop' of https://git.mesalab.cn/tango/tsg_master into develop

commit 27cb2ad3b2
Author: liuxueli <liuxueli@iie.ac.cn>
Date:   Thu Nov 21 19:21:08 2019 +0800

    获取包数、字节数与sapp头文件不一致，导致统计信息颠倒
    删除从vxlan里获取信息

commit 2b96c050d5
Merge: 0a11db6 8d954e4
Author: 崔一鸣 <cuiyiming@iie.ac.cn>
Date:   Thu Nov 21 10:22:42 2019 +0800

    Merge branch 'feature-kni' into 'develop'

    修改proto初始值为PROTO_UNKONWN

    See merge request tango/tsg_master!2

commit 46a233b782
Author: liuxueli <liuxueli@iie.ac.cn>
Date:   Thu Nov 21 10:18:08 2019 +0800

    KNI发送日志的线程不是sapp申请的，所以不能使用dictator_malloc申请内存
    增加user_region日志字段

commit 8d954e4cc5
Author: 崔一鸣 <cuiyiming@iie.ac.cn>
Date:   Wed Nov 20 19:55:59 2019 +0800

    修改proto初始值为PROTO_UNKONWN

commit 0a11db6d2e
Author: liuxueli <liuxueli@iie.ac.cn>
Date:   Wed Nov 20 18:02:54 2019 +0800

    增加总控配置文件

commit 5c04ba23aa
Author: liuxueli <liuxueli@iie.ac.cn>
Date:   Wed Nov 20 18:00:35 2019 +0800

    SSL_SNI和HTTP_HOST的deny动作由总控负责处理
    修复获取全局流ID时的错误

commit 291c05ea79
Author: liuxueli <liuxueli@iie.ac.cn>
Date:   Wed Nov 20 16:40:19 2019 +0800

    增加获取全局流ID的接口
    调整总控注册为TCP入口

commit 28da97b53b
Author: liuxueli <liuxueli@iie.ac.cn>
Date:   Wed Nov 20 14:45:41 2019 +0800

    支持平台提供的APP_STATE_KILL_OTHER
    修复BUG，天剑流标签时未对协议进行赋值，导致tfe颁发证书失败

commit 7912ec54b2
Author: liuxueli <liuxueli@iie.ac.cn>
Date:   Tue Nov 19 18:41:31 2019 +0800

    修复BUG，命中拦截策略时未对类型进行赋值，导致KNI获取不到命中结果
    添加总控配置文件
    添加FS2统计日志
    添加错误日志
    构造适用sapp4.0的rpm

commit 484feaed94
Author: liuxueli <liuxueli@iie.ac.cn>
Date:   Tue Nov 19 13:39:37 2019 +0800

    增加获取deny结果规则的接口
    修复自测试过程中遇到的BUG

commit 5165173025
Merge: 186e591 6b6cbef
Author: 崔一鸣 <cuiyiming@iie.ac.cn>
Date:   Mon Nov 18 15:46:03 2019 +0800

    Merge branch 'feature-fast-path' into feature-kni

commit 6b6cbeffa1
Author: liuxueli <liuxueli@iie.ac.cn>
Date:   Mon Nov 18 14:50:32 2019 +0800

    修复初始化错误

commit e7532d6280
Author: liuxueli <liuxueli@iie.ac.cn>
Date:   Mon Nov 18 13:33:29 2019 +0800

    添加遗漏头文件tsg_entry.h

commit f33d11c50b
Author: liuxueli <liuxueli@iie.ac.cn>
Date:   Mon Nov 18 13:29:28 2019 +0800

    修复编译错误

commit 88d9368cee
Author: liuxueli <liuxueli@iie.ac.cn>
Date:   Mon Nov 18 13:25:38 2019 +0800

    合并头文件，删除无用头文件

commit 382ca0a793
Author: 刘学利 <liuxueli@iie.ac.cn>
Date:   Mon Nov 18 13:21:25 2019 +0800

    Delete tsg_types.h

commit ab58f918e5
Author: 刘学利 <liuxueli@iie.ac.cn>
Date:   Mon Nov 18 13:21:05 2019 +0800

    Delete tsg_log_id.h

commit 3b13075701
Author: 刘学利 <liuxueli@iie.ac.cn>
Date:   Mon Nov 18 13:20:57 2019 +0800

    Delete tsg_entry.h

commit 186e591dbf
Merge: 39f82ed 91c511c
Author: 崔一鸣 <cuiyiming@iie.ac.cn>
Date:   Fri Nov 15 21:22:44 2019 +0800

    Merge branch 'master' into feature-kni

commit 39f82ed3f7
Author: 崔一鸣 <cuiyiming@iie.ac.cn>
Date:   Fri Nov 15 21:22:39 2019 +0800

    接口适配

commit 8913a7d22e
Author: liuxueli <liuxueli@iie.ac.cn>
Date:   Fri Nov 15 19:29:54 2019 +0800

    规则排序功能
    统一发送日志功能

commit 1cb2358fea
Merge: c76bc55 bd65acc
Author: 崔一鸣 <cuiyiming@iie.ac.cn>
Date:   Thu Nov 14 15:06:35 2019 +0800

    合并master修改

commit c76bc5534d
Author: 崔一鸣 <cuiyiming@iie.ac.cn>
Date:   Wed Nov 13 19:56:19 2019 +0800

    接口增加protocol参数

commit db7282dab7
Author: 崔一鸣 <cuiyiming@iie.ac.cn>
Date:   Wed Nov 13 19:24:22 2019 +0800

    扫描接口增加domain

commit 56fadd73b2
Merge: f9aee05 9e738ae
Author: 崔一鸣 <cuiyiming@iie.ac.cn>
Date:   Wed Nov 13 19:08:55 2019 +0800

    Merge branch 'master' into feature-kni

commit f9aee05bcc
Author: 崔一鸣 <cuiyiming@iie.ac.cn>
Date:   Wed Nov 13 16:17:59 2019 +0800

    修改ssl_utils文件名

commit 452a08790c
Merge: 3abe8a2 c1ffc53
Author: 崔一鸣 <cuiyiming@iie.ac.cn>
Date:   Wed Nov 13 16:12:14 2019 +0800

    Merge branch 'master' into feature-kni

commit 3abe8a2fc5
Author: 崔一鸣 <cuiyiming@iie.ac.cn>
Date:   Wed Nov 13 16:11:38 2019 +0800

    添加kni模块扫描部分
This commit is contained in:
杨威
2019-12-09 18:58:05 +08:00
parent 91c511c3ed
commit 64c1dcb159
22 changed files with 2184 additions and 357 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
.gitignore vendored Normal file
View File

@@ -0,0 +1 @@
.vscode/*

35
CMakeLists.txt
View File

@@ -8,6 +8,7 @@ include(Version)
set(CMAKE_MACOSX_RPATH 0)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall")
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -g")
include_directories(${PROJECT_SOURCE_DIR}/inc/)
include_directories(/opt/MESA/include/)
@@ -28,27 +29,45 @@ elseif(ENABLE_SANITIZE_THREAD)
set(CMAKE_SHARED_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -lasan")
endif()
set(CMAKE_INSTALL_PREFIX /home/mesasoft/sapp_run)
add_subdirectory (src)
set(CONFLIST /plug/platform/conflist_platform.inf)
set(CONFLIST /plug/conflist.inf)
set(MASTER_INF "./plug/platform/tsg_master/tsg_master.inf")
set(PROJECT_LIST /etc/project_list.conf)
file(WRITE ${PROJECT_SOURCE_DIR}/install.sh "#!/bin/sh\r\n")
file(APPEND ${PROJECT_SOURCE_DIR}/install.sh "DST=\${RPM_INSTALL_PREFIX}\r\n")
file(APPEND ${PROJECT_SOURCE_DIR}/install.sh "mkdir -p \${DST}/plug/platform/\r\n")
file(APPEND ${PROJECT_SOURCE_DIR}/install.sh "mkdir -p \${DST}/etc/\r\n")
file(APPEND ${PROJECT_SOURCE_DIR}/install.sh "touch \${DST}${CONFLIST}\r\n")
file(APPEND ${PROJECT_SOURCE_DIR}/install.sh "touch \${DST}${PROJECT_LIST}\r\n")
file(APPEND ${PROJECT_SOURCE_DIR}/install.sh "if [[ -z `grep -rn 'POLICY_PRIORITY' \${DST}${PROJECT_LIST}` ]];then\r\n")
file(APPEND ${PROJECT_SOURCE_DIR}/install.sh "echo 'POLICY_PRIORITY struct' >> \${DST}${PROJECT_LIST}\r\nfi\r\n")
file(APPEND ${PROJECT_SOURCE_DIR}/install.sh "if [[ -z `grep -rn 'tsg_master.inf' \${DST}${CONFLIST}` ]];then\r\n")
file(APPEND ${PROJECT_SOURCE_DIR}/install.sh "echo './plug/platform/tsg_master/tsg_master.inf' >> \${DST}/plug/platform/conflist_platform.inf\r\nfi\r\n")
file(APPEND ${PROJECT_SOURCE_DIR}/install.sh "sed -i '/\\[platform\\]/a\\${MASTER_INF}' \${DST}${CONFLIST}\r\nfi\r\n")
#file(APPEND ${PROJECT_SOURCE_DIR}/install.sh "echo './plug/platform/tsg_master/tsg_master.inf' >> \${DST}\${CONFLIST}\r\nfi\r\n")
SET(CPACK_RPM_PRE_INSTALL_SCRIPT_FILE "${PROJECT_SOURCE_DIR}/install.sh")
file(WRITE ${PROJECT_SOURCE_DIR}/uninstall.sh "#!/bin/sh\r\n")
file(APPEND ${PROJECT_SOURCE_DIR}/uninstall.sh "DST=\${RPM_INSTALL_PREFIX}\r\n")
file(APPEND ${PROJECT_SOURCE_DIR}/uninstall.sh "mkdir -p \${DST}/plug/platform/\r\n")
file(APPEND ${PROJECT_SOURCE_DIR}/uninstall.sh "sed -i '/tsg_master/d' \${DST}/plug/platform/conflist_platform.inf\r\n")
file(APPEND ${PROJECT_SOURCE_DIR}/uninstall.sh "sed -i '/tsg_master.inf/d' \${DST}${CONFLIST}\r\n")
file(APPEND ${PROJECT_SOURCE_DIR}/uninstall.sh "sed -i '/POLICY_PRIORITY/d' \${DST}${PROJECT_LIST}\r\n")
SET(CPACK_RPM_PRE_UNINSTALL_SCRIPT_FILE "${PROJECT_SOURCE_DIR}/uninstall.sh")
install(FILES inc/soq_sendlog.h DESTINATION /opt/MESA/include/soq)
install(FILES inc/soq_types.h DESTINATION /opt/MESA/include/soq)
install(FILES inc/t1_public.h DESTINATION /opt/MESA/include/soq)
install(FILES inc/t2_public.h DESTINATION /opt/MESA/include/soq)
install(FILES ddp/inc/ddp.h DESTINATION /opt/MESA/include/soq)
install(FILES bin/main.conf DESTINATION ${CMAKE_INSTALL_PREFIX}/tsgconf)
install(FILES bin/tsg_tableinfo.conf DESTINATION ${CMAKE_INSTALL_PREFIX}/tsgconf)
install(FILES bin/tsg_log_field.conf DESTINATION ${CMAKE_INSTALL_PREFIX}/tsgconf)
install(FILES bin/tsg_maat.json DESTINATION ${CMAKE_INSTALL_PREFIX}/tsgconf)
install(FILES bin/tsg_maat_ip_deny.json DESTINATION ${CMAKE_INSTALL_PREFIX}/tsgconf)
install(FILES inc/tsg_send_log.h DESTINATION /opt/MESA/include/tsg)
install(FILES inc/tsg_rule.h DESTINATION /opt/MESA/include/tsg)
include(Package)

39
bin/main.conf Normal file
View File

@@ -0,0 +1,39 @@
[MAAT]
MAAT_MODE=1
#EFFECTIVE_FLAG=
STAT_SWITCH=1
PERF_SWITCH=1
TABLE_INFO=tsgconf/tsg_tableinfo.conf
STAT_FILE=tsg_maat.status
EFFECT_INTERVAL_S=1
REDIS_IP=127.0.0.1
REDIS_PORT_NUM=10
REDIS_PORT=6380
REDIS_INDEX=2
JSON_CFG_FILE=tsgconf/tsg_maat.json
INC_CFG_DIR=tsgrule/inc/index/
FULL_CFG_DIR=tsgrule/full/index/
IP_ADDR_TABLE=TSG_OBJ_IP_ADDR
SUBSCRIBER_ID_TABLE=TSG_OBJ_SUBSCRIBER_ID
[TSG_LOG]
MODE=1
NIC_NAME=eth1
MAX_SERVICE=0
LOG_LEVEL=10
LOG_PATH=./tsglog/tsglog
BROKER_LIST=127.0.0.1:9092
COMMON_FIELD_FILE=tsgconf/tsg_log_field.conf
[FIELD_STAT]
CYCLE=3
TELEGRAF_PORT=8125
TELEGRAF_IP=127.0.0.1
OUTPUT_PATH=./tsg_stat.log
APP_NAME=tsg_master
[SYSTEM]
LOG_LEVEL=10
LOG_PATH=./tsglog/tsg_master
POLICY_PRIORITY_LABEL=POLICY_PRIORITY

41
bin/tsg_log_field.conf Normal file
View File

@@ -0,0 +1,41 @@
#TYPE：1:UCHAR,2:USHORT,3:ULONG,4:ULOG,5:USTRING,6:FILE,7:UBASE64,8:PACKET
#TYPE TOPIC SERVICE
TOPIC SECURITY-EVENT-LOG 0
#TYPE FIELD VALUE
LONG common_policy_id 1
LONG common_service 2
LONG common_action 3
LONG common_start_time 4
LONG common_end_time 5
STRING common_l4_protocol 6
LONG common_address_type 7
STRING common_server_ip 8
STRING common_client_ip 9
LONG common_server_port 10
LONG common_client_port 11
LONG common_stream_dir 12
STRING common_address_list 13
LONG common_entrance_id 14
LONG common_device_id 15
LONG common_link_id 16
STRING common_isp 17
LONG common_encapsulation 18
LONG common_direction 19
STRING common_sled_ip 20
STRING common_user_tags 21
STRING common_user_region 22
STRING common_app_label 23
LONG common_app_id 24
LONG common_protocol_id 25
LONG common_c2s_pkt_num 26
LONG common_s2c_pkt_num 27
LONG common_c2s_byte_num 28
LONG common_s2c_byte_num 29
LONG common_con_duration_ms 30
LONG common_has_dup_traffic 31
STRING common_stream_error 32
STRING common_stream_trace_id 33
STRING common_schema_type 34
STRING http_host 35
STRING ssl_sni 36

84
bin/tsg_maat.json Normal file
View File

@@ -0,0 +1,84 @@
{
"compile_table": "TSG_SECURITY_COMPILE",
"group_table": "POLICY_OBJECT",
"rules": [
{
"compile_id": 1,
"service": 0,
"action": 16,
"do_blacklist": 0,
"do_log": 1,
"effective_rage": 0,
"user_region": "anything",
"is_valid": "yes",
"groups": [
{
"group_name": "group_1",
"regions": [
{
"table_name": "TSG_OBJ_IP_ADDR",
"table_type": "ip",
"table_content": {
"addr_type": "ipv4",
"src_ip": "61.135.169.125",
"mask_src_ip": "255.255.255.255",
"src_port": "80",
"mask_src_port": "65535",
"dst_ip": "192.168.41.228",
"mask_dst_ip": "255.255.255.255",
"dst_port": "0",
"mask_dst_port": "65535",
"protocol": 6,
"direction": "double"
}
}
]
}
]
},
{
"compile_id": 2,
"service": 0,
"action": 128,
"do_blacklist": 0,
"do_log": 1,
"effective_rage": 0,
"user_region": "anything",
"is_valid": "yes",
"groups": [
{
"group_name": "FQDN_SNI",
"regions": [
{
"table_name": "TSG_OBJ_FQDN",
"table_type": "expr",
"table_content": {
"keywords": "baidu.com",
"expr_type": "and",
"match_method": "sub",
"format": "uncase plain"
}
}
]
}
]
},
{
"compile_id": 3,
"service": 0,
"action": 128,
"do_blacklist": 0,
"do_log": 1,
"effective_rage": 0,
"user_region": "Virtual",
"is_valid": "yes",
"groups": [
{
"group_name":"FQDN_SNI",
"virtual_table":"TSG_FIELD_SSL_SNI",
"not_flag" : 0
}
]
}
]
}

40
bin/tsg_maat_ip_deny.json Normal file
View File

@@ -0,0 +1,40 @@
{
"compile_table": "TSG_SECURITY_COMPILE",
"group_table": "POLICY_OBJECT",
"rules": [
{
"compile_id": 1,
"service": 0,
"action": 16,
"do_blacklist": 0,
"do_log": 1,
"effective_rage": 0,
"user_region": "anything",
"is_valid": "yes",
"groups": [
{
"group_name": "group_1",
"regions": [
{
"table_name": "TSG_OBJ_IP_ADDR",
"table_type": "ip",
"table_content": {
"addr_type": "ipv4",
"src_ip": "117.18.237.29",
"mask_src_ip": "255.255.255.255",
"src_port": "80",
"mask_src_port": "65535",
"dst_ip": "192.168.41.228",
"mask_dst_ip": "255.255.255.255",
"dst_port": "0",
"mask_dst_port": "65535",
"protocol": 6,
"direction": "double"
}
}
]
}
]
}
]
}

9
bin/tsg_master.inf Normal file
View File

@@ -0,0 +1,9 @@
[PLUGINFO]
PLUGNAME=TSG_MASTER
SO_PATH=./plug/platform/tsg_master/tsg_master.so
INIT_FUNC=TSG_MASTER_INIT
DESTROY_FUNC=TSG_MASTER_UNLOAD
[TCP]
FUNC_FLAG=ALL
FUNC_NAME=TSG_MASTER_TCP_ENTRY

40
bin/tsg_tableinfo.conf Normal file
View File

@@ -0,0 +1,40 @@
#each collumn seperate with '\t'
#id (0~65535)
#name string
#type one of ip,expr,expr_plus,digest,intval,compile or plugin
#src_charset one of GBK,BIG5,UNICODE,UTF8
#dst_charset combined by GBK,BIG5,UNICODE,UTF8,seperate with '/'
#do_merege yes or no
#cross cache 0~max
#quickswitch quickon or quick off
#id name type src_charset dst_charset do_merge cross_cache quickswitch
0 TSG_SECURITY_COMPILE compile escape --
1 GROUP_COMPILE_RELATION group UTF8 UTF8 no 0
2 TSG_OBJ_IP_ADDR ip_plus UTF8 UTF8 no 0
3 TSG_OBJ_SUBSCRIBER_ID expr UTF8 UTF8 no 0
4 TSG_OBJ_ACCOUNT expr UTF8 UTF8 no 0
5 TSG_OBJ_URL expr UTF8 UTF8/GBK no 0
6 TSG_OBJ_FQDN expr UTF8 UTF8 no 0
7 TSG_OBJ_KEYWORDS expr UTF8 UTF8 no 0
8 TSG_OBJ_HTTP_SIGNATURE expr_plus UTF8 UTF8/GBK no 0
9 TSG_FIELD_HTTP_HOST virtual TSG_OBJ_FQDN --
10 TSG_FIELD_HTTP_URL virtual TSG_OBJ_URL --
11 TSG_FIELD_HTTP_REQ_HDR virtual TSG_OBJ_HTTP_SIGNATURE --
12 TSG_FIELD_HTTP_RES_HDR virtual TSG_OBJ_HTTP_SIGNATURE --
13 TSG_FIELD_HTTP_REQ_CONTENT virtual TSG_OBJ_KEYWORDS --
14 TSG_FIELD_HTTP_RES_CONTENT virtual TSG_OBJ_KEYWORDS --
15 TSG_FIELD_SSL_SNI virtual TSG_OBJ_FQDN --
16 TSG_FIELD_SSL_CN virtual TSG_OBJ_FQDN --
17 TSG_FIELD_SSL_SAN virtual TSG_OBJ_FQDN --
18 TSG_FIELD_DNS_QNAME virtual TSG_OBJ_FQDN --
19 TSG_FIELD_MAIL_ACCOUNT virtual TSG_OBJ_ACCOUNT --
20 TSG_FIELD_MAIL_FROM virtual TSG_OBJ_ACCOUNT --
21 TSG_FIELD_MAIL_TO virtual TSG_OBJ_ACCOUNT --
22 TSG_FIELD_MAIL_SUBJECT virtual TSG_OBJ_KEYWORDS --
23 TSG_FIELD_MAIL_CONTENT virtual TSG_OBJ_KEYWORDS --
24 TSG_FIELD_MAIL_ATT_NAME virtual TSG_OBJ_KEYWORDS --
25 TSG_FIELD_MAIL_ATT_CONTENT virtual TSG_OBJ_KEYWORDS --
26 TSG_FIELD_FTP_URI virtual TSG_OBJ_URL --
27 TSG_FIELD_FTP_CONTENT virtual TSG_OBJ_KEYWORDS --
28 TSG_FIELD_FTP_ACCOUNT virtual TSG_OBJ_ACCOUNT --

1
inc/tsg_entry.h
View File

@@ -1 +0,0 @@

131
inc/tsg_log_id.h
View File

@@ -1,131 +0,0 @@
#ifndef __TSG_CABOT_LOG_ID_H__
#define __TSG_CABOT_LOG_ID_H__
#define CABOT_LOG_OPT_POLICY_ID 1
#define CABOT_LOG_OPT_POLICY_SERVICE 2
#define CABOT_LOG_OPT_POLICY_ACTION 3
#define CABOT_LOG_OPT_START_TIME 4
#define CABOT_LOG_OPT_END_TIME 5
#define CABOT_LOG_OPT_L4_PROTOCL 6
#define CABOT_LOG_OPT_ADDRESS_TYPE 7
#define CABOT_LOG_OPT_SERVER_IP 8
#define CABOT_LOG_OPT_CLIENT_IP 9
#define CABOT_LOG_OPT_SERVER_PORT 10
#define CABOT_LOG_OPT_CLIENT_PORT 11
#define CABOT_LOG_OPT_STREAM_DIR 12
#define CABOT_LOG_OPT_ADDRESS_LIST 13
#define CABOT_LOG_OPT_ENTRANCE_ID 14
#define CABOT_LOG_OPT_DEVICE_ID 15
#define CABOT_LOG_OPT_LINK_ID 16
#define CABOT_LOG_OPT_ISP 17
#define CABOT_LOG_OPT_ENCAPSULATION 18
#define CABOT_LOG_OPT_DIRECTION 19
#define CABOT_LOG_OPT_SLED_IP 20
#define CABOT_LOG_OPT_USER_TAGS 21
#define CABOT_LOG_OPT_USER_REGION 22
#define CABOT_LOG_OPT_APP_ID 23
#define CABOT_LOG_OPT_PROTOCIL_ID 24
#define CABOT_LOG_OPT_C2S_PKT_NUM 25
#define CABOT_LOG_OPT_S2C_PKT_NUM 26
#define CABOT_LOG_OPT_C2S_BYTE_NUM 27
#define CABOT_LOG_OPT_S2C_BYTE_NUM 28
#define CABOT_LOG_OPT_CON_DURATION_MS 29
#define CABOT_LOG_OPT_CON_LATENCY_MS 30
#define CABOT_LOG_OPT_STREAM_TRACE_ID 31
#define CABOT_LOG_OPT_HTTP_URL 32
#define CABOT_LOG_OPT_HTTP_REQUEST_LINE 33
#define CABOT_LOG_OPT_HTTP_RESPONSE_LINE 34
#define CABOT_LOG_OPT_HTTP_REQUEST_HEADER 35
#define CABOT_LOG_OPT_HTTP_RESPONSE_HEADER 36
#define CABOT_LOG_OPT_HTTP_REQUEST_BODY 37
#define CABOT_LOG_OPT_HTTP_RESPONSE_BODY 38
#define CABOT_LOG_OPT_HTTP_C2S_ISN 39
#define CABOT_LOG_OPT_HTTP_PROXY_FLAG 40
#define CABOT_LOG_OPT_HTTP_SEQUENCE 41
#define CABOT_LOG_OPT_HTTP_SNAPSHOT 42
#define CABOT_LOG_OPT_HTTP_COOKIE 43
#define CABOT_LOG_OPT_HTTP_REFERER 44
#define CABOT_LOG_OPT_HTTP_USER_AGENT 45
#define CABOT_LOG_OPT_HTTP_CONTENT_LENGTH 46
#define CABOT_LOG_OPT_HTTP_CONTENT_TYPE 47
#define CABOT_LOG_OPT_HTTP_SET_COOKIE 48
#define CABOT_LOG_OPT_MAIL_PROTOCOL 51
#define CABOT_LOG_OPT_MAIL_SENDER 52
#define CABOT_LOG_OPT_MAIL_RECEIVER 53
#define CABOT_LOG_OPT_MAIL_SUBJECT 54
#define CABOT_LOG_OPT_MAIL_CONTENT 55
#define CABOT_LOG_OPT_MAIL_ATTACHMENT_NAME 56
#define CABOT_LOG_OPT_MAIL_ATTACHMENT_CONTENT 57
#define CABOT_LOG_OPT_MAIL_EML_FILE 58
#define CABOT_LOG_OPT_MAIL_SNAPSHOT 59
#define CABOT_LOG_OPT_DNS_MESSAGE_ID 61
#define CABOT_LOG_OPT_DNS_QR 62
#define CABOT_LOG_OPT_DNS_OPCODE 63
#define CABOT_LOG_OPT_DNS_AA 64
#define CABOT_LOG_OPT_DNS_TC 65
#define CABOT_LOG_OPT_DNS_RD 66
#define CABOT_LOG_OPT_DNS_RA 67
#define CABOT_LOG_OPT_DNS_RCODE 68
#define CABOT_LOG_OPT_DNS_QDCOUNT 69
#define CABOT_LOG_OPT_DNS_ANCOUNT 70
#define CABOT_LOG_OPT_DNS_NSCOUNT 71
#define CABOT_LOG_OPT_DNS_ARCOUNT 72
#define CABOT_LOG_OPT_DNS_QNAME 73
#define CABOT_LOG_OPT_DNS_QTYPE 74
#define CABOT_LOG_OPT_DNS_QCLASS 75
#define CABOT_LOG_OPT_DNS_CNAME 76
#define CABOT_LOG_OPT_DNS_SUB 77
#define CABOT_LOG_OPT_DNS_RR 78
#define CABOT_LOG_OPT_SSL_VERSION 81
#define CABOT_LOG_OPT_SSL_SNI 82
#define CABOT_LOG_OPT_SSL_SAN 83
#define CABOT_LOG_OPT_SSL_CN 84
#define CABOT_LOG_OPT_FTP_URL 88
#define CABOT_LOG_OPT_FTP_CONTENT 89
#define CABOT_LOG_OPT_VOIP_CALLING_ACCOUNT 90
#define CABOT_LOG_OPT_VOIP_CALLED_ACCOUNT 91
#define CABOT_LOG_OPT_VOIP_CALLING_NUMBER 92
#define CABOT_LOG_OPT_VOIP_CALLED_NUMBER 93
#define CABOT_LOG_OPT_BGP_PACKET_TYPE 100
#define CABOT_LOG_OPT_BGP_AS_NUM 101
#define CABOT_LOG_OPT_BGP_ROUTE 102
#define CABOT_LOG_OPT_RADIUS_PACKET_TYPE 112
#define CABOT_LOG_OPT_RADIUS_ACCOUNT 113
#define CABOT_LOG_OPT_RADIUS_CALLBACK_NUMBER 114
#define CABOT_LOG_OPT_RADIUS_CALLBACK_ID 115
#define CABOT_LOG_OPT_RADIUS_CALLED_STATION_ID 116
#define CABOT_LOG_OPT_RADIUS_CALLING_STATION_ID 117
#define CABOT_LOG_OPT_RADIUS_ACCT_SESSION_ID 118
#define CABOT_LOG_OPT_RADIUS_ACCT_MULTI_SESSION_ID 119
#define CABOT_LOG_OPT_RADIUS_NAS_IP_ADDRESS 120
#define CABOT_LOG_OPT_RADIUS_FRAMED_IP_ADDRESS 121
#define CABOT_LOG_OPT_RADIUS_FRAMED_IP_NETMASK 122
#define CABOT_LOG_OPT_RADIUS_SERVICE_TYPE 123
#define CABOT_LOG_OPT_RADIUS_FRAMED_MTU 124
#define CABOT_LOG_OPT_RADIUS_SESSION_TIMEOUT 125
#define CABOT_LOG_OPT_RADIUS_IDLE_TIMEOUT 126
#define CABOT_LOG_OPT_RADIUS_TERMINATION_CATION 127
#define CABOT_LOG_OPT_RADIUS_PROXY_STATE 128
#define CABOT_LOG_OPT_RADIUS_ACCT_STATUS_TYPE 129
#define CABOT_LOG_OPT_RADIUS_ACCT_INPUT_OCTETS 130
#define CABOT_LOG_OPT_RADIUS_ACCT_INPUT_PACKETS 131
#define CABOT_LOG_OPT_RADIUS_ACCT_OUTPUT_OCTETS 132
#define CABOT_LOG_OPT_RADIUS_ACCT_OUTPUT_PACKETS 133
#define CABOT_LOG_OPT_RADIUS_ACCT_TERMINATE_CAUSE 134
#define CABOT_LOG_OPT_RADIUS_ACCT_LINK_COUNT 135
#define CABOT_LOG_OPT_RADIUS_ACCT_INTERIM_INTERVAL 136
#endif

30
inc/tsg_rule.h
View File

@@ -2,13 +2,27 @@
#define __TSG_RULE_H__
#include <MESA/Maat_rule.h>
#include "tsg_types.h"
#ifdef __cplusplus
extern "C"
typedef enum _tsg_protocol
{
#endif
PROTO_UNKONWN=0,
PROTO_IPv4=1,
PROTO_IPv6,
PROTO_TCP,
PROTO_UDP,
PROTO_HTTP,
PROTO_MAIL,
PROTO_DNS,
PROTO_FTP,
PROTO_SSL,
PROTO_SIP,
PROTO_BGP,
PROTO_STREAMING_MEDIA,
PROTO_MAX
}tsg_protocol_t;
#define MAX_RESULT_NUM 8
#define MAX_DOAMIN_LEN 2048
struct _identify_info
@@ -24,6 +38,8 @@ typedef enum _PULL_RESULT_TYPE
PULL_FW_RESULT
}PULL_RESULT_TYPE;
#define TSG_DOMAIN_MAX 256
extern Maat_feather_t g_tsg_maat_feather;
int tsg_rule_init(const char *conffile, void *logger);
@@ -39,9 +55,7 @@ int tsg_shared_table_init(const char *conffile, Maat_feather_t maat_feather, voi
//return value: -1: failed, 0: not hit, >0: hit count
int tsg_scan_shared_policy(Maat_feather_t maat_feather, void *pkt, int pkt_len, Maat_rule_t *result, int result_num, struct _identify_info *identify_info, scan_status_t *mid, void *logger, int thread_seq);
#ifdef __cplusplus
}
#endif
//return NULL if none exists, otherwise return one deny rule;
struct Maat_rule_t *tsg_fetch_deny_rule(Maat_rule_t *result, int result_num);
#endif

27
inc/tsg_send_log.h
View File

@@ -3,10 +3,6 @@
#include <MESA/Maat_rule.h>
#ifdef __cplusplus
extern "C"
{
#endif
typedef struct _tsg_log
{
@@ -17,26 +13,27 @@ typedef struct _tsg_log
typedef enum _tld_type
{
TLD_TYPE_UNKNOWN=0,
TLD_TYPE_LONG=1,
TLD_TYPE_STRING,
TLD_TYPE_FILE
TLD_TYPE_FILE,
TLD_TYPE_MAX
}TLD_TYPE;
typedef void* TLD_handle_t;
typedef void* tsg_log_instance_t;
struct TLD_handle_t;
struct tsg_log_instance_t;
extern tsg_log_instance_t g_tsg_log_instance;
extern struct tsg_log_instance_t *g_tsg_log_instance;
TLD_handle_t TLD_create(int thread_id);
int TLD_append(TLD_handle_t handle, char *key, void *value, TLD_TYPE type);
int TLD_cancel(TLD_handle_t handle);
struct TLD_handle_t *TLD_create(int thread_id);
int TLD_append(struct TLD_handle_t *handle, char *key, void *value, TLD_TYPE type);
int TLD_append_streaminfo(struct tsg_log_instance_t *instance, struct TLD_handle_t *handle, struct streaminfo *a_stream);
int TLD_cancel(struct TLD_handle_t *handle);
int tsg_send_log(tsg_log_instance_t instance, TLD_handle_t handle, tsg_log_t *log_msg, int thread_id);
int tsg_send_log(struct tsg_log_instance_t *instance, struct TLD_handle_t *handle, tsg_log_t *log_msg, int thread_id);
unsigned long long tsg_get_stream_id(struct streaminfo *a_stream);
#ifdef __cplusplus
}
#endif
#endif

35
inc/tsg_ssl_utils.h Normal file
View File

@@ -0,0 +1,35 @@
#pragma once
struct cipher_suite
{
int value;
const char* name;
};
enum chello_parse_result
{
CHELLO_PARSE_SUCCESS = 0,
CHELLO_PARSE_INVALID_FORMAT = -1,
CHELLO_PARSE_NOT_ENOUGH_BUFF = -2
};
struct ssl_version
{
uint8_t minor;
uint8_t major;
uint16_t ossl_format;
};
struct ssl_chello
{
struct ssl_version min_version;
struct ssl_version max_version;
char* sni;
char* alpn;
char* cipher_suites;
char* cipher_suites_tls13;
};
struct ssl_chello* ssl_chello_parse(const unsigned char* buff, size_t buff_len, enum chello_parse_result* result);
void ssl_chello_free(struct ssl_chello* chello);

109
inc/tsg_types.h
View File

@@ -1,109 +0,0 @@
#ifndef __TSG_TYPES_H__
#define __TSG_TYPES_H__
typedef enum _tsg_opt
{
LOG_OPT_HTTP_URL=1,
LOG_OPT_HTTP_HOST,
LOG_OPT_HTTP_REQUEST_LINE,
LOG_OPT_HTTP_RESPONSE_LINE,
LOG_OPT_HTTP_REQUEST_HEADER,
LOG_OPT_HTTP_RESPONSE_HEADER,
LOG_OPT_HTTP_REQUEST_BODY,
LOG_OPT_HTTP_RESPONSE_BODY,
LOG_OPT_HTTP_PROXY_FLAG,
LOG_OPT_HTTP_SEQUENCE,
LOG_OPT_HTTP_SNAPSHOT,
LOG_OPT_HTTP_COOKIE,
LOG_OPT_HTTP_REFERER,
LOG_OPT_HTTP_USER_AGENT,
LOG_OPT_HTTP_CONTENT_LENGTH,
LOG_OPT_HTTP_CONTENT_TYPE,
LOG_OPT_HTTP_SET_COOKIE,
LOG_OPT_HTTP_VERSION,
LOG_OPT_MAIL_PROTOCOL_TYPE,
LOG_OPT_MAIL_SENDER,
LOG_OPT_MAIL_RECEIVER,
LOG_OPT_MAIL_SUBJECT,
LOG_OPT_MAIL_CONTENT,
LOG_OPT_MAIL_ATTACHMENT_NAME,
LOG_OPT_MAIL_ATTACHMENT_CONTENT,
LOG_OPT_MAIL_EML_FILE,
LOG_OPT_MAIL_SNAPSHOT,
LOG_OPT_MAIL_SUBJECT_CHARSET,
LOG_OPT_DNS_MESSAGE_ID,
LOG_OPT_DNS_QR,
LOG_OPT_DNS_OPCODE,
LOG_OPT_DNS_AA,
LOG_OPT_DNS_TC,
LOG_OPT_DNS_RD,
LOG_OPT_DNS_RA,
LOG_OPT_DNS_RCODE,
LOG_OPT_DNS_QDCOUNT,
LOG_OPT_DNS_ANCOUNT,
LOG_OPT_DNS_NSCOUNT,
LOG_OPT_DNS_ARCOUNT,
LOG_OPT_DNS_QNAME,
LOG_OPT_DNS_QTYPE,
LOG_OPT_DNS_QCLASS,
LOG_OPT_DNS_CNAME,
LOG_OPT_DNS_SUB,
LOG_OPT_DNS_RR,
LOG_OPT_SSL_VERSION,
LOG_OPT_SSL_SNI,
LOG_OPT_SSL_SAN,
LOG_OPT_SSL_CN,
LOG_OPT_SSL_PINNINGST,
LOG_OPT_SSL_INTERCEPT_STATE,
LOG_OPT_SSL_SERVER_SIDE_LATENCY,
LOG_OPT_SSL_CLINET_SIDE_LATENCY,
LOG_OPT_SSL_SERVER_SIDE_VERSION,
LOG_OPT_SSL_CLIENT_SIDE_VERSION,
LOG_OPT_SSL_CERT_VERIFY,
LOG_OPT_SSL_ERROR,
LOG_OPT_SSL_CON_LATENCY_MS,
LOG_OPT_FTP_URL,
LOG_OPT_FTP_CONTENT,
LOG_OPT_BGP_TYPE,
LOG_OPT_BGP_AS_NUM,
LOG_OPT_BGP_ROUTE,
LOG_OPT_VOIP_CALLING_ACCOUNT,
LOG_OPT_VOIP_CALLED_ACCOUNT,
LOG_OPT_VOIP_CALLING_NUMBER,
LOG_OPT_VOIP_CALLED_NUMBER,
LOG_OPT_RADIUS_PACKET_TYPE,
LOG_OPT_RADIUS_NAS_IP,
LOG_OPT_RADIUS_FRAMED_IP,
LOG_OPT_RADIUS_ACCOUNT,
LOG_OPT_RADIUS_SEESION_TIMEOUT,
LOG_OPT_RADIUS_IDLE_TIMEOUT,
LOG_OPT_RADIUS_ACCT_STATUS_TYPE,
LOG_OPT_RADIUS_ACCT_TERMINATE_CAUSE,
LOG_OPT_MAX
}tsg_opt_t;
typedef enum _tsg_protocol
{
PROTO_IPv4,
PROTO_IPv6,
PROTO_TCP,
PROTO_UDP,
PROTO_HTTP,
PROTO_MAIL,
PROTO_DNS,
PROTO_FTP,
PROTO_SSL,
PROTO_SIP,
PROTO_BGP,
PROTO_STREAMING_MEDIA,
PROTO_MAX
}tsg_protocol_t;
#endif

4
src/CMakeLists.txt
View File

@@ -2,12 +2,12 @@ cmake_minimum_required(VERSION 2.8)
add_definitions(-fPIC)
set(SRC tsg_entry.cpp tsg_rule.cpp tsg_send_log.cpp)
set(SRC tsg_entry.cpp tsg_rule.cpp tsg_ssl_utils.cpp tsg_send_log.cpp)
include_directories(${CMAKE_SOURCE_DIR}/inc)
include_directories(/opt/MESA/include/MESA/)
set(TSG_MASTER_DEPEND_DYN_LIB MESA_handle_logger MESA_prof_load maatframe pthread MESA_field_stat2)
set(TSG_MASTER_DEPEND_DYN_LIB MESA_handle_logger MESA_prof_load maatframe pthread MESA_field_stat2 rdkafka cjson)
set(CMAKE_INSTALL_PREFIX /home/mesasoft/sapp_run)

364
src/tsg_entry.cpp
View File

@@ -1,6 +1,368 @@
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <assert.h>
char TSG_MASTER_VERSION_20191112=0;
#include <MESA/stream.h>
#include <MESA/MESA_prof_load.h>
#include <MESA/MESA_handle_logger.h>
#include "tsg_rule.h"
#include "tsg_entry.h"
#include "tsg_send_log.h"
#include "tsg_send_log_internal.h"
char TSG_MASTER_VERSION_20191129=0;
const char *tsg_conffile="tsgconf/main.conf";
g_tsg_para_t g_tsg_para;
id2field_t g_tsg_fs2_field[TSG_FS2_MAX]={{TLD_TYPE_UNKNOWN, TSG_FS2_LINKS, "links"},
{TLD_TYPE_UNKNOWN, TSG_FS2_BYPASS, "bypass"},
{TLD_TYPE_UNKNOWN, TSG_FS2_HIT_ADDR, "hit_addr"},
{TLD_TYPE_UNKNOWN, TSG_FS2_HIT_SHARE, "hit_share"},
{TLD_TYPE_UNKNOWN, TSG_FS2_INTERCEPT, "intercept"},
{TLD_TYPE_UNKNOWN, TSG_FS2_LOG, "log"},
{TLD_TYPE_UNKNOWN, TSG_FS2_DENY, "deny"}
};
static void free_policy_label(int thread_seq, void *project_req_value)
{
dictator_free(thread_seq, project_req_value);
project_req_value=NULL;
}
#if 0
static int is_ip_policy(Maat_rule_t *p_result, char *protocol, int len, int thread_seq)
{
int ret=0;
cJSON *item=NULL;
char *service_defined=NULL;
cJSON *user_define_object=NULL;
if(p_result->serv_def_len>MAX_SERVICE_DEFINE_LEN)
{
service_defined=dictator_malloc(thread_seq, p_result->serv_def_len+1);
ret=Maat_read_rule(g_tsg_maat_feather, p_result, MAAT_RULE_SERV_DEFINE, service_defined, p_result->serv_def_len+1);
assert(ret==p_result->serv_def_len+1);
user_define_object=cJSON_Parse(service_defined);
}
else
{
user_define_object=cJSON_Parse(p_result->service_defined);
}
if(user_define_object!=NULL)
{
item=cJSON_GetObjectItem(user_define_object, "protocol");
if(item!=NULL && item->valuestring!=NULL)
{
memcpy(protocol, item->valuestring, (len>strlen(item->valuestring)) ? strlen(item->valuestring): len);
}
item=cJSON_GetObjectItem(user_define_object, "method");
if((item==NULL) || ((strncasecmp(item->valuestring, "http", strlen(item->valuestring)))!=0 && (strncasecmp(item->valuestring, "ssl", strlen(item->valuestring)))!=0))
{
ret=1;
}
cJSON_Delete(user_define_object);
user_define_object=NULL;
}
if(service_defined!=NULL)
{
dictator_free(thread_seq, service_defined);
service_defined=NULL;
}
return ret;
}
#endif
static struct Maat_rule_t *tsg_policy_decision_criteria(Maat_rule_t *result, int result_num)
{
int i=0;
Maat_rule_t *p_result=NULL;
if(result==NULL || result_num<=0)
{
return NULL;
}
p_result=&result[0];
for(i=1; i<result_num; i++)
{
if((unsigned char)result[i].action>(unsigned char)p_result->action)
{
p_result=&result[i];
continue;
}
if(result[i].action==p_result->action)
{
if(result[i].config_id>p_result->config_id)
{
p_result=&result[i];
}
}
}
return p_result;
}
extern "C" char TSG_MASTER_TCP_ENTRY(struct streaminfo *a_tcp, void **pme, int thread_seq,void *a_packet)
{
int send_log=0,identify_flag=0;
int ret=0,hit_num=0;
int state=APP_STATE_DROPME;
scan_status_t mid=NULL;
char *domain_field_name=NULL;
char *schema_field_name=NULL;
Maat_rule_t *p_result=NULL;
Maat_rule_t *q_result=NULL;
tsg_log_t log_msg;
struct TLD_handle_t *TLD_handle=NULL;
struct _identify_info identify_info;
Maat_rule_t all_result[MAX_RESULT_NUM];
policy_priority_label_t *priority_label=NULL;
switch(a_tcp->opstate)
{
case OP_STATE_PENDING:
FS_operate(g_tsg_para.fs2_handle, g_tsg_para.fs2_field_id[TSG_FS2_LINKS], 0, FS_OP_ADD, 1);
ret=tsg_scan_nesting_addr(g_tsg_maat_feather, a_tcp, PROTO_MAX, &mid, all_result+hit_num, MAX_RESULT_NUM-hit_num);
if(ret>0)
{
hit_num+=ret;
q_result=tsg_policy_decision_criteria(all_result, hit_num);
FS_operate(g_tsg_para.fs2_handle, g_tsg_para.fs2_field_id[TSG_FS2_HIT_ADDR], 0, FS_OP_ADD, 1);
MESA_handle_runtime_log(g_tsg_para.logger,
RLOG_LV_DEBUG,
"SCAN_IP",
"Hit addr: %s scan ret: %d policy_id: %d service: %d action: %d",
printaddr(&a_tcp->addr, thread_seq),
ret,
q_result->config_id,
q_result->service_id,
q_result->action);
}
else
{
MESA_handle_runtime_log(g_tsg_para.logger, RLOG_LV_DEBUG, "SCAN_IP", "Not hit %s scan ret: %d",
printaddr(&a_tcp->addr, thread_seq), ret);
}
memset(&identify_info, 0, sizeof(identify_info));
ret=tsg_scan_shared_policy(g_tsg_maat_feather,
a_tcp->ptcpdetail->pdata,
a_tcp->ptcpdetail->datalen,
all_result+hit_num,
MAX_RESULT_NUM-hit_num,
&identify_info,
&mid,
g_tsg_para.logger,
thread_seq);
if(ret>0)
{
FS_operate(g_tsg_para.fs2_handle, g_tsg_para.fs2_field_id[TSG_FS2_HIT_SHARE], 0, FS_OP_ADD, 1);
MESA_handle_runtime_log(g_tsg_para.logger,
RLOG_LV_DEBUG,
"SCAN_FQDN",
"Hit %s: %s policy_id: %d service: %d action: %d addr: %s",
(identify_info.proto==PROTO_HTTP) ? "host" : "sni",
identify_info.domain,
all_result[hit_num].config_id,
all_result[hit_num].service_id,
all_result[hit_num].action,
printaddr(&a_tcp->addr, thread_seq)
);
hit_num+=ret;
identify_flag=1;
}
else
{
MESA_handle_runtime_log(g_tsg_para.logger,
RLOG_LV_DEBUG,
"SCAN_FQDN",
"Not hit %s: %s addr: %s",
(ret==-1) ? "NULL" : ((identify_info.proto==PROTO_HTTP) ? "host" : "sni"),
(ret==-1) ? "NULL" : identify_info.domain,
printaddr(&a_tcp->addr, thread_seq)
);
}
p_result=tsg_policy_decision_criteria(all_result, hit_num);
if(p_result!=NULL)
{
switch((unsigned char)p_result->action)
{
case TSG_ACTION_DENY:
send_log=1;
MESA_kill_tcp(a_tcp, a_packet);
state|=APP_STATE_DROPPKT|APP_STATE_KILL_OTHER;
FS_operate(g_tsg_para.fs2_handle, g_tsg_para.fs2_field_id[TSG_FS2_DENY], 0, FS_OP_ADD, 1);
MESA_handle_runtime_log(g_tsg_para.logger, RLOG_LV_DEBUG, "DENY", "Hit deny policy, policy_id: %d action: %d addr: %s",
p_result[0].config_id, p_result[0].action, printaddr(&a_tcp->addr, thread_seq));
break;
case TSG_ACTION_MONITOR:
if(q_result!=NULL && (p_result==q_result))
{
send_log=1;
}
break;
case TSG_ACTION_BYPASS:
send_log=1;
state|=APP_STATE_KILL_OTHER; //TODO
FS_operate(g_tsg_para.fs2_handle, g_tsg_para.fs2_field_id[TSG_FS2_BYPASS], 0, FS_OP_ADD, 1);
break;
case TSG_ACTION_INTERCEPT:
FS_operate(g_tsg_para.fs2_handle, g_tsg_para.fs2_field_id[TSG_FS2_INTERCEPT], 0, FS_OP_ADD, 1);
priority_label=(policy_priority_label_t *)dictator_malloc(thread_seq, sizeof(policy_priority_label_t));
priority_label->result_num=1;
priority_label->result_type=PULL_KNI_RESULT;
priority_label->proto=identify_info.proto;
priority_label->domain_len=identify_info.domain_len;
memcpy(priority_label->domain, identify_info.domain, identify_info.domain_len);
memcpy(priority_label->result, p_result, sizeof(struct Maat_rule_t));
ret=project_req_add_struct(a_tcp, g_tsg_para.priority_project_id, (void *)priority_label);
if(ret<0)
{
free_policy_label(thread_seq, (void *)priority_label);
MESA_handle_runtime_log(g_tsg_para.logger, RLOG_LV_FATAL, "PROJECT_ADD", "Add policy_priority_label failed ...");
}
MESA_handle_runtime_log(g_tsg_para.logger, RLOG_LV_DEBUG, "INTERCEPT", "Hit intercept policy, policy_id: %d action: %d addr: %s",
priority_label->result[0].config_id, priority_label->result[0].action, printaddr(&a_tcp->addr, thread_seq));
break;
case TSG_ACTION_NONE:
default:
assert(0);
break;
}
if(send_log==1 && p_result->do_log>0)
{
TLD_handle=TLD_create(thread_seq);
if(identify_flag==1)
{
schema_field_name=log_field_id2name(g_tsg_log_instance, LOG_COMMON_SCHAME_TYPE);
TLD_append(TLD_handle, schema_field_name, (void *)((identify_info.proto==PROTO_HTTP) ? "HTTP" : "SSL"), TLD_TYPE_STRING);
domain_field_name=log_field_id2name(g_tsg_log_instance, ((identify_info.proto==PROTO_HTTP) ? LOG_HTTP_HOST : LOG_SSL_SNI));
TLD_append(TLD_handle, domain_field_name, (void *)identify_info.domain, TLD_TYPE_STRING);
}
log_msg.a_stream=a_tcp;
log_msg.result=p_result;
log_msg.result_num=1;
tsg_send_log(g_tsg_log_instance, TLD_handle, &log_msg, thread_seq);
}
}
break;
case OP_STATE_DATA:
case OP_STATE_CLOSE:
default:
break;
}
return state;
}
extern "C" int TSG_MASTER_INIT()
{
int i=0,ret=0,cycle=0;
int value=0,level=30;
unsigned short fs_server_port=0;
char app_name[MAX_STRING_LEN]={0};
char log_path[MAX_STRING_LEN*4]={0};
char label_buff[MAX_STRING_LEN*4]={0};
char fs_server_ip[MAX_IPV4_LEN]={0};
char fs_output_path[MAX_STRING_LEN*4]={0};
memset(&g_tsg_para, 0, sizeof(g_tsg_para));
MESA_load_profile_int_def(tsg_conffile, "SYSTEM","LOG_LEVEL", &level, 30);
MESA_load_profile_string_def(tsg_conffile, "SYSTEM","LOG_PATH", log_path, sizeof(log_path), NULL);
g_tsg_para.logger=MESA_create_runtime_log_handle(log_path, level);
if(g_tsg_para.logger==NULL)
{
printf("MESA_create_runtime_log_handle failed ...\n");
return -1;
}
MESA_load_profile_int_def(tsg_conffile, "SYSTEM", "DEVICE_ID", &g_tsg_para.device_id, 0);
MESA_load_profile_string_def(tsg_conffile, "SYSTEM", "POLICY_PRIORITY_LABEL", label_buff, sizeof(label_buff), "POLICY_PRIORITY");
g_tsg_para.priority_project_id=project_producer_register(label_buff, PROJECT_VAL_TYPE_STRUCT, free_policy_label);
if(g_tsg_para.priority_project_id<0)
{
MESA_handle_runtime_log(g_tsg_para.logger, RLOG_LV_FATAL, "PROJECT_REGISTER", "Register %s failed ...", label_buff);
return -1;
}
ret=tsg_rule_init(tsg_conffile, g_tsg_para.logger);
if(ret<0)
{
MESA_handle_runtime_log(g_tsg_para.logger, RLOG_LV_FATAL, "INIT_MAAT", "tsg_rule_init failed ...");
return -1;
}
g_tsg_log_instance=tsg_sendlog_init(tsg_conffile);
if(g_tsg_log_instance==NULL)
{
MESA_handle_runtime_log(g_tsg_para.logger, RLOG_LV_FATAL, "INIT_SENDLOG", "tsg_sendlog_init failed ...");
return -1;
}
MESA_load_profile_int_def(tsg_conffile, "FIELD_STAT", "CYCLE", &cycle, 30);
MESA_load_profile_short_nodef(tsg_conffile, "FIELD_STAT","TELEGRAF_PORT", (short *)&(fs_server_port));
MESA_load_profile_string_nodef(tsg_conffile,"FIELD_STAT","TELEGRAF_IP",fs_server_ip, sizeof(fs_server_ip));
MESA_load_profile_string_def(tsg_conffile,"FIELD_STAT","OUTPUT_PATH",fs_output_path, sizeof(fs_output_path), "tsg_stat.log");
MESA_load_profile_string_def(tsg_conffile,"FIELD_STAT","APP_NAME", app_name, sizeof(app_name), "tsg_master");
g_tsg_para.fs2_handle=FS_create_handle();
value=1;//Rewrite
FS_set_para(g_tsg_para.fs2_handle, PRINT_MODE, &value, sizeof(value));
value=1;//Do not create stat thread
FS_set_para(g_tsg_para.fs2_handle, CREATE_THREAD, &value, sizeof(value));
FS_set_para(g_tsg_para.fs2_handle, STAT_CYCLE, &cycle, sizeof(cycle));
FS_set_para(g_tsg_para.fs2_handle, APP_NAME, app_name, strlen(app_name)+1);
FS_set_para(g_tsg_para.fs2_handle, OUTPUT_DEVICE, fs_output_path, strlen(fs_output_path)+1);
if(fs_server_port > 0 && strlen(fs_server_ip) > 0)
{
FS_set_para(g_tsg_para.fs2_handle, STATS_SERVER_IP,fs_server_ip, strlen(fs_server_ip)+1);
FS_set_para(g_tsg_para.fs2_handle, STATS_SERVER_PORT,&(fs_server_port), sizeof(fs_server_port));
}
for(i=0; i<TSG_FS2_MAX; i++)
{
g_tsg_para.fs2_field_id[g_tsg_fs2_field[i].id]=FS_register(g_tsg_para.fs2_handle, FS_STYLE_FIELD, FS_CALC_CURRENT, g_tsg_fs2_field[i].name);
}
FS_start(g_tsg_para.fs2_handle);
return 0;
}

52
src/tsg_entry.h Normal file
View File

@@ -0,0 +1,52 @@
#ifndef __TSG_ENTRY_H__
#define __TSG_ENTRY_H__
#include <MESA/Maat_rule.h>
#include <MESA/field_stat2.h>
#include "tsg_rule.h"
#define TSG_ACTION_NONE 0x00
#define TSG_ACTION_MONITOR 0x01
#define TSG_ACTION_INTERCEPT 0x02
#define TSG_ACTION_DENY 0x10
#define TSG_ACTION_MANIPULATE 0x30
#define TSG_ACTION_BYPASS 0x80
enum TSG_FS2_TYPE{
TSG_FS2_LINKS=0,
TSG_FS2_BYPASS,
TSG_FS2_HIT_ADDR,
TSG_FS2_HIT_SHARE,
TSG_FS2_INTERCEPT,
TSG_FS2_LOG,
TSG_FS2_DENY,
TSG_FS2_MAX
};
typedef struct _policy_priority_label
{
tsg_protocol_t proto; //enum _tsg_protocol (tsg_types.h)
int domain_len;
int result_type; //enum _PULL_RESULT_TYPE (tsg_rule.h)
int result_num;
char domain[MAX_DOAMIN_LEN];
Maat_rule_t result[MAX_RESULT_NUM];
}policy_priority_label_t;
typedef struct _tsg_para
{
int device_id;
int ip_addr_table_id;
int subscribe_id_table_id;
int priority_project_id;
int fs2_field_id[TSG_FS2_MAX];
void *logger;
screen_stat_handle_t fs2_handle;
}g_tsg_para_t;
extern g_tsg_para_t g_tsg_para;
#endif

386
src/tsg_rule.cpp
View File

@@ -1,18 +1,398 @@
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <assert.h>
#include <time.h>
#include <MESA/stream.h>
#include <MESA/MESA_prof_load.h>
#include "MESA/MESA_handle_logger.h"
#include "Maat_rule.h"
#include "Maat_command.h"
#include "MESA/http.h"
#include "tsg_ssl_utils.h"
#include "tsg_rule.h"
#include "tsg_entry.h"
extern Maat_feather_t g_tsg_maat_feather;
Maat_feather_t g_tsg_maat_feather;
int tsg_pull_policy_result(PULL_RESULT_TYPE pull_result_type, Maat_rule_t*result, int result_num)
#define MAX_PATH_LEN 1024
#ifndef MIN
#define MIN(a, b) (((a) < (b)) ? (a) : (b))
#endif
enum kni_scan_table{
TSG_FIELD_SSL_SNI,
TSG_FIELD_HTTP_HOST,
SCAN_TABLE_MAX
};
const char *g_kni_scan_table_name[SCAN_TABLE_MAX];
int g_kni_scan_tableid[SCAN_TABLE_MAX] = {0};
int tsg_rule_init(const char* conffile, void *logger)
{
unsigned short redis_port = 0;
int ret=0,scan_detail=0,effect_interval=60;
const char* instance_name="TSG",*module="MAAT";
int factor=0, redis_port_num=0,redis_index=0;
char redis_ip[16]={0}, effective_flag[1024]={0};
int maat_mode=0,maat_stat_on=0,maat_perf_on=0,thread_max=0;
char ip_addr_table[32]={0},subscriber_id_table[32]={0};
char json_cfg_file[MAX_PATH_LEN]={0},maat_stat_file[MAX_PATH_LEN]={0};
char table_info[MAX_PATH_LEN]={0},inc_cfg_dir[MAX_PATH_LEN]={0},ful_cfg_dir[MAX_PATH_LEN]={0};
memset(effective_flag, 0, sizeof(effective_flag));
MESA_load_profile_string_def(conffile,module,"EFFECTIVE_FLAG",effective_flag, sizeof(effective_flag),"");
MESA_load_profile_int_def(conffile, module,"MAAT_MODE", &(maat_mode),0);
MESA_load_profile_int_def(conffile, module,"STAT_SWITCH", &(maat_stat_on),1);
MESA_load_profile_int_def(conffile, module,"PERF_SWITCH", &(maat_perf_on),1);
MESA_load_profile_string_def(conffile,module,"TABLE_INFO",table_info, sizeof(table_info), "");
MESA_load_profile_string_def(conffile,module,"STAT_FILE",maat_stat_file, sizeof(maat_stat_file), "");
MESA_load_profile_int_def(conffile, module,"EFFECT_INTERVAL_S", &(effect_interval), 60);
effect_interval*=1000;//convert s to ms
thread_max=get_thread_count();
g_tsg_maat_feather=Maat_feather(thread_max, table_info, logger);
if(maat_mode==2)
{
MESA_load_profile_string_def(conffile,module,"REDIS_IP", redis_ip, sizeof(redis_ip),"");
MESA_load_profile_int_def(conffile, module,"REDIS_PORT_NUM", &(redis_port_num), 1);
MESA_load_profile_short_def(conffile, module,"REDIS_PORT", (short*)&(redis_port), 6379);
MESA_load_profile_int_def(conffile, module,"REDIS_INDEX", &redis_index, 0);
if(strlen(effective_flag)!=0)
{
Maat_set_feather_opt(g_tsg_maat_feather,MAAT_OPT_ACCEPT_TAGS,effective_flag, strlen(effective_flag)+1);
}
Maat_set_feather_opt(g_tsg_maat_feather, MAAT_OPT_EFFECT_INVERVAL_MS, &effect_interval, sizeof(effect_interval));
srand((unsigned int)time(NULL));
factor = rand()%redis_port_num;
redis_port = redis_port+factor;
Maat_set_feather_opt(g_tsg_maat_feather, MAAT_OPT_REDIS_IP, redis_ip, strlen(redis_ip)+1);
Maat_set_feather_opt(g_tsg_maat_feather, MAAT_OPT_REDIS_PORT, (void *)&redis_port, sizeof(redis_port));
Maat_set_feather_opt(g_tsg_maat_feather, MAAT_OPT_STAT_FILE_PATH, maat_stat_file, strlen(maat_stat_file)+1);
Maat_set_feather_opt(g_tsg_maat_feather, MAAT_OPT_STAT_ON, NULL, 0);
Maat_set_feather_opt(g_tsg_maat_feather, MAAT_OPT_PERF_ON, NULL, 0);
Maat_set_feather_opt(g_tsg_maat_feather, MAAT_OPT_REDIS_INDEX, &redis_index, sizeof(redis_index));
Maat_set_feather_opt(g_tsg_maat_feather, MAAT_OPT_SCAN_DETAIL, &scan_detail, sizeof(scan_detail));
//Maat_set_feather_opt(g_tsg_maat_feather, MAAT_OPT_DEFERRED_LOAD, NULL,0);
}
else
{
if(strlen(effective_flag)!=0)
{
ret=Maat_set_feather_opt(g_tsg_maat_feather,MAAT_OPT_ACCEPT_TAGS,effective_flag, strlen(effective_flag)+1);
assert(ret>=0);
}
Maat_set_feather_opt(g_tsg_maat_feather,MAAT_OPT_INSTANCE_NAME,instance_name, strlen(instance_name)+1);
if(maat_mode==1)
{
MESA_load_profile_string_def(conffile,module,"JSON_CFG_FILE",json_cfg_file, sizeof(json_cfg_file),"");
Maat_set_feather_opt(g_tsg_maat_feather, MAAT_OPT_JSON_FILE_PATH, json_cfg_file, strlen(json_cfg_file)+1);
}
else
{
MESA_load_profile_string_def(conffile,module,"INC_CFG_DIR",inc_cfg_dir, sizeof(inc_cfg_dir),"");
MESA_load_profile_string_def(conffile,module,"FULL_CFG_DIR",ful_cfg_dir, sizeof(ful_cfg_dir),"");
assert(strlen(inc_cfg_dir)!=0&&strlen(ful_cfg_dir)!=0);
Maat_set_feather_opt(g_tsg_maat_feather, MAAT_OPT_FULL_CFG_DIR, ful_cfg_dir, strlen(ful_cfg_dir)+1);
Maat_set_feather_opt(g_tsg_maat_feather, MAAT_OPT_INC_CFG_DIR, inc_cfg_dir, strlen(inc_cfg_dir)+1);
}
if(maat_stat_on)
{
Maat_set_feather_opt(g_tsg_maat_feather, MAAT_OPT_STAT_FILE_PATH, maat_stat_file, strlen(maat_stat_file)+1);
Maat_set_feather_opt(g_tsg_maat_feather, MAAT_OPT_STAT_ON, NULL, 0);
if(maat_perf_on)
{
Maat_set_feather_opt(g_tsg_maat_feather, MAAT_OPT_PERF_ON, NULL, 0);
}
}
Maat_set_feather_opt(g_tsg_maat_feather, MAAT_OPT_EFFECT_INVERVAL_MS, &effect_interval, sizeof(effect_interval));
Maat_set_feather_opt(g_tsg_maat_feather, MAAT_OPT_SCAN_DETAIL, &scan_detail, sizeof(scan_detail));
}
ret=Maat_initiate_feather(g_tsg_maat_feather);
if(ret<0)
{
return -1;
}
MESA_load_profile_string_def(conffile, module, "IP_ADDR_TABLE", ip_addr_table, sizeof(ip_addr_table), "TSG_OBJ_IP_ADDR");
MESA_load_profile_string_def(conffile, module, "SUBSCRIBER_ID_TABLE", subscriber_id_table, sizeof(subscriber_id_table), "TSG_OBJ_SUBSCRIBER_ID");
g_tsg_para.ip_addr_table_id=Maat_table_register(g_tsg_maat_feather, ip_addr_table);
if(g_tsg_para.ip_addr_table_id<0)
{
MESA_handle_runtime_log(logger, RLOG_LV_FATAL, "Maat_table_register %s failed", ip_addr_table);
return -1;
}
g_tsg_para.subscribe_id_table_id=Maat_table_register(g_tsg_maat_feather, subscriber_id_table);
if(g_tsg_para.subscribe_id_table_id<0)
{
MESA_handle_runtime_log(logger, RLOG_LV_FATAL, "RULE_INIT", "Maat_table_register %s failed", subscriber_id_table);
return -1;
}
ret=tsg_shared_table_init(conffile, g_tsg_maat_feather, logger);
if(ret<0)
{
MESA_handle_runtime_log(logger, RLOG_LV_FATAL, "RULE_INIT", "tsg_shared_table_init %s failed");
return -1;
}
return 0;
}
int tsg_scan_nesting_addr(Maat_feather_t maat_feather, struct streaminfo *a_stream, tsg_protocol_t proto, scan_status_t *mid, Maat_rule_t*result, int result_num)
static void protocol_identify(char *buff, int buff_len, struct _identify_info *result){
result->proto = PROTO_UNKONWN;
//http
char *host = NULL;
int ret = http_host_parser(buff, (uint32_t)buff_len, DIR_C2S, &host);
//printf("http_host_parse: ret = %d, buff_len = %d, buff = %s\n", ret, buff_len, buff);
if(ret >= 0){
result->proto = PROTO_HTTP;
if(ret == 0){
result->domain_len = 0;
}
else{
result->domain_len = MIN(ret, (int)sizeof(result->domain) - 1);
strncpy(result->domain, host, result->domain_len);
}
return;
}
//ssl
enum chello_parse_result chello_status = CHELLO_PARSE_INVALID_FORMAT;
struct ssl_chello *chello = NULL;
chello = ssl_chello_parse((const unsigned char*)buff, buff_len, &chello_status);
if(chello_status == CHELLO_PARSE_SUCCESS){
result->proto = PROTO_SSL;
if(chello->sni == NULL){
result->domain_len = 0;
}
else{
result->domain_len = strnlen(chello->sni, sizeof(result->domain) - 1);
strncpy(result->domain, chello->sni, result->domain_len);
}
}
ssl_chello_free(chello);
return;
}
//return -1 if failed, return 0 on success;
int tsg_shared_table_init(const char *conffile, Maat_feather_t maat_feather, void *logger){
g_tsg_maat_feather = maat_feather;
g_kni_scan_table_name[TSG_FIELD_HTTP_HOST] = "TSG_FIELD_HTTP_HOST";
g_kni_scan_table_name[TSG_FIELD_SSL_SNI] = "TSG_FIELD_SSL_SNI";
int i;
for(i = 0; i < SCAN_TABLE_MAX; i++){
g_kni_scan_tableid[i] = Maat_table_register(maat_feather, g_kni_scan_table_name[i]);
if(g_kni_scan_tableid[i] < 0){
MESA_handle_runtime_log(logger, RLOG_LV_FATAL, "Failed at Maat_table_register, tablename = %s, ret = %d",
g_kni_scan_table_name[i], g_kni_scan_tableid[i]);
return -1;
}
}
return 0;
}
int tsg_pull_policy_result(struct streaminfo *a_stream, PULL_RESULT_TYPE pull_result_type, Maat_rule_t*result, int result_num, struct _identify_info *identify_info)
{
int num=0;
policy_priority_label_t *label=NULL;
label=(policy_priority_label_t *)project_req_get_struct(a_stream, g_tsg_para.priority_project_id);
if(label!=NULL && result!=NULL && result_num>0 && identify_info!=NULL)
{
if(label->result_type==pull_result_type)
{
num=(label->result_num>result_num) ? result_num : label->result_num;
memcpy(result, label->result, num*sizeof(Maat_rule_t));
memcpy(identify_info->domain, label->domain, label->domain_len);
identify_info->domain_len=label->domain_len;
identify_info->proto = label->proto;
return num;
}
else
{
MESA_handle_runtime_log(g_tsg_para.logger, RLOG_LV_DEBUG, "PULL_RESULT",
"pull policy failed, hit: %s %s: %s policy_id: %d service: %d action: %d addr: %s",
(label->result_type==PULL_KNI_RESULT) ? "KNI" : "FW",
label->proto==PROTO_HTTP ? "host" : "sni",
label->domain,
label->result->config_id,
label->result->service_id,
label->result->action,
printaddr(&a_stream->addr, a_stream->threadnum));
}
}
else
{
MESA_handle_runtime_log(g_tsg_para.logger, RLOG_LV_DEBUG, "PULL_RESULT",
"pull policy failed, Not hit, label is %s addr: %s",
(label==NULL) ? "NULL" : label->domain,
printaddr(&a_stream->addr, a_stream->threadnum));
}
return 0;
}
int tsg_scan_nesting_addr(Maat_feather_t maat_feather, const struct streaminfo *a_stream, tsg_protocol_t proto, scan_status_t *mid, Maat_rule_t*result, int result_num)
{
struct ipaddr t_addr;
struct ipaddr* p_addr = NULL;
char subscribe_id[64]={0};
int hit_num=0,tans_proto=0;
int is_scan_addr=1, maat_ret=0,found_pos=0;
const struct streaminfo *cur_stream = a_stream;
if(result == NULL || result_num <= 0 || a_stream == NULL || maat_feather == NULL)
{
return -1;
}
do
{
if(cur_stream->addr.addrtype == __ADDR_TYPE_IP_PAIR_V4 || cur_stream->addr.addrtype == ADDR_TYPE_IPV4 || cur_stream->addr.addrtype == __ADDR_TYPE_IP_PAIR_V6 || cur_stream->addr.addrtype == ADDR_TYPE_IPV6)
{
is_scan_addr = 1;
if(cur_stream->addr.addrtype == __ADDR_TYPE_IP_PAIR_V4 || cur_stream->addr.addrtype == __ADDR_TYPE_IP_PAIR_V6)
{
memcpy(&t_addr, &cur_stream->addr, sizeof(t_addr));
if(cur_stream->addr.addrtype == __ADDR_TYPE_IP_PAIR_V4)
t_addr.addrtype = ADDR_TYPE_IPV4;
else
t_addr.addrtype = ADDR_TYPE_IPV6;
p_addr = &t_addr;
}
else
{
p_addr = (struct ipaddr *)&cur_stream->addr;
}
}
else
{
is_scan_addr = 0;
p_addr = NULL;
}
if(is_scan_addr==1 && p_addr!=NULL)
{
switch(cur_stream->type)
{
case STREAM_TYPE_TCP:
tans_proto=6;
break;
case STREAM_TYPE_UDP:
tans_proto=17;
break;
default:
tans_proto=255;
break;
}
maat_ret=Maat_scan_proto_addr(maat_feather,
g_tsg_para.ip_addr_table_id,
p_addr,
tans_proto,
result+hit_num,
result_num-hit_num,
mid,
cur_stream->threadnum);
if(maat_ret > 0)
{
hit_num+=maat_ret;
}
}
cur_stream = cur_stream->pfather;
}while(cur_stream != NULL && hit_num < result_num);
if(hit_num<result_num && subscribe_id!=NULL && strlen(subscribe_id)>0)
{
maat_ret=Maat_full_scan_string(maat_feather,
g_tsg_para.subscribe_id_table_id,
CHARSET_GBK,
subscribe_id,
strlen(subscribe_id),
result+hit_num,
&found_pos,
result_num-hit_num,
mid,
a_stream->threadnum);
if(maat_ret > 0)
{
hit_num+=maat_ret;
}
}
return hit_num;
}
//return value: -1: failed, 0: not hit, >0: hit count
int tsg_scan_shared_policy(Maat_feather_t maat_feather, void *pkt, int pkt_len, Maat_rule_t *result, int result_num,
struct _identify_info *identify_info, scan_status_t *mid, void *logger, int thread_seq)
{
memset(identify_info, 0, sizeof(*identify_info));
protocol_identify((char*)pkt, pkt_len, identify_info);
if(identify_info->proto != PROTO_SSL && identify_info->proto != PROTO_HTTP){
return -1;
}
int tableid;
if(identify_info->proto == PROTO_SSL){
tableid = g_kni_scan_tableid[TSG_FIELD_SSL_SNI];
}
else{
tableid = g_kni_scan_tableid[TSG_FIELD_HTTP_HOST];
}
return Maat_full_scan_string(g_tsg_maat_feather, tableid, CHARSET_UTF8, identify_info->domain, identify_info->domain_len,
result, NULL, result_num, mid, thread_seq);
}
struct Maat_rule_t *tsg_fetch_deny_rule(Maat_rule_t *result, int result_num)
{
int i=0;
Maat_rule_t *p_result=NULL;
for(i=0; i< result_num; i++)
{
if(result[i].action==TSG_ACTION_DENY || result[i].action==TSG_ACTION_BYPASS)
{
if(p_result==NULL)
{
p_result=&result[i];
continue;
}
if(result[i].action > p_result->action)
{
p_result=&result[i];
continue;
}
if((result[i].action==p_result->action) && (result[i].config_id > p_result->config_id))
{
p_result=&result[i];
}
}
}
return p_result;
}

531
src/tsg_send_log.cpp
View File

@@ -2,6 +2,7 @@
#include <string.h>
#include <stdlib.h>
#include <assert.h>
#include <time.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <arpa/inet.h>
@@ -11,100 +12,502 @@
#include <unistd.h>
#include <MESA/stream.h>
#include <MESA/cabot_send_log.h>
#include <MESA/MESA_prof_load.h>
#include <MESA/MESA_handle_logger.h>
#include "tsg_types.h"
#include "tsg_entry.h"
#include "tsg_send_log.h"
#include "tsg_send_log_internal.h"
tsg_logger_info_t tsg_logger_info;
char TSG_SEND_LOG_VERSION_20191129=0;
struct tsg_log_instance_t *g_tsg_log_instance;
static unsigned int get_ip_by_eth_name(const char *ifname)
const id2field_t tld_type[TLD_TYPE_MAX]={{TLD_TYPE_UNKNOWN, TLD_TYPE_UNKNOWN, "UNKOWN"},
{TLD_TYPE_LONG, TLD_TYPE_LONG, "LONG"},
{TLD_TYPE_STRING, TLD_TYPE_STRING, "STRING"},
{TLD_TYPE_FILE, TLD_TYPE_FILE, "FILE"}
};
extern "C" int MESA_get_dev_ipv4(const char *device, int *ip_add);
unsigned long long tsg_get_stream_id(struct streaminfo * a_stream)
{
int sockfd;
struct ifreq ifr;
unsigned int ip;
int ret=0;
int device_id_size=sizeof(unsigned long long);
unsigned long long device_id=(unsigned long long)g_tsg_para.device_id;
sockfd = socket(AF_INET, SOCK_DGRAM, 0);
if (-1 == sockfd) {
goto error;
}
ret=MESA_get_stream_opt(a_stream, MSO_GLOBAL_STREAM_ID, (void *)&device_id, &device_id_size);
if(ret==0)
{
return device_id;
}
strcpy(ifr.ifr_name,ifname);
if (ioctl(sockfd, SIOCGIFADDR, &ifr) < 0) {
goto error;
}
ip = ((struct sockaddr_in*)&(ifr.ifr_addr))->sin_addr.s_addr;
close(sockfd);
return ip;
error:
close(sockfd);
return INADDR_NONE;
return -1;
}
int steaminfo2opt()
int TLD_cancel(struct TLD_handle_t *handle)
{
struct TLD_handle_t *_handle=handle;
if(_handle!=NULL)
{
if(_handle->object!=NULL)
{
cJSON_Delete(_handle->object);
_handle->object=NULL;
}
free(handle);
handle=NULL;
}
return 0;
}
int tsg_sendlog_init(char *filename)
int TLD_delete(struct TLD_handle_t *handle, char *key)
{
int ret=0;
unsigned int local_ip_nr=0;
char nic_name[32];
struct TLD_handle_t *_handle=handle;
memset(&tsg_logger_info, 0, sizeof(tsg_logger_info));
MESA_load_profile_int_def(filename, "TSG_LOG", "MODE",&(tsg_logger_info.mode), 0);
MESA_load_profile_string_def(filename, "TSG_LOG", "FIELD_FILE", tsg_logger_info.field_file, sizeof(tsg_logger_info.field_file), NULL);
MESA_load_profile_string_def(filename, "TSG_LOG", "BROKER_LIST", tsg_logger_info.broker_list, sizeof(tsg_logger_info.broker_list), NULL);
MESA_load_profile_int_def(filename, "TSG_LOG", "LEVEL",&(tsg_logger_info.level), 30);
MESA_load_profile_string_def(filename, "TSG_LOG", "LOG_PATH", tsg_logger_info.log_path, sizeof(tsg_logger_info.log_path), NULL);
tsg_logger_info.logger=MESA_create_runtime_log_handle(tsg_logger_info.log_path, tsg_logger_info.level);
if(tsg_logger_info.logger==NULL)
if(_handle!=NULL && key!=NULL)
{
cJSON_DeleteItemFromObject(_handle->object, key);
}
return 0;
}
int TLD_append(struct TLD_handle_t *handle, char *key, void *value, TLD_TYPE type)
{
struct TLD_handle_t *_handle=handle;
if(_handle==NULL || key==NULL || (value==NULL && type!=TLD_TYPE_LONG))
{
printf("MESA_create_runtime_log_handle failed ..., path: %s level: %d", tsg_logger_info.log_path, tsg_logger_info.level);
return -1;
}
tsg_logger_info.cabot_handle=cabot_sendlog_create();
cabot_sendlog_set(tsg_logger_info.cabot_handle, SENDLOG_MODE,(void *)&(tsg_logger_info.mode));
cabot_sendlog_set(tsg_logger_info.cabot_handle, CONFIG_FILE,(void *)tsg_logger_info.field_file);
cabot_sendlog_set(tsg_logger_info.cabot_handle, BROKER_LIST,(void *)&(tsg_logger_info.mode));
ret=cabot_sendlog_init(tsg_logger_info.cabot_handle, tsg_logger_info.logger);
if(ret<0)
switch(type)
{
MESA_handle_runtime_log(tsg_logger_info.logger, RLOG_LV_FATAL, "CABOT_INIT", "cabot_sendlog_init failed ...");
return -2;
case TLD_TYPE_LONG:
cJSON_AddNumberToObject(_handle->object, key, (long)value);
break;
case TLD_TYPE_FILE:
break;
case TLD_TYPE_STRING:
cJSON_AddStringToObject(_handle->object, key, (char *)value);
break;
default:
return -1;
break;
}
MESA_load_profile_string_def(filename, "TSG_LOG", "NIC_NAME", nic_name, sizeof(nic_name), "eth0");
local_ip_nr=get_ip_by_eth_name(nic_name);
if(local_ip_nr==INADDR_NONE)
{
MESA_handle_runtime_log(tsg_logger_info.logger,RLOG_LV_FATAL, "GET_LOCAL_IP","get NIC_NAME: %s error.", nic_name);
return -3;
}
inet_ntop(AF_INET,&(local_ip_nr),tsg_logger_info.local_ip_str,sizeof(tsg_logger_info.local_ip_str));
//maat
return 0;
}
void tsg_send_log(const tsg_log_t* log_msg, struct _opt_unit_t* log_opt, int opt_num, int thread_id)
struct TLD_handle_t *TLD_create(int thread_id)
{
//struct _tld_handle *_handle=(struct _tld_handle *)dictator_malloc(thread_id, sizeof(struct _tld_handle));
struct TLD_handle_t *_handle=(struct TLD_handle_t *)calloc(1, sizeof(struct TLD_handle_t));
_handle->thread_id = thread_id;
_handle->object = cJSON_CreateObject();
return _handle;
}
char *log_field_id2name(struct tsg_log_instance_t *instance, tsg_log_field_id_t id)
{
struct tsg_log_instance_t *_instance=instance;
if(_instance!=NULL)
{
return _instance->id2field[id].name;
}
return NULL;
}
int TLD_append_streaminfo(struct tsg_log_instance_t *instance, struct TLD_handle_t *handle, struct streaminfo *a_stream)
{
int ret=0,addr_type=0;
unsigned short tunnel_type=0;
char nest_addr_buf[1024];
char *addr_proto=NULL;
time_t cur_time;
long common_con_duration_ms=0;
unsigned long long stream_id=0;
unsigned short c_port=0, s_port=0;
int tunnel_type_size=sizeof(tunnel_type);
struct layer_addr_ipv4 *ipv4=NULL;
struct layer_addr_ipv6 *ipv6=NULL;
char server_ip[MAX_IPV4_LEN*8]={0};
char client_ip[MAX_IPV4_LEN*8]={0};
struct TLD_handle_t *_handle=handle;
struct tsg_log_instance_t *_instance=instance;
if(_instance==NULL || _handle==NULL || a_stream==NULL)
{
MESA_handle_runtime_log(_instance->logger,
((a_stream==NULL) ? RLOG_LV_DEBUG: RLOG_LV_FATAL),
"TLD_APPEND_STREAM",
"instance==NULL || TLD_handle==NULL || addr: %s",
((a_stream==NULL) ? "NULL": (printaddr(&a_stream->addr, a_stream->threadnum)))
);
return -1;
}
switch(a_stream->addr.addrtype)
{
case ADDR_TYPE_IPV4:
case __ADDR_TYPE_IP_PAIR_V4:
ipv4=a_stream->addr.ipv4;
addr_type=4;
c_port=ntohs(ipv4->source);
s_port=ntohs(ipv4->dest);
inet_ntop(AF_INET, (void *)&ipv4->saddr, client_ip, sizeof(client_ip));
inet_ntop(AF_INET, (void *)&ipv4->daddr, server_ip, sizeof(server_ip));
break;
case ADDR_TYPE_IPV6:
case __ADDR_TYPE_IP_PAIR_V6:
ipv6=a_stream->addr.ipv6;
addr_type=6;
c_port=ntohs(ipv6->source);
s_port=ntohs(ipv6->dest);
inet_ntop(AF_INET6, (void *)ipv6->saddr, client_ip, sizeof(client_ip));
inet_ntop(AF_INET6, (void *)ipv6->daddr, server_ip, sizeof(server_ip));
break;
default:
break;
}
TLD_append(_handle, _instance->id2field[LOG_COMMON_SERVER_IP].name, (void *)server_ip, TLD_TYPE_STRING);
TLD_append(_handle, _instance->id2field[LOG_COMMON_CLIENT_IP].name, (void *)client_ip, TLD_TYPE_STRING);
TLD_append(_handle, _instance->id2field[LOG_COMMON_SERVER_PORT].name, (void *)(long)s_port, TLD_TYPE_LONG);
TLD_append(_handle, _instance->id2field[LOG_COMMON_CLIENT_PORT].name, (void *)(long)c_port, TLD_TYPE_LONG);
TLD_append(_handle, _instance->id2field[LOG_COMMON_STREAM_DIR].name, (void *)(long)a_stream->dir, TLD_TYPE_LONG);
TLD_append(_handle, _instance->id2field[LOG_COMMON_ADDRESS_TYPE].name, (void *)(long)addr_type, TLD_TYPE_LONG);
TLD_append(_handle, _instance->id2field[LOG_COMMON_S2C_PKT_NUM].name, (void *)(long)a_stream->ptcpdetail->clientpktnum, TLD_TYPE_LONG);
TLD_append(_handle, _instance->id2field[LOG_COMMON_S2C_BYTE_NUM].name, (void *)(long)a_stream->ptcpdetail->clientbytes, TLD_TYPE_LONG);
TLD_append(_handle, _instance->id2field[LOG_COMMON_C2S_PKT_NUM].name, (void *)(long)a_stream->ptcpdetail->serverpktnum, TLD_TYPE_LONG);
TLD_append(_handle, _instance->id2field[LOG_COMMON_C2S_BYTE_NUM].name, (void *)(long)a_stream->ptcpdetail->serverbytes, TLD_TYPE_LONG);
if(a_stream!=NULL && a_stream->ptcpdetail!=NULL)
{
TLD_append(_handle, _instance->id2field[LOG_COMMON_START_TIME].name, (void *)(a_stream->ptcpdetail->createtime), TLD_TYPE_LONG);
TLD_append(_handle, _instance->id2field[LOG_COMMON_END_TIME].name, (void *)(a_stream->ptcpdetail->lastmtime), TLD_TYPE_LONG);
common_con_duration_ms=(a_stream->ptcpdetail->lastmtime-a_stream->ptcpdetail->createtime)*1000;
TLD_append(_handle, _instance->id2field[LOG_COMMON_CON_DURATION_MS].name, (void *)(common_con_duration_ms), TLD_TYPE_LONG);
}
else
{
cur_time=time(NULL);
TLD_append(_handle, _instance->id2field[LOG_COMMON_START_TIME].name, (void *)cur_time, TLD_TYPE_LONG);
TLD_append(_handle, _instance->id2field[LOG_COMMON_END_TIME].name, (void *)cur_time, TLD_TYPE_LONG);
TLD_append(_handle, _instance->id2field[LOG_COMMON_CON_DURATION_MS].name, (void *)(common_con_duration_ms), TLD_TYPE_LONG);
}
stream_id=tsg_get_stream_id(a_stream);
TLD_append(_handle, _instance->id2field[LOG_COMMON_STREAM_TRACE_ID].name, (void *)(long)stream_id, TLD_TYPE_LONG);
addr_proto=(char *)layer_addr_prefix_ntop(a_stream);
TLD_append(_handle, _instance->id2field[LOG_COMMON_L4_PROTOCOL].name, (void *)addr_proto, TLD_TYPE_STRING);
ret=MESA_get_stream_opt(a_stream, MSO_STREAM_TUNNEL_TYPE, &tunnel_type, &tunnel_type_size);
assert(ret==0);
if(tunnel_type==STREAM_TUNNLE_NON)
{
layer_addr_ntop_r(a_stream,nest_addr_buf, sizeof(nest_addr_buf));
}
else
{
stream_addr_list_ntop(a_stream,nest_addr_buf, sizeof(nest_addr_buf));
}
TLD_append(_handle, _instance->id2field[LOG_COMMON_ADDRESS_LIST].name, (void *)nest_addr_buf, TLD_TYPE_STRING);
return 0;
}
static int load_log_common_field(const char *filename, id2field_t *id2field, id2field_t *service2topic)
{
int i=0;
int ret=0,id=0;
FILE *fp=NULL;
char line[1024]={0};
char field_name[64]={0};
char type_name[32]={0};
fp=fopen(filename, "r");
if(fp==NULL)
{
printf("Open %s failed ...", filename);
return -1;
}
memset(line, 0, sizeof(line));
while((fgets(line, sizeof(line), fp))!=NULL)
{
if(line[0]=='#' || line[0]=='\n' || line[0]=='\r' ||line[0]=='\0')
{
continue;
}
memset(type_name, 0, sizeof(type_name));
ret=sscanf(line, "%s %s %d", type_name, field_name, &id);
assert(ret==3 && id<LOG_COMMON_MAX);
for(i=0; i<LOG_COMMON_MAX; i++)
{
if((strncasecmp(tld_type[i].name, type_name, strlen(tld_type[i].name)))==0)
{
switch(tld_type[i].type)
{
case TLD_TYPE_FILE:
case TLD_TYPE_LONG:
case TLD_TYPE_STRING:
id2field[id].type = tld_type[i].type;
id2field[id].id = id;
memcpy(id2field[id].name, field_name, strlen(field_name));
break;
default:
if((strncasecmp("TOPIC", type_name, strlen("TOPIC")))==0)
{
service2topic[id].type = TLD_TYPE_MAX;
service2topic[id].id = id;
memcpy(service2topic[id].name, field_name, strlen(field_name));
}
break;
}
}
}
memset(line, 0, sizeof(line));
}
fclose(fp);
fp=NULL;
return 0;
}
struct tsg_log_instance_t *tsg_sendlog_init(const char *conffile)
{
int i=0,ret=0,level=30;
char nic_name[32]={0};
char kafka_errstr[1024]={0};
unsigned int local_ip_nr=0;
char log_path[MAX_STRING_LEN*4]={0};
rd_kafka_t *kafka_handle = NULL;
rd_kafka_conf_t *rdkafka_conf = NULL;
rd_kafka_topic_conf_t *topic_conf;
struct tsg_log_instance_t *_instance=NULL;
_instance=(struct tsg_log_instance_t *)calloc(1, sizeof(struct tsg_log_instance_t));
MESA_load_profile_int_def(conffile, "TSG_LOG", "MODE",&(_instance->mode), 0);
MESA_load_profile_string_def(conffile, "TSG_LOG", "COMMON_FIELD_FILE", _instance->common_field_file, sizeof(_instance->common_field_file), NULL);
MESA_load_profile_string_def(conffile, "TSG_LOG", "BROKER_LIST", _instance->broker_list, sizeof(_instance->broker_list), NULL);
MESA_load_profile_int_def(conffile, "TSG_LOG", "LOG_LEVEL",&(level), 30);
MESA_load_profile_string_def(conffile, "TSG_LOG", "LOG_PATH", log_path, sizeof(log_path), NULL);
_instance->logger=MESA_create_runtime_log_handle(log_path, level);
if(_instance->logger==NULL)
{
printf("MESA_create_runtime_log_handle failed ..., path: %s level: %d", log_path, level);
return NULL;
}
MESA_load_profile_string_def(conffile, "TSG_LOG", "NIC_NAME", nic_name, sizeof(nic_name), "eth0");
ret=MESA_get_dev_ipv4(nic_name, (int *)&local_ip_nr);
if(ret<0)
{
MESA_handle_runtime_log(_instance->logger, RLOG_LV_FATAL, "GET_LOCAL_IP", "MESA_get_dev_ipv4 is error, nic_name: %s", nic_name);
return NULL;
}
inet_ntop(AF_INET,&(local_ip_nr),_instance->local_ip_str,sizeof(_instance->local_ip_str));
rdkafka_conf = rd_kafka_conf_new();
rd_kafka_conf_set(rdkafka_conf, "queue.buffering.max.messages", "1000000", kafka_errstr, sizeof(kafka_errstr));
rd_kafka_conf_set(rdkafka_conf, "topic.metadata.refresh.interval.ms", "600000",kafka_errstr, sizeof(kafka_errstr));
rd_kafka_conf_set(rdkafka_conf, "request.required.acks", "1", kafka_errstr, sizeof(kafka_errstr));
if(!(kafka_handle=rd_kafka_new(RD_KAFKA_PRODUCER, rdkafka_conf, kafka_errstr, sizeof(kafka_errstr))))
{
MESA_handle_runtime_log(_instance->logger, RLOG_LV_FATAL, "KAFKA_INIT", "rd_kafka_new is error");
return NULL;
}
if(rd_kafka_brokers_add(kafka_handle, _instance->broker_list) == 0)
{
MESA_handle_runtime_log(_instance->logger, RLOG_LV_FATAL, "KAFKA_INIT", "rd_kafka_brokers_add is error, broker_list: %s", _instance->broker_list);
return NULL;
}
MESA_load_profile_int_def(conffile, "TSG_LOG", "MAX_SERVICE",&(_instance->max_service), 0);
(_instance->topic_rkt)=(rd_kafka_topic_t **)calloc(1, sizeof(rd_kafka_topic_t*));
*(_instance->topic_rkt)=(rd_kafka_topic_t *)calloc(1, (1+_instance->max_service)*sizeof(rd_kafka_topic_t*));
_instance->service2topic=(id2field_t *)calloc(1, (1+_instance->max_service)*sizeof(id2field_t));
load_log_common_field(_instance->common_field_file, _instance->id2field, _instance->service2topic);
for(i=0; i<_instance->max_service+1; i++)
{
if(_instance->service2topic[i].type==TLD_TYPE_MAX)
{
topic_conf=rd_kafka_topic_conf_new();
_instance->topic_rkt[_instance->service2topic[i].id]=rd_kafka_topic_new(kafka_handle, _instance->service2topic[i].name, topic_conf);
}
}
return _instance;
}
int tsg_send_log(struct tsg_log_instance_t *instance, struct TLD_handle_t *handle, tsg_log_t *log_msg, int thread_id)
{
int i=0,ret=0,status=0;
char *payload=NULL;
struct TLD_handle_t *_handle=handle;
struct tsg_log_instance_t *_instance=instance;
if(_instance==NULL || _handle==NULL || log_msg==NULL)
{
TLD_cancel(handle);
MESA_handle_runtime_log(_instance->logger, RLOG_LV_FATAL, "TSG_SEND_LOG", " instance==NULL || TLD_handle==NULL || log_msg==NULL ");
return -1;
}
if(_instance->mode==CLOSE)
{
TLD_cancel(handle);
MESA_handle_runtime_log(_instance->logger, RLOG_LV_INFO, "TSG_SEND_LOG", "Disable tsg_send_log.");
return 0;
}
//TODO
//common_user_tags
//common_isp
//common_app_label
//common_app_id
//common_protocol_id
//common_has_dup_traffic
//common_stream_error
TLD_append_streaminfo(instance, handle, log_msg->a_stream);
TLD_append(_handle, _instance->id2field[LOG_COMMON_SLED_IP].name, (void *)(_instance->local_ip_str), TLD_TYPE_STRING);
#if 0
struct vxlan_info vinfo;
int opt_val_len = sizeof(vinfo);
status=MESA_get_stream_opt(log_msg->a_stream, MSO_STREAM_VXLAN_INFO, &vinfo, &opt_val_len);
if(status < 0)
{
MESA_handle_runtime_log(_instance->logger, RLOG_LV_DEBUG, "TSG_SEND_LOG", "tsg log: get vxlan info error, tuple4: %s", printaddr(&log_msg->a_stream->addr, thread_id));
}
else
{
TLD_append((TLD_handle_t)_handle, _instance->id2field[LOG_COMMON_LINK_ID].name, (void *)(long)vinfo.link_id, TLD_TYPE_LONG);
TLD_append((TLD_handle_t)_handle, _instance->id2field[LOG_COMMON_DIRECTION].name, (void *)(long)vinfo.link_dir, TLD_TYPE_LONG);
TLD_append((TLD_handle_t)_handle, _instance->id2field[LOG_COMMON_DEVICE_ID].name, (void *)(long)vinfo.dev_id, TLD_TYPE_LONG);
TLD_append((TLD_handle_t)_handle, _instance->id2field[LOG_COMMON_ENTRANCE_ID].name, (void *)(long)vinfo.entrance_id, TLD_TYPE_LONG);
TLD_append((TLD_handle_t)_handle, _instance->id2field[LOG_COMMON_ENCAPSULATION].name, (void *)(long)vinfo.encap_type, TLD_TYPE_LONG);
}
#endif
for(i=0;i<log_msg->result_num; i++)
{
switch(log_msg->result[i].do_log)
{
case LOG_ABORT:
MESA_handle_runtime_log(_instance->logger, RLOG_LV_INFO,
"TSG_SEND_LOG",
"tsg abort log:cfg_id=%d service=%d addr=%s",
log_msg->result[i].config_id,
log_msg->result[i].service_id,
printaddr(&(log_msg->a_stream->addr), thread_id));
continue;
break;
case LOG_ALL:
break;
case LOG_NOFILE:
break;
default:
break;
}
TLD_append(_handle, _instance->id2field[LOG_COMMON_POLICY_ID].name, (void *)(long)(log_msg->result[i].config_id), TLD_TYPE_LONG);
TLD_append(_handle, _instance->id2field[LOG_COMMON_SERVICE].name, (void *)(long)(log_msg->result[i].service_id), TLD_TYPE_LONG);
TLD_append(_handle, _instance->id2field[LOG_COMMON_ACTION].name, (void *)(long)((unsigned char)log_msg->result[i].action), TLD_TYPE_LONG);
if(log_msg->result[i].serv_def_len<128)
{
TLD_append(_handle, _instance->id2field[LOG_COMMON_USER_REGION].name, (void *)(log_msg->result[i].service_defined), TLD_TYPE_STRING);
}
else
{
char *service_defined=(char *)calloc(1, log_msg->result[i].serv_def_len+1);
ret=Maat_read_rule(g_tsg_maat_feather, &log_msg->result[i], MAAT_RULE_SERV_DEFINE, service_defined, log_msg->result[i].serv_def_len);
if(ret==log_msg->result[i].serv_def_len)
{
TLD_append(_handle, _instance->id2field[LOG_COMMON_USER_REGION].name, (void *)service_defined, TLD_TYPE_STRING);
}
else
{
MESA_handle_runtime_log(_instance->logger,
RLOG_LV_FATAL,
"TSG_SEND_LOG",
"Fetch service_defined failed, policy_id: %d service: %d action: %d addr: %s",
log_msg->result[i].config_id,
log_msg->result[i].service_id,
log_msg->result[i].action,
printaddr(&log_msg->a_stream->addr, thread_id));
}
free((void *)service_defined);
service_defined=NULL;
}
payload = cJSON_PrintUnformatted(_handle->object);
status = rd_kafka_produce(_instance->topic_rkt[log_msg->result[i].service_id], RD_KAFKA_PARTITION_UA, RD_KAFKA_MSG_F_COPY, payload, strlen(payload), NULL, 0, NULL);
if(status < 0)
{
MESA_handle_runtime_log(_instance->logger, RLOG_LV_INFO, "TSG_SEND_LOG",
"tsg_send_log to kafka is error, status: %d, topic: %s payload: %s",
status, _instance->service2topic[log_msg->result[i].service_id].name, payload);
}
else
{
MESA_handle_runtime_log(_instance->logger,RLOG_LV_INFO, "TSG_SEND_LOG",
"log send successfully %s: %s", _instance->service2topic[log_msg->result[i].service_id].name, payload);
}
free(payload);
payload=NULL;
TLD_delete(_handle, _instance->id2field[LOG_COMMON_POLICY_ID].name);
TLD_delete(_handle, _instance->id2field[LOG_COMMON_SERVICE].name);
TLD_delete(_handle, _instance->id2field[LOG_COMMON_ACTION].name);
TLD_delete(_handle, _instance->id2field[LOG_COMMON_USER_REGION].name);
FS_operate(g_tsg_para.fs2_handle, g_tsg_para.fs2_field_id[TSG_FS2_LOG], 0, FS_OP_ADD, 1);
}
TLD_cancel(handle);
return 0;
}

94
src/tsg_send_log_internal.h
View File

@@ -1,17 +1,95 @@
#ifndef __TSG_SEND_LOG_INTERNAL_H__
#define __TSG_SEND_LOG_INTERNAL_H__
typedef struct _tsg_logger_info
#include <MESA/rdkafka.h>
#include <MESA/cJSON.h>
#define MAX_IPV4_LEN 16
#define MAX_STRING_LEN 32
enum _SEND_MODE
{
CLOSE=0,
KAFKA=1,
};
enum _DO_LOG
{
LOG_ABORT=0,
LOG_NOFILE=1,
LOG_ALL=2
};
typedef enum _tsg_log_field_id
{
LOG_COMMON_POLICY_ID=1,
LOG_COMMON_SERVICE,
LOG_COMMON_ACTION,
LOG_COMMON_START_TIME,
LOG_COMMON_END_TIME,
LOG_COMMON_L4_PROTOCOL,
LOG_COMMON_ADDRESS_TYPE,
LOG_COMMON_SERVER_IP,
LOG_COMMON_CLIENT_IP,
LOG_COMMON_SERVER_PORT,
LOG_COMMON_CLIENT_PORT,
LOG_COMMON_STREAM_DIR,
LOG_COMMON_ADDRESS_LIST,
LOG_COMMON_ENTRANCE_ID,
LOG_COMMON_DEVICE_ID,
LOG_COMMON_LINK_ID,
LOG_COMMON_ISP,
LOG_COMMON_ENCAPSULATION,
LOG_COMMON_DIRECTION,
LOG_COMMON_SLED_IP,
LOG_COMMON_USER_TAG,
LOG_COMMON_USER_REGION,
LOG_COMMON_APP_LABEL,
LOG_COMMON_APP_ID,
LOG_COMMON_PROTOCOL_ID,
LOG_COMMON_C2S_PKT_NUM,
LOG_COMMON_S2C_PKT_NUM,
LOG_COMMON_C2S_BYTE_NUM,
LOG_COMMON_S2C_BYTE_NUM,
LOG_COMMON_CON_DURATION_MS,
LOG_COMMON_HAS_DUP_TRAFFIC,
LOG_COMMON_STREAM_ERROR,
LOG_COMMON_STREAM_TRACE_ID,
LOG_COMMON_SCHAME_TYPE,
LOG_HTTP_HOST,
LOG_SSL_SNI,
LOG_COMMON_MAX
}tsg_log_field_id_t;
typedef struct _id2field
{
TLD_TYPE type;
int id;
char name[MAX_STRING_LEN];
}id2field_t;
struct TLD_handle_t
{
int thread_id;
cJSON *object;
};
struct tsg_log_instance_t
{
int mode;
int level;
int max_service;
void *logger;
void *cabot_handle;
char field_file[128];
char broker_list[128];
char log_path[128];
char local_ip_str[16];
}tsg_logger_info_t;
char common_field_file[MAX_STRING_LEN*4];
char broker_list[MAX_STRING_LEN*4];
char local_ip_str[MAX_IPV4_LEN];
id2field_t id2field[LOG_COMMON_MAX];
rd_kafka_topic_t **topic_rkt;
id2field_t *service2topic;
};
char *log_field_id2name(struct tsg_log_instance_t *instance, tsg_log_field_id_t id);
struct tsg_log_instance_t *tsg_sendlog_init(const char *filename);
#endif

473
src/tsg_ssl_utils.cpp Normal file
View File

@@ -0,0 +1,473 @@
#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>
#include <string.h>
#include "tsg_ssl_utils.h"
#define ALLOC(type, number) ((type *)calloc(sizeof(type), number))
#define FREE(p) {free(*p);*p=NULL;}
#define KNI_STRING_MAX 2048
struct cipher_suite cipher_suite_list[] =
{
{0xC030, "ECDHE-RSA-AES256-GCM-SHA384"},
{0xC02C, "ECDHE-ECDSA-AES256-GCM-SHA384"},
{0xC028, "ECDHE-RSA-AES256-SHA384"},
{0xC024, "ECDHE-ECDSA-AES256-SHA384"},
{0xC014, "ECDHE-RSA-AES256-SHA"},
{0xC00A, "ECDHE-ECDSA-AES256-SHA"},
{0x00A5, "DH-DSS-AES256-GCM-SHA384"},
{0x00A3, "DHE-DSS-AES256-GCM-SHA384"},
{0x00A1, "DH-RSA-AES256-GCM-SHA384"},
{0x009F, "DHE-RSA-AES256-GCM-SHA384"},
{0x006B, "DHE-RSA-AES256-SHA256"},
{0x006A, "DHE-DSS-AES256-SHA256"},
{0x0069, "DH-RSA-AES256-SHA256"},
{0x0068, "DH-DSS-AES256-SHA256"},
{0x0039, "DHE-RSA-AES256-SHA"},
{0x0038, "DHE-DSS-AES256-SHA"},
{0x0037, "DH-RSA-AES256-SHA"},
{0x0036, "DH-DSS-AES256-SHA"},
{0x0088, "DHE-RSA-CAMELLIA256-SHA"},
{0x0087, "DHE-DSS-CAMELLIA256-SHA"},
{0x0086, "DH-RSA-CAMELLIA256-SHA"},
{0x0085, "DH-DSS-CAMELLIA256-SHA"},
{0xC019, "AECDH-AES256-SHA"},
{0x00A7, "ADH-AES256-GCM-SHA384"},
{0x006D, "ADH-AES256-SHA256"},
{0x003A, "ADH-AES256-SHA"},
{0x0089, "ADH-CAMELLIA256-SHA"},
{0xC032, "ECDH-RSA-AES256-GCM-SHA384"},
{0xC02E, "ECDH-ECDSA-AES256-GCM-SHA384"},
{0xC02A, "ECDH-RSA-AES256-SHA384"},
{0xC026, "ECDH-ECDSA-AES256-SHA384"},
{0xC00F, "ECDH-RSA-AES256-SHA"},
{0xC005, "ECDH-ECDSA-AES256-SHA"},
{0x009D, "AES256-GCM-SHA384"},
{0x003D, "AES256-SHA256"},
{0x0035, "AES256-SHA"},
{0x0084, "CAMELLIA256-SHA"},
{0x008D, "PSK-AES256-CBC-SHA"},
{0xC02F, "ECDHE-RSA-AES128-GCM-SHA256"},
{0xC02B, "ECDHE-ECDSA-AES128-GCM-SHA256"},
{0xC027, "ECDHE-RSA-AES128-SHA256"},
{0xC023, "ECDHE-ECDSA-AES128-SHA256"},
{0xC013, "ECDHE-RSA-AES128-SHA"},
{0xC009, "ECDHE-ECDSA-AES128-SHA"},
{0x00A4, "DH-DSS-AES128-GCM-SHA256"},
{0x00A2, "DHE-DSS-AES128-GCM-SHA256"},
{0x00A0, "DH-RSA-AES128-GCM-SHA256"},
{0x009E, "DHE-RSA-AES128-GCM-SHA256"},
{0x0067, "DHE-RSA-AES128-SHA256"},
{0x0040, "DHE-DSS-AES128-SHA256"},
{0x003F, "DH-RSA-AES128-SHA256"},
{0x003E, "DH-DSS-AES128-SHA256"},
{0x0033, "DHE-RSA-AES128-SHA"},
{0x0032, "DHE-DSS-AES128-SHA"},
{0x0031, "DH-RSA-AES128-SHA"},
{0x0030, "DH-DSS-AES128-SHA"},
{0x009A, "DHE-RSA-SEED-SHA"},
{0x0099, "DHE-DSS-SEED-SHA"},
{0x0098, "DH-RSA-SEED-SHA"},
{0x0097, "DH-DSS-SEED-SHA"},
{0x0045, "DHE-RSA-CAMELLIA128-SHA"},
{0x0044, "DHE-DSS-CAMELLIA128-SHA"},
{0x0043, "DH-RSA-CAMELLIA128-SHA"},
{0x0042, "DH-DSS-CAMELLIA128-SHA"},
{0xC018, "AECDH-AES128-SHA"},
{0x00A6, "ADH-AES128-GCM-SHA256"},
{0x006C, "ADH-AES128-SHA256"},
{0x0034, "ADH-AES128-SHA"},
{0x009B, "ADH-SEED-SHA"},
{0x0046, "ADH-CAMELLIA128-SHA"},
{0xC031, "ECDH-RSA-AES128-GCM-SHA256"},
{0xC02D, "ECDH-ECDSA-AES128-GCM-SHA256"},
{0xC029, "ECDH-RSA-AES128-SHA256"},
{0xC025, "ECDH-ECDSA-AES128-SHA256"},
{0xC00E, "ECDH-RSA-AES128-SHA"},
{0xC004, "ECDH-ECDSA-AES128-SHA"},
{0x009C, "AES128-GCM-SHA256"},
{0x003C, "AES128-SHA256"},
{0x002F, "AES128-SHA"},
{0x0096, "SEED-SHA"},
{0x0041, "CAMELLIA128-SHA"},
{0x008C, "PSK-AES128-CBC-SHA"},
{0xC012, "ECDHE-RSA-DES-CBC3-SHA"},
{0xC008, "ECDHE-ECDSA-DES-CBC3-SHA"},
{0x0016, "EDH-RSA-DES-CBC3-SHA"},
{0x0013, "EDH-DSS-DES-CBC3-SHA"},
{0x0010, "DH-RSA-DES-CBC3-SHA"},
{0x000D, "DH-DSS-DES-CBC3-SHA"},
{0xC017, "AECDH-DES-CBC3-SHA"},
{0x001B, "ADH-DES-CBC3-SHA"},
{0xC00D, "ECDH-RSA-DES-CBC3-SHA"},
{0xC003, "ECDH-ECDSA-DES-CBC3-SHA"},
{0x000A, "DES-CBC3-SHA"},
{0x0007, "IDEA-CBC-SHA"},
{0x008B, "PSK-3DES-EDE-CBC-SHA"},
{0x0021, "KRB5-IDEA-CBC-SHA"},
{0x001F, "KRB5-DES-CBC3-SHA"},
{0x0025, "KRB5-IDEA-CBC-MD5"},
{0x0023, "KRB5-DES-CBC3-MD5"},
{0xC011, "ECDHE-RSA-RC4-SHA"},
{0xC007, "ECDHE-ECDSA-RC4-SHA"},
{0xC016, "AECDH-RC4-SHA"},
{0x0018, "ADH-RC4-MD5"},
{0xC00C, "ECDH-RSA-RC4-SHA"},
{0xC002, "ECDH-ECDSA-RC4-SHA"},
{0x0005, "RC4-SHA"},
{0x0004, "RC4-MD5"},
{0x008A, "PSK-RC4-SHA"},
{0x0020, "KRB5-RC4-SHA"},
{0x0024, "KRB5-RC4-MD5"},
{0xC010, "ECDHE-RSA-NULL-SHA"},
{0xC006, "ECDHE-ECDSA-NULL-SHA"},
{0xC015, "AECDH-NULL-SHA"},
{0xC00B, "ECDH-RSA-NULL-SHA"},
{0xC001, "ECDH-ECDSA-NULL-SHA"},
{0x003B, "NULL-SHA256"},
{0x0002, "NULL-SHA"},
{0x0001, "NULL-MD5"}
};
struct cipher_suite cipher_suite_list_tls13[] =
{
{0x1301, "TLS_AES_128_GCM_SHA256"},
{0x1302, "TLS_AES_256_GCM_SHA384"},
{0x1303, "TLS_CHACHA20_POLY1305_SHA256"},
{0x1304, "TLS_AES_128_CCM_SHA256"},
{0x1305, "TLS_AES_128_CCM_8_SHA256"}
};
void ssl_chello_free(struct ssl_chello* chello)
{
if(chello==NULL)
{
return;
}
free(chello->sni);
chello->sni = NULL;
free(chello->alpn);
chello->alpn = NULL;
free(chello->cipher_suites);
chello->cipher_suites = NULL;
free(chello->cipher_suites_tls13);
chello->cipher_suites_tls13 = NULL;
free(chello);
}
static char* parse_alpn_extension(const unsigned char* buff, size_t buff_len, enum chello_parse_result* result)
{
size_t pos = 0;
size_t len = ((size_t)buff[pos] << 8) + (size_t)buff[pos + 1];
if(2 + len != buff_len)
{
*result = CHELLO_PARSE_INVALID_FORMAT;
return NULL;
}
char* alpn = ALLOC(char, len + 1);
strncpy((char*)alpn, (const char*)buff + 2, len);
alpn[len] = '\0';
*result = CHELLO_PARSE_SUCCESS;
return alpn;
}
static char* parse_server_name_extension(const unsigned char* buff, size_t buff_len, enum chello_parse_result* result)
{
size_t pos = 2; /* skip server name list length */
size_t len;
char* sni = NULL;
while (pos + 3 < buff_len)
{
len = ((size_t)buff[pos + 1] << 8) + (size_t)buff[pos + 2];
if (pos + 3 + len > buff_len)
{
*result = CHELLO_PARSE_INVALID_FORMAT;
return NULL;
}
switch (buff[pos])
{
case 0x00: /* host_name */
sni = (char*)malloc(len + 1);
strncpy(sni, (const char*)buff + pos + 3, len);
sni[len] = '\0';
*result = CHELLO_PARSE_SUCCESS;
}
pos += 3 + len;
}
if (pos != buff_len)
{
*result = CHELLO_PARSE_INVALID_FORMAT;
}
return sni;
}
static enum chello_parse_result parse_extensions(const unsigned char* buff, size_t buff_len, struct ssl_chello* chello) {
size_t pos = 0;
/* Parse each 4 bytes for the extension header */
while (pos + 4 <= buff_len)
{
size_t len = ((size_t)buff[pos + 2] << 8) + (size_t)buff[pos + 3];
/* Check if it's a server name extension */
if (buff[pos] == 0x00 && buff[pos + 1] == 0x00)
{
if (pos + 4 + len > buff_len)
{
return CHELLO_PARSE_INVALID_FORMAT;
}
enum chello_parse_result result = CHELLO_PARSE_SUCCESS;
chello->sni = parse_server_name_extension(buff + pos + 4, len, &result);
if(result != CHELLO_PARSE_SUCCESS)
{
return result;
}
}
/* Check if it's a alpn extension */
if (buff[pos] == 0x00 && buff[pos + 1] == 0x10)
{
if (pos + 4 + len > buff_len)
{
return CHELLO_PARSE_INVALID_FORMAT;
}
enum chello_parse_result result = CHELLO_PARSE_SUCCESS;
chello->alpn = parse_alpn_extension(buff + pos + 4, len, &result);
if(result != CHELLO_PARSE_SUCCESS)
{
return result;
}
}
pos += (4 + len);
}
/* Check we ended where we expected to */
if (pos != buff_len)
{
return CHELLO_PARSE_INVALID_FORMAT;
}
return CHELLO_PARSE_SUCCESS;
}
static char* parse_cipher_suites(struct cipher_suite* _cipher_suite_list, int n, const unsigned char* buff, size_t buff_len, enum chello_parse_result* result)
{
char* cipher_suites_str = (char* )malloc(KNI_STRING_MAX);
cipher_suites_str[0] = '\0';
size_t pos = 0;
int flag = 0;
while(pos < buff_len)
{
int i = 0;
for(i = 0;i < n; i++)
{
int val = (buff[pos] << 8) + buff[pos + 1];
if(_cipher_suite_list[i].value == val)
{
if(strnlen(_cipher_suite_list[i].name, KNI_STRING_MAX) + strnlen(cipher_suites_str, KNI_STRING_MAX) + 1 > KNI_STRING_MAX)
{
flag = 1;
break;
}
strncat(cipher_suites_str, _cipher_suite_list[i].name, KNI_STRING_MAX);
strncat(cipher_suites_str, ":", KNI_STRING_MAX);
}
}
pos += 2;
if(flag == 1)
{
break;
}
}
int len = strnlen(cipher_suites_str, KNI_STRING_MAX);
if(len > 0)
{
cipher_suites_str[len-1] = '\0';
}
if(pos != buff_len && flag == 0)
{
*result = CHELLO_PARSE_INVALID_FORMAT;
free(cipher_suites_str);
return NULL;
}
*result = CHELLO_PARSE_SUCCESS;
return cipher_suites_str;
}
struct ssl_chello* ssl_chello_parse(const unsigned char* buff, size_t buff_len, enum chello_parse_result* result)
{
if(buff == NULL)
{
*result = CHELLO_PARSE_INVALID_FORMAT;
return NULL;
}
if(buff_len < 1)
{
*result = CHELLO_PARSE_NOT_ENOUGH_BUFF;
return NULL;
}
if(buff[0] != 0x80 && buff[0] != 0x16)
{
*result = CHELLO_PARSE_INVALID_FORMAT;
return NULL;
}
/* SSL 2.0 compatible Client Hello
* High bit of first byte (length) and content type is Client Hello
* See RFC5246 Appendix E.2
* if it is SSL 2.0, only parse version
*/
if(buff[0] == 0x80)
{
struct ssl_chello* _chello = (struct ssl_chello*)ALLOC(struct ssl_chello, 1);
_chello->min_version.major = 0x02;
if(buff_len < 2)
{
*result = CHELLO_PARSE_NOT_ENOUGH_BUFF;
return _chello;
}
size_t len = (size_t)buff[1];
if (buff_len < len + 2)
{
*result = CHELLO_PARSE_NOT_ENOUGH_BUFF;
return _chello;
}
buff_len = len + 2;
size_t pos = 2;
/* Handshark Message Type: Client Hello */
if (pos + 1 > buff_len)
{
*result = CHELLO_PARSE_INVALID_FORMAT;
return _chello;
}
if (buff[pos] != 0x01)
{
*result = CHELLO_PARSE_INVALID_FORMAT;
return _chello;
}
pos += 1;
/* Version */
if(pos + 2 > buff_len)
{
*result = CHELLO_PARSE_INVALID_FORMAT;
return _chello;
}
_chello->max_version.major = buff[pos];
_chello->max_version.minor = buff[pos + 1];
_chello->max_version.ossl_format=(uint16_t)_chello->max_version.major<<8|_chello->max_version.minor;
*result = CHELLO_PARSE_SUCCESS;
return _chello;
}
else
{
if (buff_len < 5)
{
*result = CHELLO_PARSE_NOT_ENOUGH_BUFF;
return NULL;
}
if(buff[1] != 3 || buff[2] > 4 || buff[2] < 0)
{
*result = CHELLO_PARSE_INVALID_FORMAT;
return NULL;
}
struct ssl_chello* _chello = (struct ssl_chello*)ALLOC(struct ssl_chello, 1);
_chello->min_version.major = buff[1];
_chello->min_version.minor = buff[2];
_chello->min_version.ossl_format=(uint16_t)_chello->min_version.major<<8|_chello->min_version.minor;
_chello->max_version.major = -1;
_chello->max_version.minor = -1;
_chello->sni = NULL;
_chello->alpn = NULL;
_chello->cipher_suites = NULL;
_chello->cipher_suites_tls13 = NULL;
/* TLS record length */
size_t len = ((size_t)buff[3] << 8) + (size_t)buff[4] + 5;
if (buff_len < len)
{
*result = CHELLO_PARSE_NOT_ENOUGH_BUFF;
return _chello;
}
buff_len = len;
size_t pos = 5;
if (pos + 1 > buff_len)
{
*result = CHELLO_PARSE_INVALID_FORMAT;
return _chello;
}
if (buff[pos] != 0x01)
{
*result = CHELLO_PARSE_INVALID_FORMAT;
return _chello;
}
pos += 4;
if(pos + 2 > buff_len)
{
*result = CHELLO_PARSE_INVALID_FORMAT;
return _chello;
}
_chello->max_version.major = buff[pos];
_chello->max_version.minor = buff[pos+1];
_chello->max_version.ossl_format=(uint16_t)_chello->max_version.major<<8|_chello->max_version.minor;
pos += 34;
/* Session ID */
if (pos + 1 > buff_len)
{
*result = CHELLO_PARSE_INVALID_FORMAT;
return _chello;
}
len = (size_t)buff[pos];
pos += 1 + len;
/* Cipher Suites */
if (pos + 2 > buff_len)
{
*result = CHELLO_PARSE_INVALID_FORMAT;
return _chello;
}
len = ((size_t)buff[pos] << 8) + (size_t)buff[pos + 1];
pos += 2;
if(pos + len > buff_len)
{
*result = CHELLO_PARSE_INVALID_FORMAT;
return _chello;
}
int n = sizeof(cipher_suite_list) / sizeof(struct cipher_suite);
_chello->cipher_suites = parse_cipher_suites(cipher_suite_list, n, buff + pos, len, result);
if(*result != CHELLO_PARSE_SUCCESS)
{
return _chello;
}
n = sizeof(cipher_suite_list_tls13) / sizeof(struct cipher_suite);
_chello->cipher_suites_tls13 = parse_cipher_suites(cipher_suite_list_tls13, n, buff + pos, len, result);
if(*result != CHELLO_PARSE_SUCCESS)
{
return _chello;
}
pos += len;
/* Compression Methods */
if (pos >= buff_len)
{
*result = CHELLO_PARSE_INVALID_FORMAT;
return _chello;
}
len = (size_t)buff[pos];
pos += 1 + len;
/* no extensions */
if(pos == buff_len)
{
*result = CHELLO_PARSE_SUCCESS;
return _chello;
}
/* Extensions */
if (pos + 2 > buff_len)
{
*result = CHELLO_PARSE_INVALID_FORMAT;
return _chello;
}
len = ((size_t)buff[pos] << 8) + (size_t)buff[pos + 1];
pos += 2;
if (pos + len > buff_len)
{
*result = CHELLO_PARSE_INVALID_FORMAT;
return _chello;
}
enum chello_parse_result ret = parse_extensions(buff + pos, len, _chello);
*result = ret;
return _chello;
}
}

5
src/version.map
View File

@@ -3,11 +3,12 @@ global:
extern "C++" {
g_*;
*TSG_MASTER_INIT*;
*TSG_BWLIST_TCP_ENTRY*;
*TSG_MASTER_TCP_ENTRY*;
*TSG_MASTER_UNLOAD*;
*tsg_send_log*;
*tsg_scan_nesting_addr*;
*tsg_pull_policy_result*;
*tsg_*;
*TLD_*;
};
local: *;
};