gfwleak/tango-tsg-master
Archived
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Squashed commit of the following:

Browse Source 
commit 91574444c3
Author: liuxueli <liuxueli@iie.ac.cn>
Date:   Tue Dec 3 11:44:16 2019 +0800

    公共字段中的通信市时长单位是：ms；转换单位
    KNI发日志需等待TFE结束，需提前构造streaminfo日志字段，调整包数字节数统计位置
    细化调整日志

commit ff053e3e2c
Author: liuxueli <liuxueli@iie.ac.cn>
Date:   Wed Nov 27 11:38:35 2019 +0800

    增加调试日志信息

commit 2d1a530165
Author: liuxueli <liuxueli@iie.ac.cn>
Date:   Wed Nov 27 11:27:06 2019 +0800

    使用前置声明结构体，修改接口函数原型

commit 27f0cfc91a
Author: liuxueli <liuxueli@iie.ac.cn>
Date:   Wed Nov 27 10:33:48 2019 +0800

    提供TLD_append_streaminfo函数接口供KNI使用
    修改配置文件

commit ed0b17a49d
Merge: 132d807 04963fe
Author: 崔一鸣 <cuiyiming@iie.ac.cn>
Date:   Tue Nov 26 17:44:28 2019 +0800

    Merge branch 'feature-kni' into 'develop'

    增加http解析部分

    See merge request tango/tsg_master!3

commit 04963fe1bf
Author: 崔一鸣 <cuiyiming@iie.ac.cn>
Date:   Tue Nov 26 17:41:43 2019 +0800

    增加http解析部分

commit 132d807af1
Author: liuxueli <liuxueli@iie.ac.cn>
Date:   Tue Nov 26 16:58:54 2019 +0800

    调整调试日志内容

commit 8cfc6b32fd
Author: liuxueli <liuxueli@iie.ac.cn>
Date:   Mon Nov 25 19:30:08 2019 +0800

    增加调试日志信息

commit 4aef0628e5
Author: liuxueli <liuxueli@iie.ac.cn>
Date:   Mon Nov 25 18:36:48 2019 +0800

    增加调试信息

commit 7c9dabdab5
Author: liuxueli <liuxueli@iie.ac.cn>
Date:   Mon Nov 25 10:51:37 2019 +0800

    矫正tableinfo配置项

commit 34ec2059ea
Merge: 27cb2ad 2b96c05
Author: liuxueli <liuxueli@iie.ac.cn>
Date:   Mon Nov 25 10:00:18 2019 +0800

    Merge branch 'develop' of https://git.mesalab.cn/tango/tsg_master into develop

commit 27cb2ad3b2
Author: liuxueli <liuxueli@iie.ac.cn>
Date:   Thu Nov 21 19:21:08 2019 +0800

    获取包数、字节数与sapp头文件不一致，导致统计信息颠倒
    删除从vxlan里获取信息

commit 2b96c050d5
Merge: 0a11db6 8d954e4
Author: 崔一鸣 <cuiyiming@iie.ac.cn>
Date:   Thu Nov 21 10:22:42 2019 +0800

    Merge branch 'feature-kni' into 'develop'

    修改proto初始值为PROTO_UNKONWN

    See merge request tango/tsg_master!2

commit 46a233b782
Author: liuxueli <liuxueli@iie.ac.cn>
Date:   Thu Nov 21 10:18:08 2019 +0800

    KNI发送日志的线程不是sapp申请的，所以不能使用dictator_malloc申请内存
    增加user_region日志字段

commit 8d954e4cc5
Author: 崔一鸣 <cuiyiming@iie.ac.cn>
Date:   Wed Nov 20 19:55:59 2019 +0800

    修改proto初始值为PROTO_UNKONWN

commit 0a11db6d2e
Author: liuxueli <liuxueli@iie.ac.cn>
Date:   Wed Nov 20 18:02:54 2019 +0800

    增加总控配置文件

commit 5c04ba23aa
Author: liuxueli <liuxueli@iie.ac.cn>
Date:   Wed Nov 20 18:00:35 2019 +0800

    SSL_SNI和HTTP_HOST的deny动作由总控负责处理
    修复获取全局流ID时的错误

commit 291c05ea79
Author: liuxueli <liuxueli@iie.ac.cn>
Date:   Wed Nov 20 16:40:19 2019 +0800

    增加获取全局流ID的接口
    调整总控注册为TCP入口

commit 28da97b53b
Author: liuxueli <liuxueli@iie.ac.cn>
Date:   Wed Nov 20 14:45:41 2019 +0800

    支持平台提供的APP_STATE_KILL_OTHER
    修复BUG，天剑流标签时未对协议进行赋值，导致tfe颁发证书失败

commit 7912ec54b2
Author: liuxueli <liuxueli@iie.ac.cn>
Date:   Tue Nov 19 18:41:31 2019 +0800

    修复BUG，命中拦截策略时未对类型进行赋值，导致KNI获取不到命中结果
    添加总控配置文件
    添加FS2统计日志
    添加错误日志
    构造适用sapp4.0的rpm

commit 484feaed94
Author: liuxueli <liuxueli@iie.ac.cn>
Date:   Tue Nov 19 13:39:37 2019 +0800

    增加获取deny结果规则的接口
    修复自测试过程中遇到的BUG

commit 5165173025
Merge: 186e591 6b6cbef
Author: 崔一鸣 <cuiyiming@iie.ac.cn>
Date:   Mon Nov 18 15:46:03 2019 +0800

    Merge branch 'feature-fast-path' into feature-kni

commit 6b6cbeffa1
Author: liuxueli <liuxueli@iie.ac.cn>
Date:   Mon Nov 18 14:50:32 2019 +0800

    修复初始化错误

commit e7532d6280
Author: liuxueli <liuxueli@iie.ac.cn>
Date:   Mon Nov 18 13:33:29 2019 +0800

    添加遗漏头文件tsg_entry.h

commit f33d11c50b
Author: liuxueli <liuxueli@iie.ac.cn>
Date:   Mon Nov 18 13:29:28 2019 +0800

    修复编译错误

commit 88d9368cee
Author: liuxueli <liuxueli@iie.ac.cn>
Date:   Mon Nov 18 13:25:38 2019 +0800

    合并头文件，删除无用头文件

commit 382ca0a793
Author: 刘学利 <liuxueli@iie.ac.cn>
Date:   Mon Nov 18 13:21:25 2019 +0800

    Delete tsg_types.h

commit ab58f918e5
Author: 刘学利 <liuxueli@iie.ac.cn>
Date:   Mon Nov 18 13:21:05 2019 +0800

    Delete tsg_log_id.h

commit 3b13075701
Author: 刘学利 <liuxueli@iie.ac.cn>
Date:   Mon Nov 18 13:20:57 2019 +0800

    Delete tsg_entry.h

commit 186e591dbf
Merge: 39f82ed 91c511c
Author: 崔一鸣 <cuiyiming@iie.ac.cn>
Date:   Fri Nov 15 21:22:44 2019 +0800

    Merge branch 'master' into feature-kni

commit 39f82ed3f7
Author: 崔一鸣 <cuiyiming@iie.ac.cn>
Date:   Fri Nov 15 21:22:39 2019 +0800

    接口适配

commit 8913a7d22e
Author: liuxueli <liuxueli@iie.ac.cn>
Date:   Fri Nov 15 19:29:54 2019 +0800

    规则排序功能
    统一发送日志功能

commit 1cb2358fea
Merge: c76bc55 bd65acc
Author: 崔一鸣 <cuiyiming@iie.ac.cn>
Date:   Thu Nov 14 15:06:35 2019 +0800

    合并master修改

commit c76bc5534d
Author: 崔一鸣 <cuiyiming@iie.ac.cn>
Date:   Wed Nov 13 19:56:19 2019 +0800

    接口增加protocol参数

commit db7282dab7
Author: 崔一鸣 <cuiyiming@iie.ac.cn>
Date:   Wed Nov 13 19:24:22 2019 +0800

    扫描接口增加domain

commit 56fadd73b2
Merge: f9aee05 9e738ae
Author: 崔一鸣 <cuiyiming@iie.ac.cn>
Date:   Wed Nov 13 19:08:55 2019 +0800

    Merge branch 'master' into feature-kni

commit f9aee05bcc
Author: 崔一鸣 <cuiyiming@iie.ac.cn>
Date:   Wed Nov 13 16:17:59 2019 +0800

    修改ssl_utils文件名

commit 452a08790c
Merge: 3abe8a2 c1ffc53
Author: 崔一鸣 <cuiyiming@iie.ac.cn>
Date:   Wed Nov 13 16:12:14 2019 +0800

    Merge branch 'master' into feature-kni

commit 3abe8a2fc5
Author: 崔一鸣 <cuiyiming@iie.ac.cn>
Date:   Wed Nov 13 16:11:38 2019 +0800

    添加kni模块扫描部分
This commit is contained in:
杨威
2019-12-09 18:58:05 +08:00
parent 91c511c3ed
commit 64c1dcb159
22 changed files with 2184 additions and 357 deletions
Download Patch File Download Diff File Expand all files Collapse all files

374
src/tsg_entry.cpp
View File

@@ -1,6 +1,368 @@
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
char TSG_MASTER_VERSION_20191112=0;
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <assert.h>
#include <MESA/stream.h>
#include <MESA/MESA_prof_load.h>
#include <MESA/MESA_handle_logger.h>
#include "tsg_rule.h"
#include "tsg_entry.h"
#include "tsg_send_log.h"
#include "tsg_send_log_internal.h"
char TSG_MASTER_VERSION_20191129=0;
const char *tsg_conffile="tsgconf/main.conf";
g_tsg_para_t g_tsg_para;
id2field_t g_tsg_fs2_field[TSG_FS2_MAX]={{TLD_TYPE_UNKNOWN, TSG_FS2_LINKS, "links"},
{TLD_TYPE_UNKNOWN, TSG_FS2_BYPASS, "bypass"},
{TLD_TYPE_UNKNOWN, TSG_FS2_HIT_ADDR, "hit_addr"},
{TLD_TYPE_UNKNOWN, TSG_FS2_HIT_SHARE, "hit_share"},
{TLD_TYPE_UNKNOWN, TSG_FS2_INTERCEPT, "intercept"},
{TLD_TYPE_UNKNOWN, TSG_FS2_LOG, "log"},
{TLD_TYPE_UNKNOWN, TSG_FS2_DENY, "deny"}
};
static void free_policy_label(int thread_seq, void *project_req_value)
{
dictator_free(thread_seq, project_req_value);
project_req_value=NULL;
}
#if 0
static int is_ip_policy(Maat_rule_t *p_result, char *protocol, int len, int thread_seq)
{
int ret=0;
cJSON *item=NULL;
char *service_defined=NULL;
cJSON *user_define_object=NULL;
if(p_result->serv_def_len>MAX_SERVICE_DEFINE_LEN)
{
service_defined=dictator_malloc(thread_seq, p_result->serv_def_len+1);
ret=Maat_read_rule(g_tsg_maat_feather, p_result, MAAT_RULE_SERV_DEFINE, service_defined, p_result->serv_def_len+1);
assert(ret==p_result->serv_def_len+1);
user_define_object=cJSON_Parse(service_defined);
}
else
{
user_define_object=cJSON_Parse(p_result->service_defined);
}
if(user_define_object!=NULL)
{
item=cJSON_GetObjectItem(user_define_object, "protocol");
if(item!=NULL && item->valuestring!=NULL)
{
memcpy(protocol, item->valuestring, (len>strlen(item->valuestring)) ? strlen(item->valuestring): len);
}
item=cJSON_GetObjectItem(user_define_object, "method");
if((item==NULL) || ((strncasecmp(item->valuestring, "http", strlen(item->valuestring)))!=0 && (strncasecmp(item->valuestring, "ssl", strlen(item->valuestring)))!=0))
{
ret=1;
}
cJSON_Delete(user_define_object);
user_define_object=NULL;
}
if(service_defined!=NULL)
{
dictator_free(thread_seq, service_defined);
service_defined=NULL;
}
return ret;
}
#endif
static struct Maat_rule_t *tsg_policy_decision_criteria(Maat_rule_t *result, int result_num)
{
int i=0;
Maat_rule_t *p_result=NULL;
if(result==NULL || result_num<=0)
{
return NULL;
}
p_result=&result[0];
for(i=1; i<result_num; i++)
{
if((unsigned char)result[i].action>(unsigned char)p_result->action)
{
p_result=&result[i];
continue;
}
if(result[i].action==p_result->action)
{
if(result[i].config_id>p_result->config_id)
{
p_result=&result[i];
}
}
}
return p_result;
}
extern "C" char TSG_MASTER_TCP_ENTRY(struct streaminfo *a_tcp, void **pme, int thread_seq,void *a_packet)
{
int send_log=0,identify_flag=0;
int ret=0,hit_num=0;
int state=APP_STATE_DROPME;
scan_status_t mid=NULL;
char *domain_field_name=NULL;
char *schema_field_name=NULL;
Maat_rule_t *p_result=NULL;
Maat_rule_t *q_result=NULL;
tsg_log_t log_msg;
struct TLD_handle_t *TLD_handle=NULL;
struct _identify_info identify_info;
Maat_rule_t all_result[MAX_RESULT_NUM];
policy_priority_label_t *priority_label=NULL;
switch(a_tcp->opstate)
{
case OP_STATE_PENDING:
FS_operate(g_tsg_para.fs2_handle, g_tsg_para.fs2_field_id[TSG_FS2_LINKS], 0, FS_OP_ADD, 1);
ret=tsg_scan_nesting_addr(g_tsg_maat_feather, a_tcp, PROTO_MAX, &mid, all_result+hit_num, MAX_RESULT_NUM-hit_num);
if(ret>0)
{
hit_num+=ret;
q_result=tsg_policy_decision_criteria(all_result, hit_num);
FS_operate(g_tsg_para.fs2_handle, g_tsg_para.fs2_field_id[TSG_FS2_HIT_ADDR], 0, FS_OP_ADD, 1);
MESA_handle_runtime_log(g_tsg_para.logger,
RLOG_LV_DEBUG,
"SCAN_IP",
"Hit addr: %s scan ret: %d policy_id: %d service: %d action: %d",
printaddr(&a_tcp->addr, thread_seq),
ret,
q_result->config_id,
q_result->service_id,
q_result->action);
}
else
{
MESA_handle_runtime_log(g_tsg_para.logger, RLOG_LV_DEBUG, "SCAN_IP", "Not hit %s scan ret: %d",
printaddr(&a_tcp->addr, thread_seq), ret);
}
memset(&identify_info, 0, sizeof(identify_info));
ret=tsg_scan_shared_policy(g_tsg_maat_feather,
a_tcp->ptcpdetail->pdata,
a_tcp->ptcpdetail->datalen,
all_result+hit_num,
MAX_RESULT_NUM-hit_num,
&identify_info,
&mid,
g_tsg_para.logger,
thread_seq);
if(ret>0)
{
FS_operate(g_tsg_para.fs2_handle, g_tsg_para.fs2_field_id[TSG_FS2_HIT_SHARE], 0, FS_OP_ADD, 1);
MESA_handle_runtime_log(g_tsg_para.logger,
RLOG_LV_DEBUG,
"SCAN_FQDN",
"Hit %s: %s policy_id: %d service: %d action: %d addr: %s",
(identify_info.proto==PROTO_HTTP) ? "host" : "sni",
identify_info.domain,
all_result[hit_num].config_id,
all_result[hit_num].service_id,
all_result[hit_num].action,
printaddr(&a_tcp->addr, thread_seq)
);
hit_num+=ret;
identify_flag=1;
}
else
{
MESA_handle_runtime_log(g_tsg_para.logger,
RLOG_LV_DEBUG,
"SCAN_FQDN",
"Not hit %s: %s addr: %s",
(ret==-1) ? "NULL" : ((identify_info.proto==PROTO_HTTP) ? "host" : "sni"),
(ret==-1) ? "NULL" : identify_info.domain,
printaddr(&a_tcp->addr, thread_seq)
);
}
p_result=tsg_policy_decision_criteria(all_result, hit_num);
if(p_result!=NULL)
{
switch((unsigned char)p_result->action)
{
case TSG_ACTION_DENY:
send_log=1;
MESA_kill_tcp(a_tcp, a_packet);
state|=APP_STATE_DROPPKT|APP_STATE_KILL_OTHER;
FS_operate(g_tsg_para.fs2_handle, g_tsg_para.fs2_field_id[TSG_FS2_DENY], 0, FS_OP_ADD, 1);
MESA_handle_runtime_log(g_tsg_para.logger, RLOG_LV_DEBUG, "DENY", "Hit deny policy, policy_id: %d action: %d addr: %s",
p_result[0].config_id, p_result[0].action, printaddr(&a_tcp->addr, thread_seq));
break;
case TSG_ACTION_MONITOR:
if(q_result!=NULL && (p_result==q_result))
{
send_log=1;
}
break;
case TSG_ACTION_BYPASS:
send_log=1;
state|=APP_STATE_KILL_OTHER; //TODO
FS_operate(g_tsg_para.fs2_handle, g_tsg_para.fs2_field_id[TSG_FS2_BYPASS], 0, FS_OP_ADD, 1);
break;
case TSG_ACTION_INTERCEPT:
FS_operate(g_tsg_para.fs2_handle, g_tsg_para.fs2_field_id[TSG_FS2_INTERCEPT], 0, FS_OP_ADD, 1);
priority_label=(policy_priority_label_t *)dictator_malloc(thread_seq, sizeof(policy_priority_label_t));
priority_label->result_num=1;
priority_label->result_type=PULL_KNI_RESULT;
priority_label->proto=identify_info.proto;
priority_label->domain_len=identify_info.domain_len;
memcpy(priority_label->domain, identify_info.domain, identify_info.domain_len);
memcpy(priority_label->result, p_result, sizeof(struct Maat_rule_t));
ret=project_req_add_struct(a_tcp, g_tsg_para.priority_project_id, (void *)priority_label);
if(ret<0)
{
free_policy_label(thread_seq, (void *)priority_label);
MESA_handle_runtime_log(g_tsg_para.logger, RLOG_LV_FATAL, "PROJECT_ADD", "Add policy_priority_label failed ...");
}
MESA_handle_runtime_log(g_tsg_para.logger, RLOG_LV_DEBUG, "INTERCEPT", "Hit intercept policy, policy_id: %d action: %d addr: %s",
priority_label->result[0].config_id, priority_label->result[0].action, printaddr(&a_tcp->addr, thread_seq));
break;
case TSG_ACTION_NONE:
default:
assert(0);
break;
}
if(send_log==1 && p_result->do_log>0)
{
TLD_handle=TLD_create(thread_seq);
if(identify_flag==1)
{
schema_field_name=log_field_id2name(g_tsg_log_instance, LOG_COMMON_SCHAME_TYPE);
TLD_append(TLD_handle, schema_field_name, (void *)((identify_info.proto==PROTO_HTTP) ? "HTTP" : "SSL"), TLD_TYPE_STRING);
domain_field_name=log_field_id2name(g_tsg_log_instance, ((identify_info.proto==PROTO_HTTP) ? LOG_HTTP_HOST : LOG_SSL_SNI));
TLD_append(TLD_handle, domain_field_name, (void *)identify_info.domain, TLD_TYPE_STRING);
}
log_msg.a_stream=a_tcp;
log_msg.result=p_result;
log_msg.result_num=1;
tsg_send_log(g_tsg_log_instance, TLD_handle, &log_msg, thread_seq);
}
}
break;
case OP_STATE_DATA:
case OP_STATE_CLOSE:
default:
break;
}
return state;
}
extern "C" int TSG_MASTER_INIT()
{
int i=0,ret=0,cycle=0;
int value=0,level=30;
unsigned short fs_server_port=0;
char app_name[MAX_STRING_LEN]={0};
char log_path[MAX_STRING_LEN*4]={0};
char label_buff[MAX_STRING_LEN*4]={0};
char fs_server_ip[MAX_IPV4_LEN]={0};
char fs_output_path[MAX_STRING_LEN*4]={0};
memset(&g_tsg_para, 0, sizeof(g_tsg_para));
MESA_load_profile_int_def(tsg_conffile, "SYSTEM","LOG_LEVEL", &level, 30);
MESA_load_profile_string_def(tsg_conffile, "SYSTEM","LOG_PATH", log_path, sizeof(log_path), NULL);
g_tsg_para.logger=MESA_create_runtime_log_handle(log_path, level);
if(g_tsg_para.logger==NULL)
{
printf("MESA_create_runtime_log_handle failed ...\n");
return -1;
}
MESA_load_profile_int_def(tsg_conffile, "SYSTEM", "DEVICE_ID", &g_tsg_para.device_id, 0);
MESA_load_profile_string_def(tsg_conffile, "SYSTEM", "POLICY_PRIORITY_LABEL", label_buff, sizeof(label_buff), "POLICY_PRIORITY");
g_tsg_para.priority_project_id=project_producer_register(label_buff, PROJECT_VAL_TYPE_STRUCT, free_policy_label);
if(g_tsg_para.priority_project_id<0)
{
MESA_handle_runtime_log(g_tsg_para.logger, RLOG_LV_FATAL, "PROJECT_REGISTER", "Register %s failed ...", label_buff);
return -1;
}
ret=tsg_rule_init(tsg_conffile, g_tsg_para.logger);
if(ret<0)
{
MESA_handle_runtime_log(g_tsg_para.logger, RLOG_LV_FATAL, "INIT_MAAT", "tsg_rule_init failed ...");
return -1;
}
g_tsg_log_instance=tsg_sendlog_init(tsg_conffile);
if(g_tsg_log_instance==NULL)
{
MESA_handle_runtime_log(g_tsg_para.logger, RLOG_LV_FATAL, "INIT_SENDLOG", "tsg_sendlog_init failed ...");
return -1;
}
MESA_load_profile_int_def(tsg_conffile, "FIELD_STAT", "CYCLE", &cycle, 30);
MESA_load_profile_short_nodef(tsg_conffile, "FIELD_STAT","TELEGRAF_PORT", (short *)&(fs_server_port));
MESA_load_profile_string_nodef(tsg_conffile,"FIELD_STAT","TELEGRAF_IP",fs_server_ip, sizeof(fs_server_ip));
MESA_load_profile_string_def(tsg_conffile,"FIELD_STAT","OUTPUT_PATH",fs_output_path, sizeof(fs_output_path), "tsg_stat.log");
MESA_load_profile_string_def(tsg_conffile,"FIELD_STAT","APP_NAME", app_name, sizeof(app_name), "tsg_master");
g_tsg_para.fs2_handle=FS_create_handle();
value=1;//Rewrite
FS_set_para(g_tsg_para.fs2_handle, PRINT_MODE, &value, sizeof(value));
value=1;//Do not create stat thread
FS_set_para(g_tsg_para.fs2_handle, CREATE_THREAD, &value, sizeof(value));
FS_set_para(g_tsg_para.fs2_handle, STAT_CYCLE, &cycle, sizeof(cycle));
FS_set_para(g_tsg_para.fs2_handle, APP_NAME, app_name, strlen(app_name)+1);
FS_set_para(g_tsg_para.fs2_handle, OUTPUT_DEVICE, fs_output_path, strlen(fs_output_path)+1);
if(fs_server_port > 0 && strlen(fs_server_ip) > 0)
{
FS_set_para(g_tsg_para.fs2_handle, STATS_SERVER_IP,fs_server_ip, strlen(fs_server_ip)+1);
FS_set_para(g_tsg_para.fs2_handle, STATS_SERVER_PORT,&(fs_server_port), sizeof(fs_server_port));
}
for(i=0; i<TSG_FS2_MAX; i++)
{
g_tsg_para.fs2_field_id[g_tsg_fs2_field[i].id]=FS_register(g_tsg_para.fs2_handle, FS_STYLE_FIELD, FS_CALC_CURRENT, g_tsg_fs2_field[i].name);
}
FS_start(g_tsg_para.fs2_handle);
return 0;
}