gfwleak/tango-maat
Archived
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

third draft

Browse Source
This commit is contained in:
liuwentan
2023-07-06 18:58:15 +08:00
parent 2d6ffdd166
commit 9d373ad454
41 changed files with 81287 additions and 455 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
docs/history.md
View File

@@ -1,7 +1,7 @@
# 更新记录
| 版本 | 时间 | 说明 | 修订人 |
| ------- | ---------- | ------------------------------------------------------------ | ------------ |
| ------- | ---------- | --------------------------------------------------------- | -------- |
| v4.0 | 2023-5-2 | Maat4.0，使用Markdown重新组织文档 | 郑超、刘文坛 |
| V3.7.0 | 2022-8-23 | 增加Boolean Expression 回调表；修正分组于子分组中的描述。| 郑超 |
| V3.2.3 | 2021-7-15 | 增加组合扫描一节 | 郑超 |
@@ -71,5 +71,3 @@
| v1.0.1 | 2014-11-19 | IP类配置支持协议扫描 | 郑超 |
| v1.0 | 2014-08-19 | 第一个稳定版 | 郑超 |
| v0.1 | 2014-06-16 | 创建文档，包含表格式说明 | 郑超 |

0
docs/imgs/configuration_diagram.png → docs/imgs/rule_diagram.png
View File

Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 117 KiB

After

Width:  |  Height:  |  Size: 117 KiB

29
docs/introduction.md
View File

@@ -3,18 +3,15 @@
**Item**: As a filter for network attributes, the smallest unit of a rule
- Eg1: specify that the UserAgent field in the HTTP protocol contains substrings "Chrome" and "11.8.1",
   HTTP UserAgent: Chrome & 11.8.1
- Eg2: specify that the domain name in the HTTP protocol ends with ".emodao.com"
   HTTP HOST: *.emodao.com
- Eg3: specify client IP address belongs to the C segment of 202.118.101.*
   Source IP: 202.11.101.0/24
There are multiple types of items stored in corresponding tables such as string, IP and numerical range, more details can be found in [Item table](./table_schema.md#item-table)
There are multiple types of items stored in corresponding tables such as string, IP and numerical range, more details can be found in [Item table](./table_schema.md#item-table).
**Group(Object)**: Collection of Items, the constraints of group are as follows:
@@ -31,13 +28,14 @@ The relationship between group and group is stored in the [group2group table](./
**Compile(Policy)**: A conjunctive normal form(CNF) consisting of multiple groups and virtual tables
- A Compile can contain up to 8 clauses and multiple clauses in the same compile can be logical 'AND' and logical 'NOT' relationships.
- A Clause consists of several Literals and the relationship between them is a logical 'OR'. A Literal consists of virtual table and group. During the configuration loading process, a unique Clause ID will be generated based on the combination of virtual table ID and group ID in the same clause.
- A Clause consists of several Literals and the relationship between them is a logical 'OR'. A Literal consists of virtual table and group. During the rules loading process, a unique Clause ID will be generated based on the combination of virtual table ID and group ID in the same clause.
![CNF](./imgs/CNF.jpg)
Configuration Diagram
![configuration Diagram](./imgs/configuration_diagram.png)
Rule Diagram
![Rule Diagram](./imgs/rule_diagram.png)
## Group/Object Nesting and Hierarchies
@@ -50,16 +48,22 @@ An object defines a subset of an object type, such as network addresses or port
There are rules of precedence to take into account when defining objects:
- Excluding has precedence over including in the same object.
- Items in a superior object have precedence over items in a subordinate object.
- Items in a superior object are not taken into account in a subordinate object, if the subordinate object is used directly in a rule.
- Peer objects (different subordinate objects of the same superior object) do not affect each other.
In short, to determine the set defined by an object, perform the following calculation:
1. For each subordinate object (remember sibling objects do not affect each other):
1. Add included items.
2. Subtract excluded items.
- Add included items.
- Subtract excluded items.
2. Add included items in the object itself, overriding any excludes in the subordinate objects.
3. Subtract excluded items in the object itself.
The following figure shows an object with an included set and an excluded subset.
@@ -87,7 +91,6 @@ Now, let's see a graph of hierarchy example, where the dotted line means exclude
![object-hierarchy-example](./imgs/object-hierarchy-example.png)
| **Matched subordinate objects** | **Activated superior Objects** |
| ------------------------------- | ------------------------------ |
| g1, g3 | g6, g9 |
@@ -104,10 +107,10 @@ Now, let's see a graph of hierarchy example, where the dotted line means exclude
| **Term** | **Description** |
| --------------------------- | ------------------------------------------------------------ |
| Instance | Maat handle |
| Item | Configuration of a certain field，such as URL field in HTTP protocol, client ip address field in DNS protocol etc.|
| Item | Rule of a certain field，such as URL field in HTTP protocol, client ip address field in DNS protocol etc.|
| Group(Object) | A collection of one or more Items, the multiple items under the same Group are logical 'OR' relationships |
| Compile(Policy) | A rule for several Groups logical AND or NOT operations |
| Table | Different types of configurations have different tables, such as ip table, keywords table, group2compile table, compile table etc. |
| Table | Different types of rules have different tables, such as ip table, keywords table, group2compile table, compile table etc. |
| Physical Table | The actual table in the database |
| Virtual Table | Table that do not exist in the database and only references physical table |
| Table Schema | Define the table type and the meaning of each column in the table |
@@ -115,4 +118,4 @@ Now, let's see a graph of hierarchy example, where the dotted line means exclude
| Maat State | Store intermediate state of multiple scans |
| Maat Stream | Handle of streaming file scanning |
| Hit Path | From the perspective of data to be scanned, describe the hit ID sequence: item_id -> sub_group_id -> superior_group_id(virtual_table_id) -> compile_id |
| Redis | In-memory data store，see https://redis.io/. It has a leader follower replication to ensure the high availability of configuration |
| Redis | In-memory data store，see https://redis.io/. It has a leader follower replication to ensure the high availability of rules |

304
docs/logic (AND | OR | NOT).md Normal file
View File

@@ -0,0 +1,304 @@
# logic AND OR NOT
- [logic AND](#logic-and)
- [logic OR](#logic-or)
- [logic NOT](#logic-not)
- [group exclude](#group-exclude)
## logic AND
rule = China & USA
```json
{
"compile_id": 123,
"service": 1,
"action": 1,
"do_blacklist": 1,
"do_log": 1,
"user_region": "null",
"is_valid": "yes",
"groups": [
{
"clause_index": 0,
"regions": [
{
"table_name": "HTTP_URL",
"table_type": "expr",
"table_content": {
"keywords": "China",
"expr_type": "none",
"match_method": "sub",
"format": "uncase plain"
}
}
]
},
{
"clause_index": 1,
"regions": [
{
"table_name": "HTTP_URL",
"table_type": "expr",
"table_content": {
"keywords": "USA",
"expr_type": "none",
"match_method": "sub",
"format": "uncase plain"
}
}
]
}
]
}
```
scan sample
```c
const char *string1 = "China";
const char *string2 = "USA";
long long results[ARRAY_SIZE] = {0};
size_t n_hit_result = 0;
int thread_id = 0;
const char *table_name = "HTTP_URL";
struct maat_state *state = maat_state_new(maat_inst, thread_id);
int table_id = maat_get_table_id(maat_inst, table_name);
ASSERT_GT(table_id, 0);
int ret = maat_scan_string(maat_inst, table_id, string1, strlen(string1),
results, ARRAY_SIZE, &n_hit_result, state);
EXPECT_EQ(ret, MAAT_SCAN_HALF_HIT);
ret = maat_scan_string(maat_inst, table_id, string2, strlen(string2),
results, ARRAY_SIZE, &n_hit_result, state);
EXPECT_EQ(ret, MAAT_SCAN_HIT);
EXPECT_EQ(n_hit_result, 1);
EXPECT_EQ(results[0], 123);
maat_state_free(state);
```
## logic OR
rule = China | USA
```json
{
"compile_id": 124,
"service": 0,
"action": 0,
"do_blacklist": 0,
"do_log": 0,
"user_region": "null",
"is_valid": "yes",
"groups": [
{
"regions": [
{
"table_type": "expr",
"table_name": "HTTP_URL",
"table_content": {
"keywords": "China",
"expr_type": "none",
"format": "uncase plain",
"match_method": "prefix"
}
},
{
"table_type": "expr",
"table_name": "HTTP_URL",
"table_content": {
"keywords": "USA",
"expr_type": "none",
"format": "uncase plain",
"match_method": "prefix"
}
}
]
}
]
}
```
scan sample
```c
const char *string1 = "China";
const char *string2 = "USA";
long long results[ARRAY_SIZE] = {0};
size_t n_hit_result = 0;
int thread_id = 0;
const char *table_name = "HTTP_URL";
struct maat_state *state = maat_state_new(maat_inst, thread_id);
int table_id = maat_get_table_id(maat_inst, table_name);
ASSERT_GT(table_id, 0);
ret = maat_scan_string(maat_inst, table_id, string1, strlen(string1),
results, ARRAY_SIZE, &n_hit_result, state);
EXPECT_EQ(ret, MAAT_SCAN_HIT);
EXPECT_EQ(n_hit_result, 1);
EXPECT_EQ(results[0], 124);
maat_state_reset(state);
ret = maat_scan_string(maat_inst, table_id, string2, strlen(string2),
results, ARRAY_SIZE, &n_hit_result, state);
EXPECT_EQ(ret, MAAT_SCAN_HIT);
EXPECT_EQ(n_hit_result, 1);
EXPECT_EQ(results[0], 124);
maat_state_free(state);
```
## logic NOT
rule = China & !USA
```json
{
"compile_id": 125,
"service": 1,
"action": 1,
"do_blacklist": 1,
"do_log": 1,
"user_region": "null",
"is_valid": "yes",
"groups": [
{
"clause_index": 0,
"not_flag": 0,
"regions": [
{
"table_name": "HTTP_URL",
"table_type": "expr",
"table_content": {
"keywords": "China",
"expr_type": "none",
"match_method": "sub",
"format": "uncase plain"
}
}
]
},
{
"clause_index": 1,
"not_flag": 1,
"regions": [
{
"table_name": "HTTP_URL",
"table_type": "expr",
"table_content": {
"keywords": "USA",
"expr_type": "none",
"match_method": "sub",
"format": "uncase plain"
}
}
]
}
]
}
```
scan sample
```c
const char *string1 = "China";
const char *string2 = "England";
long long results[ARRAY_SIZE] = {0};
size_t n_hit_result = 0;
int thread_id = 0;
const char *table_name = "HTTP_URL";
struct maat_state *state = maat_state_new(maat_inst, thread_id);
int table_id = maat_get_table_id(maat_inst, table_name);
ASSERT_GT(table_id, 0);
int ret = maat_scan_string(maat_inst, table_id, string1, strlen(string1),
results, ARRAY_SIZE, &n_hit_result, state);
EXPECT_EQ(ret, MAAT_SCAN_HALF_HIT);
maat_state_set_last_scan(state);
ret = maat_scan_string(maat_inst, table_id, string2, strlen(string2),
results, ARRAY_SIZE, &n_hit_result, state);
EXPECT_EQ(ret, MAAT_SCAN_HIT);
EXPECT_EQ(n_hit_result, 1);
EXPECT_EQ(results[0], 125);
maat_state_free(state);
```
## group exclude
```json
{
"compile_id": 126,
"service": 1,
"action": 1,
"do_blacklist": 1,
"do_log": 1,
"user_region": "null",
"is_valid": "yes",
"groups": [
{
"group_name": "ExcludeGroup199",
"sub_groups":[
{
"group_name": "ExcludeGroup199_1",
"is_exclude": 0,
"clause_index": 0,
"regions": [
{
"table_name": "HTTP_URL",
"table_type": "expr",
"table_content": {
"keywords": "baidu.com",
"expr_type": "none",
"match_method": "suffix",
"format": "uncase plain"
}
}
]
},
{
"group_name": "ExcludeGroup199_2",
"is_exclude": 1,
"clause_index": 0,
"regions": [
{
"table_name": "HTTP_URL",
"table_type": "expr",
"table_content": {
"keywords": "www.baidu.com",
"expr_type": "none",
"match_method": "exact",
"format": "uncase plain"
}
}
]
}
]
}
]
}
```
scan sample
```c
const char *string_not_hit = "www.baidu.com";
const char *string_hit = "mail.baidu.com";
long long results[ARRAY_SIZE] = {0};
size_t n_hit_result = 0;
int thread_id = 0;
const char *table_name = "HTTP_URL";
struct maat_state *state = maat_state_new(maat_inst, thread_id);
int table_id = maat_get_table_id(maat_inst, table_name);
ASSERT_GT(not_hit_table_id, 0);
int ret = maat_scan_string(maat_inst, table_id, string_not_hit, strlen(string_not_hit),
results, ARRAY_SIZE, &n_hit_result, state);
EXPECT_EQ(ret, MAAT_SCAN_HALF_HIT);
ret = maat_scan_string(maat_inst, table_id, string_hit, strlen(string_hit),
results, ARRAY_SIZE, &n_hit_result, state);
EXPECT_EQ(ret, MAAT_SCAN_HIT);
EXPECT_EQ(n_hit_result, 1);
EXPECT_EQ(results[0], 126);
maat_state_free(state);
```

4
docs/scan_api.md
View File

@@ -4,9 +4,9 @@
多命中情况下，按包含分组数由多到少返回，分组数相同时，按编译配置ID由大到小的顺序返回。
多命中扫描的最大命中次数，受MAX_SCANNER_HIT_NUM宏控制，当前为64条。
多命中扫描的最大命中次数，受MAX_SCANNER_HIT_NUM宏控制，当前为4096条。
如果命中条数超出64，则按照配置在IRIS库表文件出现的顺序返回。
如果命中条数超出4096，则按照配置在IRIS库表文件出现的顺序返回。
## 流式跨包扫描

4
docs/table_data.md
View File

@@ -2,7 +2,7 @@
Input must use UTF-8 without BOM encoding, for example, MySQL use utf8mb4.
Maat supports three configuration loading modes.
Maat supports three rule loading modes.
- [Redis mode](#1-redis-mode)
- [Iris mode](#2-iris-mode)
- [Json mode](#3-json-mode)
@@ -392,8 +392,6 @@ Maat实例的工作线程定时轮询Redis中MAAT_VERSION，如果大于实例
```
## 组合配置写入示例
已知IP类域配置表IP_TABLE、关键字类配置表KEYWORD、编译配置表COMPILE、分组表GROUP。

22
docs/table_schema.md
View File

@@ -29,7 +29,7 @@ Item tables are further subdivided into different types of subtables as follows:
Each item table must has the following columns:
- item_id: In a maat instance, the item id is globally unique, meaning that the item IDs of different tables must not be duplicate.
- item_id: In a maat instance, the item id is globally unique, meaning that the item id of different tables must not be duplicate.
- group_id: Indicate the group to which the item belongs, an item belongs to only one group.
@@ -48,7 +48,7 @@ Describe matching rules for strings.
| **group_id** | LONG LONG | N | group2group or group2compile table's group_id |
| **keywords** | VARCHAR2(1024) | N | field to match during scanning |
| **expr_type** | INT | N | 0(keywords), 1(AND expr), 2(regular expr), 3(substring with offset)
| **match_method** | INT | N | only useful when expr_type is 0 |
| **match_method** | INT | N | only useful when expr_type is 0. 0(sub), 1(suffix), 2(prefix), 3(exactly) |
| **is_hexbin** | INT | N | 0(not HEX & case insensitive, this is default value) 1(HEX & case sensitive) 2(not HEX & case sensitive) |
| **is_valid** | INT | N | 0(invalid), 1(valid) |
@@ -123,7 +123,7 @@ Describe matching rules for strings.
For example: substring expr: "1-1:48&3-4:4C4C", scan_data: "HELLO" will hit, "HLLO" will not hit.
**Note**: 48('H') 4C('L')
  Since Maat4.0，only support UTF-8，no more encoding conversion。For binary format configurations, the keyword is hexadecimal, such as the keyword "hello" is represented as "68656C6C6F". A keyword can't contain invisible characters such as spaces, tabs, and CR, which are ASCII codes 0x00 to 0x1F and 0x7F. If these characters need to be used, they must be escaped, refer to the "keywords escape table". Characters led by backslashes outside this table are processed as ordinary strings, such as '\t' will be processed as the string "\t".
  Since Maat4.0，only support UTF-8，no more encoding conversion。For binary format rules, the keyword is hexadecimal, such as the keyword "hello" is represented as "68656C6C6F". A keyword can't contain invisible characters such as spaces, tabs, and CR, which are ASCII codes 0x00 to 0x1F and 0x7F. If these characters need to be used, they must be escaped, refer to the "keywords escape table". Characters led by backslashes outside this table are processed as ordinary strings, such as '\t' will be processed as the string "\t".
The symbol '&' means conjunction operation in AND expression. So if the keywords has '&', it must be escaped by '\&'.
@@ -603,18 +603,18 @@ Describe the specific policy, One maat instance can has multiple compile tables
### 5. <a name='plugintable'></a> plugin table
There is no fixed format for configuration of the plugin table, which is determined by business side. The plugin table supports two sets of callback functions, registered with **maat_table_callback_register** and **maat_plugin_table_ex_schema_register** respectively.
There is no fixed rule format of the plugin table, which is determined by business side. The plugin table supports two sets of callback functions, registered with **maat_table_callback_register** and **maat_plugin_table_ex_schema_register** respectively.
maat_table_callback_register
```c
/*
When the plugin table configurations are updated, start will be called first and only once, then update will be called by each configuration item, and finish will be called last and only once.
When the plugin table rules are updated, start will be called first and only once, then update will be called by each rule item, and finish will be called last and only once.
If configurations have been loaded but maat_table_callback_register has not yet been called, maat will cache the loaded configurations and perform the callbacks(start, update, finish) when registration is complete.
If rules have been loaded but maat_table_callback_register has not yet been called, maat will cache the loaded rules and perform the callbacks(start, update, finish) when registration is complete.
*/
typedef void maat_start_callback_t(int update_type, ...);
//table_line points to one complete configuration line, such as: "1\tHeBei\tShijiazhuang\t1\t0"
//table_line points to one complete rule line, such as: "1\tHeBei\tShijiazhuang\t1\t0"
typedef void maat_update_callback_t(..., const char *table_line, ...);
typedef void maat_finish_callback_t(...);
@@ -643,7 +643,7 @@ int maat_plugin_table_ex_schema_register(...,
...);
```
three types of keys(pointer, integer and ip_addr) for ex_data callback.
Plugin table supports three types of keys(pointer, integer and ip_addr) for ex_data callback.
**pointer key(compatible with maat3)**
@@ -662,7 +662,7 @@ three types of keys(pointer, integer and ip_addr) for ex_data callback.
}
```
(2) plugin table configuration
(2) plugin table rules
```json
{
"table_name": "TEST_PLUGIN_POINTER_KEY_TYPE",
@@ -722,7 +722,7 @@ support integers of different lengths, such as int(4 bytes), long long(8 bytes).
}
```
(2) plugin table configuration
(2) plugin table rules
```
{
"table_name": "TEST_PLUGIN_INT_KEY_TYPE",
@@ -782,7 +782,7 @@ support ip address(ipv4 or ipv6) as key.
```
The addr_type column indicates whether the key is a v4 or v6 address.
(2) plugin table configuration
(2) plugin table rules
```
{
"table_name": "TEST_PLUGIN_IP_KEY_TYPE",

3
docs/thread_mode.md
View File

@@ -46,10 +46,11 @@ int main()
maat_options_set_caller_thread_number(opts, thread_num);
struct maat *maat_inst = maat_new(opts, table_info_path);
size_t i = 0;
pthread_t threads[thread_num];
struct thread_param thread_params[thread_num];
for (size_t i = 0; i < thread_num; i++) {
for (i = 0; i < thread_num; i++) {
thread_params[i].maat_inst = maat_inst;
thread_params[i].thread_id = i;
thread_params[i].table_name = table_name;

56
include/maat_command.h Normal file
View File

@@ -0,0 +1,56 @@
/*
**********************************************************************************************
* File: maat_command.h
* Description:
* Authors: Liu WenTan <liuwentan@geedgenetworks.com>
* Date: 2022-10-31
* Copyright: (c) Since 2022 Geedge Networks, Ltd. All rights reserved.
***********************************************************************************************
*/
#ifndef _MAAT_COMMAND_H_
#define _MAAT_COMMAND_H_
#ifdef __cplusplus
extern "C"
{
#endif
#include <limits.h>
#include "maat.h"
enum maat_operation {
MAAT_OP_DEL = 0,
MAAT_OP_ADD,
MAAT_OP_RENEW_TIMEOUT //Rule expire time is changed to now+cmd->expire_after
};
struct maat_cmd_line {
const char *table_name;
const char *table_line;
long long rule_id; // for MAAT_OP_DEL, only rule_id and table_name are necessary.
int expire_after; //expired after $timeout$ seconds, set to 0 for never timeout.
};
/**
* @brief write one line to redis
*
* @retval
* success: number of successfully updated rule.
* failed: -1
*/
int maat_cmd_set_line(struct maat *maat_instance, const struct maat_cmd_line *line_rule);
int maat_cmd_set_file(struct maat *maat_instance, const char *key, const char *value,
size_t size, enum maat_operation op);
long long maat_cmd_incrby(struct maat *maat_instance, const char *key, int increment);
int maat_cmd_flushDB(struct maat *maat_instance);
#ifdef __cplusplus
}
#endif
#endif

2
readme.md
View File

@@ -193,6 +193,8 @@ int main()
* [Rules](./docs/table_data.md)
* [Logic (AND | OR | NOT)](./docs/logic%20(AND%20%7C%20OR%20%7C%20NOT).md)
* [Scan API](./docs/scan_api.md)
* [Thread mode](./docs/thread_mode.md)

1
src/CMakeLists.txt
View File

@@ -57,4 +57,5 @@ target_link_libraries(maat_frame_shared adapter-static)
# install
set(CMAKE_INSTALL_PREFIX /opt/MESA/)
install(FILES ${PROJECT_SOURCE_DIR}/include/maat.h DESTINATION ${CMAKE_INSTALL_PREFIX}/include/MESA/ COMPONENT HEADER)
install(FILES ${PROJECT_SOURCE_DIR}/include/maat_command.h DESTINATION ${CMAKE_INSTALL_PREFIX}/include/MESA/ COMPONENT HEADER)
install(TARGETS maat_frame_shared LIBRARY DESTINATION ${CMAKE_INSTALL_PREFIX}/lib/ COMPONENT LIBRARIES)

130
src/inc_internal/maat_command.h
View File

@@ -1,130 +0,0 @@
/*
**********************************************************************************************
* File: maat_command.h
* Description:
* Authors: Liu WenTan <liuwentan@geedgenetworks.com>
* Date: 2022-10-31
* Copyright: (c) Since 2022 Geedge Networks, Ltd. All rights reserved.
***********************************************************************************************
*/
#ifndef _MAAT_COMMAND_H_
#define _MAAT_COMMAND_H_
#ifdef __cplusplus
extern "C"
{
#endif
#include <limits.h>
#include <sys/queue.h>
#include "maat.h"
#include "maat_table.h"
#include "log/log.h"
#include "uthash/uthash.h"
#include "hiredis/hiredis.h"
#include "maat_limits.h"
enum maat_operation {
MAAT_OP_DEL = 0,
MAAT_OP_ADD,
MAAT_OP_RENEW_TIMEOUT //Rule expire time is changed to now+cmd->expire_after
};
struct maat_cmd_line {
const char *table_name;
const char *table_line;
long long rule_id; // for MAAT_OP_DEL, only rule_id and table_name are necessary.
int expire_after; //expired after $timeout$ seconds, set to 0 for never timeout.
};
struct foreign_key {
int column;
char *key;
size_t key_len;
char *filename;
};
//rm= Redis Maat
struct serial_rule {
enum maat_operation op;//0: delete, 1: add.
long long rule_id;
long long timeout; // absolute unix time.
char table_name[MAX_NAME_STR_LEN];
char *table_line;
int n_foreign;
struct foreign_key *f_keys;
redisContext *ref_ctx;
TAILQ_ENTRY(serial_rule) entries;
UT_hash_handle hh;
};
/**
* @brief write one line to redis
*
* @retval
* success: number of successfully updated rule.
* failed: -1
*/
int maat_cmd_set_line(struct maat *maat_instance, const struct maat_cmd_line *line_rule);
int maat_cmd_set_file(struct maat *maat_instance, const char *key, const char *value,
size_t size, enum maat_operation op);
long long maat_cmd_incrby(struct maat *maat_instance, const char *key, int increment);
long long maat_cmd_get_config_version(struct maat *maat_instance);
/* True(1), False(0) */
int maat_cmd_config_is_updating(struct maat *maat_instance);
char *maat_cmd_str_escape(char *dst, int size, const char *src);
int maat_cmd_flushDB(struct maat *maat_instance);
/* maat command API for internal */
redisContext *maat_cmd_connect_redis(const char *redis_ip, int redis_port,
int redis_db, struct log_handle *logger);
redisReply *maat_cmd_wrap_redis_command(redisContext *c, const char *format, ...);
int maat_cmd_wrap_redis_get_reply(redisContext *c, redisReply **reply);
long long maat_cmd_redis_server_time_s(redisContext *c);
long long maat_cmd_read_redis_integer(const redisReply *reply);
int maat_cmd_get_valid_flag_offset(const char *line, int column_seq);
int maat_cmd_write_rule(redisContext *c, struct serial_rule *s_rule,
size_t serial_rule_num, long long server_time,
struct log_handle *logger);
void maat_cmd_clear_rule_cache(struct serial_rule *s_rule);
int maat_cmd_get_redis_value(redisContext *c, struct serial_rule *rule_list,
int rule_num, int print_process, struct log_handle *logger);
int maat_cmd_get_foreign_keys_by_prefix(redisContext *ctx, struct serial_rule *rule_list,
int rule_num, const char* dir, struct log_handle *logger);
void maat_cmd_get_foreign_conts(redisContext *c, struct serial_rule *rule_list,
int rule_num, int print_fn, struct log_handle *logger);
void maat_cmd_rewrite_table_line_with_foreign(struct serial_rule *s_rule);
void maat_cmd_set_serial_rule(struct serial_rule *rule, enum maat_operation op,
long long rule_id, const char *table_name,
const char *line, long long timeout);
int maat_cmd_get_rm_key_list(redisContext *c, long long instance_version,
long long desired_version, long long *new_version,
struct table_manager *tbl_mgr, struct serial_rule **list,
int *update_type, int cumulative_off, struct log_handle *logger);
#ifdef __cplusplus
}
#endif
#endif

65
src/inc_internal/maat_redis_monitor.h
View File

@@ -17,6 +17,71 @@ extern "C"
#endif
#include <stdint.h>
#include <sys/queue.h>
#include "hiredis/hiredis.h"
#include "maat_command.h"
#include "uthash/uthash.h"
struct foreign_key {
int column;
char *key;
size_t key_len;
char *filename;
};
//rm= Redis Maat
struct serial_rule {
enum maat_operation op;//0: delete, 1: add.
long long rule_id;
long long timeout; // absolute unix time.
char table_name[MAX_NAME_STR_LEN];
char *table_line;
int n_foreign;
struct foreign_key *f_keys;
redisContext *ref_ctx;
TAILQ_ENTRY(serial_rule) entries;
UT_hash_handle hh;
};
void maat_clear_rule_cache(struct serial_rule *s_rule);
void maat_set_serial_rule(struct serial_rule *rule, enum maat_operation op,
long long rule_id, const char *table_name,
const char *line, long long timeout);
redisContext *maat_connect_redis(const char *redis_ip, int redis_port,
int redis_db, struct log_handle *logger);
redisReply *maat_wrap_redis_command(redisContext *c, const char *format, ...);
int maat_wrap_redis_get_reply(redisContext *c, redisReply **reply);
long long maat_redis_server_time_s(redisContext *c);
long long maat_read_redis_integer(const redisReply *reply);
int maat_get_valid_flag_offset(const char *line, int column_seq);
int maat_cmd_write_rule(redisContext *c, struct serial_rule *s_rule,
size_t serial_rule_num, long long server_time,
struct log_handle *logger);
int maat_get_redis_value(redisContext *c, struct serial_rule *rule_list,
int rule_num, int print_process, struct log_handle *logger);
int maat_get_foreign_keys_by_prefix(redisContext *ctx, struct serial_rule *rule_list,
int rule_num, const char* dir, struct log_handle *logger);
void maat_get_foreign_conts(redisContext *c, struct serial_rule *rule_list,
int rule_num, int print_fn, struct log_handle *logger);
void maat_rewrite_table_line_with_foreign(struct serial_rule *s_rule);
int maat_get_rm_key_list(redisContext *c, long long instance_version,
long long desired_version, long long *new_version,
struct table_manager *tbl_mgr, struct serial_rule **list,
int *update_type, int cumulative_off, struct log_handle *logger);
void redis_monitor_traverse(long long version, struct source_redis_ctx *mr_ctx,
void (*start_fn)(long long, int, void *),

1
src/inc_internal/maat_utils.h
View File

@@ -93,6 +93,7 @@ int load_file_to_memory(const char *file_name, unsigned char **pp_out, size_t *o
char *strtok_r_esc(char *s, const char delim, char **save_ptr);
char *str_escape(char *dst, int size, const char *src);
char *str_unescape(char *s);
char *md5_file(const char *filename, char *md5string);

5
src/json2iris.c
View File

@@ -21,6 +21,7 @@
#include "maat_utils.h"
#include "maat_rule.h"
#include "uthash/uthash.h"
#include "maat_redis_monitor.h"
#define MODULE_JSON2IRIS module_name_str("maat.json2iris")
@@ -300,7 +301,7 @@ static int get_group_seq(struct iris_description *iris_cfg)
if (NULL == iris_cfg->redis_write_ctx) {
sequence = iris_cfg->group_cnt;
} else {
data_reply = maat_cmd_wrap_redis_command(iris_cfg->redis_write_ctx,
data_reply = maat_wrap_redis_command(iris_cfg->redis_write_ctx,
"INCRBY %s 1", mr_group_id_var);
sequence = (int)data_reply->integer - 1;
freeReplyObject(data_reply);
@@ -337,7 +338,7 @@ static int get_region_seq(struct iris_description *iris_cfg)
if (NULL == iris_cfg->redis_write_ctx) {
sequence = iris_cfg->region_cnt;
} else {
redisReply *data_reply = maat_cmd_wrap_redis_command(iris_cfg->redis_write_ctx,
redisReply *data_reply = maat_wrap_redis_command(iris_cfg->redis_write_ctx,
"INCRBY %s 1", mr_region_id_var);
sequence = (int)data_reply->integer - 1;
freeReplyObject(data_reply);

138
src/maat_command.c
View File

@@ -18,6 +18,7 @@
#include "maat_rule.h"
#include "hiredis/hiredis.h"
#include "maat_config_monitor.h"
#include "maat_redis_monitor.h"
#define MODULE_MAAT_COMMAND module_name_str("maat.command")
@@ -34,7 +35,7 @@ extern const char *mr_label_sset;
extern const int MAAT_REDIS_SYNC_TIME;
redisReply *maat_cmd_wrap_redis_command(redisContext *c, const char *format, ...)
redisReply *maat_wrap_redis_command(redisContext *c, const char *format, ...)
{
va_list ap;
void *reply = NULL;
@@ -54,7 +55,7 @@ redisReply *maat_cmd_wrap_redis_command(redisContext *c, const char *format, ...
return (redisReply *)reply;
}
redisContext *maat_cmd_connect_redis(const char *redis_ip, int redis_port,
redisContext *maat_connect_redis(const char *redis_ip, int redis_port,
int redis_db, struct log_handle *logger)
{
struct timeval connect_timeout;
@@ -80,14 +81,14 @@ redisContext *maat_cmd_connect_redis(const char *redis_ip, int redis_port,
}
redisEnableKeepAlive(c);
redisReply *reply = maat_cmd_wrap_redis_command(c, "select %d", redis_db);
redisReply *reply = maat_wrap_redis_command(c, "select %d", redis_db);
freeReplyObject(reply);
reply = NULL;
return c;
}
long long maat_cmd_read_redis_integer(const redisReply *reply)
long long maat_read_redis_integer(const redisReply *reply)
{
switch (reply->type) {
case REDIS_REPLY_INTEGER:
@@ -112,26 +113,26 @@ static int redis_flushDB(redisContext *ctx, int db_index,
{
long long maat_redis_version = 0;
redisReply *data_reply = maat_cmd_wrap_redis_command(ctx, "WATCH MAAT_VERSION");
redisReply *data_reply = maat_wrap_redis_command(ctx, "WATCH MAAT_VERSION");
freeReplyObject(data_reply);
data_reply = NULL;
data_reply = maat_cmd_wrap_redis_command(ctx, "GET MAAT_VERSION");
data_reply = maat_wrap_redis_command(ctx, "GET MAAT_VERSION");
if (data_reply->type == REDIS_REPLY_NIL) {
maat_redis_version = 0;
} else {
maat_redis_version = maat_cmd_read_redis_integer(data_reply);
maat_redis_version = maat_read_redis_integer(data_reply);
maat_redis_version++;
freeReplyObject(data_reply);
data_reply = NULL;
}
data_reply = maat_cmd_wrap_redis_command(ctx, "DBSIZE");
long long dbsize = maat_cmd_read_redis_integer(data_reply);
data_reply = maat_wrap_redis_command(ctx, "DBSIZE");
long long dbsize = maat_read_redis_integer(data_reply);
freeReplyObject(data_reply);
data_reply = NULL;
data_reply = maat_cmd_wrap_redis_command(ctx, "MULTI");
data_reply = maat_wrap_redis_command(ctx, "MULTI");
freeReplyObject(data_reply);
data_reply = NULL;
@@ -152,7 +153,7 @@ static int redis_flushDB(redisContext *ctx, int db_index,
int ret = 0;
int redis_transaction_success = 1;
for (int i = 0; i < append_cmd_cnt; i++) {
ret = maat_cmd_wrap_redis_get_reply(ctx, &data_reply);
ret = maat_wrap_redis_get_reply(ctx, &data_reply);
if (ret == REDIS_OK) {
if (data_reply->type == REDIS_REPLY_NIL) {
redis_transaction_success = 0;
@@ -175,7 +176,7 @@ static int connect_redis_for_write(struct source_redis_ctx *redis_ctx,
struct log_handle *logger)
{
assert(redis_ctx->write_ctx == NULL);
redis_ctx->write_ctx = maat_cmd_connect_redis(redis_ctx->redis_ip,
redis_ctx->write_ctx = maat_connect_redis(redis_ctx->redis_ip,
redis_ctx->redis_port,
redis_ctx->redis_db, logger);
if (NULL == redis_ctx->write_ctx) {
@@ -215,40 +216,7 @@ int maat_cmd_flushDB(struct maat *maat_inst)
return 0;
}
void maat_cmd_clear_rule_cache(struct serial_rule *s_rule)
{
if (s_rule->table_line != NULL) {
FREE(s_rule->table_line);
}
if (s_rule->n_foreign > 0) {
for (int i = 0; i < s_rule->n_foreign; i++) {
FREE(s_rule->f_keys[i].filename);
FREE(s_rule->f_keys[i].key);
}
FREE(s_rule->f_keys);
}
memset(s_rule, 0, sizeof(struct serial_rule));
}
void maat_cmd_set_serial_rule(struct serial_rule *rule, enum maat_operation op,
long long rule_id, const char *table_name,
const char *line, long long timeout)
{
memset(rule, 0, sizeof(struct serial_rule));
rule->op = op;
rule->rule_id = rule_id;
rule->timeout = timeout;
assert(strlen(table_name) < sizeof(rule->table_name));
strncpy(rule->table_name, table_name, sizeof(rule->table_name));
if (line != NULL) {
rule->table_line = maat_strdup(line);
}
}
int maat_cmd_get_valid_flag_offset(const char *line, int column_seq)
int maat_get_valid_flag_offset(const char *line, int column_seq)
{
size_t offset = 0;
size_t len = 0;
@@ -263,11 +231,11 @@ int maat_cmd_get_valid_flag_offset(const char *line, int column_seq)
return offset;
}
long long maat_cmd_redis_server_time_s(redisContext *c)
long long maat_redis_server_time_s(redisContext *c)
{
long long server_time = 0;
redisReply *data_reply = maat_cmd_wrap_redis_command(c, "TIME");
redisReply *data_reply = maat_wrap_redis_command(c, "TIME");
if (data_reply->type == REDIS_REPLY_ARRAY) {
server_time = atoll(data_reply->element[0]->str);
freeReplyObject(data_reply);
@@ -277,7 +245,7 @@ long long maat_cmd_redis_server_time_s(redisContext *c)
return server_time;
}
int maat_cmd_wrap_redis_get_reply(redisContext *c, redisReply **reply)
int maat_wrap_redis_get_reply(redisContext *c, redisReply **reply)
{
return redisGetReply(c, (void **)reply);
}
@@ -293,7 +261,7 @@ int maat_cmd_set_line(struct maat *maat_inst, const struct maat_cmd_line *line_r
return -1;
}
long long server_time = maat_cmd_redis_server_time_s(write_ctx);
long long server_time = maat_redis_server_time_s(write_ctx);
if(!server_time) {
return -1;
}
@@ -318,7 +286,7 @@ int maat_cmd_set_line(struct maat *maat_inst, const struct maat_cmd_line *line_r
return -1;
}
int valid_offset = maat_cmd_get_valid_flag_offset(line_rule->table_line, valid_column);
int valid_offset = maat_get_valid_flag_offset(line_rule->table_line, valid_column);
if (valid_offset < 0) {
log_error(maat_inst->logger, MODULE_MAAT_COMMAND,
"[%s:%d] Command set line id %lld failed: table %s valid_offset error",
@@ -332,7 +300,7 @@ int maat_cmd_set_line(struct maat *maat_inst, const struct maat_cmd_line *line_r
absolute_expire_time = server_time + line_rule->expire_after;
}
maat_cmd_set_serial_rule(s_rule + i, (enum maat_operation)is_valid, line_rule->rule_id,
maat_set_serial_rule(s_rule + i, (enum maat_operation)is_valid, line_rule->rule_id,
line_rule->table_name, line_rule->table_line, absolute_expire_time);
int success_cnt = maat_cmd_write_rule(write_ctx, s_rule, 1, server_time, maat_inst->logger);
@@ -345,7 +313,7 @@ int maat_cmd_set_line(struct maat *maat_inst, const struct maat_cmd_line *line_r
maat_inst->stat->line_cmd_acc_num += success_cnt;
error_out:
maat_cmd_clear_rule_cache(s_rule);
maat_clear_rule_cache(s_rule);
FREE(s_rule);
return ret;
@@ -387,7 +355,7 @@ int maat_cmd_set_file(struct maat *maat_inst, const char *key, const char *value
arg_vec, len_vec);
break;
case MAAT_OP_DEL:
reply = maat_cmd_wrap_redis_command(ctx, "EXPIRE %s %d", key, MAAT_REDIS_SYNC_TIME);
reply = maat_wrap_redis_command(ctx, "EXPIRE %s %d", key, MAAT_REDIS_SYNC_TIME);
break;
default:
return -1;
@@ -416,7 +384,7 @@ long long maat_cmd_incrby(struct maat *maat_inst, const char *key, int increment
return -1;
}
redisReply *data_reply = maat_cmd_wrap_redis_command(write_ctx, "INCRBY %s %d", key, increment);
redisReply *data_reply = maat_wrap_redis_command(write_ctx, "INCRBY %s %d", key, increment);
if (data_reply->type == REDIS_REPLY_INTEGER) {
result = data_reply->integer;
} else {
@@ -428,63 +396,3 @@ long long maat_cmd_incrby(struct maat *maat_inst, const char *key, int increment
return result;
}
long long maat_cmd_get_config_version(struct maat *maat_inst)
{
long long new_version = -1;
if (maat_inst->new_version != INVALID_VERSION) {
new_version = maat_inst->new_version;
} else {
new_version = maat_inst->maat_version;
}
return new_version;
}
int maat_cmd_config_is_updating(struct maat *maat_inst)
{
int ret = 0;
if (0 == pthread_mutex_trylock(&(maat_inst->background_update_mutex))) {
ret = 0;
pthread_mutex_unlock(&(maat_inst->background_update_mutex));
} else {
ret = 1;
}
return ret;
}
char *maat_cmd_str_escape(char *dst, int size, const char *src)
{
int i = 0, j = 0;
int len = strlen(src);
for (i = 0, j = 0; i < len && j < size; i++) {
switch (src[i]) {
case '&':
dst[j] = '\\';
dst[j+1] = '&';
j += 2;
break;
case ' ':
dst[j] = '\\';
dst[j+1] = 'b';//space,0x20;
j += 2;
break;
case '\\':
dst[j] = '\\';
dst[j+1] = '\\';
j += 2;
break;
default:
dst[j] = src[i];
j++; //undo the followed i++
break;
}
}
dst[j] = '\0';
return dst;
}

113
src/maat_redis_monitor.c
View File

@@ -184,7 +184,7 @@ static int get_foreign_keys_define(redisContext *ctx, struct serial_rule *rule_l
return rule_with_foreign_key;
}
int maat_cmd_get_foreign_keys_by_prefix(redisContext *ctx, struct serial_rule *rule_list,
int maat_get_foreign_keys_by_prefix(redisContext *ctx, struct serial_rule *rule_list,
int rule_num, const char* dir, struct log_handle *logger)
{
int j = 0;
@@ -243,7 +243,7 @@ static int _get_maat_redis_value(redisContext *c, struct serial_rule *rule_list,
}
for (i = 0; i < rule_num; i++) {
ret = maat_cmd_wrap_redis_get_reply(c, &reply);
ret = maat_wrap_redis_get_reply(c, &reply);
if (ret == REDIS_ERR) {
log_error(logger, MODULE_REDIS_MONITOR,
"[%s:%d] Redis GET %s:%s,%lld failed, redis server error",
@@ -289,7 +289,7 @@ static int _get_maat_redis_value(redisContext *c, struct serial_rule *rule_list,
for (i = 0; i < failed_cnt; i++) {
idx = retry_ids[i];
ret = maat_cmd_wrap_redis_get_reply(c, &reply);
ret = maat_wrap_redis_get_reply(c, &reply);
if (ret == REDIS_ERR) {
log_error(logger, MODULE_REDIS_MONITOR,
"[%s:%d] redis command %s failed, redis server error",
@@ -321,7 +321,7 @@ static int _get_maat_redis_value(redisContext *c, struct serial_rule *rule_list,
return 0;
}
int maat_cmd_get_redis_value(redisContext *c, struct serial_rule *rule_list,
int maat_get_redis_value(redisContext *c, struct serial_rule *rule_list,
int rule_num, int print_process, struct log_handle *logger)
{
int max_redis_batch = 4096;
@@ -377,7 +377,7 @@ static int get_inc_key_list(long long instance_version, long long target_version
return 0;
}
redisReply *tmp_reply= maat_cmd_wrap_redis_command(c, "ZSCORE %s %s",
redisReply *tmp_reply= maat_wrap_redis_command(c, "ZSCORE %s %s",
mr_status_sset,
reply->element[0]->str);
if (tmp_reply->type != REDIS_REPLY_STRING) {
@@ -393,7 +393,7 @@ static int get_inc_key_list(long long instance_version, long long target_version
return -1;
}
long long nearest_rule_version = maat_cmd_read_redis_integer(tmp_reply);
long long nearest_rule_version = maat_read_redis_integer(tmp_reply);
freeReplyObject(tmp_reply);
tmp_reply = NULL;
@@ -444,6 +444,39 @@ static int get_inc_key_list(long long instance_version, long long target_version
return rule_num;
}
void maat_clear_rule_cache(struct serial_rule *s_rule)
{
if (s_rule->table_line != NULL) {
FREE(s_rule->table_line);
}
if (s_rule->n_foreign > 0) {
for (int i = 0; i < s_rule->n_foreign; i++) {
FREE(s_rule->f_keys[i].filename);
FREE(s_rule->f_keys[i].key);
}
FREE(s_rule->f_keys);
}
memset(s_rule, 0, sizeof(struct serial_rule));
}
void maat_set_serial_rule(struct serial_rule *rule, enum maat_operation op,
long long rule_id, const char *table_name,
const char *line, long long timeout)
{
memset(rule, 0, sizeof(struct serial_rule));
rule->op = op;
rule->rule_id = rule_id;
rule->timeout = timeout;
assert(strlen(table_name) < sizeof(rule->table_name));
strncpy(rule->table_name, table_name, sizeof(rule->table_name));
if (line != NULL) {
rule->table_line = maat_strdup(line);
}
}
static void serial_rule_free(struct serial_rule *s_rule)
{
if (NULL == s_rule) {
@@ -559,7 +592,7 @@ static int recovery_history_version(const struct serial_rule *current, int curre
return ret;
}
int maat_cmd_get_rm_key_list(redisContext *c, long long instance_version,
int maat_get_rm_key_list(redisContext *c, long long instance_version,
long long desired_version, long long *new_version,
struct table_manager *tbl_mgr, struct serial_rule **list,
int *update_type, int cumulative_off, struct log_handle *logger)
@@ -585,7 +618,7 @@ int maat_cmd_get_rm_key_list(redisContext *c, long long instance_version,
return -1;
}
long long redis_version = maat_cmd_read_redis_integer(reply);
long long redis_version = maat_read_redis_integer(reply);
if (redis_version < 0) {
if (reply->type == REDIS_REPLY_ERROR) {
log_error(logger, MODULE_REDIS_MONITOR,
@@ -667,12 +700,12 @@ FULL_UPDATE:
size_t i = 0;
//consume reply "OK" and "QUEUED".
for (i = 0; i < append_cmd_cnt; i++) {
maat_cmd_wrap_redis_get_reply(c, &reply);
maat_wrap_redis_get_reply(c, &reply);
freeReplyObject(reply);
reply = NULL;
}
reply = maat_cmd_wrap_redis_command(c, "EXEC");
reply = maat_wrap_redis_command(c, "EXEC");
if (NULL == reply) {
log_error(logger, MODULE_REDIS_MONITOR,
"[%s:%d] Redis Communication error: %s",
@@ -689,7 +722,7 @@ FULL_UPDATE:
return -1;
}
*new_version = maat_cmd_read_redis_integer(reply->element[0]);
*new_version = maat_read_redis_integer(reply->element[0]);
redisReply *sub_reply = reply->element[1];
if (sub_reply->type != REDIS_REPLY_ARRAY) {
log_error(logger, MODULE_REDIS_MONITOR,
@@ -829,7 +862,7 @@ static void _get_foreign_conts(redisContext *c, struct serial_rule *rule_list,
redisReply *reply = NULL;
for (i = 0; i < key_num; i++) {
ret = maat_cmd_wrap_redis_get_reply(c, &reply);
ret = maat_wrap_redis_get_reply(c, &reply);
if (ret == REDIS_ERR) {
log_error(logger, MODULE_REDIS_MONITOR,
"[%s:%d] Get %s,%lld foreign key %s content failed, redis server error",
@@ -874,7 +907,7 @@ static void _get_foreign_conts(redisContext *c, struct serial_rule *rule_list,
return;
}
void maat_cmd_get_foreign_conts(redisContext *c, struct serial_rule *rule_list,
void maat_get_foreign_conts(redisContext *c, struct serial_rule *rule_list,
int rule_num, int print_fn, struct log_handle *logger)
{
int max_redis_batch = 4096;
@@ -893,7 +926,7 @@ static int invalidate_line(char *line, int column_seq)
return -1;
}
int offset = maat_cmd_get_valid_flag_offset(line, column_seq);
int offset = maat_get_valid_flag_offset(line, column_seq);
if (offset < 0) {
return -1;
}
@@ -903,7 +936,7 @@ static int invalidate_line(char *line, int column_seq)
return 0;
}
void maat_cmd_rewrite_table_line_with_foreign(struct serial_rule *s_rule)
void maat_rewrite_table_line_with_foreign(struct serial_rule *s_rule)
{
int i = 0;
size_t fn_size = 0;
@@ -950,7 +983,7 @@ static int redlock_try_lock(redisContext *c, const char *lock_name,
{
int ret = 0;
redisReply *reply = maat_cmd_wrap_redis_command(c, "SET %s locked NX PX %lld",
redisReply *reply = maat_wrap_redis_command(c, "SET %s locked NX PX %lld",
lock_name, expire);
if (reply->type == REDIS_REPLY_NIL) {
ret = 0;
@@ -979,8 +1012,8 @@ static long long exec_serial_rule_begin(redisContext* c, size_t rule_num,
}
if (rule_num > renew_rule_num) {
data_reply = maat_cmd_wrap_redis_command(c, "INCRBY MAAT_PRE_VER 1");
*transaction_version = maat_cmd_read_redis_integer(data_reply);
data_reply = maat_wrap_redis_command(c, "INCRBY MAAT_PRE_VER 1");
*transaction_version = maat_read_redis_integer(data_reply);
freeReplyObject(data_reply);
data_reply = NULL;
if (*transaction_version < 0) {
@@ -989,7 +1022,7 @@ static long long exec_serial_rule_begin(redisContext* c, size_t rule_num,
}
if (*renew_allowed == 1 || rule_num > renew_rule_num) {
data_reply = maat_cmd_wrap_redis_command(c, "MULTI");
data_reply = maat_wrap_redis_command(c, "MULTI");
freeReplyObject(data_reply);
data_reply = NULL;
ret = 0;
@@ -1000,7 +1033,7 @@ static long long exec_serial_rule_begin(redisContext* c, size_t rule_num,
static void redlock_unlock(redisContext *c, const char *lock_name)
{
redisReply *reply = maat_cmd_wrap_redis_command(c, "DEL %s", lock_name);
redisReply *reply = maat_wrap_redis_command(c, "DEL %s", lock_name);
freeReplyObject(reply);
reply = NULL;
}
@@ -1027,7 +1060,7 @@ static redisReply* exec_serial_rule_end(redisContext *c, const char *transaction
}
if (strlen(transaction_list) > 0) {
data_reply = maat_cmd_wrap_redis_command(c, "eval %s 4 MAAT_VERSION %s %s %s %lld",
data_reply = maat_wrap_redis_command(c, "eval %s 4 MAAT_VERSION %s %s %s %lld",
lua_exec_done,
mr_status_sset,
mr_version_sset,
@@ -1039,7 +1072,7 @@ static redisReply* exec_serial_rule_end(redisContext *c, const char *transaction
(*cnt)++;
}
data_reply = maat_cmd_wrap_redis_command(c, "EXEC");
data_reply = maat_wrap_redis_command(c, "EXEC");
return data_reply;
}
@@ -1154,7 +1187,7 @@ static void exec_serial_rule(redisContext *c, const char *transaction_list,
}
for (i = 0; i < append_cmd_cnt; i++) {
maat_cmd_wrap_redis_get_reply(c, &data_reply);
maat_wrap_redis_get_reply(c, &data_reply);
freeReplyObject(data_reply);
data_reply=NULL;
}
@@ -1260,7 +1293,7 @@ int maat_cmd_write_rule(redisContext *c, struct serial_rule *s_rule,
}
if (transaction_version > 0) {
transaction_finished_version = maat_cmd_read_redis_integer(transaction_reply->element[multi_cmd_cnt - 1]);
transaction_finished_version = maat_read_redis_integer(transaction_reply->element[multi_cmd_cnt - 1]);
log_info(logger, MODULE_REDIS_MONITOR,
"Redis transaction MAAT_PRE_VER = %lld , MAAT_VERSION = %lld",
transaction_version, transaction_finished_version);
@@ -1297,12 +1330,12 @@ static void cleanup_update_status(redisContext *c, struct log_handle *logger)
long long version_num = 0;
long long entry_num = 0;
long long server_time = maat_cmd_redis_server_time_s(c);
long long server_time = maat_redis_server_time_s(c);
if (!server_time) {
return;
}
redisReply *reply = maat_cmd_wrap_redis_command(c, "MULTI");
redisReply *reply = maat_wrap_redis_command(c, "MULTI");
freeReplyObject(reply);
reply = NULL;
@@ -1317,13 +1350,13 @@ static void cleanup_update_status(redisContext *c, struct log_handle *logger)
//consume reply "OK" and "QUEUED".
for(int i = 0; i < append_cmd_cnt; i++) {
maat_cmd_wrap_redis_get_reply(c, &reply);
maat_wrap_redis_get_reply(c, &reply);
freeReplyObject(reply);
reply = NULL;
}
redisReply *sub_reply = NULL;
reply = maat_cmd_wrap_redis_command(c, "EXEC");
reply = maat_wrap_redis_command(c, "EXEC");
if (reply->type != REDIS_REPLY_ARRAY) {
goto error_out;
}
@@ -1338,16 +1371,16 @@ static void cleanup_update_status(redisContext *c, struct log_handle *logger)
goto error_out;
}
version_lower_bound = maat_cmd_read_redis_integer(sub_reply->element[0]);
version_upper_bound = maat_cmd_read_redis_integer(sub_reply->element[sub_reply->elements-1]);
version_lower_bound = maat_read_redis_integer(sub_reply->element[0]);
version_upper_bound = maat_read_redis_integer(sub_reply->element[sub_reply->elements-1]);
freeReplyObject(reply);
reply = NULL;
//To deal with maat_version reset to 0, do NOT use -inf as lower bound intentionally.
reply = maat_cmd_wrap_redis_command(c, "ZREMRANGEBYSCORE %s %lld %lld",
reply = maat_wrap_redis_command(c, "ZREMRANGEBYSCORE %s %lld %lld",
mr_status_sset, version_lower_bound,
version_upper_bound);
entry_num = maat_cmd_read_redis_integer(reply);
entry_num = maat_read_redis_integer(reply);
freeReplyObject(reply);
reply = NULL;
@@ -1365,12 +1398,12 @@ static void check_maat_expiration(redisContext *c, struct log_handle *logger)
{
UNUSED int ret = 0;
long long server_time = maat_cmd_redis_server_time_s(c);
long long server_time = maat_redis_server_time_s(c);
if (!server_time) {
return;
}
redisReply *data_reply= maat_cmd_wrap_redis_command(c, "ZRANGEBYSCORE %s -inf %lld",
redisReply *data_reply= maat_wrap_redis_command(c, "ZRANGEBYSCORE %s -inf %lld",
mr_expire_sset, server_time);
if (data_reply->type != REDIS_REPLY_ARRAY || 0 == data_reply->elements) {
freeReplyObject(data_reply);
@@ -1442,7 +1475,7 @@ void redis_monitor_traverse(long long version, struct source_redis_ctx *mr_ctx,
}
log_info(maat_inst->logger, MODULE_REDIS_MONITOR, "Reconnecting...");
mr_ctx->read_ctx = maat_cmd_connect_redis(mr_ctx->redis_ip,
mr_ctx->read_ctx = maat_connect_redis(mr_ctx->redis_ip,
mr_ctx->redis_port,
mr_ctx->redis_db,
maat_inst->logger);
@@ -1457,7 +1490,7 @@ void redis_monitor_traverse(long long version, struct source_redis_ctx *mr_ctx,
long long new_version = 0;
int update_type = MAAT_UPDATE_TYPE_INC;
int rule_num = maat_cmd_get_rm_key_list(mr_ctx->read_ctx, version,
int rule_num = maat_get_rm_key_list(mr_ctx->read_ctx, version,
maat_inst->load_specific_version,
&new_version, maat_inst->tbl_mgr,
&rule_list, &update_type,
@@ -1477,7 +1510,7 @@ void redis_monitor_traverse(long long version, struct source_redis_ctx *mr_ctx,
}
if (rule_num > 0) {
ret = maat_cmd_get_redis_value(mr_ctx->read_ctx, rule_list, rule_num,
ret = maat_get_redis_value(mr_ctx->read_ctx, rule_list, rule_num,
0, maat_inst->logger);
//redis communication error
if (ret < 0) {
@@ -1504,7 +1537,7 @@ void redis_monitor_traverse(long long version, struct source_redis_ctx *mr_ctx,
ret = get_foreign_keys_define(mr_ctx->read_ctx, rule_list, rule_num,
maat_inst, maat_inst->opts.foreign_cont_dir);
if (ret > 0) {
maat_cmd_get_foreign_conts(mr_ctx->read_ctx, rule_list, rule_num, 0,
maat_get_foreign_conts(mr_ctx->read_ctx, rule_list, rule_num, 0,
maat_inst->logger);
}
}
@@ -1539,7 +1572,7 @@ void redis_monitor_traverse(long long version, struct source_redis_ctx *mr_ctx,
}
if (rule_list[i].n_foreign > 0) {
maat_cmd_rewrite_table_line_with_foreign(rule_list+i);
maat_rewrite_table_line_with_foreign(rule_list+i);
}
update_fn(rule_list[i].table_name, rule_list[i].table_line, u_param);
@@ -1556,7 +1589,7 @@ void redis_monitor_traverse(long long version, struct source_redis_ctx *mr_ctx,
clean_up:
for (i = 0; i < rule_num; i++) {
maat_cmd_clear_rule_cache(rule_list + i);
maat_clear_rule_cache(rule_list + i);
}
FREE(rule_list);

2
src/maat_rule.c
View File

@@ -276,7 +276,7 @@ void maat_read_full_config(struct maat *maat_inst)
log_info(maat_inst->logger, MODULE_MAAT_RULE,
"Maat initiate from Redis %s:%hu db%d",
redis_ctx->redis_ip, redis_ctx->redis_port, redis_ctx->redis_db);
redis_ctx->read_ctx = maat_cmd_connect_redis(redis_ctx->redis_ip,
redis_ctx->read_ctx = maat_connect_redis(redis_ctx->redis_ip,
redis_ctx->redis_port,
redis_ctx->redis_db,
maat_inst->logger);

33
src/maat_utils.c
View File

@@ -169,6 +169,39 @@ char *strtok_r_esc(char *s, const char delim, char **save_ptr)
return s;
}
char *str_escape(char *dst, int size, const char *src)
{
int i = 0, j = 0;
int len = strlen(src);
for (i = 0, j = 0; i < len && j < size; i++) {
switch (src[i]) {
case '&':
dst[j] = '\\';
dst[j+1] = '&';
j += 2;
break;
case ' ':
dst[j] = '\\';
dst[j+1] = 'b';//space,0x20;
j += 2;
break;
case '\\':
dst[j] = '\\';
dst[j+1] = '\\';
j += 2;
break;
default:
dst[j] = src[i];
j++; //undo the followed i++
break;
}
}
dst[j] = '\0';
return dst;
}
char *str_unescape(char *s)
{
size_t i=0;

1
src/version.map
View File

@@ -16,6 +16,7 @@ global:
maat_state_*;
maat_helper*;
maat_stream_*;
maat_cmd_*;
};
local: *;
};

22
test/maat_demo/CMakeLists.txt Normal file
View File

@@ -0,0 +1,22 @@
include_directories(./)
add_library(gtest-static STATIC IMPORTED GLOBAL)
add_dependencies(gtest-static gtest)
set_property(TARGET gtest-static PROPERTY IMPORTED_LOCATION ${PROJECT_SOURCE_DIR}/lib/libgtest.a)
set_property(TARGET gtest-static PROPERTY INTERFACE_INCLUDE_DIRECTORIES ${PROJECT_SOURCE_DIR}/include)
add_library(maat-shared SHARED IMPORTED GLOBAL)
add_dependencies(maat-shared maat)
set_property(TARGET maat-shared PROPERTY IMPORTED_LOCATION ${PROJECT_SOURCE_DIR}/lib/libmaat4.so)
set_property(TARGET maat-shared PROPERTY INTERFACE_INCLUDE_DIRECTORIES ${PROJECT_SOURCE_DIR}/include)
add_executable(maat_demo_gtest maat_demo_gtest.cpp)
target_link_libraries(maat_demo_gtest maat-shared gtest-static pthread)
file(COPY demo_table_info.conf DESTINATION ./)
file(COPY maat_demo.json DESTINATION ./)
file(COPY testdata DESTINATION ./)

129
test/maat_demo/demo_table_info.conf Normal file
View File

@@ -0,0 +1,129 @@
[
{
"table_id":0,
"table_name":"COMPILE",
"table_type":"compile",
"valid_column":8,
"custom": {
"gc_timeout_s": 3,
"compile_id":1,
"tags":6,
"clause_num":9
}
},
{
"table_id":1,
"table_name":"GROUP2COMPILE",
"table_type":"group2compile",
"associated_compile_table_id":0,
"valid_column":3,
"custom": {
"group_id":1,
"compile_id":2,
"not_flag":4,
"virtual_table_name":5,
"clause_index":6
}
},
{
"table_id":2,
"table_name":"HTTP_URL",
"table_type":"expr",
"valid_column":7,
"custom": {
"item_id":1,
"group_id":2,
"keywords":3,
"expr_type":4,
"match_method":5,
"is_hexbin":6
}
},
{
"table_id":3,
"table_name":"KEYWORDS_TABLE",
"table_type":"expr",
"valid_column":7,
"custom": {
"item_id":1,
"group_id":2,
"keywords":3,
"expr_type":4,
"match_method":5,
"is_hexbin":6
}
},
{
"table_id":4,
"table_name":"HTTP_SIGNATURE",
"table_type":"expr_plus",
"valid_column":8,
"custom": {
"item_id":1,
"group_id":2,
"district":3,
"keywords":4,
"expr_type":5,
"match_method":6,
"is_hexbin":7
}
},
{
"table_id":5,
"table_name":"IMAGE_FP",
"table_type":"expr",
"valid_column":7,
"custom": {
"item_id":1,
"group_id":2,
"keywords":3,
"expr_type":4,
"match_method":5,
"is_hexbin":6
}
},
{
"table_id":6,
"table_name":"APP_PAYLOAD",
"table_type":"expr_plus",
"valid_column":8,
"custom": {
"item_id":1,
"group_id":2,
"district":3,
"keywords":4,
"expr_type":5,
"match_method":6,
"is_hexbin":7
}
},
{
"table_id":7,
"table_name":"TROJAN_PAYLOAD",
"table_type":"expr",
"valid_column":7,
"custom": {
"item_id":1,
"group_id":2,
"keywords":3,
"expr_type":4,
"match_method":5,
"is_hexbin":6
}
},
{
"table_id":8,
"table_name":"MAIL_ADDR",
"table_type":"expr",
"valid_column":7,
"custom": {
"item_id":1,
"group_id":2,
"keywords":3,
"expr_type":4,
"match_method":5,
"is_hexbin":6
}
}
]

284
test/maat_demo/include/maat.h Normal file
View File

@@ -0,0 +1,284 @@
/*
**********************************************************************************************
* Maat: Deep Packet Inspection Policy Framework
* Maat is the Goddess of truth and justice in ancient Egyptian concept.
* Her feather was the measure that determined whether the souls (considered
* to reside in the heart) of the departed would reach the paradise of afterlife
* successfully.
* Authors: Liu WenTan <liuwentan@geedgenetworks.com>
* Date: 2022-10-31
* Copyright: (c) 2018-2023 Geedge Networks, Inc. All rights reserved.
***********************************************************************************************
*/
#ifndef _MAAT_H_
#define _MAAT_H_
#ifdef __cplusplus
extern "C"
{
#endif
#include <stdint.h>
#include <netinet/in.h>
/* maat instance handle */
struct maat;
struct maat_hit_path {
int Nth_scan;
int vtable_id; // 0 is not a virtual table.
long long item_id;
long long sub_group_id;
long long top_group_id;
long long compile_id;
};
struct maat_hit_group {
long long group_id;
int vtable_id;
};
enum maat_scan_status {
MAAT_SCAN_ERR = -1, //scan error
MAAT_SCAN_OK, //scan but not hit(group or compile)
MAAT_SCAN_HALF_HIT, //half hit: hit group, not hit compile
MAAT_SCAN_HIT //scan hit compile
};
enum maat_update_type {
MAAT_UPDATE_TYPE_INVALID = 0,
MAAT_UPDATE_TYPE_FULL,
MAAT_UPDATE_TYPE_INC
};
struct ip_addr {
int ip_type; //4: IPv4, 6: IPv6
union {
unsigned int ipv4; //network order
unsigned int ipv6[4];
};
};
enum log_level {
LOG_LEVEL_TRACE,
LOG_LEVEL_DEBUG,
LOG_LEVEL_INFO,
LOG_LEVEL_WARN,
LOG_LEVEL_ERROR,
LOG_LEVEL_FATAL
};
/* update_type: MAAT_UPDATE_TYPE_FULL or MAAT_UPDATE_TYPE_INC */
typedef void maat_start_callback_t(int update_type, void *u_param);
typedef void maat_update_callback_t(int table_id, const char *table_line, void *u_para);
typedef void maat_finish_callback_t(void *u_para);
typedef void maat_ex_new_func_t(const char *table_name, int table_id, const char *key,
const char *table_line, void **ad, long argl, void *argp);
typedef void maat_ex_free_func_t(int table_id, void **ad, long argl, void *argp);
typedef void maat_ex_dup_func_t(int table_id, void **to, void **from, long argl, void *argp);
/* maat_instance options API */
struct maat_options;
struct maat_options *maat_options_new(void);
void maat_options_free(struct maat_options *opts);
/**
* @brief set maat instance name
*
* @note The maximum length of instance_name is 15 bytes
*/
int maat_options_set_instance_name(struct maat_options *opts, const char *instance_name);
int maat_options_set_caller_thread_number(struct maat_options *opts, size_t n_thread);
int maat_options_set_accept_tags(struct maat_options *opts, const char *accept_tags);
int maat_options_set_rule_effect_interval_ms(struct maat_options *opts, int interval_ms);
int maat_options_set_rule_update_checking_interval_ms(struct maat_options *opts, int interval_ms);
int maat_options_set_gc_timeout_ms(struct maat_options *opts, int interval_ms);
int maat_options_set_deferred_load_on(struct maat_options *opts);
int maat_options_set_stat_on(struct maat_options *opts);
int maat_options_set_perf_on(struct maat_options *opts);
int maat_options_set_foreign_cont_dir(struct maat_options *opts, const char *dir);
int maat_options_set_logger(struct maat_options *opts, const char *log_path,
enum log_level level);
int maat_options_set_iris(struct maat_options *opts, const char *full_directory,
const char *increment_directory);
int maat_options_set_json_file(struct maat_options *opts, const char *json_filename);
/**
* Indicate whether the JSON file is compressed by gzip
* flag: 1(compressed) 0(uncompressed)
* */
int maat_options_set_json_file_gzip_flag(struct maat_options *opts, int flag);
/* Specify the decryption key for the JSON file to be decrypted */
int maat_options_set_json_file_decrypt_key(struct maat_options *opts, const char *decrypt_key);
int maat_options_set_redis(struct maat_options *opts, const char *redis_ip,
uint16_t redis_port, int redis_db);
int maat_options_set_stat_file(struct maat_options *opts, const char *stat_filename);
/* maat_instance API */
struct maat *maat_new(struct maat_options *opts, const char *table_info_path);
void maat_free(struct maat *instance);
/* maat helper API */
int maat_helper_read_column(const char *table_line, int Nth_column,
size_t *column_offset, size_t *column_len);
/**
* verify if regex expression is legal
*
* @param The NULL-terminated expression to parse.
* @retval 1(legal) 0(illegal)
**/
int maat_helper_verify_regex_expression(const char *expression);
/* maat table API */
int maat_get_table_id(struct maat *instance, const char *table_name);
/* return 0 if success, otherwise return -1 */
int maat_table_callback_register(struct maat *instance, int table_id,
maat_start_callback_t *start,
maat_update_callback_t *update,
maat_finish_callback_t *finish,
void *u_para);
/* maat plugin table API */
int maat_plugin_table_ex_schema_register(struct maat *instance, const char *table_name,
maat_ex_new_func_t *new_func,
maat_ex_free_func_t *free_func,
maat_ex_dup_func_t *dup_func,
long argl, void *argp);
/**
* xx_plugin_table_get_ex_data
* returned data is duplicated by dup_func of maat_plugin_table_ex_schema_register,
* caller is responsible to free the data.
*
* free_func support gargbage collection(gc), gc timeout(default 0) can be configured
* in table_info which means maat will not call free_func until the timeout expires
*/
/**
* NOTE: only plugin table support three key type(integer, pointer, ip_addr)
* specified in table_info.conf. If use ip_addr key type, then key should be
* ip address in network order.
*/
void *maat_plugin_table_get_ex_data(struct maat *instance, int table_id,
const char *key, size_t key_len);
int maat_ip_plugin_table_get_ex_data(struct maat *instance, int table_id,
const struct ip_addr *ip, void **ex_data_array,
size_t n_ex_data);
int maat_fqdn_plugin_table_get_ex_data(struct maat *instance, int table_id,
const char *fqdn, void **ex_data_array,
size_t n_ex_data);
int maat_bool_plugin_table_get_ex_data(struct maat *instance, int table_id,
unsigned long long *item_ids, size_t n_item,
void **ex_data_array, size_t n_ex_data);
/* maat scan API */
struct maat_state;
/**
* @param instance: maat instance created by maat_new()
* @param table_id: the id of table which to be scanned
* @param thread_id: thread index
* @param results: array to store hit compile id
* @param n_result: the array size
* @param n_hit_result: the number of hit compile id
* @param state: scan mid status
*
* @retval MAAT_SCAN_ERR
* MAAT_SCAN_OK
* MAAT_SCAN_HALF_HIT
* MAAT_SCAN_HIT
*/
int maat_scan_flag(struct maat *instance, int table_id,
long long flag, long long *results, size_t n_result,
size_t *n_hit_result, struct maat_state *state);
int maat_scan_integer(struct maat *instance, int table_id,
long long integer, long long *results, size_t n_result,
size_t *n_hit_result, struct maat_state *state);
/**
* @param ip_addr: network ipv4 address
* @param port: network port
* @param protocol: -1(ANY protocol) 1(ICMP) 6(TCP) 17(UDP)
*/
int maat_scan_ipv4(struct maat *instance, int table_id,
uint32_t ip_addr, uint16_t port, int protocol,
long long *results, size_t n_result,
size_t *n_hit_result, struct maat_state *state);
int maat_scan_ipv6(struct maat *instance, int table_id,
uint8_t *ip_addr, uint16_t port, int protocol,
long long *results, size_t n_result,
size_t *n_hit_result, struct maat_state *state);
int maat_scan_string(struct maat *instance, int table_id,
const char *data, size_t data_len, long long *results,
size_t n_result, size_t *n_hit_result,
struct maat_state *state);
struct maat_stream;
struct maat_stream *maat_stream_new(struct maat *instance, int table_id,
struct maat_state *state);
int maat_stream_scan(struct maat_stream *stream, const char *data, int data_len,
long long *results, size_t n_result, size_t *n_hit_result,
struct maat_state *state);
void maat_stream_free(struct maat_stream *stream);
/* maat state API */
struct maat_state *maat_state_new(struct maat *instance, int thread_id);
void maat_state_reset(struct maat_state *state);
void maat_state_free(struct maat_state *state);
int maat_state_set_scan_district(struct maat_state *state, int table_id,
const char *district, size_t district_len);
int maat_state_set_last_scan(struct maat_state *state);
int maat_state_set_scan_compile_table(struct maat_state *state, int compile_table_id);
int maat_state_get_hit_paths(struct maat_state *state, struct maat_hit_path *paths,
size_t n_path);
size_t maat_state_get_scan_count(struct maat_state *state);
int maat_state_get_hit_groups(struct maat_state *state, struct maat_hit_group *groups,
size_t n_group);
/* return hit object compile_id */
int maat_hit_group_compile_id(struct maat *instance, struct maat_hit_group *group);
#ifdef __cplusplus
}
#endif
#endif

56
test/maat_demo/include/maat_command.h Normal file
View File

@@ -0,0 +1,56 @@
/*
**********************************************************************************************
* File: maat_command.h
* Description:
* Authors: Liu WenTan <liuwentan@geedgenetworks.com>
* Date: 2022-10-31
* Copyright: (c) Since 2022 Geedge Networks, Ltd. All rights reserved.
***********************************************************************************************
*/
#ifndef _MAAT_COMMAND_H_
#define _MAAT_COMMAND_H_
#ifdef __cplusplus
extern "C"
{
#endif
#include <limits.h>
#include "maat.h"
enum maat_operation {
MAAT_OP_DEL = 0,
MAAT_OP_ADD,
MAAT_OP_RENEW_TIMEOUT //Rule expire time is changed to now+cmd->expire_after
};
struct maat_cmd_line {
const char *table_name;
const char *table_line;
long long rule_id; // for MAAT_OP_DEL, only rule_id and table_name are necessary.
int expire_after; //expired after $timeout$ seconds, set to 0 for never timeout.
};
/**
* @brief write one line to redis
*
* @retval
* success: number of successfully updated rule.
* failed: -1
*/
int maat_cmd_set_line(struct maat *maat_instance, const struct maat_cmd_line *line_rule);
int maat_cmd_set_file(struct maat *maat_instance, const char *key, const char *value,
size_t size, enum maat_operation op);
long long maat_cmd_incrby(struct maat *maat_instance, const char *key, int increment);
int maat_cmd_flushDB(struct maat *maat_instance);
#ifdef __cplusplus
}
#endif
#endif

BIN
test/maat_demo/lib/libgtest.a Normal file
View File

Binary file not shown.

1
test/maat_demo/lib/libmaat4.so Symbolic link
View File

@@ -0,0 +1 @@
libmaat4.so.4

1
test/maat_demo/lib/libmaat4.so.4 Symbolic link
View File

@@ -0,0 +1 @@
libmaat4.so.4.0

BIN
test/maat_demo/lib/libmaat4.so.4.0 Executable file
View File

Binary file not shown.

432
test/maat_demo/maat_demo.json Normal file
View File

@@ -0,0 +1,432 @@
{
"compile_table": "COMPILE",
"group2compile_table": "GROUP2COMPILE",
"group2group_table": "GROUP2GROUP",
"rules": [
{
"compile_id": 125,
"service": 1,
"action": 1,
"do_blacklist": 1,
"do_log": 1,
"user_region": "anything",
"is_valid": "yes",
"groups": [
{
"regions": [
{
"table_name": "HTTP_URL",
"table_type": "expr",
"table_content": {
"keywords": "action=search\\&query=(.*)",
"expr_type": "regex",
"match_method": "sub",
"format": "uncase plain"
}
}
]
}
]
},
{
"compile_id": 128,
"service": 1,
"action": 1,
"do_blacklist": 1,
"do_log": 1,
"user_region": "StringScan.ExprPlus",
"is_valid": "yes",
"groups": [
{
"group_name": "Untitled",
"regions": [
{
"table_name": "HTTP_SIGNATURE",
"table_type": "expr_plus",
"table_content": {
"district": "HtTP\\bUrL",
"keywords": "abckkk&123",
"expr_type": "and",
"match_method": "sub",
"format": "uncase plain"
}
}
]
}
]
},
{
"compile_id": 132,
"service": 1,
"action": 1,
"do_blacklist": 1,
"do_log": 1,
"user_region": "string\\bunescape",
"is_valid": "yes",
"groups": [
{
"group_name": "TakeMeHome",
"regions": [
{
"table_name": "KEYWORDS_TABLE",
"table_type": "expr",
"table_content": {
"keywords": "Take\\bme\\bHome&Batman\\",
"expr_type": "and",
"match_method": "sub",
"format": "uncase plain"
}
}
]
}
]
},
{
"compile_id": 136,
"service": 1,
"action": 1,
"do_blacklist": 1,
"do_log": 1,
"user_region": "offset_string",
"is_valid": "yes",
"groups": [
{
"group_name": "Untitled",
"regions": [
{
"table_name": "IMAGE_FP",
"table_type": "expr",
"table_content": {
"keywords": "4362-4458:323031333A30333A30372032333A35363A313000323031333A30333A30372032333A35363A3130000000FFE20C584943435F50524F46494C4500010100000C484C696E6F021000006D6E74725247422058595A2007CE00020009000600310000",
"expr_type": "offset",
"match_method": "none",
"format": "hexbin"
}
}
]
}
]
},
{
"compile_id": 146,
"service": 1,
"action": 1,
"do_blacklist": 1,
"do_log": 1,
"user_region": "StringScan.Regex",
"is_valid": "yes",
"groups": [
{
"regions": [
{
"table_name": "HTTP_URL",
"table_type": "expr",
"table_content": {
"keywords": "Cookie:\\s.*head",
"expr_type": "regex",
"match_method": "sub",
"format": "uncase plain"
}
}
]
}
]
},
{
"compile_id": 148,
"service": 0,
"action": 0,
"do_blacklist": 0,
"do_log": 0,
"user_region": "StringScan.ExprPlusWithOffset",
"effective_rage": 0,
"is_valid": "yes",
"groups": [
{
"group_name": "Untitled",
"regions": [
{
"table_name": "APP_PAYLOAD",
"table_type": "expr_plus",
"table_content": {
"format": "hexbin",
"match_method": "sub",
"district": "Payload",
"keywords": "1-1:03&9-10:2d&14-16:2d34&19-21:2d&24-25:2d",
"expr_type": "offset"
}
}
]
}
]
},
{
"compile_id": 150,
"service": 0,
"action": 0,
"do_blacklist": 0,
"do_log": 0,
"effective_rage": 0,
"user_region": "StringScan.BugReport20190325",
"is_valid": "yes",
"groups": [
{
"regions": [
{
"table_type": "expr",
"table_name": "TROJAN_PAYLOAD",
"table_content": {
"keywords": "0-4:01000000",
"expr_type": "offset",
"format": "hexbin",
"match_method": "sub"
}
}
],
"group_name": "billgates_regist1"
},
{
"regions": [
{
"table_type": "expr",
"table_name": "TROJAN_PAYLOAD",
"table_content": {
"keywords": "1:G2.40",
"expr_type": "none",
"format": "uncase plain",
"match_method": "sub"
}
}
],
"group_name": "billgates_regist2"
}
]
},
{
"compile_id": 151,
"service": 0,
"action": 0,
"do_blacklist": 0,
"do_log": 0,
"effective_rage": 0,
"user_region": "StringScan.PrefixAndSuffix",
"is_valid": "yes",
"groups": [
{
"regions": [
{
"table_type": "expr",
"table_name": "MAIL_ADDR",
"table_content": {
"keywords": "ceshi3@mailhost.cn",
"expr_type": "none",
"format": "uncase plain",
"match_method": "suffix"
}
}
],
"group_name": "Untitled"
}
]
},
{
"compile_id": 156,
"service": 1,
"action": 1,
"do_blacklist": 1,
"do_log": 1,
"user_region": "ExprPlusWithHex",
"is_valid": "yes",
"groups": [
{
"group_name": "Untitled",
"regions": [
{
"table_name": "HTTP_SIGNATURE",
"table_type": "expr_plus",
"table_content": {
"district": "Content-Type",
"keywords": "2f68746d6c",
"expr_type": "none",
"match_method": "sub",
"format": "hexbin"
}
}
]
}
]
},
{
"compile_id": 157,
"service": 0,
"action": 0,
"do_blacklist": 0,
"do_log": 0,
"effective_rage": 0,
"user_region": "StringScan.StreamScanUTF8",
"is_valid": "yes",
"groups": [
{
"regions": [
{
"table_type": "expr",
"table_name": "TROJAN_PAYLOAD",
"table_content": {
"keywords": "我的订单",
"expr_type": "none",
"format": "none",
"match_method": "sub"
}
}
]
}
]
},
{
"compile_id": 182,
"service": 1,
"action": 1,
"do_blacklist": 1,
"do_log": 1,
"user_region": "8-expr",
"is_valid": "yes",
"groups": [
{
"regions": [
{
"table_name": "KEYWORDS_TABLE",
"table_type": "expr",
"table_content": {
"keywords": "string1&string2&string3&string4&string5&string6&string7&string8",
"expr_type": "and",
"match_method": "sub",
"format": "uncase plain"
}
}
]
}
]
},
{
"compile_id": 190,
"service": 1,
"action": 1,
"do_blacklist": 1,
"do_log": 1,
"user_region": "StringScan.ExprPlus",
"is_valid": "yes",
"groups": [
{
"group_name": "Untitled",
"regions": [
{
"table_name": "HTTP_SIGNATURE",
"table_type": "expr_plus",
"table_content": {
"district": "我的DistrIct",
"keywords": "addis&sapphire",
"expr_type": "and",
"match_method": "sub",
"format": "uncase plain"
}
}
]
}
]
},
{
"compile_id": 191,
"service": 0,
"action": 0,
"do_blacklist": 0,
"do_log": 0,
"effective_rage": 0,
"user_region": "StringScan.HexBinCaseSensitive",
"is_valid": "yes",
"groups": [
{
"regions": [
{
"table_type": "expr",
"table_name": "KEYWORDS_TABLE",
"table_content": {
"keywords": "54455354",
"expr_type": "none",
"format": "hexbin",
"match_method": "sub"
}
}
],
"group_name": "Untitled"
}
]
},
{
"compile_id": 195,
"service": 0,
"action": 0,
"do_blacklist": 0,
"do_log": 0,
"user_region": "anything",
"is_valid": "yes",
"groups": [
{
"regions": [
{
"table_name": "HTTP_SIGNATURE",
"table_type": "expr_plus",
"table_content": {
"district": "I love China",
"keywords": "today&yesterday",
"expr_type": "and",
"match_method": "sub",
"format": "uncase plain"
}
}
]
},
{
"regions": [
{
"table_name": "HTTP_URL",
"table_type": "expr",
"table_content": {
"keywords": "Monday",
"expr_type": "none",
"match_method": "sub",
"format": "uncase plain"
}
}
]
}
]
},
{
"compile_id": 206,
"service": 0,
"action": 0,
"do_blacklist": 0,
"do_log": 0,
"effective_rage": 0,
"user_region": "duplicateRuleFor191",
"is_valid": "yes",
"groups": [
{
"regions": [
{
"table_type": "expr",
"table_name": "KEYWORDS_TABLE",
"table_content": {
"keywords": "54455354",
"expr_type": "none",
"format": "hexbin",
"match_method": "sub"
}
}
],
"group_name": "Untitled"
}
]
}
]
}

718
test/maat_demo/maat_demo_gtest.cpp Normal file
View File

@@ -0,0 +1,718 @@
#include <gtest/gtest.h>
#include <dirent.h>
#include <openssl/md5.h>
#include "include/maat.h"
#include "include/maat_command.h"
#define MODULE_FRAMEWORK_GTEST module_name_str("maat.framework_gtest")
#define ARRAY_SIZE 10
#define HIT_PATH_SIZE 128
#define WAIT_FOR_EFFECTIVE_S 2
#define ALLOC(type, number) ((type *)calloc(sizeof(type), number))
const char *table_info_path = "./demo_table_info.conf";
const char *maat_json_file = "./maat_demo.json";
int compile_table_set_line(struct maat *maat_inst, const char *table_name,
enum maat_operation op, long long compile_id,
const char *user_region, int clause_num,
int expire_after)
{
char table_line[1024 * 16] = {0};
sprintf(table_line, "%lld\t0\t0\t0\t0\t0\t%s\t%d\t%d\t0.0",
compile_id, user_region, op, clause_num);
struct maat_cmd_line line_rule;
line_rule.rule_id = compile_id;
line_rule.table_line = table_line;
line_rule.table_name = table_name;
line_rule.expire_after = expire_after;
return maat_cmd_set_line(maat_inst, &line_rule);
}
#define TO_GROUP2X_KEY(group_id, parent_id) ((unsigned long)group_id<<32|parent_id)
int group2compile_table_set_line(struct maat *maat_inst, const char *table_name,
enum maat_operation op, long long group_id,
long long compile_id, int not_flag,
const char *vtable_name, int clause_index,
int expire_after)
{
char table_line[128] = {0};
sprintf(table_line, "%lld\t%lld\t%d\t%d\t%s\t%d",
group_id, compile_id, op, not_flag, vtable_name, clause_index);
struct maat_cmd_line line_rule;
line_rule.rule_id = TO_GROUP2X_KEY(group_id, compile_id);
line_rule.table_line = table_line;
line_rule.table_name = table_name;
line_rule.expire_after = expire_after;
return maat_cmd_set_line(maat_inst, &line_rule);
}
int expr_table_set_line(struct maat *maat_inst, const char *table_name,
enum maat_operation op, long long item_id,
long long group_id, const char *keywords,
int expr_type, int match_method, int is_hexbin,
int expire_after)
{
char table_line[1024] = {0};
int table_id = maat_get_table_id(maat_inst, table_name);
if (table_id < 0) {
return 0;
}
sprintf(table_line, "%lld\t%lld\t%s\t%d\t%d\t%d\t%d", item_id, group_id,
keywords, expr_type, match_method, is_hexbin, op);
struct maat_cmd_line line_rule;
line_rule.rule_id = item_id;
line_rule.table_line = table_line;
line_rule.table_name = table_name;
line_rule.expire_after = expire_after;
return maat_cmd_set_line(maat_inst, &line_rule);
}
class JsonMode : public testing::Test
{
protected:
static void SetUpTestCase() {
struct maat_options *opts = maat_options_new();
maat_options_set_json_file(opts, maat_json_file);
maat_options_set_logger(opts, "./maat_sample_gtest.log", LOG_LEVEL_INFO);
_shared_maat_inst = maat_new(opts, table_info_path);
maat_options_free(opts);
if (NULL == _shared_maat_inst) {
assert(0);
}
}
static void TearDownTestCase() {
maat_free(_shared_maat_inst);
}
static struct maat *_shared_maat_inst;
};
struct maat *JsonMode::_shared_maat_inst;
TEST_F(JsonMode, ScanDataOnlyOneByte) {
const char *table_name = "HTTP_URL";
struct maat *maat_inst = JsonMode::_shared_maat_inst;
int table_id = maat_get_table_id(maat_inst, table_name);
ASSERT_GT(table_id, 0);
long long results[ARRAY_SIZE] = {0};
size_t n_hit_result = 0;
int thread_id = 0;
struct maat_state *state = maat_state_new(maat_inst, thread_id);
const char scan_data = 0x20;
int ret = maat_scan_string(maat_inst, table_id, &scan_data, sizeof(scan_data),
results, ARRAY_SIZE, &n_hit_result, state);
EXPECT_EQ(ret, MAAT_SCAN_OK);
EXPECT_EQ(n_hit_result, 0);
maat_state_free(state);
state = NULL;
}
TEST_F(JsonMode, literal) {
const char *table_name = "HTTP_URL";
struct maat *maat_inst = JsonMode::_shared_maat_inst;
int table_id = maat_get_table_id(maat_inst, table_name);
ASSERT_GT(table_id, 0);
long long results[ARRAY_SIZE] = {0};
size_t n_hit_result = 0;
int thread_id = 0;
struct maat_state *state = maat_state_new(maat_inst, thread_id);
const char *scan_data = "http://www.cyberessays.com/search_results.php?action=search&query=username,abckkk,1234567";
int ret = maat_scan_string(maat_inst, table_id, scan_data, strlen(scan_data),
results, ARRAY_SIZE, &n_hit_result, state);
EXPECT_EQ(ret, MAAT_SCAN_HIT);
EXPECT_EQ(n_hit_result, 1);
EXPECT_EQ(results[0], 125);
maat_state_free(state);
state = NULL;
}
TEST_F(JsonMode, Regex) {
int ret = 0;
long long results[ARRAY_SIZE] = {0};
size_t n_hit_result = 0;
int thread_id = 0;
const char *cookie = "Cookie: Txa123aheadBCAxd";
const char *table_name = "HTTP_URL";
struct maat *maat_inst = JsonMode::_shared_maat_inst;
struct maat_state *state = maat_state_new(maat_inst, thread_id);
int table_id = maat_get_table_id(maat_inst, table_name);
ret = maat_scan_string(maat_inst, table_id, cookie, strlen(cookie),
results, ARRAY_SIZE, &n_hit_result, state);
EXPECT_EQ(ret, MAAT_SCAN_HIT);
EXPECT_EQ(results[0], 146);
maat_state_free(state);
state = NULL;
}
TEST_F(JsonMode, ExprPlus) {
long long results[ARRAY_SIZE] = {0};
size_t n_hit_result = 0;
int thread_id = 0;
const char *region_name1 ="HTTP URL";
const char *region_name2 ="我的diStricT";
const char *scan_data1 = "http://www.cyberessays.com/search_results.php?action=search&query=abckkk,1234567";
const char *scan_data2 = "Addis Sapphire Hotel";
const char *table_name = "HTTP_SIGNATURE";
struct maat *maat_inst = JsonMode::_shared_maat_inst;
struct maat_state *state = maat_state_new(maat_inst, thread_id);
int table_id = maat_get_table_id(maat_inst, table_name);
int ret = maat_scan_string(maat_inst, table_id, scan_data1, strlen(scan_data1),
results, ARRAY_SIZE, &n_hit_result, state);
EXPECT_EQ(ret, MAAT_SCAN_ERR);//Should return error for district not setting.
ret = maat_state_set_scan_district(state, table_id, region_name1, strlen(region_name1));
ASSERT_EQ(ret, 0);
ret = maat_scan_string(maat_inst, table_id, scan_data1, strlen(scan_data1),
results, ARRAY_SIZE, &n_hit_result, state);
EXPECT_EQ(ret, MAAT_SCAN_HIT);
EXPECT_EQ(results[0], 128);
maat_state_reset(state);
ret = maat_state_set_scan_district(state, table_id, region_name2, strlen(region_name2));
ASSERT_EQ(ret, 0);
ret = maat_scan_string(maat_inst, table_id, scan_data2, strlen(scan_data2),
results, ARRAY_SIZE, &n_hit_result, state);
EXPECT_EQ(ret, MAAT_SCAN_HIT);
EXPECT_EQ(results[0], 190);
maat_state_free(state);
state = NULL;
}
TEST_F(JsonMode, ExprPlusWithOffset)
{
long long results[ARRAY_SIZE] = {0};
size_t n_hit_result = 0;
int thread_id = 0;
struct maat *maat_inst = JsonMode::_shared_maat_inst;
struct maat_state *state = maat_state_new(maat_inst, thread_id);
const char *region_name = "Payload";
unsigned char udp_payload_not_hit[] = { /* Stun packet */
0x00, 0x03, 0x00, 0x4a, 0x21, 0x12, 0xa4, 0x42,
0x4f, 0xc2, 0xc2, 0x70, 0xb3, 0xa8, 0x4e, 0x22,
0xf5, 0x22, 0x87, 0x4c, 0x40, 0x00, 0x00, 0x46,
0x03, 0x02, 0xab, 0x39, 0xbb, 0x97, 0xe5, 0x01,
0x3a, 0x46, 0x1c, 0x28, 0x5b, 0xab, 0xfa, 0x9a,
0xab, 0x2e, 0x71, 0x39, 0x66, 0xa0, 0xd7, 0xb9,
0xd8, 0x41, 0xa7, 0xa0, 0x84, 0xa9, 0xf3, 0x1b,
0x03, 0x7f, 0xa8, 0x28, 0xa2, 0xd3, 0x64, 0xc2,
0x3d, 0x20, 0xe0, 0xb1, 0x41, 0x12, 0x6c, 0x2f,
0xc5, 0xbb, 0xc3, 0xba, 0x69, 0x73, 0x52, 0x64,
0xf6, 0x30, 0x81, 0xf4, 0x3f, 0xc2, 0x19, 0x6a,
0x68, 0x61, 0x93, 0x08, 0xc0, 0x0a };
unsigned char udp_payload_hit[] = { /* Stun packet */ //rule:"1-1:03&9-10:2d&14-16:2d34&19-21:2d&24-25:2d"
0x00, 0x03, 0x00, 0x4a, 0x21, 0x12, 0xa4, 0x42, //1-1:03
0x4f, 0xc2, 0x2d, 0x70, 0xb3, 0xa8, 0x4e, 0x2d, //10-10:2d
0x34, 0x22, 0x87, 0x4c, 0x2d, 0x00, 0x00, 0x46, //15-16:2d34
0x2d, 0x34, 0xab, 0x39, 0xbb, 0x97, 0xe5, 0x01, //20-20:2d
0x03, 0x46, 0x1c, 0x28, 0x5b, 0xab, 0xfa, 0x9a, //24-24:2d
0xab, 0x2e, 0x71, 0x39, 0x66, 0xa0, 0xd7, 0xb9,
0xd8, 0x41, 0xa7, 0xa0, 0x84, 0xa9, 0xf3, 0x1b,
0x03, 0x7f, 0xa8, 0x28, 0xa2, 0xd3, 0x64, 0xc2,
0x3d, 0x20, 0xe0, 0xb1, 0x41, 0x12, 0x6c, 0x2f,
0xc5, 0xbb, 0xc3, 0xba, 0x69, 0x73, 0x52, 0x64,
0xf6, 0x30, 0x81, 0xf4, 0x3f, 0xc2, 0x19, 0x6a,
0x68, 0x61, 0x93, 0x08, 0xc0, 0x0a };
int table_id = maat_get_table_id(maat_inst, "APP_PAYLOAD");
ASSERT_GT(table_id, 0);
int ret = maat_state_set_scan_district(state, table_id, region_name, strlen(region_name));
EXPECT_EQ(ret, 0);
ret = maat_scan_string(maat_inst, table_id, (char*)udp_payload_not_hit, sizeof(udp_payload_not_hit),
results, ARRAY_SIZE, &n_hit_result, state);
EXPECT_EQ(ret, MAAT_SCAN_OK);
ret = maat_scan_string(maat_inst, table_id, (char*)udp_payload_hit, sizeof(udp_payload_hit),
results, ARRAY_SIZE, &n_hit_result, state);
EXPECT_EQ(ret, MAAT_SCAN_HIT);
EXPECT_EQ(results[0], 148);
maat_state_free(state);
state = NULL;
}
TEST_F(JsonMode, ExprPlusWithHex) {
long long results[ARRAY_SIZE] = {0};
size_t n_hit_result = 0;
int thread_id = 0;
struct maat *maat_inst = JsonMode::_shared_maat_inst;
struct maat_state *state = maat_state_new(maat_inst, thread_id);
const char *scan_data1 = "text/html; charset=UTF-8";
const char *scan_data2 = "Batman\\:Take me Home.Superman/:Fine,stay with me.";
const char *region_name1 = "Content-Type";
const char *region_name2 = "User-Agent";
int table_id = maat_get_table_id(maat_inst, "HTTP_SIGNATURE");
ASSERT_GT(table_id, 0);
int ret = maat_state_set_scan_district(state, table_id, region_name1, strlen(region_name1));
ASSERT_EQ(ret, 0);
ret = maat_scan_string(maat_inst, table_id, scan_data1, strlen(scan_data1),
results, ARRAY_SIZE, &n_hit_result, state);
EXPECT_EQ(ret, MAAT_SCAN_HIT);
EXPECT_EQ(results[0], 156);
ret = maat_state_set_scan_district(state, table_id, region_name2, strlen(region_name2));
ASSERT_EQ(ret, 0);
ret = maat_scan_string(maat_inst, table_id, scan_data1, strlen(scan_data1),
results, ARRAY_SIZE, &n_hit_result, state);
EXPECT_EQ(ret, MAAT_SCAN_OK); //maat-v3 consider as half hit, it's unreasonable
table_id = maat_get_table_id(maat_inst, "KEYWORDS_TABLE");
ret = maat_scan_string(maat_inst, table_id, scan_data2, strlen(scan_data2),
results, ARRAY_SIZE, &n_hit_result, state);
EXPECT_EQ(ret, MAAT_SCAN_HIT);
EXPECT_EQ(results[0], 132);
maat_state_free(state);
state = NULL;
}
TEST_F(JsonMode, ExprAndExprPlus) {
long long results[ARRAY_SIZE] = {0};
size_t n_hit_result = 0;
int thread_id = 0;
struct maat *maat_inst = JsonMode::_shared_maat_inst;
struct maat_state *state = maat_state_new(maat_inst, thread_id);
const char *expr_table_name = "HTTP_URL";
const char *expr_plus_table_name = "HTTP_SIGNATURE";
const char *region_name = "I love China";
const char *scan_data = "today is Monday and yesterday is Tuesday";
int expr_table_id = maat_get_table_id(maat_inst, expr_table_name);
int expr_plus_table_id = maat_get_table_id(maat_inst, expr_plus_table_name);
int ret = maat_scan_string(maat_inst, expr_plus_table_id, scan_data, strlen(scan_data),
results, ARRAY_SIZE, &n_hit_result, state);
EXPECT_EQ(ret, MAAT_SCAN_ERR);
ret = maat_state_set_scan_district(state, expr_plus_table_id, region_name, strlen(region_name));
ASSERT_EQ(ret, 0);
ret = maat_scan_string(maat_inst, expr_plus_table_id, scan_data, strlen(scan_data),
results, ARRAY_SIZE, &n_hit_result, state);
EXPECT_EQ(ret, MAAT_SCAN_HALF_HIT);
ret = maat_scan_string(maat_inst, expr_table_id, scan_data, strlen(scan_data),
results, ARRAY_SIZE, &n_hit_result, state);
EXPECT_EQ(ret, MAAT_SCAN_HIT);
EXPECT_EQ(results[0], 195);
maat_state_free(state);
state = NULL;
}
TEST_F(JsonMode, ShouldNotHitExprPlus) {
long long results[ARRAY_SIZE] = {0};
size_t n_hit_result = 0;
int thread_id = 0;
struct maat *maat_inst = JsonMode::_shared_maat_inst;
struct maat_state *state = maat_state_new(maat_inst, thread_id);
const char *region_name = "tcp.payload";
unsigned char udp_payload_not_hit[] = { /* Stun packet */
0x00, 0x03, 0x00, 0x4a, 0x21, 0x12, 0xa4, 0x42,
0x4f, 0xc2, 0xc2, 0x70, 0xb3, 0xa8, 0x4e, 0x22,
0xf5, 0x22, 0x87, 0x4c, 0x40, 0x00, 0x00, 0x46,
0x03, 0x02, 0xab, 0x39, 0xbb, 0x97, 0xe5, 0x01,
0x3a, 0x46, 0x1c, 0x28, 0x5b, 0xab, 0xfa, 0x9a,
0xab, 0x2e, 0x71, 0x39, 0x66, 0xa0, 0xd7, 0xb9,
0xd8, 0x41, 0xa7, 0xa0, 0x84, 0xa9, 0xf3, 0x1b,
0x03, 0x7f, 0xa8, 0x28, 0xa2, 0xd3, 0x64, 0xc2,
0x3d, 0x20, 0xe0, 0xb1, 0x41, 0x12, 0x6c, 0x2f,
0xc5, 0xbb, 0xc3, 0xba, 0x69, 0x73, 0x52, 0x64,
0xf6, 0x30, 0x81, 0xf4, 0x3f, 0xc2, 0x19, 0x6a,
0x68, 0x61, 0x93, 0x08, 0xc0, 0x0a, 0xab, 0x00 };
int table_id = maat_get_table_id(maat_inst, "APP_PAYLOAD");
ASSERT_GT(table_id, 0);
int ret = maat_state_set_scan_district(state, table_id, region_name, strlen(region_name));
ASSERT_EQ(ret, 0);
ret = maat_scan_string(maat_inst, table_id, (char *)udp_payload_not_hit, sizeof(udp_payload_not_hit),
results, ARRAY_SIZE, &n_hit_result, state);
EXPECT_EQ(ret, MAAT_SCAN_OK); //maat-v3 consider as half hit, it's unreasonable
maat_state_free(state);
state = NULL;
}
TEST_F(JsonMode, Expr8) {
const char *table_name = "KEYWORDS_TABLE";
int thread_id = 0;
struct maat *maat_inst = JsonMode::_shared_maat_inst;
struct maat_state *state = maat_state_new(maat_inst, thread_id);
int table_id = maat_get_table_id(maat_inst, table_name);
char scan_data[128] = "string1, string2, string3, string4, string5, string6, string7, string8";
long long results[ARRAY_SIZE] = {0};
size_t n_hit_result = 0;
int ret = maat_scan_string(maat_inst, table_id, scan_data, strlen(scan_data),
results, ARRAY_SIZE, &n_hit_result, state);
EXPECT_EQ(ret, MAAT_SCAN_HIT);
EXPECT_EQ(n_hit_result, 1);
EXPECT_EQ(results[0], 182);
struct maat_hit_path hit_path[HIT_PATH_SIZE] = {0};
int n_read = 0;
n_read = maat_state_get_hit_paths(state, hit_path, HIT_PATH_SIZE);
EXPECT_NE(n_read, 0);
maat_state_free(state);
state = NULL;
}
TEST_F(JsonMode, HexBinCaseSensitive) {
const char *table_name = "KEYWORDS_TABLE";
const char *scan_data1 = "String TeST should not hit.";
const char *scan_data2 = "String TEST should hit";
struct maat *maat_inst = JsonMode::_shared_maat_inst;
int thread_id = 0;
int table_id = maat_get_table_id(maat_inst, table_name);
ASSERT_GT(table_id, 0);
long long results[ARRAY_SIZE] = {0};
size_t n_hit_result = 0;
struct maat_state *state = maat_state_new(maat_inst, thread_id);
int ret = maat_scan_string(maat_inst, table_id, scan_data1, strlen(scan_data1),
results, ARRAY_SIZE, &n_hit_result, state);
EXPECT_EQ(ret, MAAT_SCAN_OK);
maat_state_reset(state);
ret = maat_scan_string(maat_inst, table_id, scan_data2, strlen(scan_data2),
results, ARRAY_SIZE, &n_hit_result, state);
EXPECT_EQ(ret, MAAT_SCAN_HIT);
EXPECT_EQ(n_hit_result, 2);
EXPECT_EQ(results[0], 206);
EXPECT_EQ(results[1], 191);
maat_state_free(state);
}
TEST_F(JsonMode, BugReport20190325) {
unsigned char scan_data[] = {/* Packet 1 */
0x01, 0x00, 0x00, 0x00, 0x79, 0x00, 0x00, 0x00,
0x00, 0xf4, 0x01, 0x00, 0x00, 0x32, 0x00, 0x00,
0x00, 0xe8, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x01, 0x01, 0x00, 0x00, 0x00, 0x00,
0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0xff, 0xff, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00,
0x2d, 0x3d, 0x3d, 0x20, 0x48, 0x3d, 0x48, 0x20,
0x3d, 0x3d, 0x2d, 0x3a, 0x00, 0x02, 0x00, 0x00,
0x00, 0x07, 0x0e, 0x00, 0x00, 0xe8, 0x03, 0x00,
0x00, 0x4c, 0x69, 0x6e, 0x75, 0x78, 0x20, 0x33,
0x2e, 0x31, 0x39, 0x2e, 0x30, 0x2d, 0x31, 0x35,
0x2d, 0x67, 0x65, 0x6e, 0x65, 0x72, 0x69, 0x63,
0x00, 0x31, 0x3a, 0x47, 0x32, 0x2e, 0x34, 0x30,
0x00};
const char *table_name = "TROJAN_PAYLOAD";
struct maat *maat_inst = JsonMode::_shared_maat_inst;
int thread_id = 0;
int table_id = maat_get_table_id(maat_inst, table_name);
ASSERT_GT(table_id, 0);
long long results[ARRAY_SIZE] = {0};
size_t n_hit_result = 0;
struct maat_state *state = maat_state_new(maat_inst, thread_id);
int ret = maat_scan_string(maat_inst, table_id, (char *)scan_data, sizeof(scan_data),
results, ARRAY_SIZE, &n_hit_result, state);
EXPECT_EQ(ret, MAAT_SCAN_HIT);
EXPECT_EQ(n_hit_result, 1);
EXPECT_EQ(results[0], 150);
maat_state_free(state);
state = NULL;
}
TEST_F(JsonMode, MaatUnescape) {
const char *scan_data = "Batman\\:Take me Home.Superman/:Fine,stay with me.";
const char *table_name = "KEYWORDS_TABLE";
struct maat *maat_inst = JsonMode::_shared_maat_inst;
int thread_id = 0;
int table_id = maat_get_table_id(maat_inst, table_name);
ASSERT_GT(table_id, 0);
long long results[ARRAY_SIZE] = {0};
size_t n_hit_result = 0;
struct maat_state *state = maat_state_new(maat_inst, thread_id);
int ret = maat_scan_string(maat_inst, table_id, scan_data, strlen(scan_data),
results, ARRAY_SIZE, &n_hit_result, state);
EXPECT_EQ(ret, MAAT_SCAN_HIT);
EXPECT_EQ(n_hit_result, 1);
EXPECT_EQ(results[0], 132);
maat_state_free(state);
state = NULL;
}
TEST_F(JsonMode, OffsetChunk64) {
const char *table_name = "IMAGE_FP";
const char *file_name = "./testdata/mesa_logo.jpg";
long long results[ARRAY_SIZE] = {0};
size_t n_hit_result = 0;
int thread_id = 0;
struct maat *maat_inst = JsonMode::_shared_maat_inst;
struct maat_state *state = maat_state_new(maat_inst, thread_id);
FILE *fp = fopen(file_name, "r");
ASSERT_FALSE(fp==NULL);
char scan_data[64];
int table_id = maat_get_table_id(maat_inst, table_name);
ASSERT_GT(table_id, 0);
struct maat_stream *sp = maat_stream_new(maat_inst, table_id, state);
ASSERT_TRUE(sp != NULL);
int ret = 0;
int read_size = 0;
int pass_flag = 0;
while (0 == feof(fp)) {
read_size = fread(scan_data, 1, sizeof(scan_data), fp);
ret = maat_stream_scan(sp, scan_data, read_size,
results, ARRAY_SIZE, &n_hit_result, state);
if (ret > 0) {
pass_flag = 1;
break;
}
}
EXPECT_EQ(pass_flag, 1);
EXPECT_EQ(results[0], 136);
maat_stream_free(sp);
fclose(fp);
maat_state_free(state);
state = NULL;
}
TEST_F(JsonMode, OffsetChunk1460) {
const char *table_name = "IMAGE_FP";
const char *file_name = "./testdata/mesa_logo.jpg";
long long results[ARRAY_SIZE] = {0};
size_t n_hit_result = 0;
int thread_id = 0;
struct maat *maat_inst = JsonMode::_shared_maat_inst;
struct maat_state *state = maat_state_new(maat_inst, thread_id);
FILE *fp = fopen(file_name, "r");
ASSERT_FALSE(fp==NULL);
char scan_data[1460];
int table_id = maat_get_table_id(maat_inst, table_name);
ASSERT_GT(table_id, 0);
struct maat_stream *sp = maat_stream_new(maat_inst, table_id, state);
ASSERT_TRUE(sp != NULL);
int ret = 0;
int read_size = 0;
int pass_flag = 0;
while (0 == feof(fp)) {
read_size = fread(scan_data, 1, sizeof(scan_data), fp);
ret = maat_stream_scan(sp, scan_data, read_size,
results, ARRAY_SIZE, &n_hit_result, state);
if (ret > 0) {
pass_flag = 1;
break;
}
}
EXPECT_EQ(pass_flag, 1);
EXPECT_EQ(results[0], 136);
maat_stream_free(sp);
fclose(fp);
maat_state_free(state);
state = NULL;
}
TEST_F(JsonMode, StreamScanUTF8) {
const char *table_name = "TROJAN_PAYLOAD";
const char* file_name = "./testdata/jd.com.html";
long long results[ARRAY_SIZE] = {0};
size_t n_hit_result = 0;
int thread_id = 0;
char scan_data[2048];
struct maat *maat_inst = JsonMode::_shared_maat_inst;
struct maat_state *state = maat_state_new(maat_inst, thread_id);
FILE *fp = fopen(file_name, "r");
ASSERT_FALSE(fp == NULL);
int table_id = maat_get_table_id(maat_inst, table_name);
ASSERT_GT(table_id, 0);
struct maat_stream *sp = maat_stream_new(maat_inst, table_id, state);
ASSERT_FALSE(sp == NULL);
int pass_flag = 0;
while (0 == feof(fp)) {
size_t read_size = fread(scan_data, 1, sizeof(scan_data), fp);
int ret = maat_stream_scan(sp, scan_data, read_size, results, ARRAY_SIZE,
&n_hit_result, state);
if (ret == MAAT_SCAN_HIT) {
pass_flag = 1;
break;
}
}
EXPECT_EQ(pass_flag, 1);
EXPECT_EQ(results[0], 157);
maat_stream_free(sp);
fclose(fp);
maat_state_free(state);
state = NULL;
}
TEST_F(JsonMode, StreamInput) {
long long results[ARRAY_SIZE] = {0};
size_t n_hit_result = 0;
int thread_id = 0;
struct maat *maat_inst = JsonMode::_shared_maat_inst;
struct maat_state *state = maat_state_new(maat_inst, thread_id);
const char *scan_data = "http://www.cyberessays.com/search_results.php?action=search&query=yulingjing,abckkk,1234567";
const char *table_name = "HTTP_URL";
int table_id = maat_get_table_id(maat_inst, table_name);
ASSERT_GT(table_id, 0);
struct maat_stream *sp = maat_stream_new(maat_inst, table_id, state);
ASSERT_TRUE(sp != NULL);
int ret = maat_stream_scan(sp, "www.cyberessays.com", strlen("www.cyberessays.com"),
results, ARRAY_SIZE, &n_hit_result, state);
EXPECT_EQ(ret, MAAT_SCAN_OK);
ret = maat_stream_scan(sp, scan_data, strlen(scan_data), results, ARRAY_SIZE,
&n_hit_result, state);
maat_stream_free(sp);
EXPECT_EQ(ret, MAAT_SCAN_HIT);
EXPECT_EQ(results[0], 125);
maat_state_free(state);
state = NULL;
}
class RedisMode : public testing::Test
{
protected:
static void SetUpTestCase() {
const char *redis_ip = "127.0.0.1";
uint16_t redis_port = 6379;
int redis_db = 0;
struct maat_options *opts = maat_options_new();
maat_options_set_redis(opts, redis_ip, redis_port, redis_db);
maat_options_set_logger(opts, "./maat_sample_gtest.log", LOG_LEVEL_INFO);
_shared_maat_inst = maat_new(opts, table_info_path);
maat_options_free(opts);
if (NULL == _shared_maat_inst) {
assert(0);
}
}
static void TearDownTestCase() {
maat_free(_shared_maat_inst);
}
static struct maat *_shared_maat_inst;
};
struct maat *RedisMode::_shared_maat_inst;
TEST_F(RedisMode, dynamic_config) {
const char *table_name = "HTTP_URL";
char data[128] = "welcome to maat version4, it's funny.";
long long results[ARRAY_SIZE] = {0};
size_t n_hit_result = 0;
int thread_id = 0;
struct maat *maat_inst = RedisMode::_shared_maat_inst;
struct maat_state *state = maat_state_new(maat_inst, thread_id);
int table_id = maat_get_table_id(maat_inst, table_name);
int ret = maat_scan_string(maat_inst, table_id, data, strlen(data), results,
ARRAY_SIZE, &n_hit_result, state);
EXPECT_EQ(ret, MAAT_SCAN_OK);
maat_state_reset(state);
const char *compile_table_name = "COMPILE";
const char *g2c_table_name = "GROUP2COMPILE";
/* compile table add line */
long long compile_id = maat_cmd_incrby(maat_inst, "TEST_SEQ", 1);
ret = compile_table_set_line(maat_inst, compile_table_name, MAAT_OP_ADD, compile_id, "null", 1, 0);
EXPECT_EQ(ret, 1);
/* group2compile table add line */
long long group_id = maat_cmd_incrby(maat_inst, "SEQUENCE_GROUP", 1);
ret = group2compile_table_set_line(maat_inst, g2c_table_name, MAAT_OP_ADD, group_id,
compile_id, 0, "null", 1, 0);
EXPECT_EQ(ret, 1);
/* expr table add line */
long long item_id = maat_cmd_incrby(maat_inst, "SEQUENCE_REGION", 1);
const char *keywords = "welcome to maat";
ret = expr_table_set_line(maat_inst, table_name, MAAT_OP_ADD, item_id, group_id,
keywords, 1, 0, 0, 0); /* EXPR_TYPE_AND MATCH_METHOD_SUB */
EXPECT_EQ(ret, 1);
sleep(WAIT_FOR_EFFECTIVE_S);
ret = maat_scan_string(maat_inst, table_id, data, strlen(data), results,
ARRAY_SIZE, &n_hit_result, state);
EXPECT_EQ(ret, MAAT_SCAN_HIT);
EXPECT_EQ(n_hit_result, 1);
EXPECT_EQ(results[0], compile_id);
maat_state_reset(state);
/* expr table del line */
ret = expr_table_set_line(maat_inst, table_name, MAAT_OP_DEL, item_id, group_id,
keywords, 1, 0, 0, 0); /* EXPR_TYPE_AND MATCH_METHOD_SUB */
EXPECT_EQ(ret, 1);
/* group2compile table del line */
ret = group2compile_table_set_line(maat_inst, g2c_table_name, MAAT_OP_DEL, group_id,
compile_id, 0, "null", 1, 0);
EXPECT_EQ(ret, 1);
/* compile table del line */
ret = compile_table_set_line(maat_inst, compile_table_name, MAAT_OP_DEL, compile_id, "null", 1, 0);
EXPECT_EQ(ret, 1);
sleep(WAIT_FOR_EFFECTIVE_S);
ret = maat_scan_string(maat_inst, table_id, data, strlen(data), results,
ARRAY_SIZE, &n_hit_result, state);
EXPECT_EQ(ret, MAAT_SCAN_OK);
maat_state_free(state);
state = NULL;
}
int main(int argc, char ** argv)
{
int ret=0;
::testing::InitGoogleTest(&argc, argv);
ret=RUN_ALL_TESTS();
return ret;
}

18
test/maat_demo/readme.md Normal file
View File

@@ -0,0 +1,18 @@
依赖 crypto库，需提前安装
安装 redis-server 并按默认配置启动即可，maat_demo_gtest 会用到 redis
## 编译 & 运行单测
1. 当前目录，mkdir build
2. cd build
3. cmake ..
4. make
5. ./maat_demo_gtest
## 文件说明：
- include 目录存放 maat 库头文件
- lib 目录存放 maat 动态库及 gtest 静态库
- testdata 为单测所需测试数据
- maat_demo.json 为json 格式的匹配规则，运行时会转为 iris 格式，位于maat_demo.json_iris_tmp目录(运行时生成)
- demo_table_info.conf用于表示 iris 格式规则每列代表的含义，maat解析对应列的数据
- maat_demo_gtest.cpp 为单测文件，字符串匹配相关测试用例可供参考

77893
test/maat_demo/testdata/bool-matcher-test-exprs.txt vendored Normal file
View File

File diff suppressed because it is too large Load Diff

48
test/maat_demo/testdata/charsetWindows1251.txt vendored Normal file
View File

@@ -0,0 +1,48 @@
<EFBFBD><EFBFBD> <20><><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>!','JS_CORE_WINDOW_AUTH':'<27><><EFBFBD><EFBFBD><EFBFBD>','JS_CORE_IMAGE_FULL':'<27><><EFBFBD><EFBFBD><EFBFBD><EFBFBD> <20><><EFBFBD><EFBFBD><EFBFBD><EFBFBD>'});</script>
<script type="text/javascript">(window.BX||top.BX).message({'LANGUAGE_ID':'ru','FORMAT_DATE':'DD.MM.YYYY','FORMAT_DATETIME':'DD.MM.YYYY HH:MI:SS','COOKIE_PREFIX':'BITRIX_SM','USER_ID':'','SERVER_TIME':'1578340589','SERVER_TZ_OFFSET':'18000','USER_TZ_OFFSET':'0','USER_TZ_AUTO':'Y','bitrix_sessid':'fadf964e9f5bb819f212e5abf5ffb255','SITE_ID':'s1'});</script>
<script type="text/javascript" src="/bitrix/cache/js/s1/web20/kernel_main/kernel_main.js?1402043622360126"></script>
<script type="text/javascript" src="/bitrix/js/main/rsasecurity.js?136876011925044"></script>
<script type="text/javascript">BX.setCSSList(['/bitrix/js/main/core/css/core.css','/bitrix/js/main/core/css/core_popup.css','/bitrix/js/main/core/css/core_date.css','/bitrix/js/main/core/css/core.css','/bitrix/js/main/core/css/core_popup.css','/bitrix/js/main/core/css/core_date.css']); </script>
<script type="text/javascript">BX.setJSList(['/bitrix/js/main/core/core.js','/bitrix/js/main/core/core_ajax.js','/bitrix/js/main/session.js','/bitrix/js/main/json/json2.min.js','/bitrix/js/main/core/core_ls.js','/bitrix/js/main/core/core_window.js','/bitrix/js/main/utils.js','/bitrix/js/main/core/core_popup.js','/bitrix/js/main/core/core_date.js','/bitrix/js/main/dd.js']); </script>
<script type="text/javascript">
bxSession.Expand(1440, 'fadf964e9f5bb819f212e5abf5ffb255', false, '35a74b06af8f9ea55ffbda20075b0894');
</script>
<script>new Image().src='http://www.sgaice.ru/bitrix/spread.php?s=QklUUklYX1NNX0dVRVNUX0lEATY4MTg5NQExNjA5NDQ0NTg5AS8BAQECQklUUklYX1NNX0xBU1RfVklTSVQBMDcuMDEuMjAyMCAwMDo1NjoyOQExNjA5NDQ0NTg5AS8BAQEC&k=71d3b79b44f9716b27b47feab4a206cf';
</script>
<script type="text/javascript" src="/bitrix/cache/js/s1/web20/template_1e341eb2f86845c7519566374f51d35a/template_1e341eb2f86845c7519566374f51d35a_368c1a68876fd1c32b307a10695f3654.js?14010848191120"></script>
<script type="text/javascript" src="/bitrix/js/imgzoom/thumb.js"></script>
<meta name="google-site-verification" content="gL_64SaiDgQcX5z-pvPZmBJ-exN-wS6KZNoDMcPtYtM" />
<title><3E><><EFBFBD> <20><><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD> <20><>ѻ</title>
</head>
<body>
<div id="maintop">
<table align="left1" width="100%" border="0" cellpadding="0" cellspacing="0">
<tr><td valign="top">
<script type="text/javascript">
top.BX.defer(top.rsasec_form_bind)({'formid':'system_auth_form6zOUGO','key':{'M':'HazQxsgvQCIFPf30iHR40R22fp7P9YLPXFhQu6uus68RZxf2IpMo9v0KDpxkgg43WXaZaXrTRvjg1e2126IOo66vH5bphkMP/69MSPlEoaXYzWjTokd+Yzy30WR6HEOyB9tJwADGyjysqoE4+jUfHZQv2JMaVZS0U4SHWOUPwNU=','E':'AQAB','chunk':'128'},'rsa_rand':'5e1390ed8a8e19.17355178','params':['USER_PASSWORD']});
</script>
<div id="login-form-window">
<a href="" onclick="return CloseLoginForm()" style="float:right;"><3E><><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD></a>
<form method="post" target="_top" action="/index.php?login=yes">
<input type="hidden" name="backurl" value="/index.php" />
<input type="hidden" name="AUTH_FORM" value="Y" />
<input type="hidden" name="TYPE" value="AUTH" />
<table width="95%">
<tr>
<td colspan="2">
<09><><EFBFBD><EFBFBD><EFBFBD>:<br />
<input type="text" name="USER_LOGIN" maxlength="50" value="

BIN
test/maat_demo/testdata/digest_test.data vendored Normal file
View File

Binary file not shown.

968
test/maat_demo/testdata/jd.com.html vendored Normal file
View File

File diff suppressed because one or more lines are too long

BIN
test/maat_demo/testdata/mesa_logo.jpg vendored Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 105 KiB

59
test/maat_framework_gtest.cpp
View File

@@ -10,6 +10,7 @@
#include "json2iris.h"
#include "log/log.h"
#include "maat_config_monitor.h"
#include "maat_redis_monitor.h"
#define MODULE_FRAMEWORK_GTEST module_name_str("maat.framework_gtest")
@@ -23,21 +24,6 @@ const char *json_filename = "maat_json.json";
size_t g_thread_num = 4;
void wait_for_cmd_effective(struct maat *maat_inst, long long version_before)
{
long long version_after = version_before;
int is_updating = 1;
long long wating_us = 0, sleep_us = 1000 * 100;
while (is_updating || version_before == version_after) {
is_updating = maat_cmd_config_is_updating(maat_inst);
version_after = maat_cmd_get_config_version(maat_inst);
usleep(sleep_us);//waiting for commands go into effect
wating_us += sleep_us;
}
}
int count_line_num_cb(const char *table_name, const char *line, void *u_para)
{
(*((unsigned int *)u_para))++;
@@ -58,17 +44,17 @@ int make_serial_rule(const char *table_name, const char *line, void *u_para)
}
const char *redis_rule_key = "TEST_RULE_KEY";
redisReply *reply = maat_cmd_wrap_redis_command(ctx, "INCRBY %s %d", redis_rule_key, 1);
redisReply *reply = maat_wrap_redis_command(ctx, "INCRBY %s %d", redis_rule_key, 1);
if (reply->type == REDIS_REPLY_NIL) {
printf("incrby redis_rule_key:%s failed.", redis_rule_key);
return -1;
} else {
s_rule->rule_id = maat_cmd_read_redis_integer(reply);
s_rule->rule_id = maat_read_redis_integer(reply);
freeReplyObject(reply);
reply = NULL;
}
maat_cmd_set_serial_rule(s_rule + line_idx, MAAT_OP_ADD, s_rule->rule_id, table_name,
maat_set_serial_rule(s_rule + line_idx, MAAT_OP_ADD, s_rule->rule_id, table_name,
buff, absolute_expire_time);
(s_rule + line_idx)->ref_ctx = ctx;
line_idx++;
@@ -85,12 +71,12 @@ int write_config_to_redis(char *redis_ip, int redis_port, int redis_db,
snprintf(json_iris_path, sizeof(json_iris_path), "./%s_iris_tmp", json_filename);
redisContext *c = maat_cmd_connect_redis(redis_ip, redis_port, redis_db, logger);
redisContext *c = maat_connect_redis(redis_ip, redis_port, redis_db, logger);
if (NULL == c) {
return -1;
}
redisReply *reply = maat_cmd_wrap_redis_command(c, "flushdb");
redisReply *reply = maat_wrap_redis_command(c, "flushdb");
if (NULL == reply) {
return -1;
} else {
@@ -128,7 +114,7 @@ int write_config_to_redis(char *redis_ip, int redis_port, int redis_db,
struct serial_rule *s_rule = ALLOC(struct serial_rule, total_line_cnt);
s_rule->ref_ctx = c;
long long server_time = maat_cmd_redis_server_time_s(c);
long long server_time = maat_redis_server_time_s(c);
if (server_time < 0) {
return -1;
}
@@ -148,7 +134,7 @@ int write_config_to_redis(char *redis_ip, int redis_port, int redis_db,
EXPECT_EQ(success_cnt, (int)total_line_cnt);
for (size_t i = 0; i < total_line_cnt; i++) {
maat_cmd_clear_rule_cache(s_rule + i);
maat_clear_rule_cache(s_rule + i);
}
FREE(s_rule);
redisFree(c);
@@ -4027,7 +4013,6 @@ struct maat *MaatCmdTest::_shared_maat_inst;
int *MaatCmdTest::_ex_data_counter;
TEST_F(MaatCmdTest, SetIP) {
long long version_before = 0;
long long results[ARRAY_SIZE] = {0};
size_t n_hit_result = 0;
int thread_id = 0;
@@ -4059,8 +4044,7 @@ TEST_F(MaatCmdTest, SetIP) {
IPv4, ip1, ip2, port_min, port_max, 0);
EXPECT_EQ(ret, 1);
version_before = maat_cmd_get_config_version(maat_inst);
wait_for_cmd_effective(maat_inst, version_before);
sleep(WAIT_FOR_EFFECTIVE_S);
uint32_t sip;
ret = inet_pton(AF_INET, ip1, &sip);
@@ -4096,16 +4080,15 @@ TEST_F(MaatCmdTest, SetExpr) {
struct maat *maat_inst = MaatCmdTest::_shared_maat_inst;
struct maat_state *state = maat_state_new(maat_inst, thread_id);
maat_cmd_str_escape(escape_buff1, sizeof(escape_buff1), keywords1);
maat_cmd_str_escape(escape_buff2, sizeof(escape_buff2), keywords2);
str_escape(escape_buff1, sizeof(escape_buff1), keywords1);
str_escape(escape_buff2, sizeof(escape_buff2), keywords2);
snprintf(keywords, sizeof(keywords), "%s&%s", escape_buff1, escape_buff2);
long long compile_id = maat_cmd_incrby(maat_inst, "TEST_SEQ", 2);
test_add_expr_command(maat_inst, table_name, compile_id - 1, 0, keywords);
test_add_expr_command(maat_inst, table_name, compile_id, 0, keywords);
long long version_before = maat_cmd_get_config_version(maat_inst);
wait_for_cmd_effective(maat_inst, version_before);
sleep(WAIT_FOR_EFFECTIVE_S);
int table_id = maat_get_table_id(maat_inst, table_name);
ASSERT_GT(table_id, 0);
@@ -5473,7 +5456,7 @@ TEST_F(MaatCmdTest, HitPath) {
long long item4_id = maat_cmd_incrby(maat_inst, "SEQUENCE_REGION", 1);
long long group4_id = maat_cmd_incrby(maat_inst, "SEQUENCE_GROUP", 1);
ret = expr_table_set_line(maat_inst, keywords_table_name, MAAT_OP_ADD, item4_id, group4_id,
maat_cmd_str_escape(temp, sizeof(temp), "a finite or infinite"),
str_escape(temp, sizeof(temp), "a finite or infinite"),
NULL, 0, 0, 0, 0); /*EXPR_TYPE_STRING MATCH_METHOD_SUB*/
EXPECT_EQ(ret, 1);
@@ -5714,7 +5697,7 @@ TEST_F(MaatCmdTest, SameSuperGroupRefByMultiCompile) {
long long item5_id = maat_cmd_incrby(maat_inst, "SEQUENCE_REGION", 1);
long long group5_id = maat_cmd_incrby(maat_inst, "SEQUENCE_GROUP", 1);
int ret = expr_table_set_line(maat_inst, http_sig_table_name, MAAT_OP_ADD, item5_id, group5_id,
maat_cmd_str_escape(temp, sizeof(temp), "same supergroup referenced by multi compile"),
str_escape(temp, sizeof(temp), "same supergroup referenced by multi compile"),
"KEY", 0, 0, 0, 0); /*EXPR_TYPE_STRING MATCH_METHOD_SUB*/
EXPECT_EQ(ret, 1);
@@ -6679,7 +6662,7 @@ struct log_handle *MaatRollbackTest::logger;
static int clear_config_in_redis(redisContext *c, struct log_handle *logger)
{
long long redis_version = 0;
redisReply *reply = maat_cmd_wrap_redis_command(c, "GET MAAT_VERSION");
redisReply *reply = maat_wrap_redis_command(c, "GET MAAT_VERSION");
if (reply != NULL) {
if (reply->type == REDIS_REPLY_NIL || reply->type == REDIS_REPLY_ERROR) {
log_error(logger, MODULE_FRAMEWORK_GTEST,
@@ -6696,7 +6679,7 @@ static int clear_config_in_redis(redisContext *c, struct log_handle *logger)
return -1;
}
redis_version = maat_cmd_read_redis_integer(reply);
redis_version = maat_read_redis_integer(reply);
if (redis_version < 0) {
if (reply->type == REDIS_REPLY_ERROR) {
log_error(logger, MODULE_FRAMEWORK_GTEST,
@@ -6711,7 +6694,7 @@ static int clear_config_in_redis(redisContext *c, struct log_handle *logger)
freeReplyObject(reply);
reply = NULL;
reply = maat_cmd_wrap_redis_command(c, "MULTI");
reply = maat_wrap_redis_command(c, "MULTI");
freeReplyObject(reply);
reply = NULL;
@@ -6731,7 +6714,7 @@ static int clear_config_in_redis(redisContext *c, struct log_handle *logger)
int redis_transaction_success = 1;
for (int i = 0; i < append_cmd_cnt; i++) {
int ret = maat_cmd_wrap_redis_get_reply(c, &reply);
int ret = maat_wrap_redis_get_reply(c, &reply);
if (ret == REDIS_OK) {
if (reply->type == REDIS_REPLY_NIL) {
redis_transaction_success = 0;
@@ -6750,7 +6733,7 @@ static int clear_config_in_redis(redisContext *c, struct log_handle *logger)
static int rollback_redis_version(redisContext *c, struct log_handle *logger)
{
redisReply *reply = maat_cmd_wrap_redis_command(c, "SET MAAT_VERSION 0");
redisReply *reply = maat_wrap_redis_command(c, "SET MAAT_VERSION 0");
if (NULL == reply) {
log_error(logger, MODULE_FRAMEWORK_GTEST,
"[%s:%d] set MAAT_VERSION failed, Redis Communication error: %s",
@@ -6788,7 +6771,7 @@ TEST_F(MaatRollbackTest, FullConfigRollback) {
char redis_ip[64] = "127.0.0.1";
int redis_port = 6379;
int redis_db = 0;
redisContext *c = maat_cmd_connect_redis(redis_ip, redis_port, redis_db, logger);
redisContext *c = maat_connect_redis(redis_ip, redis_port, redis_db, logger);
EXPECT_TRUE(c != NULL);
ret = clear_config_in_redis(c, logger);
@@ -6837,7 +6820,7 @@ TEST_F(MaatRollbackTest, FullConfigRollbackWhenScanUnfinished) {
char redis_ip[64] = "127.0.0.1";
int redis_port = 6379;
int redis_db = 0;
redisContext *c = maat_cmd_connect_redis(redis_ip, redis_port, redis_db, logger);
redisContext *c = maat_connect_redis(redis_ip, redis_port, redis_db, logger);
EXPECT_TRUE(c != NULL);
ret = clear_config_in_redis(c, logger);

12
test/maat_framework_perf_gtest.cpp
View File

@@ -8,6 +8,7 @@
#include "maat_command.h"
#include "json2iris.h"
#include "maat_config_monitor.h"
#include "maat_redis_monitor.h"
#define MODULE_FRAMEWORK_PERF_GTEST module_name_str("maat.framework_perf_gtest")
@@ -78,7 +79,8 @@ int make_serial_rule(const char *table_name, const char *line, void *u_para)
buff[strlen(buff)-1]='\0';
}
maat_cmd_set_serial_rule(s_rule + line_idx, MAAT_OP_ADD, rule_id, table_name, buff, absolute_expire_time);
maat_set_serial_rule(s_rule + line_idx, MAAT_OP_ADD, rule_id, table_name,
buff, absolute_expire_time);
line_idx++;
FREE(buff);
@@ -93,12 +95,12 @@ static int write_config_to_redis(char *redis_ip, int redis_port, int redis_db,
snprintf(json_iris_path, sizeof(json_iris_path), "./%s_iris_tmp", json_filename);
redisContext *c = maat_cmd_connect_redis(redis_ip, redis_port, redis_db, logger);
redisContext *c = maat_connect_redis(redis_ip, redis_port, redis_db, logger);
if (NULL == c) {
return -1;
}
redisReply *reply = maat_cmd_wrap_redis_command(c, "flushdb");
redisReply *reply = maat_wrap_redis_command(c, "flushdb");
if (NULL == reply) {
return -1;
} else {
@@ -135,7 +137,7 @@ static int write_config_to_redis(char *redis_ip, int redis_port, int redis_db,
&total_line_cnt, NULL, logger);
struct serial_rule *s_rule = ALLOC(struct serial_rule, total_line_cnt);
long long server_time = maat_cmd_redis_server_time_s(c);
long long server_time = maat_redis_server_time_s(c);
if (server_time < 0) {
return -1;
}
@@ -154,7 +156,7 @@ static int write_config_to_redis(char *redis_ip, int redis_port, int redis_db,
EXPECT_EQ(success_cnt, (int)total_line_cnt);
for (size_t i = 0; i < total_line_cnt; i++) {
maat_cmd_clear_rule_cache(s_rule + i);
maat_clear_rule_cache(s_rule + i);
}
FREE(s_rule);
redisFree(c);

15
test/maat_input_mode_gtest.cpp
View File

@@ -4,6 +4,7 @@
#include "json2iris.h"
#include "maat_table.h"
#include "maat_config_monitor.h"
#include "maat_redis_monitor.h"
#include <gtest/gtest.h>
@@ -135,17 +136,17 @@ int make_serial_rule(const char *table_name, const char *line, void *u_para)
}
const char *redis_rule_key = "TEST_RULE_KEY";
redisReply *reply = maat_cmd_wrap_redis_command(ctx, "INCRBY %s %d", redis_rule_key, 1);
redisReply *reply = maat_wrap_redis_command(ctx, "INCRBY %s %d", redis_rule_key, 1);
if (reply->type == REDIS_REPLY_NIL) {
printf("incrby redis_rule_key:%s failed.", redis_rule_key);
return -1;
} else {
s_rule->rule_id = maat_cmd_read_redis_integer(reply);
s_rule->rule_id = maat_read_redis_integer(reply);
freeReplyObject(reply);
reply = NULL;
}
maat_cmd_set_serial_rule(s_rule + line_idx, MAAT_OP_ADD, s_rule->rule_id, table_name,
maat_set_serial_rule(s_rule + line_idx, MAAT_OP_ADD, s_rule->rule_id, table_name,
buff, absolute_expire_time);
(s_rule + line_idx)->ref_ctx = ctx;
line_idx++;
@@ -163,10 +164,10 @@ TEST(redis_mode, maat_scan_string) {
snprintf(json_iris_path, sizeof(json_iris_path), "./%s_iris_tmp", json_filename);
redisContext *c = maat_cmd_connect_redis(redis_ip, redis_port, redis_db, g_logger);
redisContext *c = maat_connect_redis(redis_ip, redis_port, redis_db, g_logger);
EXPECT_TRUE(c != NULL);
redisReply *reply = maat_cmd_wrap_redis_command(c, "flushdb");
redisReply *reply = maat_wrap_redis_command(c, "flushdb");
EXPECT_TRUE(reply != NULL);
freeReplyObject(reply);
reply = NULL;
@@ -193,7 +194,7 @@ TEST(redis_mode, maat_scan_string) {
struct serial_rule *s_rule = ALLOC(struct serial_rule, total_line_cnt);
s_rule->ref_ctx = c;
long long server_time = maat_cmd_redis_server_time_s(c);
long long server_time = maat_redis_server_time_s(c);
EXPECT_NE(server_time, -1);
absolute_expire_time = server_time + 300;
@@ -208,7 +209,7 @@ TEST(redis_mode, maat_scan_string) {
EXPECT_EQ(success_cnt, (int)total_line_cnt);
for (size_t i = 0; i < total_line_cnt; i++) {
maat_cmd_clear_rule_cache(s_rule + i);
maat_clear_rule_cache(s_rule + i);
}
FREE(s_rule);
redisFree(c);

39
tools/maat_redis_tool.cpp
View File

@@ -10,6 +10,7 @@
#include "maat_command.h"
#include "cJSON/cJSON.h"
#include "maat_config_monitor.h"
#include "maat_redis_monitor.h"
#include "json2iris.h"
#include "hiredis/hiredis.h"
@@ -82,7 +83,7 @@ void read_rule_from_redis(redisContext *c, long long desire_version, const char
FILE *index_fp = NULL;
struct serial_rule *rule_list = NULL;
int rule_num = maat_cmd_get_rm_key_list(c, 0, desire_version, &version, NULL, &rule_list, &update_type, 0, logger);
int rule_num = maat_get_rm_key_list(c, 0, desire_version, &version, NULL, &rule_list, &update_type, 0, logger);
if (0 == rule_num) {
if (desire_version != 0) {
printf("Read desired version %lld failed.\n", desire_version);
@@ -104,7 +105,7 @@ void read_rule_from_redis(redisContext *c, long long desire_version, const char
}
printf("Reading value: \n");
ret = maat_cmd_get_redis_value(c, rule_list, rule_num, 1, logger);
ret = maat_get_redis_value(c, rule_list, rule_num, 1, logger);
if (ret < 0) {
goto clean_up;
}
@@ -125,10 +126,10 @@ void read_rule_from_redis(redisContext *c, long long desire_version, const char
}
}
ret = maat_cmd_get_foreign_keys_by_prefix(c, rule_list, rule_num, foreign_files_dir, logger);
ret = maat_get_foreign_keys_by_prefix(c, rule_list, rule_num, foreign_files_dir, logger);
if (ret > 0) {
printf("%d lines has foreign content.\n", ret);
maat_cmd_get_foreign_conts(c, rule_list, rule_num, 1, NULL);
maat_get_foreign_conts(c, rule_list, rule_num, 1, NULL);
}
snprintf(index_path,sizeof(index_path), "%s/full_config_index.%020lld", output_path, version);
@@ -140,7 +141,7 @@ void read_rule_from_redis(redisContext *c, long long desire_version, const char
for (i = 0; i < rule_num; i++) {
if (rule_list[i].n_foreign > 0) {
maat_cmd_rewrite_table_line_with_foreign(rule_list+i);
maat_rewrite_table_line_with_foreign(rule_list+i);
}
if (NULL == cur_table || 0 != strcmp(cur_table,rule_list[i].table_name)) {
@@ -185,7 +186,7 @@ void read_rule_from_redis(redisContext *c, long long desire_version, const char
clean_up:
for (i = 0; i < rule_num; i++) {
maat_cmd_clear_rule_cache(rule_list+i);
maat_clear_rule_cache(rule_list+i);
}
FREE(rule_list);
@@ -223,17 +224,17 @@ int make_serial_rule(const char *table_name, const char *line, void *u_para)
}
const char *redis_rule_key = "TEST_RULE_KEY";
redisReply *reply = maat_cmd_wrap_redis_command(ctx, "INCRBY %s %d", redis_rule_key, 1);
redisReply *reply = maat_wrap_redis_command(ctx, "INCRBY %s %d", redis_rule_key, 1);
if (reply->type == REDIS_REPLY_NIL) {
printf("incrby redis_rule_key:%s failed.", redis_rule_key);
return -1;
} else {
s_rule->rule_id = maat_cmd_read_redis_integer(reply);
s_rule->rule_id = maat_read_redis_integer(reply);
freeReplyObject(reply);
reply = NULL;
}
maat_cmd_set_serial_rule(s_rule + line_idx, MAAT_OP_ADD, s_rule->rule_id, table_name,
maat_set_serial_rule(s_rule + line_idx, MAAT_OP_ADD, s_rule->rule_id, table_name,
buff, absolute_expire_time);
(s_rule + line_idx)->ref_ctx = ctx;
line_idx++;
@@ -266,7 +267,7 @@ int write_config_to_redis(redisContext *c, char *json_filename, struct log_handl
struct serial_rule *s_rule = ALLOC(struct serial_rule, total_line_cnt);
s_rule->ref_ctx = c;
long long server_time = maat_cmd_redis_server_time_s(c);
long long server_time = maat_redis_server_time_s(c);
if (server_time < 0) {
return -1;
}
@@ -284,7 +285,7 @@ int write_config_to_redis(redisContext *c, char *json_filename, struct log_handl
assert(success_cnt == (int)total_line_cnt);
for (size_t i = 0; i < total_line_cnt; i++) {
maat_cmd_clear_rule_cache(s_rule + i);
maat_clear_rule_cache(s_rule + i);
}
FREE(s_rule);
@@ -294,7 +295,7 @@ int write_config_to_redis(redisContext *c, char *json_filename, struct log_handl
int rollback_redis_version(redisContext *c, struct log_handle *logger)
{
redisReply *reply = maat_cmd_wrap_redis_command(c, "SET MAAT_VERSION 0");
redisReply *reply = maat_wrap_redis_command(c, "SET MAAT_VERSION 0");
if (NULL == reply) {
log_error(logger, MODULE_REDIS_TOOL,
"[%s:%d] set MAAT_VERSION failed, Redis Communication error: %s",
@@ -310,7 +311,7 @@ int rollback_redis_version(redisContext *c, struct log_handle *logger)
int clear_config_in_redis(redisContext *c, struct log_handle *logger)
{
long long redis_version = 0;
redisReply *reply = maat_cmd_wrap_redis_command(c, "GET MAAT_VERSION");
redisReply *reply = maat_wrap_redis_command(c, "GET MAAT_VERSION");
if (reply != NULL) {
if (reply->type == REDIS_REPLY_NIL || reply->type == REDIS_REPLY_ERROR) {
log_error(logger, MODULE_REDIS_TOOL,
@@ -327,7 +328,7 @@ int clear_config_in_redis(redisContext *c, struct log_handle *logger)
return -1;
}
redis_version = maat_cmd_read_redis_integer(reply);
redis_version = maat_read_redis_integer(reply);
if (redis_version < 0) {
if (reply->type == REDIS_REPLY_ERROR) {
log_error(logger, MODULE_REDIS_TOOL,
@@ -342,7 +343,7 @@ int clear_config_in_redis(redisContext *c, struct log_handle *logger)
freeReplyObject(reply);
reply = NULL;
reply = maat_cmd_wrap_redis_command(c, "MULTI");
reply = maat_wrap_redis_command(c, "MULTI");
freeReplyObject(reply);
reply = NULL;
@@ -362,7 +363,7 @@ int clear_config_in_redis(redisContext *c, struct log_handle *logger)
int redis_transaction_success = 1;
for (int i = 0; i < append_cmd_cnt; i++) {
int ret = maat_cmd_wrap_redis_get_reply(c, &reply);
int ret = maat_wrap_redis_get_reply(c, &reply);
if (ret == REDIS_OK) {
if (reply->type == REDIS_REPLY_NIL) {
redis_transaction_success = 0;
@@ -440,7 +441,7 @@ int main(int argc, char * argv[])
}
}
redisContext *c = maat_cmd_connect_redis(redis_ip, redis_port, redis_db, logger);
redisContext *c = maat_connect_redis(redis_ip, redis_port, redis_db, logger);
if (NULL == c) {
return -1;
}
@@ -473,7 +474,7 @@ int main(int argc, char * argv[])
struct serial_rule *s_rule = ALLOC(struct serial_rule, total_line_cnt);
s_rule->ref_ctx = c;
long long server_time = maat_cmd_redis_server_time_s(c);
long long server_time = maat_redis_server_time_s(c);
if (!server_time) {
log_error(logger, MODULE_REDIS_TOOL, "Get Redis Time failed.");
FREE(s_rule);
@@ -498,7 +499,7 @@ int main(int argc, char * argv[])
}
for (size_t i = 0; i < total_line_cnt; i++) {
maat_cmd_clear_rule_cache(s_rule+i);
maat_clear_rule_cache(s_rule+i);
}
FREE(s_rule);
redisFree(c);