gfwleak/tango-maat
Archived
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

third draft

Browse Source
This commit is contained in:
liuwentan
2023-07-06 18:58:15 +08:00
parent 2d6ffdd166
commit 9d373ad454
41 changed files with 81287 additions and 455 deletions
Download Patch File Download Diff File Expand all files Collapse all files

90
docs/history.md
View File

@@ -1,25 +1,25 @@
# 更新记录
| 版本 | 时间 | 说明 | 修订人 |
| ------- | ---------- | ------------------------------------------------------------ | ------------ |
| v4.0 | 2023-5-2 | Maat4.0，使用Markdown重新组织文档 | 郑超、刘文坛 |
| V3.7.0 | 2022-8-23 | 增加Boolean Expression 回调表；修正分组于子分组中的描述。 | 郑超 |
| V3.2.3 | 2021-7-15 | 增加组合扫描一节 | 郑超 |
| V3.1.20 | 2021-4-28 | 增加加载gzip压缩后的JSON文件说明 | 郑超 |
| V3.1.5 | 2021-3-12 | 增加回调类FQDN表的匹配示例 | 郑超 |
| V3.1.4 | 2020-11-04 | 内容外键为“null”时，表示空文件。 | 郑超 |
| V3.1.1 | 2020-9-27 | 增加虚拟表建立在多个不同类型物理表上的说明 | 郑超 |
| V3.1.0 | 2020-9-18 | 增加FQDN回调表 | 郑超 |
| v3.0.4 | 2020-8-17 | 增加扩展的数值类域配置interval_plus | 郑超 |
| v3.0 | 2020-6-29 | Maat 3.0, 支持子句，原分组表拆分为分组关系表和分组编译表 | 郑超 |
| v2.8 | 2020-3-13 | 支持组合表 | 郑超 |
| v2.8 | 2020-2-11 | 支持命中路径（Hit Path） | 郑超 |
| v2.8 | 2020-2-4 | 支持策略按照执行顺序（Evaluation Order）排序 | 郑超 |
| v2.8 | 2020-1-22 | Maat JSON文件支持加密，参见17.3 | 郑超 |
| v2.8 | 2019-7-28 | 1、 不再兼容无分组表的情况 2、 支持虚拟表 | 郑超 |
| v2.7.2 | 2019-7-10 | 扩展类IP配置表支持CIDR格式。 | 郑超 |
| v2.7.1 | 2019-5-23 | 将富IP类配置表改名为扩展类IP配置表。 | 郑超 |
| v2.7.0 | 2019-5-12 | 1、 增加子分组关系说明 2、 增加富IP类配置的说明 | 郑超 |
| 版本 | 时间 | 说明 | 修订人 |
| ------- | ---------- | --------------------------------------------------------- | -------- |
| v4.0 | 2023-5-2 | Maat4.0，使用Markdown重新组织文档 | 郑超、刘文坛 |
| V3.7.0 | 2022-8-23 | 增加Boolean Expression 回调表；修正分组于子分组中的描述。| 郑超 |
| V3.2.3 | 2021-7-15 | 增加组合扫描一节 | 郑超 |
| V3.1.20 | 2021-4-28 | 增加加载gzip压缩后的JSON文件说明 | 郑超 |
| V3.1.5 | 2021-3-12 | 增加回调类FQDN表的匹配示例 | 郑超 |
| V3.1.4 | 2020-11-04 | 内容外键为“null”时，表示空文件。 | 郑超 |
| V3.1.1 | 2020-9-27 | 增加虚拟表建立在多个不同类型物理表上的说明 | 郑超 |
| V3.1.0 | 2020-9-18 | 增加FQDN回调表 | 郑超 |
| v3.0.4 | 2020-8-17 | 增加扩展的数值类域配置interval_plus | 郑超 |
| v3.0 | 2020-6-29 | Maat 3.0, 支持子句，原分组表拆分为分组关系表和分组编译表 | 郑超 |
| v2.8 | 2020-3-13 | 支持组合表 | 郑超 |
| v2.8 | 2020-2-11 | 支持命中路径（Hit Path） | 郑超 |
| v2.8 | 2020-2-4 | 支持策略按照执行顺序（Evaluation Order）排序 | 郑超 |
| v2.8 | 2020-1-22 | Maat JSON文件支持加密，参见17.3 | 郑超 |
| v2.8 | 2019-7-28 | 1、 不再兼容无分组表的情况 2、 支持虚拟表 | 郑超 |
| v2.7.2 | 2019-7-10 | 扩展类IP配置表支持CIDR格式。 | 郑超 |
| v2.7.1 | 2019-5-23 | 将富IP类配置表改名为扩展类IP配置表。 | 郑超 |
| v2.7.0 | 2019-5-12 | 1、 增加子分组关系说明 2、 增加富IP类配置的说明 | 郑超 |
| v2.6.0 | 2019-1-8 | 1、增加“非”运算的说明； 2、文档和动态链接库版本号的前两位相同。 | 郑超 |
| v2.3.3 | 2018-12-24 | 增加Extra Data和read column函数的说明；多命中情况下，按照配置ID由大到小返回命中结果。 | 郑超 |
| v2.3.2 | 2018-12-03 | 支持运行态以全量通知的方式加载变化的json文件 | 郑超 |
@@ -32,44 +32,42 @@
| v2.2.1 | 2018-05-25 | 更新Redis配置加载接口的事务机制。 | 郑超 |
| v2.1.2 | 2018-03-24 | 增加概述 | 郑超 |
| v2.1.1 | 2018-02-08 | 1、 maat_redis_tool支持写入json格式的配置到redis。 2、 增加Maat_read_state函数的使用说明。 | 郑超 |
| v2.1.0 | 2017-10-04 | 提供maat_redis_tool，可以dump出redis中的maat配置。 | 郑超 |
| v2.0.1 | 2017-08-22 | 支持配置延迟初始化 | 郑超 |
| v2.1.0 | 2017-10-04 | 提供maat_redis_tool，可以dump出redis中的maat配置。 | 郑超 |
| v2.0.1 | 2017-08-22 | 支持配置延迟初始化 | 郑超 |
| v2.0.0 | 2017-08-01 | 1. 支持从Redis加载配置； 2. 提供配置增删函数接口，即Maat Command API； 3. 支持相似性字符串匹配； 4. 提供辅助工具digest_gen，具备SFH摘要计算、编辑距离计算功能； | 郑超 |
| v1.9.1 | 2017-06-24 | 可以提示MAAT组合扫描中最后一个域，可以提升与表达式运算性能。 | 郑超 |
| v1.9.0 | 2017-06-09 | 支持对个别配置表进行加密 | 郑超 |
| v1.8.4 | 2017-03-12 | 调整SNORT规则转换的目录结构 | 郑超 |
| v1.9.0 | 2017-06-09 | 支持对个别配置表进行加密 | 郑超 |
| v1.8.4 | 2017-03-12 | 调整SNORT规则转换的目录结构 | 郑超 |
| v1.8.3 | 2016-12-30 | 1、增加设置实例名的功能；2、更新运行状态数据说明。 | 郑超 |
| v1.8.2 | 2016-9-19 | JSON输入模式支持未命名分组，可将group_name配置为"Untitled"以表示非分组复用，由Maat自动生成group_id。 | 郑超 |
| v1.8.1 | 2016-9-14 | 在IP类域配置表中增加多层嵌套IP匹配的指导方案。 | 郑超 |
| v1.8.0 | 2016-09-02 | 增加配置表连接功能的说明 | 郑超 |
| v1.8.0 | 2016-09-02 | 增加配置表连接功能的说明 | 郑超 |
| v1.7.1 | 2016-06-17 | 1. 通过转义方式，支持空格作为关键字; 2. 增加快速字符串扫描开关; | 郑超 |
| v1.7.0 | 2016-04-03 | 单独设小节描述编码转换，增加UNICODE到ASCII转码功能的说明 | 郑超 |
| v1.6.1 | 2016-03-24 | 完善配置分组表的字段说明。 | 郑超 |
| v1.6.0 | 2016-02-11 | 1. 增加状态统计和性能统计描述，以及相应的函数接口调整； 2. 增加扩展的字符串类域配置的函数接口描述。 | 郑超 |
| v1.5.0 | 2016-01-05 | 增加扩展的字符串类域配置，完善IP类域配置关于IPv6掩码的描述。 | 郑超 |
| v1.4.4 | 2015-12-24 | 修改回调表的配置更新机制，以节约内存。 | 郑超 |
| v1.4.4 | 2015-12-24 | 修改回调表的配置更新机制，以节约内存。 | 郑超 |
| v1.4.3 | 2015-12-02 | 完善数据库表间关系的描述，丰富界面设计要点。 | 郑超 |
| v1.4.2 | 2015-12-01 | 合并配置更新文件接口一章 | 郑超 |
| v1.4.1 | 2015-11-23 | 完善摘要类配置的说明 | 郑超 |
| v1.4.0 | 2015-11-06 | 增加文件摘要类域配置 | 郑超 |
| v1.4.2 | 2015-12-01 | 合并配置更新文件接口一章 | 郑超 |
| v1.4.1 | 2015-11-23 | 完善摘要类配置的说明 | 郑超 |
| v1.4.0 | 2015-11-06 | 增加文件摘要类域配置 | 郑超 |
| v1.3.9 | 2015-07-24 | 提供Maat_set_feather_opt接口，Maat_20150724版本以上支持该功能。 | 郑超 |
| v1.3.8 | 2015-07-06 | 润色概述中对三类配置的说明，增加编码转换句柄复用的说明。 | 郑超 |
| v1.3.7 | 2015-06-16 | 增加常见故障处理的建议。 | 郑超 |
| v1.3.7 | 2015-06-16 | 增加常见故障处理的建议。 | 郑超 |
| v1.3.6 | 2015-06-10 | 1. 字符串域表增加相对位置的表达式类型; 2. 配置分组表支持非； 测试分支，暂不合并到主版本。 | 郑超 |
| v1.3.5 | 2015-06-08 | 添加数据库表描述示例 | 郑超 |
| v1.3.4 | 2015-05-22 | 可通过JSON加载回调类配置表 | 郑超 |
| v1.3.5 | 2015-06-08 | 添加数据库表描述示例 | 郑超 |
| v1.3.4 | 2015-05-22 | 可通过JSON加载回调类配置表 | 郑超 |
| v1.3.3 | 2015-05-13 | 增加对非组合域编译的快速处理路径，增加对命中结果返回条件的说明 | 郑超 |
| v1.3.2 | 2015-04-13 | 扫描空配置的返回值由-1改为0 | 郑超 |
| v1.3.1 | 2015-03-02 | 对于GBK到BIG5的转码，将源编码改为GB2312 | 郑超 |
| v1.3 | 2015-02-20 | 增加JSON配置加载功能 | 郑超 |
| v1.2.2 | 2015-01-06 | 增加强制卸载机制 | 郑超 |
| v1.2.1 | 2015-01-05 | 扩展字符串类域配置的hexbin字段的定义 | 郑超 |
| v1.2.0 | 2014-12-20 | 增加配置更新机制的描述 | 郑超 |
| v1.1.0 | 2014-12-12 | 增加函数接口的说明，修改引用计数避免伪共享 | 郑超 |
| v1.0.3 | 2014-12-03 | 采用垃圾回收机制保证线程安全 | 郑超 |
| v1.0.2 | 2014-11-26 | 支持正则分组 | 郑超 |
| v1.0.1 | 2014-11-19 | IP类配置支持协议扫描 | 郑超 |
| v1.0 | 2014-08-19 | 第一个稳定版 | 郑超 |
| v0.1 | 2014-06-16 | 创建文档，包含表格式说明 | 郑超 |
| v1.3.2 | 2015-04-13 | 扫描空配置的返回值由-1改为0 | 郑超 |
| v1.3.1 | 2015-03-02 | 对于GBK到BIG5的转码，将源编码改为GB2312 | 郑超 |
| v1.3 | 2015-02-20 | 增加JSON配置加载功能 | 郑超 |
| v1.2.2 | 2015-01-06 | 增加强制卸载机制 | 郑超 |
| v1.2.1 | 2015-01-05 | 扩展字符串类域配置的hexbin字段的定义 | 郑超 |
| v1.2.0 | 2014-12-20 | 增加配置更新机制的描述 | 郑超 |
| v1.1.0 | 2014-12-12 | 增加函数接口的说明，修改引用计数避免伪共享 | 郑超 |
| v1.0.3 | 2014-12-03 | 采用垃圾回收机制保证线程安全 | 郑超 |
| v1.0.2 | 2014-11-26 | 支持正则分组 | 郑超 |
| v1.0.1 | 2014-11-19 | IP类配置支持协议扫描 | 郑超 |
| v1.0 | 2014-08-19 | 第一个稳定版 | 郑超 |
| v0.1 | 2014-06-16 | 创建文档，包含表格式说明 | 郑超 |

0
docs/imgs/configuration_diagram.png → docs/imgs/rule_diagram.png
View File

Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 117 KiB

After

Width:  |  Height:  |  Size: 117 KiB

35
docs/introduction.md
View File

@@ -2,19 +2,16 @@
**Item**: As a filter for network attributes, the smallest unit of a rule
- Eg1: specify that the UserAgent field in the HTTP protocol contains substrings "Chrome" and "11.8.1",
- Eg1: specify that the UserAgent field in the HTTP protocol contains substrings "Chrome" and "11.8.1",
   HTTP UserAgent: Chrome & 11.8.1
- Eg2: specify that the domain name in the HTTP protocol ends with ".emodao.com"
- Eg2: specify that the domain name in the HTTP protocol ends with ".emodao.com"
   HTTP HOST: *.emodao.com
- Eg3: specify client IP address belongs to the C segment of 202.118.101.*
- Eg3: specify client IP address belongs to the C segment of 202.118.101.*
   Source IP: 202.11.101.0/24
There are multiple types of items stored in corresponding tables such as string, IP and numerical range, more details can be found in [Item table](./table_schema.md#item-table)
There are multiple types of items stored in corresponding tables such as string, IP and numerical range, more details can be found in [Item table](./table_schema.md#item-table).
**Group(Object)**: Collection of Items, the constraints of group are as follows:
@@ -31,13 +28,14 @@ The relationship between group and group is stored in the [group2group table](./
**Compile(Policy)**: A conjunctive normal form(CNF) consisting of multiple groups and virtual tables
- A Compile can contain up to 8 clauses and multiple clauses in the same compile can be logical 'AND' and logical 'NOT' relationships.
- A Clause consists of several Literals and the relationship between them is a logical 'OR'. A Literal consists of virtual table and group. During the configuration loading process, a unique Clause ID will be generated based on the combination of virtual table ID and group ID in the same clause.
- A Clause consists of several Literals and the relationship between them is a logical 'OR'. A Literal consists of virtual table and group. During the rules loading process, a unique Clause ID will be generated based on the combination of virtual table ID and group ID in the same clause.
![CNF](./imgs/CNF.jpg)
Configuration Diagram
![configuration Diagram](./imgs/configuration_diagram.png)
Rule Diagram
![Rule Diagram](./imgs/rule_diagram.png)
## Group/Object Nesting and Hierarchies
@@ -50,16 +48,22 @@ An object defines a subset of an object type, such as network addresses or port
There are rules of precedence to take into account when defining objects:
- Excluding has precedence over including in the same object.
- Items in a superior object have precedence over items in a subordinate object.
- Items in a superior object are not taken into account in a subordinate object, if the subordinate object is used directly in a rule.
- Peer objects (different subordinate objects of the same superior object) do not affect each other.
In short, to determine the set defined by an object, perform the following calculation:
1. For each subordinate object (remember sibling objects do not affect each other):
1. Add included items.
2. Subtract excluded items.
- Add included items.
- Subtract excluded items.
2. Add included items in the object itself, overriding any excludes in the subordinate objects.
3. Subtract excluded items in the object itself.
The following figure shows an object with an included set and an excluded subset.
@@ -87,7 +91,6 @@ Now, let's see a graph of hierarchy example, where the dotted line means exclude
![object-hierarchy-example](./imgs/object-hierarchy-example.png)
| **Matched subordinate objects** | **Activated superior Objects** |
| ------------------------------- | ------------------------------ |
| g1, g3 | g6, g9 |
@@ -104,10 +107,10 @@ Now, let's see a graph of hierarchy example, where the dotted line means exclude
| **Term** | **Description** |
| --------------------------- | ------------------------------------------------------------ |
| Instance | Maat handle |
| Item | Configuration of a certain field，such as URL field in HTTP protocol, client ip address field in DNS protocol etc.|
| Item | Rule of a certain field，such as URL field in HTTP protocol, client ip address field in DNS protocol etc.|
| Group(Object) | A collection of one or more Items, the multiple items under the same Group are logical 'OR' relationships |
| Compile(Policy) | A rule for several Groups logical AND or NOT operations |
| Table | Different types of configurations have different tables, such as ip table, keywords table, group2compile table, compile table etc. |
| Table | Different types of rules have different tables, such as ip table, keywords table, group2compile table, compile table etc. |
| Physical Table | The actual table in the database |
| Virtual Table | Table that do not exist in the database and only references physical table |
| Table Schema | Define the table type and the meaning of each column in the table |
@@ -115,4 +118,4 @@ Now, let's see a graph of hierarchy example, where the dotted line means exclude
| Maat State | Store intermediate state of multiple scans |
| Maat Stream | Handle of streaming file scanning |
| Hit Path | From the perspective of data to be scanned, describe the hit ID sequence: item_id -> sub_group_id -> superior_group_id(virtual_table_id) -> compile_id |
| Redis | In-memory data store，see https://redis.io/. It has a leader follower replication to ensure the high availability of configuration |
| Redis | In-memory data store，see https://redis.io/. It has a leader follower replication to ensure the high availability of rules |

304
docs/logic (AND | OR | NOT).md Normal file
View File

@@ -0,0 +1,304 @@
# logic AND OR NOT
- [logic AND](#logic-and)
- [logic OR](#logic-or)
- [logic NOT](#logic-not)
- [group exclude](#group-exclude)
## logic AND
rule = China & USA
```json
{
"compile_id": 123,
"service": 1,
"action": 1,
"do_blacklist": 1,
"do_log": 1,
"user_region": "null",
"is_valid": "yes",
"groups": [
{
"clause_index": 0,
"regions": [
{
"table_name": "HTTP_URL",
"table_type": "expr",
"table_content": {
"keywords": "China",
"expr_type": "none",
"match_method": "sub",
"format": "uncase plain"
}
}
]
},
{
"clause_index": 1,
"regions": [
{
"table_name": "HTTP_URL",
"table_type": "expr",
"table_content": {
"keywords": "USA",
"expr_type": "none",
"match_method": "sub",
"format": "uncase plain"
}
}
]
}
]
}
```
scan sample
```c
const char *string1 = "China";
const char *string2 = "USA";
long long results[ARRAY_SIZE] = {0};
size_t n_hit_result = 0;
int thread_id = 0;
const char *table_name = "HTTP_URL";
struct maat_state *state = maat_state_new(maat_inst, thread_id);
int table_id = maat_get_table_id(maat_inst, table_name);
ASSERT_GT(table_id, 0);
int ret = maat_scan_string(maat_inst, table_id, string1, strlen(string1),
results, ARRAY_SIZE, &n_hit_result, state);
EXPECT_EQ(ret, MAAT_SCAN_HALF_HIT);
ret = maat_scan_string(maat_inst, table_id, string2, strlen(string2),
results, ARRAY_SIZE, &n_hit_result, state);
EXPECT_EQ(ret, MAAT_SCAN_HIT);
EXPECT_EQ(n_hit_result, 1);
EXPECT_EQ(results[0], 123);
maat_state_free(state);
```
## logic OR
rule = China | USA
```json
{
"compile_id": 124,
"service": 0,
"action": 0,
"do_blacklist": 0,
"do_log": 0,
"user_region": "null",
"is_valid": "yes",
"groups": [
{
"regions": [
{
"table_type": "expr",
"table_name": "HTTP_URL",
"table_content": {
"keywords": "China",
"expr_type": "none",
"format": "uncase plain",
"match_method": "prefix"
}
},
{
"table_type": "expr",
"table_name": "HTTP_URL",
"table_content": {
"keywords": "USA",
"expr_type": "none",
"format": "uncase plain",
"match_method": "prefix"
}
}
]
}
]
}
```
scan sample
```c
const char *string1 = "China";
const char *string2 = "USA";
long long results[ARRAY_SIZE] = {0};
size_t n_hit_result = 0;
int thread_id = 0;
const char *table_name = "HTTP_URL";
struct maat_state *state = maat_state_new(maat_inst, thread_id);
int table_id = maat_get_table_id(maat_inst, table_name);
ASSERT_GT(table_id, 0);
ret = maat_scan_string(maat_inst, table_id, string1, strlen(string1),
results, ARRAY_SIZE, &n_hit_result, state);
EXPECT_EQ(ret, MAAT_SCAN_HIT);
EXPECT_EQ(n_hit_result, 1);
EXPECT_EQ(results[0], 124);
maat_state_reset(state);
ret = maat_scan_string(maat_inst, table_id, string2, strlen(string2),
results, ARRAY_SIZE, &n_hit_result, state);
EXPECT_EQ(ret, MAAT_SCAN_HIT);
EXPECT_EQ(n_hit_result, 1);
EXPECT_EQ(results[0], 124);
maat_state_free(state);
```
## logic NOT
rule = China & !USA
```json
{
"compile_id": 125,
"service": 1,
"action": 1,
"do_blacklist": 1,
"do_log": 1,
"user_region": "null",
"is_valid": "yes",
"groups": [
{
"clause_index": 0,
"not_flag": 0,
"regions": [
{
"table_name": "HTTP_URL",
"table_type": "expr",
"table_content": {
"keywords": "China",
"expr_type": "none",
"match_method": "sub",
"format": "uncase plain"
}
}
]
},
{
"clause_index": 1,
"not_flag": 1,
"regions": [
{
"table_name": "HTTP_URL",
"table_type": "expr",
"table_content": {
"keywords": "USA",
"expr_type": "none",
"match_method": "sub",
"format": "uncase plain"
}
}
]
}
]
}
```
scan sample
```c
const char *string1 = "China";
const char *string2 = "England";
long long results[ARRAY_SIZE] = {0};
size_t n_hit_result = 0;
int thread_id = 0;
const char *table_name = "HTTP_URL";
struct maat_state *state = maat_state_new(maat_inst, thread_id);
int table_id = maat_get_table_id(maat_inst, table_name);
ASSERT_GT(table_id, 0);
int ret = maat_scan_string(maat_inst, table_id, string1, strlen(string1),
results, ARRAY_SIZE, &n_hit_result, state);
EXPECT_EQ(ret, MAAT_SCAN_HALF_HIT);
maat_state_set_last_scan(state);
ret = maat_scan_string(maat_inst, table_id, string2, strlen(string2),
results, ARRAY_SIZE, &n_hit_result, state);
EXPECT_EQ(ret, MAAT_SCAN_HIT);
EXPECT_EQ(n_hit_result, 1);
EXPECT_EQ(results[0], 125);
maat_state_free(state);
```
## group exclude
```json
{
"compile_id": 126,
"service": 1,
"action": 1,
"do_blacklist": 1,
"do_log": 1,
"user_region": "null",
"is_valid": "yes",
"groups": [
{
"group_name": "ExcludeGroup199",
"sub_groups":[
{
"group_name": "ExcludeGroup199_1",
"is_exclude": 0,
"clause_index": 0,
"regions": [
{
"table_name": "HTTP_URL",
"table_type": "expr",
"table_content": {
"keywords": "baidu.com",
"expr_type": "none",
"match_method": "suffix",
"format": "uncase plain"
}
}
]
},
{
"group_name": "ExcludeGroup199_2",
"is_exclude": 1,
"clause_index": 0,
"regions": [
{
"table_name": "HTTP_URL",
"table_type": "expr",
"table_content": {
"keywords": "www.baidu.com",
"expr_type": "none",
"match_method": "exact",
"format": "uncase plain"
}
}
]
}
]
}
]
}
```
scan sample
```c
const char *string_not_hit = "www.baidu.com";
const char *string_hit = "mail.baidu.com";
long long results[ARRAY_SIZE] = {0};
size_t n_hit_result = 0;
int thread_id = 0;
const char *table_name = "HTTP_URL";
struct maat_state *state = maat_state_new(maat_inst, thread_id);
int table_id = maat_get_table_id(maat_inst, table_name);
ASSERT_GT(not_hit_table_id, 0);
int ret = maat_scan_string(maat_inst, table_id, string_not_hit, strlen(string_not_hit),
results, ARRAY_SIZE, &n_hit_result, state);
EXPECT_EQ(ret, MAAT_SCAN_HALF_HIT);
ret = maat_scan_string(maat_inst, table_id, string_hit, strlen(string_hit),
results, ARRAY_SIZE, &n_hit_result, state);
EXPECT_EQ(ret, MAAT_SCAN_HIT);
EXPECT_EQ(n_hit_result, 1);
EXPECT_EQ(results[0], 126);
maat_state_free(state);
```

4
docs/scan_api.md
View File

@@ -4,9 +4,9 @@
多命中情况下，按包含分组数由多到少返回，分组数相同时，按编译配置ID由大到小的顺序返回。
多命中扫描的最大命中次数，受MAX_SCANNER_HIT_NUM宏控制，当前为64条。
多命中扫描的最大命中次数，受MAX_SCANNER_HIT_NUM宏控制，当前为4096条。
如果命中条数超出64，则按照配置在IRIS库表文件出现的顺序返回。
如果命中条数超出4096，则按照配置在IRIS库表文件出现的顺序返回。
## 流式跨包扫描

4
docs/table_data.md
View File

@@ -2,7 +2,7 @@
Input must use UTF-8 without BOM encoding, for example, MySQL use utf8mb4.
Maat supports three configuration loading modes.
Maat supports three rule loading modes.
- [Redis mode](#1-redis-mode)
- [Iris mode](#2-iris-mode)
- [Json mode](#3-json-mode)
@@ -392,8 +392,6 @@ Maat实例的工作线程定时轮询Redis中MAAT_VERSION，如果大于实例
```
## 组合配置写入示例
已知IP类域配置表IP_TABLE、关键字类配置表KEYWORD、编译配置表COMPILE、分组表GROUP。

22
docs/table_schema.md
View File

@@ -29,7 +29,7 @@ Item tables are further subdivided into different types of subtables as follows:
Each item table must has the following columns:
- item_id: In a maat instance, the item id is globally unique, meaning that the item IDs of different tables must not be duplicate.
- item_id: In a maat instance, the item id is globally unique, meaning that the item id of different tables must not be duplicate.
- group_id: Indicate the group to which the item belongs, an item belongs to only one group.
@@ -48,7 +48,7 @@ Describe matching rules for strings.
| **group_id** | LONG LONG | N | group2group or group2compile table's group_id |
| **keywords** | VARCHAR2(1024) | N | field to match during scanning |
| **expr_type** | INT | N | 0(keywords), 1(AND expr), 2(regular expr), 3(substring with offset)
| **match_method** | INT | N | only useful when expr_type is 0 |
| **match_method** | INT | N | only useful when expr_type is 0. 0(sub), 1(suffix), 2(prefix), 3(exactly) |
| **is_hexbin** | INT | N | 0(not HEX & case insensitive, this is default value) 1(HEX & case sensitive) 2(not HEX & case sensitive) |
| **is_valid** | INT | N | 0(invalid), 1(valid) |
@@ -123,7 +123,7 @@ Describe matching rules for strings.
For example: substring expr: "1-1:48&3-4:4C4C", scan_data: "HELLO" will hit, "HLLO" will not hit.
**Note**: 48('H') 4C('L')
  Since Maat4.0，only support UTF-8，no more encoding conversion。For binary format configurations, the keyword is hexadecimal, such as the keyword "hello" is represented as "68656C6C6F". A keyword can't contain invisible characters such as spaces, tabs, and CR, which are ASCII codes 0x00 to 0x1F and 0x7F. If these characters need to be used, they must be escaped, refer to the "keywords escape table". Characters led by backslashes outside this table are processed as ordinary strings, such as '\t' will be processed as the string "\t".
  Since Maat4.0，only support UTF-8，no more encoding conversion。For binary format rules, the keyword is hexadecimal, such as the keyword "hello" is represented as "68656C6C6F". A keyword can't contain invisible characters such as spaces, tabs, and CR, which are ASCII codes 0x00 to 0x1F and 0x7F. If these characters need to be used, they must be escaped, refer to the "keywords escape table". Characters led by backslashes outside this table are processed as ordinary strings, such as '\t' will be processed as the string "\t".
The symbol '&' means conjunction operation in AND expression. So if the keywords has '&', it must be escaped by '\&'.
@@ -603,18 +603,18 @@ Describe the specific policy, One maat instance can has multiple compile tables
### 5. <a name='plugintable'></a> plugin table
There is no fixed format for configuration of the plugin table, which is determined by business side. The plugin table supports two sets of callback functions, registered with **maat_table_callback_register** and **maat_plugin_table_ex_schema_register** respectively.
There is no fixed rule format of the plugin table, which is determined by business side. The plugin table supports two sets of callback functions, registered with **maat_table_callback_register** and **maat_plugin_table_ex_schema_register** respectively.
maat_table_callback_register
```c
/*
When the plugin table configurations are updated, start will be called first and only once, then update will be called by each configuration item, and finish will be called last and only once.
When the plugin table rules are updated, start will be called first and only once, then update will be called by each rule item, and finish will be called last and only once.
If configurations have been loaded but maat_table_callback_register has not yet been called, maat will cache the loaded configurations and perform the callbacks(start, update, finish) when registration is complete.
If rules have been loaded but maat_table_callback_register has not yet been called, maat will cache the loaded rules and perform the callbacks(start, update, finish) when registration is complete.
*/
typedef void maat_start_callback_t(int update_type, ...);
//table_line points to one complete configuration line, such as: "1\tHeBei\tShijiazhuang\t1\t0"
//table_line points to one complete rule line, such as: "1\tHeBei\tShijiazhuang\t1\t0"
typedef void maat_update_callback_t(..., const char *table_line, ...);
typedef void maat_finish_callback_t(...);
@@ -643,7 +643,7 @@ int maat_plugin_table_ex_schema_register(...,
...);
```
three types of keys(pointer, integer and ip_addr) for ex_data callback.
Plugin table supports three types of keys(pointer, integer and ip_addr) for ex_data callback.
**pointer key(compatible with maat3)**
@@ -662,7 +662,7 @@ three types of keys(pointer, integer and ip_addr) for ex_data callback.
}
```
(2) plugin table configuration
(2) plugin table rules
```json
{
"table_name": "TEST_PLUGIN_POINTER_KEY_TYPE",
@@ -722,7 +722,7 @@ support integers of different lengths, such as int(4 bytes), long long(8 bytes).
}
```
(2) plugin table configuration
(2) plugin table rules
```
{
"table_name": "TEST_PLUGIN_INT_KEY_TYPE",
@@ -782,7 +782,7 @@ support ip address(ipv4 or ipv6) as key.
```
The addr_type column indicates whether the key is a v4 or v6 address.
(2) plugin table configuration
(2) plugin table rules
```
{
"table_name": "TEST_PLUGIN_IP_KEY_TYPE",

5
docs/thread_mode.md
View File

@@ -46,10 +46,11 @@ int main()
maat_options_set_caller_thread_number(opts, thread_num);
struct maat *maat_inst = maat_new(opts, table_info_path);
size_t i = 0;
pthread_t threads[thread_num];
struct thread_param thread_params[thread_num];
for (size_t i = 0; i < thread_num; i++) {
for (i = 0; i < thread_num; i++) {
thread_params[i].maat_inst = maat_inst;
thread_params[i].thread_id = i;
thread_params[i].table_name = table_name;
@@ -63,4 +64,4 @@ int main()
return 0;
}
```
```