gfwleak/tango-maat
Archived
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

rename terminology "compile" to "rule"

Browse Source
This commit is contained in:
root
2024-08-22 03:11:15 +00:00
parent a4ca92ea41
commit 678ddd718a
44 changed files with 2586 additions and 2586 deletions
Download Patch File Download Diff File Expand all files Collapse all files

52
docs/api_reference.md
View File

@@ -14,7 +14,7 @@ After that, call maat_new to create a maat instance. Next, for the two main usag
* Call maat_state_new to create a maat_state structure to save the intermediate state of the scan.
* Call the maat_scan_xx interface to perform the scan task and return the hit compile_id.
* Call the maat_scan_xx interface to perform the scan task and return the hit rule_id.
* If you are interested in the hit item, group, or path information, you can call the maat_state_get_xx interface as needed.
@@ -68,9 +68,9 @@ After that, call maat_new to create a maat instance. Next, for the two main usag
* [maat_state_reset](#maat_state_reset)
* [maat_state_free](#maat_state_free)
* [maat_state_set_scan_district](#maat_state_set_scan_district)
* [maat_state_set_scan_compile_table](#maat_state_set_scan_compile_table)
* [maat_state_set_scan_rule_table](#maat_state_set_scan_rule_table)
* [maat_state_get_hit_paths](#maat_state_get_hit_paths)
* [maat_state_get_compile_table_ids](#maat_state_get_compile_table_ids)
* [maat_state_get_rule_table_ids](#maat_state_get_rule_table_ids)
* [maat_state_get_scan_count](#maat_state_get_scan_count)
* [maat_state_get_direct_hit_groups](#maat_state_get_direct_hit_groups)
* [maat_state_get_direct_hit_group_cnt](#maat_state_get_direct_hit_group_cnt)
@@ -602,18 +602,18 @@ To set the field to be matched in the next scan, for example: if you want to mat
**Returns**: success(0) failure(-1)
### maat_state_set_scan_compile_table
### maat_state_set_scan_rule_table
```c
int maat_state_set_scan_compile_table(struct maat_state *state,
int compile_table_id);
int maat_state_set_scan_rule_table(struct maat_state *state,
int rule_table_id);
```
Maat supports loading multiple compile tables and building their respective runtime. Within the table schema, there is a compile table that has a `default_compile_table` field, indicating that maat will use the runtime of this table to calculate which compile_id will be hit by the hit group_id. If the caller wants to use another compile table runtime for the above calculation, this interface can be invoked to set it up.
Maat supports loading multiple rule tables and building their respective runtime. Within the table schema, there is a rule table that has a `default_rule_table` field, indicating that maat will use the runtime of this table to calculate which rule_id will be hit by the hit group_id. If the caller wants to use another rule table runtime for the above calculation, this interface can be invoked to set it up.
**Parameters**:
* state - Pointer to the maat state structure.
* compile_table_id - The compile table ID.
* rule_table_id - The rule table ID.
**Returns**: success(0) failure(-1)
@@ -636,24 +636,24 @@ Gets all hit paths since maat state was created.
**Returns**: The number of hit paths is returned on success, and -1 is returned on failure.
### maat_state_get_compile_table_ids
### maat_state_get_rule_table_ids
```c
int maat_state_get_compile_table_ids(struct maat_state *state,
long long *compile_ids,
size_t n_compile_ids,
int *compile_table_ids);
int maat_state_get_rule_table_ids(struct maat_state *state,
long long *rule_ids,
size_t n_rule_ids,
int *rule_table_ids);
```
Get the ID of the table that compile_id belongs to.
Get the ID of the table that rule_id belongs to.
**Parameters**:
* state - Pointer to the maat state structure.
* compile_ids - Array of compile_ids.
* n_compild_ids - The number of compile_ids.
* compile_table_ids - The array of id of the tables to which each compile id belongs.
* rule_ids - Array of rule_ids.
* n_compild_ids - The number of rule_ids.
* rule_table_ids - The array of id of the tables to which each rule id belongs.
**Returns**: The number of compile table ids is return on success, and -1 is returned on failure.
**Returns**: The number of rule table ids is return on success, and -1 is returned on failure.
### maat_state_get_scan_count
@@ -780,15 +780,15 @@ int maat_scan_flag(struct maat *instance, int table_id, long long flag,
struct maat_state *state);
```
Check if the `flag` in the scan input hits the rules in the effective compile table. If there is a hit, put the compile_id of the rules into the results array and put the number of hit compile_ids into n_hit_result.
Check if the `flag` in the scan input hits the rules in the effective rule table. If there is a hit, put the rule_id of the rules into the results array and put the number of hit rule_ids into n_hit_result.
**Parameters**:
* instance - Maat instance created by maat_new().
* table_id - ID of the table to be scanned.
* flag - Data to be scanned.
* results - Array to store hit compile_ids.
* results - Array to store hit rule_ids.
* n_result - Size of the results array.
* n_hit_result - The number of hit compile_ids.
* n_hit_result - The number of hit rule_ids.
* state - Used to store the intermediate state of the scan.
**Returns**:
@@ -850,7 +850,7 @@ int maat_scan_group(struct maat *instance, int table_id,
size_t *n_hit_result, struct maat_state *state);
```
Different from the above interface, the input to this interface is the hit item_id and group_id. Therefore, there is no need to execute the underlying scanning engine. It only needs to calculate the hit compile_id based on the hit group_id.
Different from the above interface, the input to this interface is the hit item_id and group_id. Therefore, there is no need to execute the underlying scanning engine. It only needs to calculate the hit rule_id based on the hit group_id.
### maat_scan_not_logic
@@ -861,7 +861,7 @@ int maat_scan_not_logic(struct maat *instance, int table_id,
size_t *n_hit_result, struct maat_state *state);
```
If the compile configuration contains NOT-clauses, this interface needs to be called to activate the calculation of compile_id involving NOT-clauses. This means that other scanning interfaces will not activate NOT-clauses except for calling this interface. When to call this interface is determined by the caller. For example, on a session, four packets were scanned with maat_scan_string and three compile_id were hit. The NOT-clause was expected to be activated, so this interface was called and two new compile_id were hit.
If the rule configuration contains NOT-clauses, this interface needs to be called to activate the calculation of rule_id involving NOT-clauses. This means that other scanning interfaces will not activate NOT-clauses except for calling this interface. When to call this interface is determined by the caller. For example, on a session, four packets were scanned with maat_scan_string and three rule_id were hit. The NOT-clause was expected to be activated, so this interface was called and two new rule_id were hit.
Since this interface only activates non-clauses and does not call the underlying scanning engine to perform actual scanning tasks, there is no data to be scanned.
@@ -894,15 +894,15 @@ int maat_stream_scan(struct maat_stream *stream, const char *data, int data_len,
struct maat_state *state);
```
This interface is used to perform streaming scanning tasks, for example: in a session, consecutively scanning multiple packets and providing the hit compile_ids.
This interface is used to perform streaming scanning tasks, for example: in a session, consecutively scanning multiple packets and providing the hit rule_ids.
**Parameters**:
* stream - Maat_stream handle created by maat_stream_new().
* data - Data to be scanned.
* data_len - The length of the data.
* results - Array to store hit compile_ids.
* results - Array to store hit rule_ids.
* n_result - Size of the results array.
* n_hit_result - The number of hit compile_ids.
* n_hit_result - The number of hit rule_ids.
* state - Used to store the intermediate state of the scan.
**Returns**:

6
docs/configuration_management.md
View File

@@ -9,7 +9,7 @@ Maat supports three configuration loading modes:
## 1.<a name='Redis mode'></a> Redis mode
As shown in the diagram below, the data source is stored in the relational database `MariaDB`. An adapter writes this data into the `leader redis`, which is then distributed through redis's `leader-follower synchronization` mechanism. The storage structure design in redis for configurations does not need to consider the logical relationships between compile, group, and item. Maat will construct these logical relationships internally after loading the configurations from redis.
As shown in the diagram below, the data source is stored in the relational database `MariaDB`. An adapter writes this data into the `leader redis`, which is then distributed through redis's `leader-follower synchronization` mechanism. The storage structure design in redis for configurations does not need to consider the logical relationships between rule, group, and item. Maat will construct these logical relationships internally after loading the configurations from redis.
<img src="./imgs/sync-with-redis.png" width="600" height="300" >
@@ -23,7 +23,7 @@ As shown in the diagram below, the data source is stored in the relational datab
| MAAT_UPDATE_STATUS | Configuration status | sorted set, where the member is a configuration rule and the score is the version number | MAAT reads this using ZRANGEBYSCORE command. |
| MAAT_RULE_TIMER | Main configuration timeout information | Sorted Set, where the member is a configuration rule and the score is the timeout | MAAT configuration update thread periodically checks timeout status and sets timeout state. |
| MAAT_VERSION_TIMER | Version creation time | Sorted Set | Stores the creation time of each version, where the score is the version creation time and the member is the version, used to keep MAAT_UPDATE_STATUS within a smaller scale. |
| MAAT_LABEL_INDEX | Label index | Sorted Set, where the element is the configuration table name, compile_id, and the score is the label_id | |
| MAAT_LABEL_INDEX | Label index | Sorted Set, where the element is the configuration table name, rule_id, and the score is the label_id | |
| EFFECTIVE_RULE:TableName,ID OBSOLETE_RULE:TableName,ID | Main configuration | string | Active configurations, loaded one by one by MAAT. |
| SEQUENCE_REGION | Region ID generation sequence | INTEGER | Used by producers to generate unique region_id(item_id) |
| SEQUENCE_GROUP | Group ID generation sequence | INTEGER | Used by producers to generate unique group_id |
@@ -161,6 +161,6 @@ To manually add an existing table configuration, the following files need to be
* Append a new line to the table file that needs modification. The region_id should not conflict with any existing domain tables, and update the first line of the file with the new number of rows.
* Add the summary information of the new configuration to the configuration summary table. Ensure it matches the compile_id in the table file and does not conflict with any existing compile_id. Update the first line of the file with the new number of rows.
* Add the summary information of the new configuration to the configuration summary table. Ensure it matches the rule_id in the table file and does not conflict with any existing rule_id. Update the first line of the file with the new number of rows.
* Modify the row counts for the configuration summary table and domain tables in the table index file.

30
docs/getting_started.md
View File

@@ -31,7 +31,7 @@ A complete use case consists of three parts:
### Case1
In the scanning scenario, it is necessary to configure the schema of multiple tables, including the item table, group2compile table, and compile table. If there is also group nesting involved, the schema of the group2group table needs to be configured.
In the scanning scenario, it is necessary to configure the schema of multiple tables, including the item table, group2rule table, and rule table. If there is also group nesting involved, the schema of the group2group table needs to be configured.
**(1) table schema**
@@ -41,25 +41,25 @@ Table schema is stored in a json file(such as table_info.conf), which is loaded
[
{
"table_id":0,
"table_name":"COMPILE",
"table_type":"compile",
"default_compile_table":0, /* key:indicate this is the default compile table, value:can be anything(not care) */
"table_name":"RULE",
"table_type":"rule",
"default_rule_table":0, /* key:indicate this is the default rule table, value:can be anything(not care) */
"valid_column":8,
"custom": {
"compile_id":1,
"rule_id":1,
"tags":6,
"clause_num":9
}
},
{
"table_id":1,
"table_name":"GROUP2COMPILE",
"table_type":"group2compile",
"associated_compile_table_id":0, /* associate compile table_id, group2compile table shares the same runtime with the corresponding compile table, so it needs to be mapped to the corresponding compile table. */
"table_name":"GROUP2RULE",
"table_type":"group2rule",
"associated_rule_table_id":0, /* associate rule table_id, group2rule table shares the same runtime with the corresponding rule table, so it needs to be mapped to the corresponding rule table. */
"valid_column":3,
"custom": {
"group_id":1,
"compile_id":2,
"rule_id":2,
"not_flag":4,
"virtual_table_name":5,
"clause_index":6
@@ -98,12 +98,12 @@ Table schema is stored in a json file(such as table_info.conf), which is loaded
Configurations are stored in a json file(such as maat_json.json), which is loaded when maat instance is created.
```json
{
"compile_table": "COMPILE",
"group2compile_table": "GROUP2COMPILE",
"rule_table": "RULE",
"group2rule_table": "GROUP2RULE",
"group2group_table": "GROUP2GROUP",
"rules": [
{
"compile_id": 123,
"rule_id": 123,
"service": 1,
"action": 1,
"do_blacklist": 1,
@@ -172,7 +172,7 @@ int main()
/**
* Becase maat instance has loaded rule in table_info.conf which keywords is "Hello Maat",
so maat_scan_string should return hit flag and rule's compile_id stored in results array.
so maat_scan_string should return hit flag and rule's rule_id stored in results array.
*/
int ret = maat_scan_string(maat_instance, table_id, scan_data, strlen(scan_data),
results, ARRAY_SIZE, &n_hit_result, state);
@@ -215,8 +215,8 @@ In the callback scenario, only the schema of the corresponding table needs to be
```json
{
"compile_table": "COMPILE",
"group2compile_table": "GROUP2COMPILE",
"rule_table": "RULE",
"group2rule_table": "GROUP2RULE",
"group2group_table": "GROUP2GROUP",
"plugin_table": {
"table_name": "TEST_IP_PLUGIN_WITH_EXDATA",

34
docs/logical_combinations.md
View File

@@ -33,7 +33,7 @@ Before showing how to configure the specific rules, we need some raw materials (
## AND logic
`Note`: Each compile supports a maximum of 8 clauses, which are connected by the logical 'AND' relationship, with clause index ranging from 0 to 7.
`Note`: Each rule supports a maximum of 8 clauses, which are connected by the logical 'AND' relationship, with clause index ranging from 0 to 7.
* case1: Deny hosts with source IP addresses in the range of 192.168.1.11 to 192.168.1.20 from accessing the website www.baidu.com.
@@ -41,7 +41,7 @@ Before showing how to configure the specific rules, we need some raw materials (
* www.baidu.com => group_id(201)
```bash
compile(rule) = clause1 & clause2
rule(rule) = clause1 & clause2
= {vtable1, group1, clause_index1} & {vtable2, group2, clause_index2}
= {2, 211, 1} & {1, 201, 2}
```
@@ -52,14 +52,14 @@ compile(rule) = clause1 & clause2
* 端口80 ~ 80 => group_id(221)
```bash
compile(rule) = clause1 & clause2
rule(rule) = clause1 & clause2
= {vtable1, group1, clause_index1} & {vtable2, group2, clause_index2}
= {2, 211, 1} & {3, 221, 2}
```
The JSON configuration for the logical `AND` can be referenced at [unit_test Json configuration](../test/maat_json.json) with compile_id=152.
The JSON configuration for the logical `AND` can be referenced at [unit_test Json configuration](../test/maat_json.json) with rule_id=152.
group_name: "152_mail_addr" and group_name: "interval_group_refered" are two clauses of this compile, with a logical `AND` relationship between them.
group_name: "152_mail_addr" and group_name: "interval_group_refered" are two clauses of this rule, with a logical `AND` relationship between them.
## OR logic
@@ -73,7 +73,7 @@ group_name: "152_mail_addr" and group_name: "interval_group_refered" are two cla
* www.baidu.com => group_id(201)
```bash
compile(rule) = clause1 & clause2
rule(rule) = clause1 & clause2
= {vtable1, (group1 | group2), clause_index1} & {vtable2, group3, clause_index2}
= {2, (211 | 212), 1} & {1, 201, 2}
```
@@ -85,12 +85,12 @@ compile(rule) = clause1 & clause2
* port 443 ~ 443 => group_id(222)
```bash
compile(rule) = clause1 & clause2
rule(rule) = clause1 & clause2
= {vtable1, group1, clause_index1} & {vtable2, (group2 | group3), clause_index2}
= {2, 211, 1} & {3, (221 | 222), 2}
```
The JSON configuration for the logical `OR` can be referenced at [unit_test Json configuration](../test/maat_json.json) with compile_id=152.
The JSON configuration for the logical `OR` can be referenced at [unit_test Json configuration](../test/maat_json.json) with rule_id=152.
group_name: "152_mail_addr" contains two regions(items) with a logical `OR` relationship between them.
@@ -105,7 +105,7 @@ group_name: "152_mail_addr" contains two regions(items) with a logical `OR` rela
* www.baidu.com => group_id(201)
```bash
compile(rule) = clause1 & !clause2
rule(rule) = clause1 & !clause2
= {vtable1, group1, clause_index1} & !{vtable2, group2, clause_index2}
= {2, 211, 1} & !{1, 201, 2}
```
@@ -117,14 +117,14 @@ compile(rule) = clause1 & !clause2
* port 443 ~ 443 => group_id(222)
```bash
compile(rule) = clause1 & !clause2
rule(rule) = clause1 & !clause2
= {vtable1, group1, clause_index1} & !{vtable2, (group2 | group3), clause_index2}
= {2, 211, 1} & !{3, (221 | 222), 2}
```
The JSON configuration for the logical `OR` can be referenced at [unit_test Json configuration](../test/maat_json.json) with compile_id=145.
The JSON configuration for the logical `OR` can be referenced at [unit_test Json configuration](../test/maat_json.json) with rule_id=145.
The group_name: "123_IP_group" is a NOT clause of this compile.
The group_name: "123_IP_group" is a NOT clause of this rule.
## Group exclude
@@ -145,9 +145,9 @@ super_group1 = group1 exclude group2
= 210 exclude 211
```
And then configure the compile.
And then configure the rule.
```bash
compile(rule) = clause1 & clause2
rule(rule) = clause1 & clause2
= {vtable1, super_group1, clause_index1} & {vtable2, group2, clause_index2}
= {2, (210 exclude 211), 1} & {1, 201, 2}
```
@@ -164,13 +164,13 @@ super_group2 = group2 exclude group3
= 202 exclude 201
```
And then configure the compile.
And then configure the rule.
```bash
compile(rule) = clause1 & clause2
rule(rule) = clause1 & clause2
= {vtable1, group1, clause_index1} & {vtable2, super_group2, clause_index2}
= {2, 211, 1} & {1, (202 exclude 201), 2}
```
The JSON configuration for the logical `OR` can be referenced at [unit_test Json configuration](../test/maat_json.json) with compile_id=200.
The JSON configuration for the logical `OR` can be referenced at [unit_test Json configuration](../test/maat_json.json) with rule_id=200.
The group_name: "ExcludeLogicGroup200" is a super group that includes an include group "ExcludeLogicGroup200_1" and an exclude group "ExcludeLogicGroup200_2".

74
docs/maat_table.md
View File

@@ -11,8 +11,8 @@ Maat tables are divided into two categories: physical tables that actually exist
The types of physical tables are as follows:
- [item table](#11-item-table)
- [compile table](#12-compile-table)
- [group2compile table](#13-group2compile-table)
- [rule table](#12-rule-table)
- [group2rule table](#13-group2rule-table)
- [group2group table](#14-group2group-table)
- [plugin table](#15-plugin-table)
- [ip_plugin table](#16-ip_plugin-table)
@@ -43,7 +43,7 @@ Each item table must has the following columns:
- is_valid: In incremental updates, 1(valid means add) 0(invalid means del)
The range of item_id(group_id, compile_id) is 0～2^63，which is 8 bytes.
The range of item_id(group_id, rule_id) is 0～2^63，which is 8 bytes.
#### 1.1.1 <a name='exprtable'></a> expr item table
@@ -52,7 +52,7 @@ Describe matching rules for strings.
| **FieldName** | **type** | **constraint** |
| ---------------- | -------------- | ------- |
| **item_id** | LONG LONG | primary key |
| **group_id** | LONG LONG | leaf group id, can be referenced by group2group & group2compile table |
| **group_id** | LONG LONG | leaf group id, can be referenced by group2group & group2rule table |
| **keywords** | VARCHAR2(1024) | field to match during scanning |
| **expr_type** | INT | 0(keywords), 1(AND expr), 2(regular expr), 3(substring with offset)
| **match_method** | INT | only useful when expr_type is 0. 0(sub), 1(suffix), 2(prefix), 3(exactly) |
@@ -158,7 +158,7 @@ Describe extended matching rules for strings by adding the district column.
| **FieldName** | **type** | **constraint** |
| ---------------- | -------------- | ------- |
| **item_id** | LONG LONG | primary key |
| **group_id** | LONG LONG | leaf group id, can be referenced by group2group & group2compile table |
| **group_id** | LONG LONG | leaf group id, can be referenced by group2group & group2rule table |
| **district** | VARCHAR2(1024) | describe the effective position of the keywords |
| **keywords** | VARCHAR2(1024) | field to match during scanning |
| **expr_type** | INT | 0(keywords), 1(AND expr), 2(regular expr), 3(substring with offset)
@@ -183,7 +183,7 @@ Describe matching rules for IP address. Both the address and port are represente
| **FieldName** | **type** | **constraint** |
| -------------- | ------------ | -------------- |
| **item_id** | LONG LONG | primary key |
| **group_id** | LONG LONG | leaf group id, can be referenced by group2group & group2compile table |
| **group_id** | LONG LONG | leaf group id, can be referenced by group2group & group2rule table |
| **addr_type** | INT | Ipv4 = 4 Ipv6 = 6 |
| **addr_format**| VARCHAR2(40) | ip addr format, single/range/CIDR/mask |
| **ip1** | VARCHAR2(40) | start ip |
@@ -197,7 +197,7 @@ Determine whether an integer is within a certain numerical range.
| **FieldName** | **type** | **constraint** |
| ---------------- | -------- | -------------- |
| **item_id** | INT | primary key |
| **group_id** | INT | leaf group id, can be referenced by group2group & group2compile table |
| **group_id** | INT | leaf group id, can be referenced by group2group & group2rule table |
| **low_boundary** | INT | lower bound of the numerical range(including lb), 0 ~ (2^32 - 1)|
| **up_boundary** | INT | upper bound of the numerical range(including ub), 0 ~ (2^32 - 1)|
| **is_valid** | INT | 0(invalid), 1(valid) |
@@ -209,7 +209,7 @@ Describe extended matching rules for integer by adding the district column.
| **FieldName** | **type** | **constraint** |
| ---------------- | -------- | -------------- |
| **item_id** | INT | primary key |
| **group_id** | INT | leaf group id, can be referenced by group2group & group2compile table |
| **group_id** | INT | leaf group id, can be referenced by group2group & group2rule table |
| **district** | VARCHAR2(1024)| describe the effective position of the keywords |
| **low_boundary** | INT | lower bound of the numerical range(including lb), 0 ~ (2^32 - 1)|
| **up_boundary** | INT | upper bound of the numerical range(including ub), 0 ~ (2^32 - 1)|
@@ -220,7 +220,7 @@ Describe extended matching rules for integer by adding the district column.
| **FieldName** | **type** | **constraint** |
| ------------- | -------- | -------------- |
| **item_id** | INT | primary key |
| **group_id** | INT | leaf group id, can be referenced by group2group & group2compile table |
| **group_id** | INT | leaf group id, can be referenced by group2group & group2rule table |
| **flag** | INT | flag, 0 ~ (2^32 - 1)|
| **flag_mask** | INT | flag_mask, 0 ~ (2^32 - 1)|
| **is_valid** | INT | 0(invalid), 1(valid) |
@@ -230,31 +230,31 @@ Describe extended matching rules for integer by adding the district column.
| **FieldName** | **type** | **constraint** |
| ------------- | -------- | -------------- |
| **item_id** | INT | primary key |
| **group_id** | INT | leaf group id, can be referenced by group2group & group2compile table |
| **group_id** | INT | leaf group id, can be referenced by group2group & group2rule table |
| **district** | INT | describe the effective position of the flag |
| **flag** | INT | flag, 0 ~ (2^32 - 1)|
| **flag_mask** | INT | flag_mask, 0 ~ (2^32 - 1)|
| **is_valid** | INT | 0(invalid), 1(valid) |
### 1.2 <a name='CompileTable'></a> compile table
### 1.2 <a name='RuleTable'></a> rule table
Describe the specific policy, one maat instance can has multiple compile tables with different names.
Describe the specific policy, one maat instance can has multiple rule tables with different names.
| **FieldName** | **type** | **constraint** |
| -------------- | -------------- | --------------- |
| **compile_id** | LONG LONG | primary key, compile id |
| **rule_id** | LONG LONG | primary key, rule id |
| **tags** | VARCHAR2(1024) | default 0，means no tag |
| **is_valid** | INT | 0(invalid)，1(valid) |
| **clause_num** | INT | no more than 8 clauses |
### 1.3 <a name='Group2CompileTable'></a> group2compile table
### 1.3 <a name='Group2RuleTable'></a> group2rule table
Describe the relationship between group and compile.
Describe the relationship between group and rule.
| **FieldName** | **type** | **constraint** |
| ----------------- | ------------- | -------------- |
| **group_ids** | VARCHAR(256) | group ids are separated by commas(g1,g2,g3) |
| **compile_id** | LONG LONG | compile id |
| **rule_id** | LONG LONG | rule id |
| **is_valid** | INT | 0(invalid), 1(valid) |
| **not_flag** | INT | logical 'NOT', identify a NOT clause, 0(no) 1(yes) |
| **virtual_table** | VARCHAR2(256) | virtual table name, NOT NULL |
@@ -402,7 +402,7 @@ Configuration tags are tags stored on compilation configurations or group config
Maat loads the configuration of different types of tables into memory to form the corresponding runtime for each table. We can see all table types from the table schema, and the runtime for the item table is similar, as it is an abstraction of the scanning engine. When we provide the data to be scanned and call the corresponding scanning interface, we can return whether the item is hit or not, and if it is hit, we can return the corresponding item’s group_id.
From the [configuration relationship](./overview.md#12-configuration-relationship) diagram, we can see how the hit group is referenced by other groups or compiles. If a hit group is referenced by other groups or compiles, there will be one or more hit paths that follow the `item_id -> group_id` {-> super group_id} `-> compile_id`. This requires two special runtimes: group2group_runtime and compile_runtime.
From the [configuration relationship](./overview.md#12-configuration-relationship) diagram, we can see how the hit group is referenced by other groups or rules. If a hit group is referenced by other groups or rules, there will be one or more hit paths that follow the `item_id -> group_id` {-> super group_id} `-> rule_id`. This requires two special runtimes: group2group_runtime and rule_runtime.
Based on this, we can divide the runtime into the following three categories:
@@ -412,9 +412,9 @@ Based on this, we can divide the runtime into the following three categories:
* flag_runtime
* interval_runtime
2. group & compile table runtime
2. group & rule table runtime
* group2group_runtime
* compile_runtime
* rule_runtime
3. xx_plugin table runtime
* plugin_runtime
@@ -431,47 +431,47 @@ Among the four types of runtimes mentioned above, `expr_runtime` is relatively u
**Note**: Due to the inability to unify the native rulescan usage with hyperscan, a partial refactoring has been done on rulescan. The refactored rulescan follows the same interface and usage as hyperscan, making it compatible with the design of the expr_matcher abstraction layer.
### 2.2 group & compile table runtime
### 2.2 group & rule table runtime
#### 2.2.1 group2group runtime
The `group2group_runtime` is a runtime that is built based on the reference relationships between groups, which are stored in the [group2group table](#14-group2group-table). From the [group hierarchy](./group_hierarchy.md), we can understand that if a hit occurs in a leaf group that is referenced by other groups, there may be certain super groups that are also hit. This is exactly the functionality provided by this runtime.
#### 2.2.2 compile runtime
#### 2.2.2 rule runtime
In addition to the compile table, there is also the group2compile table in the table schema. However, from a runtime perspective, the configurations of these two tables together constitute compile_runtime. This means that there is no standalone group2compile_runtime. Compile_runtime is the most complex among all runtime types because it serves multiple functions.
In addition to the rule table, there is also the group2rule table in the table schema. However, from a runtime perspective, the configurations of these two tables together constitute rule_runtime. This means that there is no standalone group2rule_runtime. Rule_runtime is the most complex among all runtime types because it serves multiple functions.
**Note:** This will involve the terminology of [clause](./terminology.md#clause).
1. For expressions without NOT-clauses, returning the matched compile_id:
1. For expressions without NOT-clauses, returning the matched rule_id:
* compile1 = clause1 & clause2 = {vtable1, g1} & {vtable2, g2}
* rule1 = clause1 & clause2 = {vtable1, g1} & {vtable2, g2}
* compile2 = clause1 & clause2 = {vtable1, g2} & {vtable2, g3}
* rule2 = clause1 & clause2 = {vtable1, g2} & {vtable2, g3}
Given the matched vtable_id and group_id, all matching compile_ids can be provided. For example, if scanning vtable1 matches g2 and vtable2 matches g3, compile_runtime will return the matched compile_id 2.
Given the matched vtable_id and group_id, all matching rule_ids can be provided. For example, if scanning vtable1 matches g2 and vtable2 matches g3, rule_runtime will return the matched rule_id 2.
2. For expressions with NOT-clauses, returning the matched compile_id:
2. For expressions with NOT-clauses, returning the matched rule_id:
* compile3 = clause1 & !clause2 = {vtable1, g1} & !{vtable2, g2}
* rule3 = clause1 & !clause2 = {vtable1, g1} & !{vtable2, g2}
* compile4 = !clause1 & clause2 = !{vtable1, g2} & {vtable2, g3}
* rule4 = !clause1 & clause2 = !{vtable1, g2} & {vtable2, g3}
If scanning vtable1 matches g1 and vtable2 matches g3, compile_runtime will return the matched compile_id 4.
If scanning vtable1 matches g1 and vtable2 matches g3, rule_runtime will return the matched rule_id 4.
3. If a compile_id is matched, the full hit path can be obtained: **item_id -> group_id ->** {super_group_id} -> clause{**vtable_id, not_flag, clause_index} -> compile_id**. If the matched group is not referenced by a compile, a half hit path can be obtained: **item_id -> group_id** -> {super_group_id}.
3. If a rule_id is matched, the full hit path can be obtained: **item_id -> group_id ->** {super_group_id} -> clause{**vtable_id, not_flag, clause_index} -> rule_id**. If the matched group is not referenced by a rule, a half hit path can be obtained: **item_id -> group_id** -> {super_group_id}.
4. Getting the matched group_ids and the count of hit groups.
The internal structure of compile_runtime is as follows, including the control plane for configuration loading and the data plane for external calls.
The internal structure of rule_runtime is as follows, including the control plane for configuration loading and the data plane for external calls.
![compile runtime](./imgs/compile-runtime.png)
![rule runtime](./imgs/rule-runtime.png)
* **Control plane**
Compile runtime loads the compile table and group2compile table configurations into memory, assigning a unique clause_id to all clauses of each compile. The following three parts are constructed based on the clause_id:
Rule runtime loads the rule table and group2rule table configurations into memory, assigning a unique clause_id to all clauses of each rule. The following three parts are constructed based on the clause_id:
1. All clause_ids under the same compile are used to construct AND expressions, and all compile AND expressions are used to build a bool_matcher.
1. All clause_ids under the same rule are used to construct AND expressions, and all rule AND expressions are used to build a bool_matcher.
2. For not_flag=0 (clauses), a `clause_id hash` is built, key:{group_id, vtable_id, not_flag}, value:clause_id.
@@ -491,12 +491,12 @@ On the data plane, services are provided externally through the maat API, primar
* Identify the group_ids in vtable_id table that appear in the NOT_clause and add them to the `NOT_clause_group` set. Ensure that this set does not contain any group_id that was hit during scanning. If any such group_id is present, remove it from the set to form the final `NOT_clause_group` for the vtable_id table.
* Use the hit clause_ids to determine if there are any hit compile_ids. If there are, populate the half-hit path which will become full-hit path.
* Use the hit clause_ids to determine if there are any hit rule_ids. If there are, populate the half-hit path which will become full-hit path.
2. **maat_scan_not_logic**: This interface is used to activate NOT-clause logic.
* Traverse the `NOT_clause_group` of `vtable_id`. For each `group_id`, form a key `{group_id, vtable_id, 1}` to obtain the `NOT_clause_id`. If it is in the `exclude clause_ids` set, ignore it; otherwise, add it to the `all hit clause_ids` set as a hit `NOT_clause_id`, and record the half-hit path of the NOT-clause.
* Use the `all hit clause_ids` to calculate if there are any newly hit compile_ids. If there are, populate the half-hit path of the NOT-clause which will become full-hit path.
* Use the `all hit clause_ids` to calculate if there are any newly hit rule_ids. If there are, populate the half-hit path of the NOT-clause which will become full-hit path.
3. **xx_get_hit_path**: This interface is used to retrieve the hit path.

4
docs/monitor_tools.md
View File

@@ -28,13 +28,13 @@ The statistical information in the above figure is divided into two parts, part1
- garbage_queue_len: number of elements in the garbage queue
- hit_compile_num: total number of hits on compile
- hit_rule_num: total number of hits on rule
- state_num: total number of current maat_state
- per_state_num: number of bytes occupied by each maat_state on average
- compile_state_num: total number of current compile_state created within maat_state during partial hits or hits
- rule_state_num: total number of current rule_state created within maat_state during partial hits or hits
- stream_num: total number of scanned streams

12
docs/overview.md
View File

@@ -7,7 +7,7 @@ Before proceeding, please make sure you are familiar with the [terminology](./te
As mentioned in the readme, maat has two typical usage patterns:
**Pattern 1**
* Update rules in the item table, group2compile table, and compile table
* Update rules in the item table, group2rule table, and rule table
* Call the maat scanning api to determine if the actual traffic hits the effective rules
* If a rule is hit, maat can provide detailed information about the hit rule
@@ -20,15 +20,15 @@ As mentioned in the readme, maat has two typical usage patterns:
Different types of configurations are stored in different tables. For all configuration types, please refer to the [table schema](./maat_table.md#1-table-schema).
The physical tables are mainly divided into three categories: the item table, group compile relationship table (compile table, group2compile table, group2group table), and xx_plugin table. The first two types of tables are used for maat traffic scanning, while the xx_plugin table is used as a callback table, which can obtain the detailed configuration information for a specific key.
The physical tables are mainly divided into three categories: the item table, group rule relationship table (rule table, group2rule table, group2group table), and xx_plugin table. The first two types of tables are used for maat traffic scanning, while the xx_plugin table is used as a callback table, which can obtain the detailed configuration information for a specific key.
### 1.2 Configuration relationship
As shown in the diagram below, maat organizes and abstracts configurations using terms such as item, group, literal, clause, compile, etc., allowing users to flexibly configure various policies. The term "literal" is an internal concept in maat and is not visible to external users.
As shown in the diagram below, maat organizes and abstracts configurations using terms such as item, group, literal, clause, rule, etc., allowing users to flexibly configure various policies. The term "literal" is an internal concept in maat and is not visible to external users.
In addition, groups support nesting. For more detailed information, please refer to [group hierarchy](./group_hierarchy.md).
If we define literal_id = {virtual_table_id, group_id}, then a literal is composed of one or more literal_ids. The multiple literal_ids that form the same clause have a logical “OR” relationship. The multiple clauses that form the same compile have a logical “AND” relationship, and there can be a maximum of 8 clauses within the same compile. In addition, the clause itself supports logical "NOT".
If we define literal_id = {virtual_table_id, group_id}, then a literal is composed of one or more literal_ids. The multiple literal_ids that form the same clause have a logical “OR” relationship. The multiple clauses that form the same rule have a logical “AND” relationship, and there can be a maximum of 8 clauses within the same rule. In addition, the clause itself supports logical "NOT".
<img src="./imgs/rule_diagram.png" width="800" height="450" >
@@ -56,7 +56,7 @@ The diagram illustrates the overall architecture of maat, including the control
* **Data Plane**
When calling the maat scanning interface, it subsequently calls the table runtime of the corresponding table, then proceeds to the scanning engine. Upon the scanning engine returning a hit group, it further searches for the matching `compile_id` through group2group runtime, group2compile runtime, and compile runtime, which is then returned to the caller. In addition, if the caller is interested in the hit path, they can also retrieve it through the interfaces provided by maat.
When calling the maat scanning interface, it subsequently calls the table runtime of the corresponding table, then proceeds to the scanning engine. Upon the scanning engine returning a hit group, it further searches for the matching `rule_id` through group2group runtime, group2rule runtime, and rule runtime, which is then returned to the caller. In addition, if the caller is interested in the hit path, they can also retrieve it through the interfaces provided by maat.
The scanning mentioned above all uses the `effective runtime`. If there are configuration changes, it will trigger the construction of `updating runtime`. Once this construction is completed, it will become effective runtime, and the original effective runtime will be put into the garbage collection queue waiting to be recycled.
@@ -117,7 +117,7 @@ All of the benchmarks are run on the same TSG-X device. Here are the details of
```bash
scan consume: The time consumption of a single scan, calculated as the average of 1,000,000 scans for each of the 5 threads.
In scenarios with different numbers of rules, each rule hits only one item → one compile.
In scenarios with different numbers of rules, each rule hits only one item → one rule.
Configuration capacity:

18
docs/terminology.md
View File

@@ -2,7 +2,7 @@
* [Item](#item)
* [Group(Object)](#groupobject)
* [Compile(Policy)](#compilepolicy)
* [Rule(Policy)](#rulepolicy)
* [Clause(Condition)](#clause)
* [Literal](#literal)
* [Physical table](#physical-table)
@@ -42,17 +42,17 @@ A group defines a set that can contain different types of items and can also ref
- Group supports multi-level nesting, see [group hierarchy](./overview.md#groupobject-nesting-and-hierarchies)
- A Group can be referenced by different compiles.
- A Group can be referenced by different rules.
The relationship between group and group is stored in the [group2group table](./maat_table.md#14-group2group-table).
## Compile(Policy)
## Rule(Policy)
A conjunctive normal form(CNF) consisting of multiple groups and virtual tables.
`Note`: A compile can contain up to 8 clauses and multiple clauses in the same compile can be logical 'AND' and logical 'NOT' relationships.
`Note`: A rule can contain up to 8 clauses and multiple clauses in the same rule can be logical 'AND' and logical 'NOT' relationships.
The relationship between group and compile is stored in the [group2compile table](./maat_table.md#13-group2compile-table).
The relationship between group and rule is stored in the [group2rule table](./maat_table.md#13-group2rule-table).
<img src="./imgs/CNF.jpg" alt="exclude" style="zoom:80%" />
@@ -68,7 +68,7 @@ A Literal consists of `vtable_id(virtual table id)` and `group_id`. During the r
## Physical table
Different rules are stored in different tables in the actual database, including [item table](./maat_table.md#11-item-table), [compile table](./maat_table.md#12-compile-table), [group2compile table](./maat_table.md#13-group2compile-table), [group2group table](./maat_table.md#14-group2group-table), and [xx_plugin table](./maat_table.md#15-plugin-table), and so on.
Different rules are stored in different tables in the actual database, including [item table](./maat_table.md#11-item-table), [rule table](./maat_table.md#12-rule-table), [group2rule table](./maat_table.md#13-group2rule-table), [group2group table](./maat_table.md#14-group2group-table), and [xx_plugin table](./maat_table.md#15-plugin-table), and so on.
## Virtual table
@@ -86,7 +86,7 @@ Defines the type of table and the configuration format, determining the specific
## Table runtime
The runtime generated by loading the configuration in the table into memory. Different tables have different runtimes. The group2compile table is merged with the corresponding compile table to generate a compile runtime, meaning there is no separate group2compile runtime.
The runtime generated by loading the configuration in the table into memory. Different tables have different runtimes. The group2rule table is merged with the corresponding rule table to generate a rule runtime, meaning there is no separate group2rule runtime.
Different scanning api use runtimes of different tables. For example, the HTTP_URL table is of type expr, and its corresponding scanning interface is maat_scan_string. Therefore, when calling this scanning interface, the API internally uses the runtime of the HTTP_URL table to perform the actual scanning task.
@@ -112,11 +112,11 @@ Maat supports not only block-based scanning but also stream-based scanning. For
## Half/Full hit
From the diagram of [configuration relationship](./overview.md#12-configuration-relationship), it can be seen that if the group that is hit is not referenced by compile, or even if referenced by compile, but after logical operations no compile is hit, then this hit is called a half hit; if a compile is hit, then this hit is called a full hit.
From the diagram of [configuration relationship](./overview.md#12-configuration-relationship), it can be seen that if the group that is hit is not referenced by rule, or even if referenced by rule, but after logical operations no rule is hit, then this hit is called a half hit; if a rule is hit, then this hit is called a full hit.
## Hit path
From the relationship of item, group, and compile mentioned above, if a scan hits a certain compile, there must be a logical path composed of item_id -> group_id -> compile_id. Maat names this path the hit path. If a group has nested references, the hit path should be item_id -> sub_group_id -> group_id -> compile_id.
From the relationship of item, group, and rule mentioned above, if a scan hits a certain rule, there must be a logical path composed of item_id -> group_id -> rule_id. Maat names this path the hit path. If a group has nested references, the hit path should be item_id -> sub_group_id -> group_id -> rule_id.
## Redis