gfwleak/tango-maat
Archived
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

[Doc] maatframe markdown documents

Browse Source
This commit is contained in:
liuwentan
2024-03-29 08:37:40 +00:00
parent 3e1acddf61
commit 93da4afe02
59 changed files with 3039 additions and 2412 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1151
docs/api_reference.md Normal file
View File

File diff suppressed because it is too large Load Diff

166
docs/configuration_management.md Normal file
View File

@@ -0,0 +1,166 @@
# Configuration management
Input must use UTF-8 without BOM encoding, for example, MySQL use utf8mb4. It supports dynamic loading and multi-machine synchronization of configurations.
Maat supports three configuration loading modes:
- [Redis mode](#1-redis-mode)
- [Json mode](#2-json-mode)
- [Iris mode](#3-iris-mode)
## 1.<a name='Redis mode'></a> Redis mode
As shown in the diagram below, the data source is stored in the relational database `MariaDB`. An adapter writes this data into the `leader redis`, which is then distributed through redis's `leader-follower synchronization` mechanism. The storage structure design in redis for configurations does not need to consider the logical relationships between compile, group, and item. Maat will construct these logical relationships internally after loading the configurations from redis.
<img src="./imgs/sync-with-redis.png" width="600" height="300" >
### 1.1 Transactional Write
| Redis KEY | Name | Structure | Purpose |
| ------------------ | ---------------- | ----------- | ---------------- |
| MAAT_VERSION | primary version | INTEGER | Identifies the version number of configurations in redis. When the version number in redis is greater than the version number configured in MAAT, it reads MAAT_UPDATE_STATUS. |
| MAAT_PRE_VERSION | pre-version | INTEGER | |
| MAAT_TRANSACTION_xx| Transaction configuration status | LIST | Temporarily stores the configuration status in the transaction. xx is one of the states in MAAT_PRE_VERSION, which will be updated to MAAT_UPDATE_STATUS after the transaction ends and will be deleted itself. |
| MAAT_UPDATE_STATUS | Configuration status | sorted set, where the member is a configuration rule and the score is the version number | MAAT reads this using ZRANGEBYSCORE command. |
| MAAT_RULE_TIMER | Main configuration timeout information | Sorted Set, where the member is a configuration rule and the score is the timeout | MAAT configuration update thread periodically checks timeout status and sets timeout state. |
| MAAT_VERSION_TIMER | Version creation time | Sorted Set | Stores the creation time of each version, where the score is the version creation time and the member is the version, used to keep MAAT_UPDATE_STATUS within a smaller scale. |
| MAAT_LABEL_INDEX | Label index | Sorted Set, where the element is the configuration table name, compile_id, and the score is the label_id | |
| EFFECTIVE_RULE:TableName,ID OBSOLETE_RULE:TableName,ID | Main configuration | string | Active configurations, loaded one by one by MAAT. |
| SEQUENCE_REGION | Region ID generation sequence | INTEGER | Used by producers to generate unique region_id(item_id) |
| SEQUENCE_GROUP | Group ID generation sequence | INTEGER | Used by producers to generate unique group_id |
| EXPIRE_OP_LOCK | Distributed lock | string "locked" | Ensures that at most one writer performs eviction. |
`Note`: The following api supports writing one line of configuration to redis.
```c
int maat_cmd_set_line(struct maat *maat_instance, const struct maat_cmd_line *line_rule);
```
### 1.2 Primary_version、Pre_version and Lua Script
When a producer wants to modify the configuration, it first increments the `preliminary version` and use it as the version for writing the configuration status. Once the configuration change is completed, it then increments the `primary version`. This method can greatly improve the writing performance and ensure successful writing except for ID conflicts.
When there are multiple producers, there may be a nconsistency between the configuration status version and the primary version. Let the primary version be v and the version declared in the configuration status during a particular update be u. Incremental updates by consumers can lead to the following scenarios:
- If v == u, the version numbers are consistent, and the configuration is loaded normally.
- The situation where v > u only occurs when there are three or more producers. This may lead to configuration loss, as illustrated in the table below.
- If v < u, it indicates that among the two producers, the one that started writing first has not yet completed. In this case, only update to version v for this time, and update to u will be carried out in the next polling.
|time | producer1 |producer2|producer3|consumer|
|-----|--------------------------------|---------|---------|-------|
|0 |ready to update mv=3924, tv=3925| | | |
|1 | |ready to update mv=3924, tv=3925| | |
|2 | | |ready to update mv=3924, tv=3927| |
|3 | | |ready to update mv=3925, tv=3927|get version 3925, zrangebyscore can't get status of version 3925|
|4 | |ready to update mv=3926, tv=3926| |maat version increased to 3926 with error noncontigous|
|5 |update completed | | |3925 is skipped|
When consumers perform a full update, they do not need to read the version of the configuration status; instead, they directly read all valid configurations. This is because the configuration write and the increment of the primary version number by 1 are executed within the same transaction. Therefore, the full configuration version read will always be consistent with the primary version.
At the end of the transaction, a Lua script is used to check the transaction version (transaction_version) against the primary version number (maat_version):
- If tv equals mv, no correction is needed.
- If tv is greater than mv, the incremental update for this transaction will be applied in the next iteration.
- If tv is less than mv, how to identify the rules for the configuration status written in this transaction and then can change the score to mv?
To address the issue of transaction version being less than maat_version at the end of the transaction, the Redis list MAAT_TRANSACTION_xx is used to store configuration update status, where xx is derived from MAAT_PRE_VERSION. At the end of the transaction, a Lua script is used to synchronize MAAT_UPDATE_STATUS and delete MAAT_TRANSACTION_xx.
<img src="./imgs/update-rule-in-redis.png" width="600" height="650" >
### 1.3 MAAT_UPDATE_STATUS
In this structure, a Sorted Set is used to store the change status of the main configuration, where the score represents the version number and the member represents the configuration status. The first two bytes of the member describe the update instruction:
1. ADD, which signifies configuration addition, has the structure ADD,TableName,ID.
2. DEL, which signifies configuration deletion, has the structure DEL,TableName,ID.
When MAAT detects a change in MAAT_VERSION, it uses ZRANGEBYSCORE to read the updated configuration status (in ascending order of VERSION) and checks the Score of the first configuration. If this Score is greater than Maat version + 1, it indicates missing updates (due to a prolonged network interruption), triggering the full update process.
For DEL status, if the corresponding main configuration status cannot be found, it also indicates missing updates (network interruption duration exceeding the OBSOLETE_RULE timeout), triggering the full update process.
### 1.4 MAAT_VERSION_TIMER
This structure uses a Sorted Set to store the creation time of each version, where the score is the version creation time and the member is the version number (version), corresponding to the score of MAAT_UPDATE_STATUS, to maintain MAAT_UPDATE_STATUS within a smaller scale.
### 1.5 Configuration Structure
There are two types of configuration naming conventions:
1. EFFECTIVE_RULE:TableName,ID indicates the currently effective configuration.
2. OBSOLETE_RULE:TableName,ID indicates configurations that have been deleted. These configurations will be removed from Redis after they expire.
### 1.6 Load From Redis
The working threads of Maat instances periodically poll MAAT_VERSION in redis. If the version in Redis is greater than the instance's MAAT_VERSION, an update is initiated.
<img src="./imgs/load-rule-from-redis.png" width="400" height="550" >
### 1.8 Read/Write Performance
To ensure transactions, Redis needs to operate in a single-node + leader-follower mode. Writing configurations with timeouts can achieve a rate of 5000 entries per second, while writing configurations without timeouts can achieve a rate of 10000 entries per second.
## 2. Json Mode
Maat supports loading encrypted and compressed json configurations, a feature that the other two modes do not have. The json mode also supports dynamic loading of configurations. After a Maat instance is created, the monitor_loop thread will periodically check if the json configuration file has changed. If a change is detected, it triggers a full update.
## 3. Iris Mode
In the iris mode, the configuration can only be loaded once when the maat instance is created and cannot be dynamically updated after that. Below is an introduction to the file format in this mode.
A configuration consists of an index file and several configuration data files. The index file is named to identify different versions.
### 3.1 Index File Naming
The full index file is named in the format full_config_index.SEQ (e.g., full_config_index.00000000000000055410), while the incremental index file is named in the format inc_config_index.SEQ. These files are stored in the designated full index directory and incremental index directory, respectively.
Where SEQ is a self-incrementing sequence maintained by the configuration line, ensuring:
1) When two index directories are not empty, newly generated index files have higher sequence numbers than subsequently generated index files (it is recommended to use the configuration version number in the database or write to local disk to ensure consistency).
2) The sequence number for generating full index files and incremental index files in the same deployment is the same.
3) The sequence number is a 20-digit number padded with zeros if it is less than 20 digits. Its value does not exceed the range of signed long long integers in C (i.e., 0 to 2^63-1).
### 3.2 Index File Format
As shown in the table below. Columns are separated by \t and rows are separated by \n. The configuration data file path is an absolute path.
|table_name |config num |table file path |
|--------------|-----------|----------------------------------------------|
|PASS_WHITE_IP |3 |/home/config/full/2024-02-23/PASS_WHITE_IP.123|
|PASS_URL |2 |/home/config/full/2024-02-23/PASS_URL.123 |
### 3.3 Configuration Data File Format
As shown below, columns are separated by \t and rows are separated by \n. The first line indicates the total number of configurations in the current data file, a value that should match the config_num in the corresponding index file.
```
3
45695 2 202.108.181.33 255.255.255.255 0 1
48697 2 202.108.181.33 255.255.255.255 0 1
66699 2 202.108.181.33 255.255.255.255 0 1
```
### 3.4 Manually Modifying IRIS Configuration
It is generally **not recommended** to directly modify IRIS configurations manually; debugging should be done using the JSON mode.
To add a new table to the configuration, the following files need to be modified:
* Add the new table path and entry count to the table file index.
* Add the table information to the table description file (table_info.conf).
* Modify the configuration summary table to apply the new configuration. Ensure that the total number of configurations in the first row matches the actual number of configuration lines.
To manually add an existing table configuration, the following files need to be modified:
* Append a new line to the table file that needs modification. The region_id should not conflict with any existing domain tables, and update the first line of the file with the new number of rows.
* Add the summary information of the new configuration to the configuration summary table. Ensure it matches the compile_id in the table file and does not conflict with any existing compile_id. Update the first line of the file with the new number of rows.
* Modify the row counts for the configuration summary table and domain tables in the table index file.

347
docs/getting_started.md Normal file
View File

@@ -0,0 +1,347 @@
# Getting started
## Build
```shell
$ mkdir build && cd build
$ cmake ..
$ make
$ make install
```
## Run test
```shell
$ cd build/test
$ ./maat_framework_gtest #Functional test set
$ ./maat_framework_perf_gtest #Performance test set
```
Maat is used as a dynamic library by applications and its API is defined in the header file(maat.h).
## Use case
A complete use case consists of three parts:
* `table schema`: Define how to parse configuration in different types of tables, that is, specify what each column in the table represents.
* `configuration`: Different types of configurations are stored in tables of the corresponding type.
* `maat_scan_xx/xx_plugin_get_ex_data`: In the scanning scenario, the caller uses the maat_scan_xx api to determine whether the traffic hits effective rules; In the callback scenario, the caller uses the xx_plugin_get_ex_data api to query the ex_data of the specified key.
### Case1
In the scanning scenario, it is necessary to configure the schema of multiple tables, including the item table, group2compile table, and compile table. If there is also group nesting involved, the schema of the group2group table needs to be configured.
**(1) table schema**
Table schema is stored in a json file(such as table_info.conf), which is loaded when maat instance is created.
```json
[
{
"table_id":0,
"table_name":"COMPILE",
"table_type":"compile",
"default_compile_table":0, /* key:indicate this is the default compile table, value:can be anything(not care) */
"valid_column":8,
"custom": {
"compile_id":1,
"tags":6,
"clause_num":9
}
},
{
"table_id":1,
"table_name":"GROUP2COMPILE",
"table_type":"group2compile",
"associated_compile_table_id":0, /* associate compile table_id, group2compile table shares the same runtime with the corresponding compile table, so it needs to be mapped to the corresponding compile table. */
"valid_column":3,
"custom": {
"group_id":1,
"compile_id":2,
"not_flag":4,
"virtual_table_name":5,
"clause_index":6
}
},
{
"table_id":2,
"table_name":"GROUP2GROUP",
"table_type":"group2group",
"valid_column":4,
"custom": {
"group_id":1,
"included_sub_group_ids":2,
"excluded_sub_group_ids":3
}
},
{
"table_id":3,
"table_name":"HTTP_URL",
"table_type":"expr",
"valid_column":7,
"custom": {
"item_id":1,
"group_id":2,
"keywords":3,
"expr_type":4,
"match_method":5,
"is_hexbin":6
}
}
]
```
**(2) configuration**
Configurations are stored in a json file(such as maat_json.json), which is loaded when maat instance is created.
```json
{
"compile_table": "COMPILE",
"group2compile_table": "GROUP2COMPILE",
"group2group_table": "GROUP2GROUP",
"rules": [
{
"compile_id": 123,
"service": 1,
"action": 1,
"do_blacklist": 1,
"do_log": 1,
"user_region": "anything",
"is_valid": "yes",
"groups": [
{
"group_name": "Untitled",
"regions": [
{
"table_name": "HTTP_URL",
"table_type": "expr",
"table_content": {
"keywords": "Hello Maat",
"expr_type": "none",
"match_method": "sub",
"format": "uncase plain"
}
}
]
}
]
}
]
}
```
**(3) scanning**
```C
#include <assert.h>
#include "maat.h"
#define ARRAY_SIZE 16
const char *json_filename = "./maat_json.json";
const char *table_info_path = "./table_info.conf";
int main()
{
/* initialize maat options which will be used by maat_new() */
struct maat_options *opts = maat_options_new();
maat_options_set_json_file(opts, json_filename);
maat_options_set_logger(opts, "./sample_test.log", LOG_LEVEL_INFO);
/* create maat instance, rules in table_info.conf will be loaded. */
struct maat *maat_instance = maat_new(opts, table_info_path);
assert(maat_instance != NULL);
maat_options_free(opts);
const char *table_name = "HTTP_URL"; /* maat_json.json has HTTP_URL rule */
int table_id = maat_get_table_id(maat_instance, table_name);
assert(table_id == 3); /* defined in table_info.conf */
int thread_id = 0;
long long results[ARRAY_SIZE] = {0};
size_t n_hit_result = 0;
/* store scanning intermediate state */
struct maat_state *state = maat_state_new(maat_instance, thread_id);
assert(state != NULL);
const char *scan_data = "Hello Maat, nice to meet you";
/**
* Becase maat instance has loaded rule in table_info.conf which keywords is "Hello Maat",
so maat_scan_string should return hit flag and rule's compile_id stored in results array.
*/
int ret = maat_scan_string(maat_instance, table_id, scan_data, strlen(scan_data),
results, ARRAY_SIZE, &n_hit_result, state);
assert(ret == MAAT_SCAN_HIT);
assert(n_hit_result == 1);
assert(results[0] == 123);
maat_state_free(state);
return 0;
}
```
### Case2
In the callback scenario, only the schema of the corresponding table needs to be configured.
**(1) table schema**
```json
[
{
"table_id":1,
"table_name":"TEST_IP_PLUGIN_WITH_EXDATA",
"table_type":"ip_plugin",
"valid_column":6,
"custom": {
"gc_timeout_s": 3,
"item_id":1,
"ip_type":2,
"start_ip":3,
"end_ip":4
}
}
]
```
**(2) configuration**
```json
{
"compile_table": "COMPILE",
"group2compile_table": "GROUP2COMPILE",
"group2group_table": "GROUP2GROUP",
"plugin_table": {
"table_name": "TEST_IP_PLUGIN_WITH_EXDATA",
"table_content": [
"101\t4\t192.168.30.99\t192.168.30.101\tSomething-like-json\t1",
"102\t4\t192.168.30.90\t192.168.30.128\tBigger-range-should-in-the-back\t1",
"103\t6\t2001:db8:1234::\t2001:db8:1235::\tBigger-range-should-in-the-back\t1",
"104\t6\t2001:db8:1234::1\t2001:db8:1234::5210\tSomething-like-json\t1",
]
},
}
```
**(3) xx_plugin_get_ex_data**
```C
#include <assert.h>
#include "maat.h"
#include "maat_utils.h"
#define ARRAY_SIZE 16
const char *json_filename = "./maat_json.json";
const char *table_info_path = "./table_info.conf";
struct ip_plugin_ud {
long long rule_id;
char *buffer;
size_t buf_len;
};
void ip_plugin_ex_new_cb(const char *table_name, int table_id, const char *key,
const char *table_line, void **ad, long argl, void *argp)
{
int *counter = (int *)argp;
size_t column_offset=0, column_len=0;
struct ip_plugin_ud *ud = ALLOC(struct ip_plugin_ud, 1);
int ret = get_column_pos(table_line, 1, &column_offset, &column_len);
EXPECT_EQ(ret, 0);
ud->rule_id = atoll(table_line + column_offset);
ret = get_column_pos(table_line, 5, &column_offset, &column_len);
EXPECT_EQ(ret, 0);
ud->buffer = ALLOC(char, column_len + 1);
strncpy(ud->buffer, table_line + column_offset, column_len);
ud->buf_len = column_len + 1;
*ad = ud;
(*counter)++;
}
void ip_plugin_ex_free_cb(int table_id, void **ad, long argl, void *argp)
{
struct ip_plugin_ud *ud = (struct ip_plugin_ud *)(*ad);
ud->rule_id = 0;
memset(ud->buffer, 0, ud->buf_len);
ud->buf_len = 0;
free(ud->buffer);
free(ud);
*ad = NULL;
}
void ip_plugin_ex_dup_cb(int table_id, void **to, void **from, long argl, void *argp)
{
struct ip_plugin_ud *ud = (struct ip_plugin_ud *)(*from);
*to = ud;
}
int main()
{
/* initialize maat options which will be used by maat_new() */
struct maat_options *opts = maat_options_new();
maat_options_set_json_file(opts, json_filename);
maat_options_set_logger(opts, "./sample_test.log", LOG_LEVEL_INFO);
/* create maat instance, rules in table_info.conf will be loaded. */
struct maat *maat_instance = maat_new(opts, table_info_path);
assert(maat_instance != NULL);
maat_options_free(opts);
const char *table_name = "TEST_IP_PLUGIN_WITH_EXDATA"; /* maat_json.json has TEST_IP_PLUGIN_WITH_EXDATA rule */
int table_id = maat_get_table_id(maat_instance, table_name);
assert(table_id == 1); /* defined in table_info.conf */
int ret = maat_plugin_table_ex_schema_register(maat_inst, table_name,
ip_plugin_ex_new_cb,
ip_plugin_ex_free_cb,
ip_plugin_ex_dup_cb,
0, &ip_plugin_ex_data_counter);
EXPECT_EQ(ret, 0);
EXPECT_EQ(ip_plugin_ex_data_counter, 4);
struct ip_addr ipv4;
ipv4.ip_type = IPv4;
ret = inet_pton(AF_INET, "192.168.30.100", &ipv4.ipv4);
EXPECT_EQ(ret, 1);
/* get ex_data */
struct ip_plugin_ud *results[ARRAY_SIZE];
ret = maat_ip_plugin_table_get_ex_data(maat_inst, table_id, &ipv4,
(void **)results, ARRAY_SIZE);
EXPECT_EQ(ret, 2);
EXPECT_EQ(results[0]->rule_id, 101);
EXPECT_EQ(results[1]->rule_id, 102);
struct ip_addr ipv6;
ipv6.ip_type = IPv6;
inet_pton(AF_INET6, "2001:db8:1234::5210", &(ipv6.ipv6));
memset(results, 0, sizeof(results));
ret = maat_ip_plugin_table_get_ex_data(maat_inst, table_id, &ipv6,
(void**)results, ARRAY_SIZE);
EXPECT_EQ(ret, 2);
EXPECT_EQ(results[0]->rule_id, 104);
EXPECT_EQ(results[1]->rule_id, 103);
maat_state_free(state);
return 0;
}
```

158
docs/group_hierarchy.md Normal file
View File

@@ -0,0 +1,158 @@
# Group(Object) hierarchies
A group can reference other groups, and can also be referenced by other groups. For example, group_A references group_B, group_A is the superior group of group_B, and group_B is the subordinate group of group_A. There are two reference relationships between groups: `include` and `exclude`.
## Include
Include is equivalent to the inclusion semantics in set theory. For example, when group_A is included by group_B, if a traffic attribute satisfies group_A, group_B is satisfied.
## Exclude
A group defines a subset of an group type, such as network addresses or port numbers. The definition is made using items, which can be used to add to or exclude from the group definition. Groups can also have subordinate groups whose definitions are included in the superior group.
There are rules of precedence to take into account when defining groups:
- Excluding has precedence over including in the same group.
- Items in a superior group have precedence over items in a subordinate group.
- Items in a superior group are not taken into account in a subordinate group, if the subordinate group is used directly in a rule.
- Peer groups (different subordinate groups of the same superior group) do not affect each other.
In short, to determine the set defined by a group, perform the following calculation:
1. For each subordinate group (remember sibling groups do not affect each other):
- Add included items.
- Subtract excluded items.
2. Add included items in the group itself, overriding any excludes in the subordinate groups.
3. Subtract excluded items in the group itself.
The following figure shows a group with an included set and an excluded subset.
<img src="./imgs/exclude.png" width="350" height="200" >
Now, consider adding a subordinate group. The subordinate group also has an included set and an excluded subset. If the superior group is used, the result is shown in the following figure “A superior and subordinate group”.
<img src="./imgs/hierarchy-and-exclude.png" width="350" height="230" >
As can be seen, the excluded item in the subordinate group is overwritten since it is in the included set of the superior group. Also, the excluded item from the superior group excludes part of the included item in the subordinate.
If only the subordinate group is used in a rule condition, the superior group items are disregarded, leaving the set shown in the next figure, “The subordinate group”.
<img src="./imgs/sub-exclude.png" width="350" height="230" >
Restrictions:
- A group can only include or exclude groups of the same type.
- A group should include at least one subordinate group. (Exclude only is not allowed)
- Traffic attribute using stream scan cannot allow use object(group) with exclude, i.e., keywords object on HTTP Response Body, Email attachment.
Now, let's see a graph of hierarchy example, where the dotted line means exclude. The matched subordinate groups and activated superiors are listed in the following table.
Now, let's see a graph of hierarchy example, where the dotted line means exclude. If the matched subordinate groups: **{g11, g13}**, then activated superiors: **{g2, g7}**.
<img src="./imgs/group-hierarchy1.png" width="750" height="350" >
The analysis process is as follows:
**matched groups {g11, g13}**
**Level 1**: g11 => incl{g6, g7} excl{null}, g13 => incl{g3} excl{g4}; new matched groups {g6, g7, g3}, `all matched groups` {g11, g13, g6, g7, g3}
**new matched groups {g6, g7, g3}**
**Level 2**: g6 => incl{g1} excl{g2, g3}, g7 => incl{g2, g4} excl{g6}; new matched groups {g1, g2, g4}, `all matched groups` {g11, g13, g7, g1, g2, g4}
**new matched groups {g1, g2, g4}**
**Level 3**: end
Check the validity of `all matched groups` {g11, g13, g7, g1, g2, g4}:
1. g11, g13 are the originally matched nodes, so keep {g11, g13}.
2. g7 is derived from the match of g11, implying that g10 is not matched (if g10 is matched, then g7 is not matched, as the exclude priority is higher), so keep g7.
3. g1 is derived from the match of g6, but since g6 is excluded due to both g11 and g7, and g6 excludes g7 with higher priority, g1 is no longer matched. Therefore, remove g1 from the set.
3. g1 is derived from the match of g6, which in turn is derived from the match of g11. However, since g7 is also matched and g6 excludes g7 with higher priority, g7's match causes g6 to be unmatched, and consequently, g1 is no longer matched. Therefore, g1 is removed from the set.
4. g2 is derived from the match of g7, with the implied condition that g6 is not matched, so keep g2.
5. Finally, `all matched groups` {g11, g13, g7, g2}, `matched super groups` {g2, g7}.
Remove the dotted line from g7 to g6, as shown in the diagram below. If the matched subordinate groups: **{g11, g13}**, then activated groups: **{g1, g6, g7}**.
<img src="./imgs/group-hierarchy2.png" width="750" height="350" >
The analysis process is as follows:
**matched groups {g11, g13}**
**Level 1**: g11 => incl{g6, g7} excl{null}, g13 => incl{g3} excl{g4}; new matched groups {g6, g7, g3}, `all matched groups` {g11, g13, g6, g7, g3}
**new matched groups {g6, g7, g3}**
**Level 2**: g6 => incl{g1} excl{g2, g3}, g7 => incl{g2, g4} excl{null}; new matched groups {g1, g2, g4}, `all matched groups` {g11, g13, g6, g1, g7, g2, g4}
**new matched groups {g1, g2, g4}**
**Level 3**: end
Check the validity of `all matched groups` {g11, g13, g6, g1, g7, g2, g4}:
1. g11, g13 are the originally matched nodes, so keep {g11, g13}.
2. g6 is derived from the match of g11, implying the g10 is not matched(if g10 is matched, then g7 is not matched, as the exclude priority is higher), so keep g6.
3. g1 is derived from the match of g6, which in turn is derived from the match of g11, implying the g5 is not matched, so keep g1.
4. g7 is derived from the match of g11, implying the g10 is not matched, so keep g7.
5. g2 is derived from the match of g7. However, since g6 is also matched and g2 excludes g6 with higher priority, g6's match causes g2 to be unmatched. Therefore, g2 is removed from the set.
6. g4 is derived from the match of g7. However, since g13 is also matched and g4 excludes g13 with higher priority, g13's match causes g4 to be unmatched. Therefore, g4 is removed from the set.
7. Finally, `all matched groups` {g11, g13, g1, g6, g7}, `matched super groups` {g1, g6, g7}.
Again remove the dotted line from g13 to g4, as shown in the diagram below. If the matched subordinate groups: **{g11, g13}**, then activated superiors: **{g1, g4, g6, g7}**.
<img src="./imgs/group-hierarchy3.png" width="750" height="350" >
The analysis process is as follows:
**matched groups {g11, g13}**
**Level 1**: g11 => incl{g6, g7} excl{null}, g13 => incl{g3} excl{null}; new matched groups {g6, g7, g3}, `all matched groups` {g11, g13, g6, g7, g3}
**new matched groups {g6, g7, g3}**
**Level 2**: g6 => incl{g1} excl{g2, g3}, g7 => incl{g2, g4} excl{null}; new matched groups {g1, g2, g4}, `all matched groups` {g11, g13, g6, g1, g7, g2, g4}
**new matched groups {g1, g2, g4}**
**Level 3**: end
Check the validity of `all matched groups` {g11, g13, g6, g1, g7, g2, g4}:
1. g11, g13 are the originally matched nodes, so keep {g11, g13}.
2. g6 is derived from the match of g11, implying the g10 is not matched(if g10 is matched, then g7 is not matched, as the exclude priority is higher), so keep g6.
3. g1 is derived from the match of g6, which in turn is derived from the match of g11, implying the g5 is not matched, so keep g1.
4. g7 is derived from the match of g11, implying the g10 is not matched, so keep g7.
5. g2 is derived from the match of g7. However, since g6 is also matched and g2 excludes g6 with higher priority, g6's match causes g2 to be unmatched. Therefore, g2 is removed from the set.
6. g4 is derived from the match of g13, so keep g4.
7. Finally, `all matched groups` {g11, g13, g6, g1, g7, g4}, `matched super groups` {g1, g4, g6, g7}.
The above example is strictly consistent with the case `GroupExcludeTest.level_3_function` in the [test/group_nesting](../test/group_nesting/) test. Please run it to see the actual execution results.

142
docs/history.md
View File

@@ -1,73 +1,73 @@
# 更新记录 # 更新记录
| 版本 | 时间 | 说明 | 修订人 | | Version| Date | Description | Author |
| ------- | ---------- | --------------------------------------------------------- | -------- | |--------|-----------|-------------|-------------|
| v4.0 | 2023-5-2 | Maat4.0，使用Markdown重新组织文档 | 郑超、刘文坛 | | v4.0 | 2023-5-2 | Maat 4.0, reorganize documentation using Markdown | Zheng Chao, Liu Wentan |
| V3.7.0 | 2022-8-23 | 增加Boolean Expression 回调表；修正分组于子分组中的描述。| 郑超 | | v3.7.0 | 2022-8-23 | Add Boolean Expression callback table; correct descriptions of grouping within subgroups| Zheng Chao|
| V3.2.3 | 2021-7-15 | 增加组合扫描一节 | 郑超 | | v3.2.3 | 2021-7-15 | Add section on composite scanning | Zheng Chao |
| V3.1.20 | 2021-4-28 | 增加加载gzip压缩后的JSON文件说明 | 郑超 | | v3.1.20| 2021-4-28 | Add explanation for loading gzip-compressed JSON files | Zheng Chao |
| V3.1.5 | 2021-3-12 | 增加回调类FQDN表的匹配示例 | 郑超 | | v3.1.5 | 2021-3-12 | Add matching examples for callback class FQDN table | Zheng Chao |
| V3.1.4 | 2020-11-04 | 内容外键为“null”时，表示空文件。 | 郑超 | | v3.1.4 | 2020-11-04| When content foreign key is "null," it represents an empty file | Zheng Chao |
| V3.1.1 | 2020-9-27 | 增加虚拟表建立在多个不同类型物理表上的说明 | 郑超 | | v3.1.1 | 2020-9-27 | Add explanation for creating virtual tables on multiple different types of physical tables | Zheng Chao |
| V3.1.0 | 2020-9-18 | 增加FQDN回调表 | 郑超 | | v3.1.0 | 2020-9-18 | Add FQDN callback table | Zheng Chao |
| v3.0.4 | 2020-8-17 | 增加扩展的数值类域配置interval_plus | 郑超 | | v3.0.4 | 2020-8-17 | Add extended numerical domain configuration interval_plus | Zheng Chao |
| v3.0 | 2020-6-29 | Maat 3.0, 支持子句，原分组表拆分为分组关系表和分组编译表 | 郑超 | | v3.0 | 2020-6-29 | Maat 3.0, support clauses, original grouping table split into grouping relationship table and grouping compilation table | Zheng Chao |
| v2.8 | 2020-3-13 | 支持组合表 | 郑超 | | v2.8 | 2020-3-13 | Support composite tables | Zheng Chao |
| v2.8 | 2020-2-11 | 支持命中路径（Hit Path） | 郑超 | | v2.8 | 2020-2-11 | Support Hit Path | Zheng Chao |
| v2.8 | 2020-2-4 | 支持策略按照执行顺序（Evaluation Order）排序 | 郑超 | | v2.8 | 2020-2-4 | Support sorting strategies by Evaluation Order | Zheng Chao |
| v2.8 | 2020-1-22 | Maat JSON文件支持加密，参见17.3 | 郑超 | | v2.8 | 2020-1-22 | Maat JSON file encryption support, see 17.3 | Zheng Chao |
| v2.8 | 2019-7-28 | 1、 不再兼容无分组表的情况 2、 支持虚拟表 | 郑超 | | v2.8 | 2019-7-28 | 1. No longer compatible with no grouping tables 2. Support virtual tables | Zheng Chao |
| v2.7.2 | 2019-7-10 | 扩展类IP配置表支持CIDR格式。 | 郑超 | | v2.7.2 | 2019-7-10 | Expand IP configuration table to support CIDR format | Zheng Chao |
| v2.7.1 | 2019-5-23 | 将富IP类配置表改名为扩展类IP配置表。 | 郑超 | | v2.7.1 | 2019-5-23 | Rename rich IP class configuration table to extended IP configuration table | Zheng Chao |
| v2.7.0 | 2019-5-12 | 1、 增加子分组关系说明 2、 增加富IP类配置的说明 | 郑超 | | v2.7.0 | 2019-5-12 | 1. Add explanation for subgroup relationships 2. Add explanation for rich IP class configuration | Zheng Chao |
| v2.6.0 | 2019-1-8 | 1、增加“非”运算的说明； 2、文档和动态链接库版本号的前两位相同。 | 郑超 | | v2.6.0 | 2019-1-8 | 1. Add explanation for "not" operation 2. Document and dynamic link library version numbers share the first two digits | Zheng Chao |
| v2.3.3 | 2018-12-24 | 增加Extra Data和read column函数的说明；多命中情况下，按照配置ID由大到小返回命中结果。 | 郑超 | | v2.3.3 | 2018-12-24| Add explanation for Extra Data and read column function; in multiple hit scenarios, return results in descending order of configuration ID | Zheng Chao |
| v2.3.2 | 2018-12-03 | 支持运行态以全量通知的方式加载变化的json文件 | 郑超 | | v2.3.2 | 2018-12-03| Support runtime loading of changing JSON files in full notification mode | Zheng Chao |
| v2.3.1 | 2018-10-15 | 完善内容外键和配置生效标签的说明。 | 郑超 | | v2.3.1 | 2018-10-15| Improve explanation of content foreign keys and configuration effective tags | Zheng Chao |
| v2.3.0 | 2018-9-26 | 增加内容外键和配置生效标签的说明。 | 郑超 | | v2.3.0 | 2018-9-26 | Add explanation of content foreign keys and configuration effective tags | Zheng Chao |
| v2.2.5 | 2018-9-12 | 更正IP类域配置关于全0掩码表示任意说明，全0掩码为精确匹配。 | 郑超 | | v2.2.5 | 2018-9-12 | Correct explanation of IP domain configuration regarding all 0 masks representing any to exact match | Zheng Chao |
| v2.2.4 | 2018-08-09 | 在“配置表描述文件”一节中，说明编译配置表中用户自定义域转码的设置方法。 | 郑超 | | v2.2.4 | 2018-08-09| In "Configuration Table Description Files" section, explain how to set custom domain transcoding in compilation configuration table | Zheng Chao |
| v2.2.3 | 2018-07-27 | 1、完善“常见故障处理->扫描结果与预想不一致”一节。 2、整理“配置表描述方式”一节。 | 郑超 | | v2.2.3 | 2018-07-27| 1. Enhance "Common Fault Handling -> Scan Results Inconsistent" section 2. Organize "Configuration Description Methods" section | Zheng Chao |
| v2.2.2 | 2018-07-05 | 支持超过128字节的用户自定义域 | 郑超 | | v2.2.2 | 2018-07-05| Support user-defined domains exceeding 128 bytes | Zheng Chao |
| v2.2.1 | 2018-05-25 | 更新Redis配置加载接口的事务机制。 | 郑超 | | v2.2.1 | 2018-05-25| Update transaction mechanism for loading Redis configuration interface | Zheng Chao |
| v2.1.2 | 2018-03-24 | 增加概述 | 郑超 | | v2.1.2 | 2018-03-24 | Add overview | Zheng Chao |
| v2.1.1 | 2018-02-08 | 1、 maat_redis_tool支持写入json格式的配置到redis。 2、 增加Maat_read_state函数的使用说明。 | 郑超 | | v2.1.1 | 2018-02-08 | 1. maat_redis_tool supports writing JSON format configurations to Redis 2. Add instructions for using Maat_read_state function | Zheng Chao |
| v2.1.0 | 2017-10-04 | 提供maat_redis_tool，可以dump出redis中的maat配置。 | 郑超 | | v2.1.0 | 2017-10-04 | Provide maat_redis_tool to dump Maat configurations from Redis | Zheng Chao |
| v2.0.1 | 2017-08-22 | 支持配置延迟初始化 | 郑超 | | v2.0.1 | 2017-08-22 | Support delayed initialization for configurations | Zheng Chao |
| v2.0.0 | 2017-08-01 | 1. 支持从Redis加载配置； 2. 提供配置增删函数接口，即Maat Command API； 3. 支持相似性字符串匹配； 4. 提供辅助工具digest_gen，具备SFH摘要计算、编辑距离计算功能； | 郑超 | | v2.0.0 | 2017-08-01 | 1. Support loading configurations from Redis; 2.Provide configuration add and delete function interfaces, i.e., Maat Command API; 3.Support similarity string matching; 4.Provide auxiliary tool digest_gen, with SFH digest calculation and edit distance calculation functions. | Zheng Chao |
| v1.9.1 | 2017-06-24 | 可以提示MAAT组合扫描中最后一个域，可以提升与表达式运算性能。 | 郑超 | | v1.9.1 | 2017-06-24 | Can prompt the last domain in MAAT combination scan, which can improve performance with expression operations. | Zheng Chao |
| v1.9.0 | 2017-06-09 | 支持对个别配置表进行加密 | 郑超 | | v1.9.0 | 2017-06-09 | Support encryption for individual configuration tables | Zheng Chao |
| v1.8.4 | 2017-03-12 | 调整SNORT规则转换的目录结构 | 郑超 | | v1.8.4 | 2017-03-12 | Adjust the directory structure of SNORT rule conversion | Zheng Chao |
| v1.8.3 | 2016-12-30 | 1、增加设置实例名的功能；2、更新运行状态数据说明。 | 郑超 | | v1.8.3 | 2016-12-30 | 1. Add the functionality to set the instance name; 2. Update the explanation of running status data. | Zheng Chao |
| v1.8.2 | 2016-9-19 | JSON输入模式支持未命名分组，可将group_name配置为"Untitled"以表示非分组复用，由Maat自动生成group_id。 | 郑超 | | v1.8.2 | 2016-9-19 | JSON input mode supports unnamed groups, where group_name can be configured as "Untitled" to indicate non-group reuse, and Maat automatically generates group_id. | Zheng Chao |
| v1.8.1 | 2016-9-14 | 在IP类域配置表中增加多层嵌套IP匹配的指导方案。 | 郑超 | | v1.8.1 | 2016-9-14 | Add guidance for nested IP matching in IP class domain configuration tables. | Zheng Chao |
| v1.8.0 | 2016-09-02 | 增加配置表连接功能的说明 | 郑超 | | v1.8.0 | 2016-09-02 | Add explanation for configuration table connections | Zheng Chao |
| v1.7.1 | 2016-06-17 | 1. 通过转义方式，支持空格作为关键字; 2. 增加快速字符串扫描开关; | 郑超 | | v1.7.1 | 2016-06-17 | 1. Support spaces as keywords through escape mode; 2. Add a switch for fast string scanning; | Zheng Chao |
| v1.7.0 | 2016-04-03 | 单独设小节描述编码转换，增加UNICODE到ASCII转码功能的说明 | 郑超 | | v1.7.0 | 2016-04-03 | Add a section describing encoding conversion, and include instructions for UNICODE to ASCII transcoding functionality. | Zheng Chao |
| v1.6.1 | 2016-03-24 | 完善配置分组表的字段说明。 | 郑超 | | v1.6.1 | 2016-03-24 | Improve the field description of configuration group tables. | Zheng Chao |
| v1.6.0 | 2016-02-11 | 1. 增加状态统计和性能统计描述，以及相应的函数接口调整； 2. 增加扩展的字符串类域配置的函数接口描述。 | 郑超 | | v1.6.0 | 2016-02-11 | 1. Add descriptions for status and performance statistics, and adjust corresponding function interfaces; 2. Add function interface descriptions for extended string class domain configurations. | Zheng Chao |
| v1.5.0 | 2016-01-05 | 增加扩展的字符串类域配置，完善IP类域配置关于IPv6掩码的描述。 | 郑超 | | v1.5.0 | 2016-01-05 | Add extended string class domain configurations, and enhance descriptions for IPv6 mask in IP class domain configurations. | Zheng Chao |
| v1.4.4 | 2015-12-24 | 修改回调表的配置更新机制，以节约内存。 | 郑超 | | v1.4.4 | 2015-12-24 | Modify the configuration update mechanism for callback tables to save memory. | Zheng Chao |
| v1.4.3 | 2015-12-02 | 完善数据库表间关系的描述，丰富界面设计要点。 | 郑超 | | v1.4.3 | 2015-12-02 | Improve descriptions of database table relationships and enrich interface design points. | Zheng Chao |
| v1.4.2 | 2015-12-01 | 合并配置更新文件接口一章 | 郑超 | | v1.4.2 | 2015-12-01 | Merge the chapter on configuration update file interfaces. | Zheng Chao |
| v1.4.1 | 2015-11-23 | 完善摘要类配置的说明 | 郑超 | | v1.4.1 | 2015-11-23 | Improve descriptions for summary class configurations | Zheng Chao |
| v1.4.0 | 2015-11-06 | 增加文件摘要类域配置 | 郑超 | | v1.4.0 | 2015-11-06 | Add file summary class domain configurations | Zheng Chao |
| v1.3.9 | 2015-07-24 | 提供Maat_set_feather_opt接口，Maat_20150724版本以上支持该功能。 | 郑超 | | v1.3.9 | 2015-07-24 | Provide Maat_set_feather_opt interface, supported in versions Maat_20150724 and above. | Zheng Chao |
| v1.3.8 | 2015-07-06 | 润色概述中对三类配置的说明，增加编码转换句柄复用的说明。 | 郑超 | | v1.3.8 | 2015-07-06 | Refine explanations of the three types of configurations in the overview, and add explanations for encoding handle reuse. | Zheng Chao |
| v1.3.7 | 2015-06-16 | 增加常见故障处理的建议。 | 郑超 | | v1.3.7 | 2015-06-16 | Add suggestions for common fault handling.| Zheng Chao |
| v1.3.6 | 2015-06-10 | 1. 字符串域表增加相对位置的表达式类型; 2. 配置分组表支持非； 测试分支，暂不合并到主版本。 | 郑超 | | v1.3.6 | 2015-06-10 | 1. Add expression types for relative positions to string domain tables; 2. Support non-testing branches in configuration group tables, not yet merged into the main version. | Zheng Chao |
| v1.3.5 | 2015-06-08 | 添加数据库表描述示例 | 郑超 | | v1.3.5 | 2015-06-08 | Add examples for database table descriptions | Zheng Chao |
| v1.3.4 | 2015-05-22 | 可通过JSON加载回调类配置表 | 郑超 | | v1.3.4 | 2015-05-22 | Load callback class configurations through JSON | Zheng Chao |
| v1.3.3 | 2015-05-13 | 增加对非组合域编译的快速处理路径，增加对命中结果返回条件的说明 | 郑超 | | v1.3.3 | 2015-05-13 | Add fast processing path for non-combined domain compilation, and explain conditions for returning hit results. | Zheng Chao |
| v1.3.2 | 2015-04-13 | 扫描空配置的返回值由-1改为0 | 郑超 | | v1.3.2 | 2015-04-13 | Change return value for scanning empty configurations from -1 to 0 | Zheng Chao |
| v1.3.1 | 2015-03-02 | 对于GBK到BIG5的转码，将源编码改为GB2312 | 郑超 | | v1.3.1 | 2015-03-02 | Change source encoding for GBK to BIG5 transcoding to GB2312 | Zheng Chao|
| v1.3 | 2015-02-20 | 增加JSON配置加载功能 | 郑超 | | v1.3 | 2015-02-20 | Add JSON configuration loading functionality | Zheng Chao |
| v1.2.2 | 2015-01-06 | 增加强制卸载机制 | 郑超 | | v1.2.2 | 2015-01-06 | Add forced unloading mechanism | Zheng Chao |
| v1.2.1 | 2015-01-05 | 扩展字符串类域配置的hexbin字段的定义 | 郑超 | | v1.2.1 | 2015-01-05 | Extend definition of hexbin field for string class domain configurations | Zheng Chao |
| v1.2.0 | 2014-12-20 | 增加配置更新机制的描述 | 郑超 | | v1.2.0 | 2014-12-20 | Add description for configuration update mechanism | Zheng Chao |
| v1.1.0 | 2014-12-12 | 增加函数接口的说明，修改引用计数避免伪共享 | 郑超 | | v1.1.0 | 2014-12-12 | Add descriptions for function interfaces, and modify reference counting to avoid false sharing | Zheng Chao |
| v1.0.3 | 2014-12-03 | 采用垃圾回收机制保证线程安全 | 郑超 | | v1.0.3 | 2014-12-03 | Use garbage collection mechanism to ensure thread safety | Zheng Ch|
| v1.0.2 | 2014-11-26 | 支持正则分组 | 郑超 | | v1.0.2 | 2014-11-26 | Support regular expression grouping | Zheng Chao |
| v1.0.1 | 2014-11-19 | IP类配置支持协议扫描 | 郑超 | | v1.0.1 | 2014-11-19 | IP scanning supports protocol parameter | Zheng Chao |
| v1.0 | 2014-08-19 | 第一个稳定版 | 郑超 | | v1.0 | 2014-08-19 | First stable version | Zheng Chao |
| v0.1 | 2014-06-16 | 创建文档，包含表格式说明 | 郑超 | | v0.1 | 2014-06-16 | Create document and define table format | Zheng Chao |

BIN
docs/imgs/A-include-B.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 20 KiB

BIN
docs/imgs/NOT-clause-groups.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 42 KiB

BIN
docs/imgs/add-del-rule-with-redis.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 184 KiB

BIN
docs/imgs/compile-runtime.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 153 KiB

BIN
docs/imgs/data-sync-with-redis.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 29 KiB

BIN
docs/imgs/exclude.png
View File

Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 34 KiB

After

Width:  |  Height:  |  Size: 24 KiB

BIN
docs/imgs/expr-runtime.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 25 KiB

BIN
docs/imgs/group-hierarchy1.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 62 KiB

BIN
docs/imgs/group-hierarchy2.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 61 KiB

BIN
docs/imgs/group-hierarchy3.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 60 KiB

BIN
docs/imgs/hierarchy-and-exclude.png
View File

Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 28 KiB

After

Width:  |  Height:  |  Size: 52 KiB

BIN
docs/imgs/load-data-from-redis.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 43 KiB

BIN
docs/imgs/load-rule-from-redis.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 56 KiB

BIN
docs/imgs/logo.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 107 KiB

BIN
docs/imgs/maat-architect.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 87 KiB

BIN
docs/imgs/maat_logo.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 6.5 KiB

BIN
docs/imgs/maat_stat.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1.5 MiB

BIN
docs/imgs/object-hierarchy-example.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 26 KiB

BIN
docs/imgs/sub-exclude.png
View File

Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 13 KiB

After

Width:  |  Height:  |  Size: 36 KiB

BIN
docs/imgs/sync-with-redis.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 52 KiB

BIN
docs/imgs/thread-model.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 122 KiB

BIN
docs/imgs/thread_mode.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 44 KiB

BIN
docs/imgs/update-rule-in-redis.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 95 KiB

BIN
docs/imgs/virt-phy-mapping.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 20 KiB

BIN
docs/imgs/xx_plugin_runtime.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 56 KiB

121
docs/introduction.md
View File

@@ -1,121 +0,0 @@
# Concepts
**Item**: As a filter for network attributes, the smallest unit of a rule
- Eg1: specify that the UserAgent field in the HTTP protocol contains substrings "Chrome" and "11.8.1",
&ensp;&ensp; HTTP UserAgent: Chrome & 11.8.1
- Eg2: specify that the domain name in the HTTP protocol ends with ".emodao.com"
&ensp;&ensp; HTTP HOST: *.emodao.com
- Eg3: specify client IP address belongs to the C segment of 202.118.101.*
&ensp;&ensp; Source IP: 202.11.101.0/24
There are multiple types of items stored in corresponding tables such as string, IP and numerical range, more details can be found in [Item table](./table_schema.md#item-table).
**Group(Object)**: Collection of Items, the constraints of group are as follows:
- An Item only belongs to one group, but one group can has multiple items. The multiple items under the same group are logical 'OR' relationships. e.g.(g1 = item1 | item2)
- A Group can be included or excluded by other groups. The multiple included groups under the same superior group are logical 'OR' relationship. e.g.(g3 = incl-g1 | incl-g2). Between included group and excluded group under the same superior group are logical 'AND' relationship. e.g.(g4 = incl-g1 & excl-g2)
- Group supports multi-level nesting
- A Group can be referenced by multiple compiles.
The relationship between group and group is stored in the [group2group table](./table_schema.md#4-group2group-table), while the relationship between group and compile is stored in the [group2compile table](./table_schema.md#5-group2compile-table).
**Compile(Policy)**: A conjunctive normal form(CNF) consisting of multiple groups and virtual tables
- A Compile can contain up to 8 clauses and multiple clauses in the same compile can be logical 'AND' and logical 'NOT' relationships.
- A Clause consists of several Literals and the relationship between them is a logical 'OR'. A Literal consists of virtual table and group. During the rules loading process, a unique Clause ID will be generated based on the combination of virtual table ID and group ID in the same clause.
![CNF](./imgs/CNF.jpg)
Rule Diagram
![Rule Diagram](./imgs/rule_diagram.png)
## Group/Object Nesting and Hierarchies
The objects and items are compared in the rule evaluation with **OR** operator. If a traffic attribute satisfies an item, the object it belongs to is satisfied.
### Exclude
An object defines a subset of an object type, such as network addresses or port numbers. The definition is made using items, which can be used to add to or exclude from the object definition. Objects can also have subordinate objects whose definitions are included in the superior object.
There are rules of precedence to take into account when defining objects:
- Excluding has precedence over including in the same object.
- Items in a superior object have precedence over items in a subordinate object.
- Items in a superior object are not taken into account in a subordinate object, if the subordinate object is used directly in a rule.
- Peer objects (different subordinate objects of the same superior object) do not affect each other.
In short, to determine the set defined by an object, perform the following calculation:
1. For each subordinate object (remember sibling objects do not affect each other):
- Add included items.
- Subtract excluded items.
2. Add included items in the object itself, overriding any excludes in the subordinate objects.
3. Subtract excluded items in the object itself.
The following figure shows an object with an included set and an excluded subset.
![exclude](./imgs/exclude.png)
Now, consider adding a subordinate object. The subordinate object also has an included set and an excluded subset. If the superior object is used, the result is shown in the following figure “A superior and subordinate object”.
![A superior and subordinate object](./imgs/hierarchy-and-exclude.png)
As can be seen, the excluded item in the subordinate object is overwritten since it is in the included set of the superior object. Also, the excluded item from the superior object excludes part of the included item in the subordinate.
If only the subordinate object is used in a rule condition, the superior object items are disregarded, leaving the set shown in the next figure, “The subordinate object”.
![The subordinate object](./imgs/sub-exclude.png)
Restrictions:
- An object can only include or exclude objects of the same type.
- An object should include at least one subordinate object. (Exclude only is not allowed)
- Traffic attribute using stream scan cannot allow use object with exclude, i.e., keywords object on HTTP Response Body, Email attachment.
Now, let's see a graph of hierarchy example, where the dotted line means exclude. The matched subordinate objects and activated superiors are listed in the following table.
![object-hierarchy-example](./imgs/object-hierarchy-example.png)
| **Matched subordinate objects** | **Activated superior Objects** |
| ------------------------------- | ------------------------------ |
| g1, g3 | g6, g9 |
| g3, g4 | none |
| g5 | g8, g10, g13 |
| g2, g3 | g9 |
| g3, g5 | none |
| g8, g11 | g10, g12 |
| g4, g11 | g7, g8, g9 |
| g1, g2 | g9 |
## Defined Terms
| **Term** | **Description** |
| --------------------------- | ------------------------------------------------------------ |
| Instance | Maat handle |
| Item | Rule of a certain field，such as URL field in HTTP protocol, client ip address field in DNS protocol etc.|
| Group(Object) | A collection of one or more Items, the multiple items under the same Group are logical 'OR' relationships |
| Compile(Policy) | A rule for several Groups logical AND or NOT operations |
| Table | Different types of rules have different tables, such as ip table, keywords table, group2compile table, compile table etc. |
| Physical Table | The actual table in the database |
| Virtual Table | Table that do not exist in the database and only references physical table |
| Table Schema | Define the table type and the meaning of each column in the table |
| Table Runtime | The runtime of each table in memory which is used for scanning |
| Maat State | Store intermediate state of multiple scans |
| Maat Stream | Handle of streaming file scanning |
| Hit Path | From the perspective of data to be scanned, describe the hit ID sequence: item_id -> sub_group_id -> superior_group_id(virtual_table_id) -> compile_id |
| Redis | In-memory data store，see https://redis.io/. It has a leader follower replication to ensure the high availability of rules |

176
docs/logical_combinations.md Normal file
View File

@@ -0,0 +1,176 @@
# Logical combinations
When you understand the [configuration relationship](./overview.md#12-configuration-relationship) and [group hierarchy](./group_hierarchy.md), you will find multiple layers of logical relationships. Are all these logical relationships necessary? If yes, what are the use cases targeted by each? This document serves as a cheat sheet for you to understand the logical combinations of maat rules.
Before describing specific rules, we need to define the syntax of logical combinations. A rule is written using infix notation, which consists of operands, operators, and parentheses. The operands are rule IDs, and operators are "!(NOT)", "&(AND)", "|(OR)".
Before showing how to configure the specific rules, we need some raw materials (items). The following is the configuration information for the different types of item tables:
**keywords(table_id=1)**
| item_id | group_id | keywords |
| ------- | -------- | ------------- |
| 101 | 201 | www.baidu.com |
| 102 | 202 | baidu.com |
**ip address(table_id=2)**
| item_id | group_id | ip range |
| ------- | -------- | --------------------------- |
| 110 | 210 | 192.168.1.1 ~ 192.168.1.255 |
| 111 | 211 | 192.168.1.11 ~ 192.168.1.20 |
| 112 | 212 | 192.168.1.21 ~ 192.168.1.30 |
**port(table_id=3)**
| item_id | group_id | interval range |
| ------- | -------- | -------------- |
| 120 | 220 | 100 ~ 200 |
| 121 | 221 | 80 ~ 80 |
| 122 | 222 | 443 ~ 443 |
* [AND logic](#and-logic)
* [OR logic](#or-logic)
* [NOT logic](#not-logic)
* [Group exclude](#group-exclude)
## AND logic
`Note`: Each compile supports a maximum of 8 clauses, which are connected by the logical 'AND' relationship, with clause index ranging from 0 to 7.
* case1: Deny hosts with source IP addresses in the range of 192.168.1.11 to 192.168.1.20 from accessing the website www.baidu.com.
* 192.168.1.11 ~ 192.168.1.20 => group_id(211)
* www.baidu.com => group_id(201)
```bash
compile(rule) = clause1 & clause2
= {vtable1, group1, clause_index1} & {vtable2, group2, clause_index2}
= {2, 211, 1} & {1, 201, 2}
```
* case2: To block the traffic whose source IP address is 192.168.1.11 to 192.168.1.20 and source port 80.
* 192.168.1.11 ~ 192.168.1.20 => group_id(211)
* 端口80 ~ 80 => group_id(221)
```bash
compile(rule) = clause1 & clause2
= {vtable1, group1, clause_index1} & {vtable2, group2, clause_index2}
= {2, 211, 1} & {3, 221, 2}
```
The JSON configuration for the logical `AND` can be referenced at [unit_test Json configuration](../test/maat_json.json) with compile_id=152.
group_name: "152_mail_addr" and group_name: "interval_group_refered" are two clauses of this compile, with a logical `AND` relationship between them.
## OR logic
`Note`: Multiple groups under the same clause have a logical 'OR' relationship.
* case1： Deny hosts with source IP addresses in the range of 192.168.1.11 to 192.168.1.30 from accessing the website www.baidu.com.
* 192.168.1.11 ~ 192.168.1.20 => group_id(211)
* 192.168.1.21 ~ 192.168.1.30 => group_id(212)
* www.baidu.com => group_id(201)
```bash
compile(rule) = clause1 & clause2
= {vtable1, (group1 | group2), clause_index1} & {vtable2, group3, clause_index2}
= {2, (211 | 212), 1} & {1, 201, 2}
```
* case2: To block the traffic whose source ip address is 192.168.1.11 to 192.168.1.20 and source port 80 or 443.
* 192.168.1.11 ~ 192.168.1.20 => group_id(211)
* port 80 ~ 80 => group_id(221)
* port 443 ~ 443 => group_id(222)
```bash
compile(rule) = clause1 & clause2
= {vtable1, group1, clause_index1} & {vtable2, (group2 | group3), clause_index2}
= {2, 211, 1} & {3, (221 | 222), 2}
```
The JSON configuration for the logical `OR` can be referenced at [unit_test Json configuration](../test/maat_json.json) with compile_id=152.
group_name: "152_mail_addr" contains two regions(items) with a logical `OR` relationship between them.
## NOT logic
`Note`: Only clauses can support NOT-logic.
* case1: Hosts with source ip addresses ranging from 192.168.1.11 to 192.168.1.20 are allowed to access websites other than www.baidu.com.
* 192.168.1.11 ~ 192.168.1.20 => group_id(211)
* www.baidu.com => group_id(201)
```bash
compile(rule) = clause1 & !clause2
= {vtable1, group1, clause_index1} & !{vtable2, group2, clause_index2}
= {2, 211, 1} & !{1, 201, 2}
```
* case2: To block the traffic whose source ip address is in 192.168.1.11 to 192.168.1.20 and the source port is not 80 or 443.
* 192.168.1.11 ~ 192.168.1.20 => group_id(211)
* port 80 ~ 80 => group_id(221)
* port 443 ~ 443 => group_id(222)
```bash
compile(rule) = clause1 & !clause2
= {vtable1, group1, clause_index1} & !{vtable2, (group2 | group3), clause_index2}
= {2, 211, 1} & !{3, (221 | 222), 2}
```
The JSON configuration for the logical `OR` can be referenced at [unit_test Json configuration](../test/maat_json.json) with compile_id=145.
The group_name: "123_IP_group" is a NOT clause of this compile.
## Group exclude
`Note`: The `exclude` relationship only applies between groups of the same type, such as super_group1 = include {group1}, exclude {group2}.
Constraint: The super group cannot consist only of exclude groups; it must contain at least one include group.
* case1: Deny hosts with source IP addresses in the range of 192.168.1.1 to 192.168.1.255 but not in the range of 192.168.1.11 to 192.168.1.20 from accessing the website www.baidu.com.
* 192.168.1.1 ~ 192.168.1.255 => group_id(210)
* 192.168.1.11 ~ 192.168.20 => group_id(211)
* www.baidu.com => group_id(201)
In this case, you need to configure super_group1 first.
```bash
super_group1 = group1 exclude group2
= 210 exclude 211
```
And then configure the compile.
```bash
compile(rule) = clause1 & clause2
= {vtable1, super_group1, clause_index1} & {vtable2, group2, clause_index2}
= {2, (210 exclude 211), 1} & {1, 201, 2}
```
* case2: Deny hosts with source IP addresses in the range of 192.168.1.11 to 192.168.1.20 from accessing subdomains of baidu.com except for www.baidu.com.
* 192.168.1.11 ~ 192.168.1.20 => group_id(211)
* www.baidu.com => group_id(201)
* baidu.com => group_id(202)
In this case, you need to configure super_group2 first.
```bash
super_group2 = group2 exclude group3
= 202 exclude 201
```
And then configure the compile.
```bash
compile(rule) = clause1 & clause2
= {vtable1, group1, clause_index1} & {vtable2, super_group2, clause_index2}
= {2, 211, 1} & {1, (202 exclude 201), 2}
```
The JSON configuration for the logical `OR` can be referenced at [unit_test Json configuration](../test/maat_json.json) with compile_id=200.
The group_name: "ExcludeLogicGroup200" is a super group that includes an include group "ExcludeLogicGroup200_1" and an exclude group "ExcludeLogicGroup200_2".

304
docs/logical_operation.md
View File

@@ -1,304 +0,0 @@
# logic AND OR NOT
- [logic AND](#logic-and)
- [logic OR](#logic-or)
- [logic NOT](#logic-not)
- [group exclude](#group-exclude)
## logic AND
rule = China & USA
```json
{
"compile_id": 123,
"service": 1,
"action": 1,
"do_blacklist": 1,
"do_log": 1,
"user_region": "null",
"is_valid": "yes",
"groups": [
{
"clause_index": 0,
"regions": [
{
"table_name": "HTTP_URL",
"table_type": "expr",
"table_content": {
"keywords": "China",
"expr_type": "none",
"match_method": "sub",
"format": "uncase plain"
}
}
]
},
{
"clause_index": 1,
"regions": [
{
"table_name": "HTTP_URL",
"table_type": "expr",
"table_content": {
"keywords": "USA",
"expr_type": "none",
"match_method": "sub",
"format": "uncase plain"
}
}
]
}
]
}
```
scan sample
```c
const char *string1 = "China";
const char *string2 = "USA";
long long results[ARRAY_SIZE] = {0};
size_t n_hit_result = 0;
int thread_id = 0;
const char *table_name = "HTTP_URL";
struct maat_state *state = maat_state_new(maat_inst, thread_id);
int table_id = maat_get_table_id(maat_inst, table_name);
ASSERT_GT(table_id, 0);
int ret = maat_scan_string(maat_inst, table_id, string1, strlen(string1),
results, ARRAY_SIZE, &n_hit_result, state);
EXPECT_EQ(ret, MAAT_SCAN_HALF_HIT);
ret = maat_scan_string(maat_inst, table_id, string2, strlen(string2),
results, ARRAY_SIZE, &n_hit_result, state);
EXPECT_EQ(ret, MAAT_SCAN_HIT);
EXPECT_EQ(n_hit_result, 1);
EXPECT_EQ(results[0], 123);
maat_state_free(state);
```
## logic OR
rule = China | USA
```json
{
"compile_id": 124,
"service": 0,
"action": 0,
"do_blacklist": 0,
"do_log": 0,
"user_region": "null",
"is_valid": "yes",
"groups": [
{
"regions": [
{
"table_type": "expr",
"table_name": "HTTP_URL",
"table_content": {
"keywords": "China",
"expr_type": "none",
"format": "uncase plain",
"match_method": "prefix"
}
},
{
"table_type": "expr",
"table_name": "HTTP_URL",
"table_content": {
"keywords": "USA",
"expr_type": "none",
"format": "uncase plain",
"match_method": "prefix"
}
}
]
}
]
}
```
scan sample
```c
const char *string1 = "China";
const char *string2 = "USA";
long long results[ARRAY_SIZE] = {0};
size_t n_hit_result = 0;
int thread_id = 0;
const char *table_name = "HTTP_URL";
struct maat_state *state = maat_state_new(maat_inst, thread_id);
int table_id = maat_get_table_id(maat_inst, table_name);
ASSERT_GT(table_id, 0);
ret = maat_scan_string(maat_inst, table_id, string1, strlen(string1),
results, ARRAY_SIZE, &n_hit_result, state);
EXPECT_EQ(ret, MAAT_SCAN_HIT);
EXPECT_EQ(n_hit_result, 1);
EXPECT_EQ(results[0], 124);
maat_state_reset(state);
ret = maat_scan_string(maat_inst, table_id, string2, strlen(string2),
results, ARRAY_SIZE, &n_hit_result, state);
EXPECT_EQ(ret, MAAT_SCAN_HIT);
EXPECT_EQ(n_hit_result, 1);
EXPECT_EQ(results[0], 124);
maat_state_free(state);
```
## logic NOT
rule = China & !USA
```json
{
"compile_id": 125,
"service": 1,
"action": 1,
"do_blacklist": 1,
"do_log": 1,
"user_region": "null",
"is_valid": "yes",
"groups": [
{
"clause_index": 0,
"not_flag": 0,
"regions": [
{
"table_name": "HTTP_URL",
"table_type": "expr",
"table_content": {
"keywords": "China",
"expr_type": "none",
"match_method": "sub",
"format": "uncase plain"
}
}
]
},
{
"clause_index": 1,
"not_flag": 1,
"regions": [
{
"table_name": "HTTP_URL",
"table_type": "expr",
"table_content": {
"keywords": "USA",
"expr_type": "none",
"match_method": "sub",
"format": "uncase plain"
}
}
]
}
]
}
```
scan sample
```c
const char *string1 = "China";
const char *string2 = "England";
long long results[ARRAY_SIZE] = {0};
size_t n_hit_result = 0;
int thread_id = 0;
const char *table_name = "HTTP_URL";
struct maat_state *state = maat_state_new(maat_inst, thread_id);
int table_id = maat_get_table_id(maat_inst, table_name);
ASSERT_GT(table_id, 0);
int ret = maat_scan_string(maat_inst, table_id, string1, strlen(string1),
results, ARRAY_SIZE, &n_hit_result, state);
EXPECT_EQ(ret, MAAT_SCAN_HALF_HIT);
maat_state_set_last_scan(state);
ret = maat_scan_string(maat_inst, table_id, string2, strlen(string2),
results, ARRAY_SIZE, &n_hit_result, state);
EXPECT_EQ(ret, MAAT_SCAN_HIT);
EXPECT_EQ(n_hit_result, 1);
EXPECT_EQ(results[0], 125);
maat_state_free(state);
```
## group exclude
```json
{
"compile_id": 126,
"service": 1,
"action": 1,
"do_blacklist": 1,
"do_log": 1,
"user_region": "null",
"is_valid": "yes",
"groups": [
{
"group_name": "ExcludeGroup199",
"sub_groups":[
{
"group_name": "ExcludeGroup199_1",
"is_exclude": 0,
"clause_index": 0,
"regions": [
{
"table_name": "HTTP_URL",
"table_type": "expr",
"table_content": {
"keywords": "baidu.com",
"expr_type": "none",
"match_method": "suffix",
"format": "uncase plain"
}
}
]
},
{
"group_name": "ExcludeGroup199_2",
"is_exclude": 1,
"clause_index": 0,
"regions": [
{
"table_name": "HTTP_URL",
"table_type": "expr",
"table_content": {
"keywords": "www.baidu.com",
"expr_type": "none",
"match_method": "exact",
"format": "uncase plain"
}
}
]
}
]
}
]
}
```
scan sample
```c
const char *string_not_hit = "www.baidu.com";
const char *string_hit = "mail.baidu.com";
long long results[ARRAY_SIZE] = {0};
size_t n_hit_result = 0;
int thread_id = 0;
const char *table_name = "HTTP_URL";
struct maat_state *state = maat_state_new(maat_inst, thread_id);
int table_id = maat_get_table_id(maat_inst, table_name);
ASSERT_GT(not_hit_table_id, 0);
int ret = maat_scan_string(maat_inst, table_id, string_not_hit, strlen(string_not_hit),
results, ARRAY_SIZE, &n_hit_result, state);
EXPECT_EQ(ret, MAAT_SCAN_HALF_HIT);
ret = maat_scan_string(maat_inst, table_id, string_hit, strlen(string_hit),
results, ARRAY_SIZE, &n_hit_result, state);
EXPECT_EQ(ret, MAAT_SCAN_HIT);
EXPECT_EQ(n_hit_result, 1);
EXPECT_EQ(results[0], 126);
maat_state_free(state);
```

488
docs/maat_table.md Normal file
View File

@@ -0,0 +1,488 @@
# Maat table
The maat table consists of two parts: `schema` and `runtime`, which is the core skeleton of maat. In a production environment, maat periodically loads the configurations from redis and parses it according to the schema, building a table-based runtime for use by the scanning interface.
* [table schema](#1-table-schema)
* [table runtime](#2-table-runtime)
## 1. Table schema
Maat tables are divided into two categories: physical tables that actually exist in the database and virtual tables that reference physical tables.
The types of physical tables are as follows:
- [item table](#11-item-table)
- [compile table](#12-compile-table)
- [group2compile table](#13-group2compile-table)
- [group2group table](#14-group2group-table)
- [plugin table](#15-plugin-table)
- [ip_plugin table](#16-ip_plugin-table)
- [fqdn_plugin table](#17-fqdn_plugin-table)
- [bool_plugin table](#18-bool_plugin-table)
- [ipport_plugin table](#19-ipport_plugin-table)
Different physical tables can be combined into one table, see [conjunction table](#110-conjunction-table)
A virtual table can only reference one physical table or conjuntion table, see [virtual table](#111-virtual-table)
### 1.1 <a name='Itemtable'></a> Item table
Item tables are further subdivided into different types of subtables as follows:
- [expr item table](#111-expr-item-table)
- [expr_plus item table](#112-expr_plus-item-table)
- [ip item table](#113-ip-item-table)
- [interval item table](#114-interval-item-table)
- [interval_plus item table](#115-interval_plus-item-table)
- [flag item table](#116-flag-item-table)
- [flag_plus item table](#117-flag_plus-item-table)
Each item table must has the following columns:
- item_id: In a maat instance, the item id is globally unique, meaning that the item id of different tables must not be duplicate.
- group_id: Indicate the group to which the item belongs, an item belongs to only one group.
- is_valid: In incremental updates, 1(valid means add) 0(invalid means del)
The range of item_id(group_id, compile_id) is 0～2^63，which is 8 bytes.
#### 1.1.1 <a name='exprtable'></a> expr item table
Describe matching rules for strings.
| **FieldName** | **type** | **constraint** |
| ---------------- | -------------- | ------- |
| **item_id** | LONG LONG | primary key |
| **group_id** | LONG LONG | leaf group id, can be referenced by group2group & group2compile table |
| **keywords** | VARCHAR2(1024) | field to match during scanning |
| **expr_type** | INT | 0(keywords), 1(AND expr), 2(regular expr), 3(substring with offset)
| **match_method** | INT | only useful when expr_type is 0. 0(sub), 1(suffix), 2(prefix), 3(exactly) |
| **is_hexbin** | INT | 0(not HEX & case insensitive, this is default value) 1(HEX & case sensitive) 2(not HEX & case sensitive) |
| **is_valid** | INT | 0(invalid), 1(valid) |
The table schema is stored in table_info.conf.
```c
{
"table_id":3, //[0 ~ 1023], don't allow duplicate
"table_name":"HTTP_URL", //db table's name
"table_type":"expr",
"valid_column":7, //7th column(is_valid field)
"custom": {
"item_id":1, //1st column(item_id field)
"group_id":2, //2nd column(group_id field)
"keywords":3, //3rd column(keywords field)
"expr_type":4, //4th column(expr_type field)
"match_method":5,//5th column(match_method field)
"is_hexbin":6 //6th column(is_hexbin field)
}
}
/* If you want to combine multiple physical tables into one table, db_tables should be added as follows.
The value of table_name can be a user-defined string, the value of db_tables is the table name that actually exists in database. */
{
"table_id":3, //[0 ~ 1023], don't allow duplicate
"table_name":"HTTP_REGION", //user-defined string
"db_tables":["HTTP_URL", "HTTP_HOST"],
"table_type":"expr",
"valid_column":7,
"custom": {
"item_id":1,
"group_id":2,
"keywords":3,
"expr_type":4,
"match_method":5,
"is_hexbin":6
}
}
```
`expr_type` column represents the expression type:
1. keywords matching(0), match_method column as follows
- substring matching (0)
For example: substring: "China", scan_data: "Hello China" will hit, "Hello World" will not hit
- suffix matching (1)
For example: suffix: ".baidu.com", scan_data: "www.baidu.com" will hit, "www.google.com" will not hit
- prefix matching (2)
For example: prefix: "^abc", scan_data: "abcdef" will hit, "1abcdef" will not hit
- exactly matching (3)
For example: string: "World", scan_data: "World" will hit, "Hello World" will not hit
2. AND expression(1), supports up to 8 substrings.
For example: AND expr: "yesterday&today", scan_data: "Goodbye yesterday, Hello today!" will hit, "Goodbye yesterday, Hello tomorrow!" will not hit.
3. Regular expression(2)
For example: Regex expr: "[W|world]", scan_data: "Hello world" will hit, "Hello World" will hit too.
4. substring matching with offset(3)
- offset start with 0, [offset_start, offset_end] closed interval
- multiple substrings with offset are logical AND
For example: substring expr: "1-1:48&3-4:4C4C", scan_data: "HELLO" will hit, "HLLO" will not hit.
**Note**: 48('H') 4C('L')
Since Maat4.0, only UTF-8 is supported, and encoding conversion is no longer supported. For binary format rules, keywords are represented in hexadecimal, such as the keyword "hello" being represented as "68656C6C6F". Keywords cannot contain invisible characters such as spaces, tabs, and carriage returns, which have ASCII codes from 0x00 to 0x1F and 0x7F. If these characters need to be used, they must be escaped, referring to the "keyword escape table". Characters led by backslashes outside this table are processed as ordinary strings, such as '\t' being processed as the string "\t".
The symbol '&' represents the conjunction operation in an AND expression. Therefore, if a keyword contains '&', it must be escaped as '\&'.
**keywords escape table**
| **symbol** | **ASCII code** | **symbol after escape** |
| ---------- | -------------- | ----------------------- |
| \ | 0x5c | \\\ |
| & | 0x26 | \\& |
| blank space| 0x20 | \b |
Length constraint：
- Single substring no less than 3 bytes
- No less than 3 bytes for a single substring in AND expression
- Support up to 8 substrings in one AND expression, expr = substr1 & substr2 & substr3 & substr4 & substr5 & substr6 & substr7 & substr8
- The length of one AND expression should not exceed 1024 bytes(including '&')
#### 1.1.2 <a name='ExprPlusItemTable'></a> expr_plus item table
Describe extended matching rules for strings by adding the district column.
| **FieldName** | **type** | **constraint** |
| ---------------- | -------------- | ------- |
| **item_id** | LONG LONG | primary key |
| **group_id** | LONG LONG | leaf group id, can be referenced by group2group & group2compile table |
| **district** | VARCHAR2(1024) | describe the effective position of the keywords |
| **keywords** | VARCHAR2(1024) | field to match during scanning |
| **expr_type** | INT | 0(keywords), 1(AND expr), 2(regular expr), 3(substring with offset)
| **match_method** | INT | only useful when expr_type is 0 |
| **is_hexbin** | INT | 0(not HEX & case insensitive, this is default value) 1(HEX & case sensitive) 2(not HEX & case sensitive) |
| **is_valid** | INT | 0(invalid), 1(valid) |
For example, if the district is User-Agent and keywords is Chrome, scanning in the following way will hit.
```c
const char *scan_data = "Chrome is fast";
const char *district = "User-Agent";
maat_state_set_scan_district(..., district, ...);
maat_scan_string(..., scan_data, ...)
```
#### 1.1.3 <a name='IPItemTable'></a> ip item table
Describe matching rules for IP address. Both the address and port are represented by string, IPv4 is dotted decimal and IPv6 is colon separated hexadecimal.
| **FieldName** | **type** | **constraint** |
| -------------- | ------------ | -------------- |
| **item_id** | LONG LONG | primary key |
| **group_id** | LONG LONG | leaf group id, can be referenced by group2group & group2compile table |
| **addr_type** | INT | Ipv4 = 4 Ipv6 = 6 |
| **addr_format**| VARCHAR2(40) | ip addr format, single/range/CIDR/mask |
| **ip1** | VARCHAR2(40) | start ip |
| **ip2** | VARCHAR2(40) | end ip |
| **is_valid** | INT | 0(invalid), 1(valid) |
#### 1.1.4 <a name='IntervalItemTable'></a> interval item table
Determine whether an integer is within a certain numerical range.
| **FieldName** | **type** | **constraint** |
| ---------------- | -------- | -------------- |
| **item_id** | INT | primary key |
| **group_id** | INT | leaf group id, can be referenced by group2group & group2compile table |
| **low_boundary** | INT | lower bound of the numerical range(including lb), 0 ~ (2^32 - 1)|
| **up_boundary** | INT | upper bound of the numerical range(including ub), 0 ~ (2^32 - 1)|
| **is_valid** | INT | 0(invalid), 1(valid) |
#### 1.1.5 <a name='IntervalPlusItemTable'></a> interval_plus item table
Describe extended matching rules for integer by adding the district column.
| **FieldName** | **type** | **constraint** |
| ---------------- | -------- | -------------- |
| **item_id** | INT | primary key |
| **group_id** | INT | leaf group id, can be referenced by group2group & group2compile table |
| **district** | VARCHAR2(1024)| describe the effective position of the keywords |
| **low_boundary** | INT | lower bound of the numerical range(including lb), 0 ~ (2^32 - 1)|
| **up_boundary** | INT | upper bound of the numerical range(including ub), 0 ~ (2^32 - 1)|
| **is_valid** | INT | 0(invalid), 1(valid) |
#### 1.1.6 <a name="FlagItemTable"></a> flag item table
| **FieldName** | **type** | **constraint** |
| ------------- | -------- | -------------- |
| **item_id** | INT | primary key |
| **group_id** | INT | leaf group id, can be referenced by group2group & group2compile table |
| **flag** | INT | flag, 0 ~ (2^32 - 1)|
| **flag_mask** | INT | flag_mask, 0 ~ (2^32 - 1)|
| **is_valid** | INT | 0(invalid), 1(valid) |
#### 1.1.7 <a name="FlagPlusItemTable"></a> flag_plus item table
| **FieldName** | **type** | **constraint** |
| ------------- | -------- | -------------- |
| **item_id** | INT | primary key |
| **group_id** | INT | leaf group id, can be referenced by group2group & group2compile table |
| **district** | INT | describe the effective position of the flag |
| **flag** | INT | flag, 0 ~ (2^32 - 1)|
| **flag_mask** | INT | flag_mask, 0 ~ (2^32 - 1)|
| **is_valid** | INT | 0(invalid), 1(valid) |
### 1.2 <a name='CompileTable'></a> compile table
Describe the specific policy, one maat instance can has multiple compile tables with different names.
| **FieldName** | **type** | **constraint** |
| -------------- | -------------- | --------------- |
| **compile_id** | LONG LONG | primary key, compile id |
| **tags** | VARCHAR2(1024) | default 0，means no tag |
| **is_valid** | INT | 0(invalid)，1(valid) |
| **clause_num** | INT | no more than 8 clauses |
### 1.3 <a name='Group2CompileTable'></a> group2compile table
Describe the relationship between group and compile.
| **FieldName** | **type** | **constraint** |
| ----------------- | ------------- | -------------- |
| **group_ids** | VARCHAR(256) | group ids are separated by commas(g1,g2,g3) |
| **compile_id** | LONG LONG | compile id |
| **is_valid** | INT | 0(invalid), 1(valid) |
| **not_flag** | INT | logical 'NOT', identify a NOT clause, 0(no) 1(yes) |
| **virtual_table** | VARCHAR2(256) | virtual table name, NOT NULL |
| **Nth_clause** | INT | the clause seq in (conjunctive normal form)CNF, from 0 to 7. groups with the same clause ID are logical 'OR' |
NOTE: If group_id is invalid in xx_item table, it must be marked as invalid in this table.
### 1.4 <a name='Group2GroupTable'></a> group2group table
Describe the relationship between groups.
| **FieldName** | **type** | **constraint** |
| ---------------------- | ------------ | ---------------|
| **group_id** | LONG LONG | reference from xx_item table's group_id |
| **incl_sub_group_ids** | VARCHAR(256) | included sub group ids are separated by commas(g1,g2,g3)|
| **excl_sub_group_ids** | VARCHAR(256) | excluded sub group ids are separated by commas(g4,g5)|
| **is_valid** | Bool | (invalid), 1(valid) |
### 1.5 <a name='PluginTable'></a> plugin table
There is no fixed rule format of the plugin table, which is determined by business side. The plugin table supports two sets of callback functions, registered with **maat_table_callback_register** and **maat_plugin_table_ex_schema_register** respectively.
```c
int maat_table_callback_register(struct maat *instance, int table_id,
maat_start_callback_t *start_cb,
maat_update_callback_t *update_cb,
maat_finish_callback_t *finish_cb,
void *u_para);
```
When the plugin table rules are updated, `start_cb` will be called first and only once, then `update_cb` will be called by each rule item, and `finish_cb` will be called last and only once.
If rules have been loaded but maat_table_callback_register has not yet been called, maat will cache the loaded rules and perform the callbacks(start, update, finish) when registration is complete.
This set of callbacks is concerned with changes to the table, including when the table starts to change (start_cb), the type of change (full or incremental), when the change ends (finish_cb), and the specific content of each change (update_cb).
```c
int maat_plugin_table_ex_schema_register(struct maat *instance, const char *table_name,
maat_ex_new_func_t *new_func,
maat_ex_free_func_t *free_func,
maat_ex_dup_func_t *dup_func,
long argl, void *argp);
```
This interface registers a set of callback functions for the xx_plugin table. Unlike the callbacks registered with `maat_table_callback_register`, when adding a configuration, the `new_func` is called immediately, and when deleting a configuration, the `free_func` is not called immediately due to the introduction of a garbage collection mechanism. Instead, the free_func is called when the garbage collection queue starts the collection process.
this set of callbacks is concerned with the specific configuration changes line by line, which configuration is added (new_func), which configuration is deleted (free_func), and which configuration can be queried for ex_data (dup_func).
```c
void *maat_plugin_table_get_ex_data(struct maat *instance, int table_id,
const char *key, size_t key_len);
```
Plugin table supports three types of keys to query ex_data.
1. Pointer key(compatible with maat3)
2. Integer key
3. Ipv4 or ipv6 address as key.
### 1.6 <a name='IpPluginTable'></a> ip_plugin table
Similar to plugin table but the key of maat_ip_plugin_table_get_ex_data is ip address.
### 1.7 <a name='FQDNPlugintable'></a> fqdn_plugin table
Scan the input string according to the domain name hierarchy '.'
Return results order: sort by decreasing the length of the hit rule
For example:
1. example.com.cn
2. com.cn
3. example.com.cn
4. cn
5. ample.com.cn
If the input string is example.com.cn, the expected result order would be: 3, 1, 2, 4. The 'ample' in rule 5 is not part of the domain hierarchy and should not be returned.
### 1.8 <a name='BoolPluginTable'></a> bool_plugin table
Scan the input integer array based on a boolean expression, such as [100, 1000, 2, 3].
The boolean expression rule is numbers separated by "&", for example, "1&2&1000".
### 1.9 <a name='IpPortPluginTable'></a>ipport_plugin table
Different from IPPlugin table, which uses ip as the key, IPPortPlugin table uses ip+port as the key, which can meet users' more refined ex_data query requirements. For example, by building a mapping from ip+port to subscriber ID, network traffic can be distributed based on subscriber ID.
### 1.10 <a name='ConjunctionTable'></a> conjunction table
By default, maat builds a separate runtime for each physical table, which can be used for rule matching by specifying the table ID during scanning. If the user wants to combine multiple physical tables of the same type into a single table for runtime build and scan, it means conjunction of multiple tables.
For example: HTTP_REGION is the conjunction of HTTP_URL and HTTP_HOST.
```json
{
"table_id":1,
"table_name":"HTTP_REGION",
"db_tables":["HTTP_URL", "HTTP_HOST"],
"table_type":"expr",
"valid_column":7,
"custom": {
"item_id":1,
"group_id":2,
"keywords":3,
"expr_type":4,
"match_method":5,
"is_hexbin":6
}
}
```
`Note`: Only physical tables support conjunction.
### 1.11 <a name='VirtualTable'></a> virtual table
A physical table refers to a table that physically exists in the database. In contrast, there are no virtual tables in the database. Virtual tables are merely references to physical tables, where one virtual table can only reference one physical table. If you want to reference multiple physical tables of the same type, you need to first combine these physical tables into a conjunction table, and then have the virtual table reference it. A physical table can be referenced by multiple virtual tables.
Virtual tables are often used for different traffic attributes, where different virtual tables represent different traffic attributes, such as HTTP_HOST, HTTP_URL, and so on.
### 1.12 <a name='ForeignFiles'></a>Foreign Files
In callback configurations, specific fields can point to external content, currently supporting pointing to a key in Redis.
The foreign key column in the callback table must have the prefix "redis://". The content stored in Redis as a foreign key must have the prefix "__FILE_". When the key is "null", it indicates that the file is empty.
For example, if the original file is ./testdata/mesa_logo.jpg, and after calculating its MD5 value, we get the Redis foreign key __FILE_795700c2e31f7de71a01e8350cf18525, the format written in the callback table would be as follows:
```
14 ./testdata/digest_test.data redis://__FILE_795700c2e31f7de71a01e8350cf18525 1
```
Each row in the callback table can have a maximum of 8 foreign keys, and the foreign key content can be set using the Maat_cmd_set_file function.
Before notifying the callback table, Maat fetches the foreign keys to local files and replaces the foreign key column with the local file path.
### 1.13 <a name='Tags'></a>Tags
By matching the tags accepted by Maat with the configuration tags, selective configuration loading is achieved. Configuration tags are a collection of tag arrays, denoted as "tag_sets", while Maat accepts tags are tag arrays denoted as "tags".
Configuration tags are tags stored on compilation configurations or group configurations, identifying where the configuration is effective in which Maat instances. It consists of multiple tag sets, where multiple tags within a set are ANDed, and multiple values of a tag are ORed.
## 2. Table runtime
Maat loads the configuration of different types of tables into memory to form the corresponding runtime for each table. We can see all table types from the table schema, and the runtime for the item table is similar, as it is an abstraction of the scanning engine. When we provide the data to be scanned and call the corresponding scanning interface, we can return whether the item is hit or not, and if it is hit, we can return the corresponding item’s group_id.
From the [configuration relationship](./overview.md#12-configuration-relationship) diagram, we can see how the hit group is referenced by other groups or compiles. If a hit group is referenced by other groups or compiles, there will be one or more hit paths that follow the `item_id -> group_id` {-> super group_id} `-> compile_id`. This requires two special runtimes: group2group_runtime and compile_runtime.
Based on this, we can divide the runtime into the following three categories:
1. item table runtime
* expr_runtime
* ip_runtime
* flag_runtime
* interval_runtime
2. group & compile table runtime
* group2group_runtime
* compile_runtime
3. xx_plugin table runtime
* plugin_runtime
* ip_plugin_runtime
* fqdn_plugin_runtime
* bool_plugin_runtime
* ipport_plugin_runtime
### 2.1 item table runtime
<img src="./imgs/expr-runtime.png" width="300" height="300" >
Among the four types of runtimes mentioned above, `expr_runtime` is relatively unique. Its expr_matcher supports two types of scanning engines: `hyperscan` and `rulescan`. Other xx_runtime directly calls the corresponding xx_matcher to obtain scanning results.
**Note**: Due to the inability to unify the native rulescan usage with hyperscan, a partial refactoring has been done on rulescan. The refactored rulescan follows the same interface and usage as hyperscan, making it compatible with the design of the expr_matcher abstraction layer.
### 2.2 group & compile table runtime
#### 2.2.1 group2group runtime
The `group2group_runtime` is a runtime that is built based on the reference relationships between groups, which are stored in the [group2group table](#14-group2group-table). From the [group hierarchy](./group_hierarchy.md), we can understand that if a hit occurs in a leaf group that is referenced by other groups, there may be certain super groups that are also hit. This is exactly the functionality provided by this runtime.
#### 2.2.2 compile runtime
In addition to the compile table, there is also the group2compile table in the table schema. However, from a runtime perspective, the configurations of these two tables together constitute compile_runtime. This means that there is no standalone group2compile_runtime. Compile_runtime is the most complex among all runtime types because it serves multiple functions.
**Note:** This will involve the terminology of [clause](./terminology.md#clause).
1. For expressions without NOT-clauses, returning the matched compile_id:
* compile1 = clause1 & clause2 = {vtable1, g1} & {vtable2, g2}
* compile2 = clause1 & clause2 = {vtable1, g2} & {vtable2, g3}
Given the matched vtable_id and group_id, all matching compile_ids can be provided. For example, if scanning vtable1 matches g2 and vtable2 matches g3, compile_runtime will return the matched compile_id 2.
2. For expressions with NOT-clauses, returning the matched compile_id:
* compile3 = clause1 & !clause2 = {vtable1, g1} & !{vtable2, g2}
* compile4 = !clause1 & clause2 = !{vtable1, g2} & {vtable2, g3}
If scanning vtable1 matches g1 and vtable2 matches g3, compile_runtime will return the matched compile_id 4.
3. If a compile_id is matched, the full hit path can be obtained: **item_id -> group_id ->** {super_group_id} -> clause{**vtable_id, not_flag, clause_index} -> compile_id**. If the matched group is not referenced by a compile, a half hit path can be obtained: **item_id -> group_id** -> {super_group_id}.
4. Getting the matched group_ids and the count of hit groups.
The internal structure of compile_runtime is as follows, including the control plane for configuration loading and the data plane for external calls.
![compile runtime](./imgs/compile-runtime.png)
* **Control plane**
Compile runtime loads the compile table and group2compile table configurations into memory, assigning a unique clause_id to all clauses of each compile. The following three parts are constructed based on the clause_id:
1. All clause_ids under the same compile are used to construct AND expressions, and all compile AND expressions are used to build a bool_matcher.
2. For not_flag=0 (clauses), a `clause_id hash` is built, key:{group_id, vtable_id, not_flag}, value:clause_id.
3. For not_flag=1 (NOT-clauses), a `NOT_clause_id hash` is built, key:{group_id, vtable_id, not_flag}, value:clause_id.
* **Data Plane**
On the data plane, services are provided externally through the maat API, primarily with the following three types of interfaces:
1. maat_scan_xx: This interface dynamically generates the matched {item_id, group_id}.
* The matched item_id and group_id form a half-hit path.
* The group_id that is matched and the scanned `vtable_id` form the key {group_id, vtable_id, 0}. This key is used to find the matched clause_id in the clause_id hash. The final matched

110
docs/monitor_tools.md Normal file
View File

@@ -0,0 +1,110 @@
# Monitor tools
## 1. maat_stat
Maat outputs statistical information in running state using fieldstat. To make it easier to view the output in a more readable format, the fieldstat_exporter.py script is provided. This script can be used to convert the JSON format output (stat.log) into a more user-friendly format.
```bash
python3 /opt/MESA/bin/fieldstat_exporter.py local -j ./stat.log -e -t '{{print_tables("TBL",["rule","reg/v6","scan_times","hit_times","scan_bytes","scan_cpu_time","hit_item_num","hit_pat_num","#Name<IN_Bps>#Ratio<scan_bytes,scan_cpu_time,100>","#Name<hit_rate>#Ratio<hit_times,scan_times,1>"],False)}}'
```
![maat_stat](./imgs/maat_stat.png)
The statistical information in the above figure is divided into two parts, part1 provides overall statistics, and part2 provides statistics for each table.
**part1**
- version: current configuration version
- thread_num: number of scan threads
- table_num: number of tables loaded by maat
- plug_cached_num: number of cached configurations for plugin tables with unregistered callback functions
- plug_acc_num: total number of configurations for plugin tables
- NOT_clause_num: total number of NOT-clauses
- excl_grp_num: total number of exclude groups
- garbage_queue_len: number of elements in the garbage queue
- hit_compile_num: total number of hits on compile
- state_num: total number of current maat_state
- per_state_num: number of bytes occupied by each maat_state on average
- compile_state_num: total number of current compile_state created within maat_state during partial hits or hits
- stream_num: total number of scanned streams
- z_stream_num: number of zombie streams
- update_error: number of configuration update errors
- scan_error: number of scan errors
**part2**
`Note`: The statistics are output in 2-second cycles, so some statistical indicators represent values for a 2-second period. The previous 2-second statistical values are cleared after output, and the next 2-second statistics are calculated.
- rule: number of rules in this table
- reg/v6: Only valid for expr/expr_plus/ip_plus tables. For expr tables, it represents the number of regular expression; for ip_plus tables, it represents the number of IPv6 rules
- scan_times: Number of times this table is scanned within 2 seconds
- hit_times: Number of times this table is scanned and hit within 2 seconds
- scan_bytes: Total number of bytes scanned in this table within 2 seconds
- scan_cpu_time: CPU time spent on scanning this table within 2 seconds
- hit_item_num: Number of items hit in this table within 2 seconds
- hit_pat_num: Only valid for expr/expr_plus tables, number of patterns hit in this table within 2 seconds (each rule in an expr/expr_plus table, if it is an AND expression, each item in the AND expression is a pattern)
- IN_Bps: Scanning bandwidth
- hit_rate: hit_times/scan_times, indicating the hit rate of the scan
## 2. maat_redis_tool
* Dump the configuration in redis to a local iris format file.
* Quickly check if there are any issues in the process of pulling full configuration from redis.
* Write the configuration in json format to redis. (Not Recommended)
For specific usage, run ./maat_redis_tool -h
## 3. parse the json configuration that has been gzip compressed
compress json configuration using gzip command
```shell
$ gzip -9 < maat_json.json > maat_json.json.gz
```
```c
maat_options_set_json_file_gzip_flag()
```
This api can indicate whether maat instance needs to decompress the json configuration before loading it. It can be used in conjunction with the encryption command below, compressed first and then encrypted.
## 4. parse the json configuration encrypted by openssl
Using openssl enc to encrypt configuration：
```shell
$ openssl enc -e -aes-128-cbc -k 123456 -p -nosalt -in inputfile.txt -out encryptfile.txt
```
decrypt：
```shell
$ openssl enc -d -aes-128-cbc -k 123456 -p -nosalt -in encryptfile.txt -out tmp.txt
```
```c
maat_options_set_json_file_decrypt_key()
```
This api can specify the decryption key for the JSON file to be decrypted.

188
docs/overview.md Normal file
View File

@@ -0,0 +1,188 @@
# Overview
## 1. Introduction
Before proceeding, please make sure you are familiar with the [terminology](./terminology.md) related to maat. In the context of maat, configuration can be equivalently understood as rule.
As mentioned in the readme, maat has two typical usage patterns:
**Pattern 1**
* Update rules in the item table, group2compile table, and compile table
* Call the maat scanning api to determine if the actual traffic hits the effective rules
* If a rule is hit, maat can provide detailed information about the hit rule
**Pattern 2**
* Register callback functions for xx_plugin table
* Update rules in the xx_plugin table
* Call xx_plugin_get_ex_data to query the ex_data for a specific key
### 1.1 Configuration
Different types of configurations are stored in different tables. For all configuration types, please refer to the [table schema](./maat_table.md#1-table-schema).
The physical tables are mainly divided into three categories: the item table, group compile relationship table (compile table, group2compile table, group2group table), and xx_plugin table. The first two types of tables are used for maat traffic scanning, while the xx_plugin table is used as a callback table, which can obtain the detailed configuration information for a specific key.
### 1.2 Configuration relationship
As shown in the diagram below, maat organizes and abstracts configurations using terms such as item, group, literal, clause, compile, etc., allowing users to flexibly configure various policies. The term "literal" is an internal concept in maat and is not visible to external users.
In addition, groups support nesting. For more detailed information, please refer to [group hierarchy](./group_hierarchy.md).
If we define literal_id = {virtual_table_id, group_id}, then a literal is composed of one or more literal_ids. The multiple literal_ids that form the same clause have a logical “OR” relationship. The multiple clauses that form the same compile have a logical “AND” relationship, and there can be a maximum of 8 clauses within the same compile. In addition, the clause itself supports logical "NOT".
<img src="./imgs/rule_diagram.png" width="800" height="450" >
### 1.3 Dynamic configuration management
Maat supports three configuration loading modes.
* **Redis mode**(for production): The data source is typically a relational database, such as Oracle, MySQL.
* **Json file mode**(for production and debugging): it's mainly used in unit test
* **Iris file mode**(for troubleshooting)
Redis mode and Json file mode support dynamic loading of configurations, where configurations for different tables are loaded into memory to generate corresponding table runtimes. If you are already familiar with the [thread model](./thread_model.md) of maat, you’ll know that when a maat instance is created, a monitor_loop thread runs in the background. This thread periodically checks for configuration changes and generates new runtimes. Additionally, since the scanning interface also requires access to the runtime, maat uses the RCU(Read-Copy-Update) mechanism to ensure high performance of the scanning. For more details, please refer to [table runtime](./maat_table.md#2-table-runtime).
## 2. High level architecture
<img src="./imgs/maat-architect.png" width="800" height="480" >
As indicated by the maat [thread model](./thread_model.md), upon creating a maat instance, a `monitor_loop` thread will be created in the background for dynamic configuration updates. Threads calling maat's scanning interface are created by the caller.
The diagram illustrates the overall architecture of maat, including the control plane for configuration updates and the data plane for external calls.
* **Control Plane**
As mentioned earlier, maat supports three configuration loading mode, using redis as an example here. The maat `monitor_loop` thread periodically checks for updates in the redis. Updating the configuration of a specific table generates the corresponding updating runtime. When updating is complete, a commit operation is triggered, transforming the updating runtime into a new effective runtime, while the original effective runtime is put into the garbage collection queue.
* **Data Plane**
When calling the maat scanning interface, it subsequently calls the table runtime of the corresponding table, then proceeds to the scanning engine. Upon the scanning engine returning a hit group, it further searches for the matching `compile_id` through group2group runtime, group2compile runtime, and compile runtime, which is then returned to the caller. In addition, if the caller is interested in the hit path, they can also retrieve it through the interfaces provided by maat.
The scanning mentioned above all uses the `effective runtime`. If there are configuration changes, it will trigger the construction of `updating runtime`. Once this construction is completed, it will become effective runtime, and the original effective runtime will be put into the garbage collection queue waiting to be recycled.
## 3. Features
* RCU (Read-Copy-Update): From the maat thread model, it is evident that maat follows a typical single-writer-multiple-readers model, making it well-suited for utilizing RCU to avoid potential lock mechanisms. This allows the maat scanning interface to offer higher performance.
* Garbage collection: To ensure high performance of the scanning interface, maat puts old runtime into a garbage collection queue to periodically reclaim memory resources.
* Per-thread scanning: The maat scanning interface operates on a per-thread basis, requiring the thread_id as input parameter when used in a multi-threaded environment. Each thread's scanning is independent and does not interfere with others, ensuring complete isolation.
* Two expression scanning engines (hyperscan & rulescan): `hyperscan` outperforms `rulescan` in terms of scanning performance, especially in regular expression matching. However, the build time becomes unacceptable when the number of configurations exceeds 50k.
**_Note_**: _Hyperscan engine is always used for regular expressions._
Maat supports two engine switching modes: auto mode(default) and user-specified mode.
**Auto Mode**
When the number of literal string configurations is less than 50k, hyperscan is used; otherwise, rulescan is used.
**User-specified Mode**
* Method1: By calling the `maat_options_set_expr_engine` interface, you can specify the engine used for all expr tables at once.
* Method2: By configuring the `expr_engine` field in the table schema, you can specify the engine used for the table.
*_Note_*: _Method2 takes precedence over Method1_.
* Streaming-based Scanning: For more information on stream-based scanning, please refer [hyperscan](https://intel.github.io/hyperscan/dev-reference/runtime.html#runtime)
## 4. Tools
Maat provides a [command-line tool](./monitor_tools.md#2-maat_redis_tool) that can pull remote redis configurations to local files in iris format, allowing for full-text search or viewing of configurations, which is extremely useful for troubleshooting. Imagine without this tool, you would need to log in to redis and could only view the contents of specified keys. Even worse, there might be someone else making changes to the redis configuration, leading to unpredictable outcomes.
## 5. Test
There are a bunch of [unit tests](../test/) that test specific features of maat.
* maat_framework_gtest: The functional test set is used to ensure the correctness of the maat function. If there are any changes to the maat code, please ensure that these existing cases can run smoothly.
maat_framework_perf_gtest: The performance test set is mainly used to test the time consumption and bandwidth of the maat scanning interface.
* benchmark: Maat performance benchmark test, testing the scanning time of different scanning interfaces under different scale rule sets.
* group_nesting: Functionality and performance test set for group nesting.
* ipport_plugin: Functionality and performance test set for the ipport_plugin table.
## 6. Performance
All of the benchmarks are run on the same TSG-X device. Here are the details of the test setup:
**CPU**: Intel(R) Xeon(R) Gold 5318Y CPU @ 2.10GHz
**RAM**: 128G
```bash
scan consume: The time consumption of a single scan, calculated as the average of 1,000,000 scans for each of the 5 threads.
In scenarios with different numbers of rules, each rule hits only one item → one compile.
Configuration capacity:
IP: 8,000,000
URL: 2,000,000
Interval: 10,000
FQDN: 2,000,000
Regex: 20,000
```
### 1. literal string scanning
Rule generation method: Random strings up to 64 bytes in length.
|rule_num|build consume(ms)|scan consume(us)|scan_count/PerSec|
|--------|-----------------|----------------|-----------------|
|1k |27 |1.0 |1,010,305 |
|5k |115 |1.1 |865,950 |
|10k |217 |1.2 |813,272 |
|50k |1,325 |1.8 |536,883 |
|100k |2,998 |1.7 |601,539 |
|500k |17,971 |1.7 |560,663 |
|1M |38,246 |3.3 |300,607 |
|2M |81,093 |3.4 |293,186 |
### 2. literal stream scanning
|rule_num|build consume(ms)|scan consume(us)|scan_count/PerSec|
|--------|-----------------|----------------|-----------------|
|1k |24 |0.9 |1,078,981 |
|5k |107 |1.0 |975,039 |
|10k |214 |1.0 |948,946 |
|50k |1,286 |1.3 |739,973 |
|100k |2,858 |1.6 |617,436 |
|500k |17,695 |2.3 |426,693 |
|1M |38,227 |4.3 |229,484 |
|2M |79,294 |4.4 |222,074 |
### 3. ip scanning
|rule_num|build consume(ms)|scan consume(us)|scan_count/PerSec|
|--------|-----------------|----------------|-----------------|
|1k |10 |0.5 |1,745,810 |
|5k |20 |0.5 |1,727,712 |
|10k |31 |0.6 |1,669,449 |
|50k |132 |0.6 |1,675,603 |
|100k |250 |0.6 |1,682,368 |
|500k |1,382 |0.6 |1,658,925 |
|1M |3,032 |0.6 |1,696,640 |
|5M |19,823 |0.6 |1,715,265 |
|10M |41,757 |0.6 |1,634,521 |
### 4. interval scanning
|rule_num|build consume(ms)|scan consume(us)|scan_count/PerSec|
|--------|-----------------|----------------|-----------------|
|1k |0.1 |0.7 |1,437,607 |
|5k |0.5 |0.7 |1,406,865 |
|10k |0.9 |0.7 |1,285,677 |
### 5. flag scanning
|rule_num|build consume(ms)|scan consume(us)|scan_count/PerSec|
|--------|-----------------|----------------|-----------------|
|1k |0.002 |4.0 |247,561 |
|5k |0.019 |15 |63,476 |
|10k |0.1 |30 |32,900 |
### 6. ipport_plugin
|rule_num|build consume(ms)|scan consume(us)|scan_count/PerSec|
|--------|-----------------|----------------|-----------------|
|50k |100 |0.21 |4,629,629 |

112
docs/scan_api.md
View File

@@ -1,112 +0,0 @@
# Scan API
## 最大返回结果限制
多命中情况下，按包含分组数由多到少返回，分组数相同时，按编译配置ID由大到小的顺序返回。
多命中扫描的最大命中次数，受MAX_HIT_NUM宏控制，当前为4096条。
如果命中条数超出4096，则按照配置在IRIS库表文件出现的顺序返回。
## 流式跨包扫描
在配置表描述文件中，如果指定某个字符串类域表设置的跨包缓存值且该值大于0，则在为该表开启流式跨包扫描功能。
进行流式扫描中，会对输入数据结束后缓存该值大小内容，以便下次扫描时进行拼接。会增加一次内存拷贝，并且该拷贝内存在下次扫描时复用。由此决定了下文中Maat_hit_detail_t结构体中命中位置指针的生存周期。
为了避免流式跨包扫描对带偏移量的字符串匹配的功能的干扰，流式扫描结构体中包含了一个当前已扫描长度的变量，用以显示表识输入数据的绝对偏移量。
## 结果缓存复用
另外为了提高系统的性能，为每个scanner分配了一个region_rslt_buff，用以缓存域扫描命中的中间结果。通过malloc分配，长度为sizeof(scan_result_t)*MAX_HIT_NUM*scan_thread_num，当外部调用扫描时，根据thread_num参数，分配region_rslt_buff+MAX_HIT_NUM*thread_num供当前线程保存域中间扫描结果。
### 命中结果返回条件
Maat_xx_scan函数输入的数据命中了某个域配置，且满足某一编译配置就会返回该编译配置ID，无论该编译配置是否已在之前扫描中命中。
### 命中结果执行顺序
多命中情况下，编译配置的按照如下顺序返回：
1. 两条编译配置的执行序号（exec_seq）都不为0，执行序号较小的在前；
2. 执行序号相同，或者有任意一条编译配置的执行序号为0时，依次按照以下顺序返回：
1. 分组数量较多的在前
2. 编译配置ID较大的在前
## 命中路径
从待扫描数据的角度，描述其域ID->子分组ID->顶级分组ID（含虚拟表名）->编译配置的ID序列，称为命中路径。
一个待扫描数据在以下情况下有多个命中路径：
- 多规则命中：输入同时命中多个region ID。如
- 输入URL: abc.com/index.html；规则r1=”abc”，r2=”index.html”；输入有两个分别以r1和r2为起点的命中路径。
- 分组复用
- 分组间“与”运算
策略的结构是从compile到item的自顶向下，而命中路径是从最底层的item到顶层的compile，是自底向上的。它将网状和层次结构的object-policy图，进行扁平化，转换为图论中的路径。例如，有如下策略：
- 字符串object1，包含 item1="abc"
- 字符串object2，包含item2=".com"
- 字符串object3，包含object2
- IP object4，包含item3=192.168.0.1/24
- IP object5，包含object4
- IP object6，包含object5
- rule1=object1 & object6
- rule2=object1
输入：url=abc.com，ua=chrome80，IP=192.168.0.1的命中路径为
| 输入 | Region ID | Sub Group ID | Top Group ID | Virutal Table | Rule ID |
| ----------- | --------- | ----------------------- | ------------ | ------------- | ------- |
| abc.com | item1 | object1 | object1 | HTTP_URL | Rule1 |
| item2 | object2 | N/A（因为没有最终命中） | N/A | N/A | |
| item1 | object1 | object1 | HTTP_URL | Rule2 | |
| 192.168.0.1 | item3 | object4 | object6 | IP | Rule2 |
| chrome80 | N/A | N/A | N/A | N/A | N/A |
命中关系还是一张图，采用邻接矩阵存储，而命中路径是邻接矩阵中的一行。
![Hit Path as a Graph](./imgs/hitpath.png)
## 组合扫描
同一网络传输单元的多个属性的扫描，是通过scan_status* mid实现的，其中缓存了已命中的子句id，该ID在增量更新中不重用。
在一个mid的生存期内，配置组合关系变化产生新的bool matcher。
- 如果是增量更新，缓存的子句id会和新的bool matcher进行运算。由于子句ID在增量更新中不变化，不会产生误命中。
- 如果是全量更新，缓存的子句id不会和新的bool matcher进行运算。
## 常见故障处理
### 初始化失败
当出现Maat初始化失败、扫描不命中时，首先查看Maat生成的日志文件，FATAL级别的日志都会导致Maat不能正确加载、扫描配置。
日志提示“config file xxx,not full or inc config”，请确认Maat_ feather/Maat_summon_feather/Maat_summon_feather_json初始化时传入的两个路径是否正确，注意路径为IRIS配置线写入的index路径，例如/home/ config/dest/index/，而不是/home /config/dest/。
该日志文件由Maat_summon_feather函数调用时，传递的日志句柄所生成，需要注意的是该句柄的日志级别过高会导致”INFO”级日志不可见。
### 扫描出现段错误
确认扫描传入的maat_state* state已在调用前创建。
确认输入的待扫描内容的内存可访问，长度参数正确。
### 扫描不命中
扫描不命中，首先查看初始化日志，判断加载的配置是否是你期待的路径。
然后，判断配置加载是否有FATAL级别的错误日志。
接下来确认扫描时传入的table_id，与期待的表名是否正确对应，避免出现用表A的配置扫描表B的字段。
对于字符串扫描，charset参数传递正确，且包含在配置表描述文件（通常为table_info.conf）的”目的编码”中。如果不能确定，可以暂时修改配置表描述文件中的do_merge列为”yes”。
在使用带有命中细节的函数扫描时，一般detail_num应与rule_num相等。
IRIS文件模式下，如待命中配置在增量索引中，需要等待延迟加载（日志中Postpone xx entries）完成。调试时，可将MAAT_OPT_SCANDIR_INTERVAL_MS 和MAAT_OPT_EFFECT_INVERVAL_MS设置为20毫秒。
另外，JSON调试只能支持分组的模式下工作，需要在配置表描述文件table_info.conf中增加分组表，否则可能导致误命中或不命中。

507
docs/table_data.md
View File

@@ -1,507 +0,0 @@
# Table Data
Input must use UTF-8 without BOM encoding, for example, MySQL use utf8mb4.
Maat supports three rule loading modes.
- [Redis mode](#1-redis-mode)
- [Iris mode](#2-iris-mode)
- [Json mode](#3-json-mode)
## 1.<a name='Redis mode'></a> Redis mode
Maat可以通过Redis的主从同步机制，实现配置的分发。本节介绍MAAT加载Redis中配置时，对存储结构的要求。和数据库一样，Redis存储结构的设计上不需要考虑编译、分组和域的逻辑层次。由配置更新线程，通过行列式配置重构各层次间的组合关系。
![Sync Rule with Redis](./imgs/data-sync-with-redis.png)
### 1.1 Transactional Write
表 26 MAAT Redis中定义的数据结构
| Redis KEY | 名称 | 结构 | 用途 |
| ------------------------------------------------------ | ---------------- | -------------- | ------------------------- |
| MAAT_VERSION | primary version | INTERGER | 标识Redis中配置的版本号。当redis中版本号大于MAAT中配置版本号时，会去读取MAAT_UPDATE_STATUS。 |
| MAAT_PRE_VERSION | 预备版本 | INTERGER | |
| MAAT_TRANSACTION_xx | 事务配置状态 | LIST | 用于临时存储事务中的配置状态，xx为MAAT_PRE_VERSION其中的状态在事务结束后会被更新到MAAT_UPDATE_STATUS，本身被删除。 |
| MAAT_UPDATE_STATUS | 配置状态 | sorted set, member是配置规则，score 为版本号，详见11.3 | MAAT会用ZRANGEBYSCORE命令读取。 |
| MAAT_RULE_TIMER | 主配置超时信息 | sorted set, member是配置规则，score为超时间，详见11.4 | MAAT配置更新线程会定时检查超时状况，并设置超时状态。 |
| MAAT_VERSION_TIMER | 版本创建时间 | sorted Set | 存储了每个版本的创建时间，score为版本创建时间，member 为version，用以将MAAT_UPDATE_STATUS维持在一个较小的规模。 |
| MAAT_LABEL_INDEX | 标签索引 | sorted set, element 是配置表名，编译配置ID，score为label_id | |
| EFFECTIVE_RULE:TableName,ID OBSOLETE_RULE:TableName,ID | 主配置 |string | 生效中的配置，结构与10.3中的行结构相同，MAAT会逐条加载。 |
| SEQUENCE_ REGION | 域ID生成序列号 | INTERGER | 用于生产者生成不重复的region_id |
| SEQUENCE_ GROUP | 分组ID生成序列号 | INTERGER | 用于生产者生成不重复的group_id |
| EXPIRE_OP_LOCK | 分布式锁 | 字符串”locked" | 用于保证最多只有一个写者进行淘汰。 |
Maat command API 可直接将配置写入 redis
```c
struct maat_cmd_line {
const char *table_name;
const char *table_line;
long long rule_id; // for MAAT_OP_DEL, only rule_id and table_name are necessary.
int expire_after; //expired after $timeout$ seconds, set to 0 for never timeout.
};
int maat_cmd_set_line(struct maat *maat_inst, const struct maat_cmd_line *line_rule);
Example:
char table_line[1024] = {0};
long long item_id = 100;
long long group_id = 200;
const char *keywords = "Hello&Maat";
int expr_type = 1; //EXPR_TYPE_AND
int match_method = 0; //MATCH_METHOD_SUB
int is_hexbin = 0;
int op = 1; //add
sprintf(table_line, "%lld\t%lld\t%s\t%d\t%d\t%d\t%d", item_id, group_id,
keywords, expr_type, match_method, is_hexbin, op);
struct maat_cmd_line line_rule;
line_rule.rule_id = item_id;
line_rule.table_line = table_line;
line_rule.table_name = table_name;
line_rule.expire_after = expire_after;
int ret = maat_cmd_set_line(maat_inst, &line_rule);
```
### 1.2 主版本号、预备版本号与Lua Script
生产者写入配置时，先对预备版本号加1，并作为写入配置状态的score，待写入完成后，再对主版本号加1。放弃WATCH MAAT_VERSION的事务。这一方法可以大幅提高写入性能，除ID冲突外，可确保写入成功。
当有多个生产者时，可能存在配置状态与主版本号不一致的问题。主版本号为v，某次更新时在配置状态中声明的版本号为u，消费者增量更新时有以下情况：
- 若v=u，则版本号一致，配置正常加载；
- 若v>u，该情况不存在。因为只有配置状态修改完成，主版本号才会增加1。换句话说，每次写入都是先增加预备版本号，后增加主版本号，所以主版本号必然小于或等于配置状态中的最大版本号。
- 错误：三个生产者情况下，有问题，如下表。
- 若v<u，说明两个生产者中，先启动写入的并没有先完成。此时，本次只更新到版本v，留待下次轮询再更新至u。
消费者全量更新时不看配置状态，直接读取全部有效配置，因为配置写入和主版本号增加1在同一个事务中执行，读取到的全量配置版本必定与主版本一致。
有多个生产者的情况下，可能丢失配置更新消息状态：
| **Time** | **Producer1** | **Producer2** | **Producer3** | **consmuer** |
| -------- | ------------------------ | ------------------------ | ------------------------ | ----------------------------------------------- |
| **0** | 准备更新mv=3924, tv=3925 | | | |
| **1** | | 准备更新mv=3924, tv=3926 | | |
| **2** | | | 准备更新mv=3924, tv=3927 | |
| **3** | | | 更新完毕mv=3925, tv=3927 | Get version 3925, zrangebyscore拿不到3925的状态 |
| **4** | | 更新完毕mv=3926, tv=3926 | | Maat版本号升到3926，报错:noncontigous |
| **5** | 更新完毕mv=3927, tv=3925 | | | 3925被跳过 |
在事务结束部分，采用lua script检查事务版本号transaction_version与主版本号maat_version：
- tv==mv，无需修正
- tv>mv，本次更新的增量将在下一次
- tv<mv，如何识别本次事务写入配置状态的规则呢？然后才能将其score改为mv
为了解决事物结束时，transaction version<maat_version的问题，使用redis list MAAT_TRANSACTION_xx存储配置更新状态，xx取自MAAT_PRE_VERSION，事务结束时再用lua script同步MAAT_UPDATE_STATUS，并删除MAAT_TRANSACTION_xx。
![Add and Delete Operation](./imgs/add-del-rule-with-redis.png)
#### 1.3 MAAT_UPDATE_STATUS
该结构中使用Sorted Set存储了主配置的变化状态，score为版本号，member为配置状态。member的0～2字节描述了更新指令：
1. ADD，即配置增加，结构为ADD,TableName,ID；
2. DEL，即配置删除，结构为DEL,TableName,ID；
MAAT在发现MAAT_VERSION变化后，会用ZRANGEBYSCORE读取更新的配置状态（按VERSION升序），并检测第一个配置的Score，如该Score>Maat版本+1，则说明有遗漏的更新（网络长时间中断），启用全量更新流程。
对于DEL状态，如果查询不到对应的主配置状态，同样说明有遗漏更新（网络中断时间超过OBSOLETE_RULE超时时间），启用全量更新流程。
#### MAAT_EXPIRE_TIMER
该结构使用Sorted Set存储了主配置的超时信息，score为绝对超时时间，member的结构为TableName,ID。
#### MAAT_VERSION_TIMER
该结构使用Sorted Set存储了每个版本的创建时间，score为版本创建时间，member为 版本号（version），即MAAT_UPDATE_STATUS的score，用以将MAAT_UPDATE_STATUS维持在一个较小的规模。
#### 主配置结构
有两类配置命名方式：
1. EFFECTIVE_RULE:TableName,ID 表示正在生效的配置；
2. OBSOLETE_RULE:TableName,ID 表示已经删除的配置，这些配置超时（EXPIRE）后会被Redis删除。
### Load From Redis
Maat实例的工作线程定时轮询Redis中MAAT_VERSION，如果大于实例的MAAT_VERSION则进行更新。
![Load from Redis](./imgs/load-data-from-redis.png)
### 读写性能
为保证事务，Redis需工作在单机+主从模式。带超时的配置写入5000条/秒，无超时配置10000条/秒。
## 2.<a name='Iris mode'></a> Iris mode
在Maat可以监听全量和增量目录下的文件来更新配置运行时变化，下面对这种模式下的文件格式进行介绍。
配置是由一个索引文件和若干配置数据文件组成的。索引文件通过索引文件命名标识不同版本。
### 索引文件命名
全量索引文件名的格式为full_config_index.SEQ（如full_config_index. 00000000000000055410），增量索引文件名的格式为inc_config_index.SEQ，分别存放在全量索引约定目录和增量索引约定目录下。
其中SEQ为配置线维护的自增序列,该序列保证：
1） 当两个索引约定目录不为空时，新生成的索引文件比后生成的索引文件序号大；（建议使用数据库内的配置版本号或通过写入本地磁盘以保证一致性）
2） 同一次下发产生全量索引文件与增量索引文件的的序列号相同；
3） 序列号的格式为20位数字，不满20位时用0补位，取值范围不超过C语言中有符号长整型（long long）的上下界，即0~2^63-1。
### 索引文件格式
如下表所示。列间以\t分割，行间以\n分割。配置数据文件路径为绝对路径。
表 24索引文件格式/示例表
| **列名** | **table name** | **Config num** | **Table file path** | 加密算法(可为空) |
| -------- | -------------- | -------------- | ---------------------------------------------- | ---------------- |
| **示例** | PASS_WHITE_IP | 3 | /home/config/full/2014-04-24/PASS_WHITE_IP.123 | |
| | PASS_URL | 2 | /home/config/full/2014-04-24/PASS_URL.123 | aes-128-cbc |
### 配置数据文件格式
如下所示。列间用\t分割，行之间用\n分割。首行为当前数据文件中包含的配置总数，此值应以索引文件中对应的config_num一致。
表 25配置数据文件格式示例
```
3
45695 2 202.108.181.33 255.255.255.255 0 1
48697 2 202.108.181.33 255.255.255.255 0 1
66699 2 202.108.181.33 255.255.255.255 0 1
```
### 手工修改IRIS配置
通常**不建议**直接手工修改IRIS配置，调试应使用JSON模式。
增加库表需要修改以下文件：
1. 在库表文件索引中增加新的库表路径及条目数
2. 在库表描述文件（table_info.conf）中增加库表的信息
3. 修改配置汇总表，使新增加配置生效，注意要保证第一行配置总数与实际的配置行数一致
手工增加已有库表配置需要修改以下文件
1. 在需要修改的库表文件中追加行，region_id不能与已有所有域表中的冲突，并修改文件第一行的行数
2. 在配置汇总表中增加该配置的汇总信息，注意要和库表文件中的compile_id一致，且不能与已有compile_id冲突，修改文件第一行的行数
3. 在库表索引文件中修改配置汇总表和域表的行数
## 3.<a name='Json mode'></a> Json mode
使用Maat_summon_feather_jsonMaat_set_feather_opt函数通过选项MAAT_OPT_JSON_FILE_PATH设置，进行JSON格式配置的加载。Maat在初始化后，一旦检测到文件MD5值的变化，则以全量更新的方式加载变化的json文件。
采用JSON文件进行描述，纯文本可以使用JSON Viewer进行编辑和格式检查。
```json
{ //JSON语法不支持注释，直接将以下内容拷贝后，应删除注释后才符合JSON语法
//规定内容区分大小写，不可混用
"compile_table": "COMPILE",//编译表表名，优先级高于配置表描述中的compile表
"group_table": "GROUP",//分组表表名，优先级高于配置表描述中的group表
"rules": [
{
"compile_id": 123, //编译配置ID，数值，注意不要重复
"service": 1, //32位有符号整数
"action": 1, //8位有符号整数
"do_blacklist": 1, //8位有符号整数
"do_log": 1, //8位有符号整数
"effective_range": 0, //生效范围，置0，已被配置生效标签取代
"tags":"{\"tag_sets\":[[{\"tag\":\"location\",\"value\":[\"北京/朝阳/华严北里\",\"上海/浦东/陆家嘴\"]},{\"tag\":\"isp\",\"value\":[\"电信\",\"联通\"]}],[{\"tag\":\"location\",\"value\":[\"北京\"]},{\"tag\":\"isp\",\"value\":[\"联通\"]}]]}",
//tags为配置生效标签
"user_region": "anything",//用户自定义域，不超过128字节的字符串，不可//包含空格或制表符
"is_valid": "yes",//有效标志，有效：”yes”，无效：”no”
"evaluation_order": "100.2",//执行顺序，双精度浮点数，为了便于转换，用//字符串形式
"table_name": "COMPILE_ALIAS", //编译配置表名，可选，
//默认使用compile_table
"groups": [ //接下来描述配置包含的分组
{
"group_name": "IP_group",//分组名，用于辅助记忆，注意不要重复
//重复分组名表示分组内容复用。
//可以使用”Untitled”作为分组名表示无需 //复用，此时由Maat自动生成group_id。
//无group_name时，视为”Untitled” 。
"not_flag":0, //非运算标志位，1表示不能包含该分组，默认为0
“virtual_table”:”HTTP_RESPONSE_KEYWORDS”, //虚拟表名，可选
"regions": [ //对分组内配置域的描述
{
"table_name": "IP_CONFIG", //域表表名和类型，应与配置表
//描述文件一致
"table_type": "ip",//类型包括 ip, string, interval 三种
"table_content": {//以下是ip类配置表的格式
"addr_type": "ipv4",//地址类型，可选ipv4和ipv6
"src_ip": "10.0.6.201",
"mask_src_ip": "255.255.255.255",
"src_port": "0",
"mask_src_port": "65535",
"dst_ip": "0.0.0.0",
"mask_dst_ip": "255.255.255.255",
"dst_port": "0",
"mask_dst_port": "65535",
"protocol": 6,
"direction": "double"//扫描方向，single或double
}
},
{
"table_name": "IP_CONFIG",
"table_type": "ip",
"table_content": {
"addr_type": "ipv4",
"src_ip": "192.168.0.1",
"mask_src_ip": "255.255.255.255",
"src_port": "0",
"mask_src_port": "65535",
"dst_ip": "0.0.0.0",
"mask_dst_ip": "255.255.255.255",
"dst_port": "0",
"mask_dst_port": "65535",
"protocol": 6,
"direction": "double"
}
}
]
},
{
"group_name": "Untitled",//用”Untitled”作为分组名表示不需 //要复用
"regions": [
{
"table_name": "HTTP_URL",
"table_type": "string",
"table_content": {
"keywords": "abc&123",//格式遵循字符串类与配置一
//节的约定，”\”需要写作”\\”
"expr_type": "and",//表达式类型，包括none, and,
// regex, offset
"match_method": "sub",//匹配模式,包括sub, right,
//left, complete
"format": "uncase plain"//格式类型，包括uncase plain,
// hexbin, case plain
}
}
]
}
]
},
{
"compile_id": 124,
"service": 1,
"action": 1,
"do_blacklist": 1,
"do_log": 1,
"effective_range": 0,
"user_region": "anything",
"is_valid": "yes",
"groups": [
{
"group_name": "IP_group"//复用上一条编译配置的分组，仅列名字，
//重复分组名的域配置不会生效
},
{
"group_name": "Untitled",//Untitled表示该分组不会被复用
"regions": [
{
"table_name": "CONTENT_SIZE",
"table_type": "interval",
"table_content": {
"low_boundary": 100, //上下界的约定参照数值类域配
"up_boundary": 500 //置表定义
}
}
]
}
]
},
{
"compile_id": 125,
"service": 1,
"action": 1,
"do_blacklist": 1,
"do_log": 1,
"effective_range": 0,
"user_region": "anything",
"is_valid": "yes",
"groups": [
{
"group_name": "group_4",
"regions": [
{
"table_name": " TARGET_FILE_DIGEST ",
"table_type": "digest",
"table_content": {
" raw_len ": 20436318, //文件的原始长度
"digest": "VY2DfSUuQ5xkNQTnCN8iwd0VM5h3WVOunA/Jk87S3kfpukO84AL1qvmq2brnBt9sPvrfeaRip9Y+VhuEk0Sy4dvdKtk8DdSFOa7ZM[0:20436317]", //通过digest_gen工具生成的摘要
"cfds_level": 3 //摘要匹配置信度
}
}
]
}
]
},
{
"compile_id": 128,
"service": 1,
"action": 1,
"do_blacklist": 1,
"do_log": 1,
"effective_range": 0,
"user_region": "anything",
"is_valid": "yes",
"groups": [
{
"group_name": "group_8",
"regions": [
{
"table_name": "HTTP_REGION",
"table_type": "expr_plus", //扩展的字符串表
"table_content": {
"district": "URL", //keywords的匹配区域，大小写敏感
"keywords": "abckkk&123",
"expr_type": "and",
"match_method": "sub",
"format": "uncase plain"
}
}
]
}
]
}
],
"plugin_table":[ //回调表，可不配置
{
"table_name":"QD_ENTRY_INFO",
"table_content":[
"1\t192.168.0.1\t101",//回调表行，使用Tab键或’\t’分隔
"2\t192.168.0.2\t101",
"3\t192.168.1.1\t102" ]
},
{
"table_name":"TEST_PLUGIN_TABLE",
"table_content":[
"1\t3388\t99\t1",
"2\t3355\t66\t1",
"3\tcccc\t11\t1"
]
}
]
}
```
## 组合配置写入示例
已知IP类域配置表IP_TABLE、关键字类配置表KEYWORD、编译配置表COMPILE、分组表GROUP。
GRP为group_id的数据库sequence，REGION为region_id的数据库sequence，COMPLIE为compile_id的数据库sequence
先要求写入规则(IP_TABLE:192.168.0.1|IP_TABLE:192.168.0.2)&(KEYWORD:abc)
写入顺序如下：
1、 从数据库中取 group_id1=GRP.next, region_id1=REGION.next, compile_id1=COMPILE.next;
2、 将"group_id1,compile_id1,1"写入GROUP表;
3、 将"group_id1,region_id1,abc"写入KEYWORD表;
4、 从数据库中取 group_id2=GRP.next
5、 将"group_id2,compile_id1,1"写入GROUP表;
6、 对于IP地址192.168.0.1，从数据库获取region_id2=REGION.next；
7、 将"group_id2,region_id2,192.168.0.1"写入IP_TABLE表；
8、 对于IP地址192.168.0.2，从数据库中取 region_id3=REGION.next
9、 将"group_id2,region_id3,192.168.0.2"写入IP_TABLE表；
10、 将"compile_id1,group_num=2"写入COMPILE表；
此时各个表中的记录如下：
```
COMPILE表：
c1,2;
GROUP表:
g1,c1
g2,c1
KEYWORD表:
g1,r1,abc
IP_TABLE表:
g2,r2,192.168.0.1
g2,r3,192.168.0.2
```
新增一个关键字"edf"，也要使用IP组g2，则增加三条记录：
COMPILE:
c2;
GROUP:
g3,c2
g2,c2
KEYWORD:
g3,r5,edf
则当前库表记录状态为
```
COMPILE表：
c1,2;
c2,2;
GROUP表:
g1,c1
g2,c1
g3,c2
g2,c2
KEYWORD表:
g1,r1,abc
g3,r5,edf
IP_TABLE表:
g2,r3,192.168.0.2
g2,r4,192.168.0.3
```
新增IP
- 如果用户在group_id2分组中增加IP，则从数据库中取 region_id4=REGION.next，将"group_id2,region_id4,192.168.0.3"写入KEYWORD表；即可
删除IP
- 如果用户要删除group_id2分组中的IP192.168.0.1，则将IP_TABLE表中g2,r2,192.168.0.1 该条记录置为失效
## 配置更新机制
### 最短加载间隔
Maat的配置管理线程会针对增量索引文件目录进行扫描，在初始化后每隔1s（默认），扫描增量配置目录。同时为了避免自动机频繁更新，耗尽内存，自动机最短每隔60s（默认）创建一次，如果两次**增量**配置更新间隔小于60s，则会累积在更新队列中，直到与上次更新自动机间隔60s才会批量加载。全量配置和回调类配置更新不受最短加载时间的限制。
文件扫描间隔和配置生效间隔，可以通过Maat_set_feather_opt设置，详见本文档“函数接口”一章。
### 延迟删除机制
Maat使用延时删除机制，在不使用锁的前提下保证线程安全。
即配置管理线程在更新操作中：
a) 需要删除group_rule、compile_rule时，将他们放入垃圾回收队列，待超时后（默认10）后再进行释放。
b) 需要读取时，直接访问；
c) 需要修改时，获得mutex后访问
扫描线程中：
a) 需要读取时，获得mutex后访问

961
docs/table_schema.md
View File

@@ -1,961 +0,0 @@
# Table Schema
Maat tables are divided into two categories: physical tables that actually exist in the database and virtual tables that reference physical tables.
The types of physical tables are as follows:
- [item table](#1-item-table)
- [compile table](#4-compile-table)
- [group2compile table](#3-group2compile-table)
- [group2group table](#2-group2group-table)
- [plugin table](#5-plugin-table)
- [ip_plugin table](#6-ip_plugin-table)
- [fqdn_plugin table](#7-fqdn_plugin-table)
- [bool_plugin table](#8-bool_plugin-table)
- [virtual table](#9-virtual-table)
- [conjunction table](#10-conjunction-table)
Different physical tables can be combined into one table, see [conjunction table](#10-conjunction-table)
A virtual table can only reference one physical table or conjuntion table, see [virtual table](#9-virtual-table)
## 1. <a name='Itemtable'></a> Item table
Item tables are further subdivided into different types of subtables as follows:
- [expr item table](#11-expr-item-table)
- [expr_plus item table](#12-expr_plus-item-table)
- [ip_plus item table](#13-ip_plus-item-table)
- [interval item table](#14-numeric-range-item-table)
- [interval_plus item table](#14-numeric-range-item-table)
- [flag item table](#15-flag-item-table)
- [flag_plus item table](#16-flag_plus-item-table)
Each item table must has the following columns:
- item_id: In a maat instance, the item id is globally unique, meaning that the item id of different tables must not be duplicate.
- group_id: Indicate the group to which the item belongs, an item belongs to only one group.
- is_valid: In incremental updates, 1(valid means add) 0(invalid means del)
The range of item_id(group_id, compile_id) is 0～2^63，which is 8 bytes.
### 1.1 <a name='exprtable'></a> expr item table
Describe matching rules for strings.
- table format
| **FieldName** | **type** | **NULL** | **constraint** |
| ---------------- | -------------- | -------- | ------- |
| **item_id** | LONG LONG | N | primary key |
| **group_id** | LONG LONG | N | group2group or group2compile table's group_id |
| **keywords** | VARCHAR2(1024) | N | field to match during scanning |
| **expr_type** | INT | N | 0(keywords), 1(AND expr), 2(regular expr), 3(substring with offset)
| **match_method** | INT | N | only useful when expr_type is 0. 0(sub), 1(suffix), 2(prefix), 3(exactly) |
| **is_hexbin** | INT | N | 0(not HEX & case insensitive, this is default value) 1(HEX & case sensitive) 2(not HEX & case sensitive) |
| **is_valid** | INT | N | 0(invalid), 1(valid) |
- table schema(stored in table_info.conf)
```c
{
"table_id":3, //[0 ~ 1023], don't allow duplicate
"table_name":"HTTP_URL", //db table's name
"table_type":"expr",
"valid_column":7, //7th column(is_valid field)
"custom": {
"item_id":1, //1st column(item_id field)
"group_id":2, //2nd column(group_id field)
"keywords":3, //3rd column(keywords field)
"expr_type":4, //4th column(expr_type field)
"match_method":5,//5th column(match_method field)
"is_hexbin":6 //6th column(is_hexbin field)
}
}
/* If you want to combine multiple physical tables into one table, db_tables should be added as follows.
The value of table_name can be a user-defined string, the value of db_tables is the table name that actually exists in database. */
{
"table_id":3, //[0 ~ 1023], don't allow duplicate
"table_name":"HTTP_REGION", //user-defined string
"db_tables":["HTTP_URL", "HTTP_HOST"],
"table_type":"expr",
"valid_column":7,
"custom": {
"item_id":1,
"group_id":2,
"keywords":3,
"expr_type":4,
"match_method":5,
"is_hexbin":6
}
}
```
**expr_type** column represents the expression type:
1. keywords matching(0), match_method column as follows
- substring matching (0)
For example: substring: "China", scan_data: "Hello China" will hit, "Hello World" will not hit
- suffix matching (1)
For example: suffix: ".baidu.com", scan_data: "www.baidu.com" will hit, "www.google.com" will not hit
- prefix matching (2)
For example: prefix: "^abc", scan_data: "abcdef" will hit, "1abcdef" will not hit
- exactly matching (3)
For example: string: "World", scan_data: "World" will hit, "Hello World" will not hit
2. AND expression(1), supports up to 8 substrings.
For example: AND expr: "yesterday&today", scan_data: "Goodbye yesterday, Hello today!" will hit, "Goodbye yesterday, Hello tomorrow!" will not hit.
3. Regular expression(2)
For example: Regex expr: "[W|world]", scan_data: "Hello world" will hit, "Hello World" will hit too.
4. substring matching with offset(3)
- offset start with 0, [offset_start, offset_end] closed interval
- multiple substrings with offset are logical AND
For example: substring expr: "1-1:48&3-4:4C4C", scan_data: "HELLO" will hit, "HLLO" will not hit.
**Note**: 48('H') 4C('L')
&ensp;&ensp;Since Maat4.0，only support UTF-8，no more encoding conversion。For binary format rules, the keyword is hexadecimal, such as the keyword "hello" is represented as "68656C6C6F". A keyword can't contain invisible characters such as spaces, tabs, and CR, which are ASCII codes 0x00 to 0x1F and 0x7F. If these characters need to be used, they must be escaped, refer to the "keywords escape table". Characters led by backslashes outside this table are processed as ordinary strings, such as '\t' will be processed as the string "\t".
The symbol '&' means conjunction operation in AND expression. So if the keywords has '&', it must be escaped by '\&'.
**keywords escape table**
| **symbol** | **ASCII code** | **symbol after escape** |
| ---------- | -------------- | ----------------------- |
| \ | 0x5c | \\\ |
| & | 0x26 | \\& |
| blank space| 0x20 | \b |
Length constraint：
- Single substring no less than 3 bytes
- No less than 3 bytes for a single substring in AND expression
- Support up to 8 substrings in one AND expression, expr = substr1 & substr2 & substr3 & substr4 & substr5 & substr6 & substr7 & substr8
- The length of one AND expression should not exceed 1024 bytes(including '&')
Sample
- table schema
- rule
- scanning
table schema stored in table_info.conf
```json
[
{
"table_id":0,
"table_name":"COMPILE",
"table_type":"compile",
"valid_column":8,
"custom": {
"compile_id":1,
"tags":6,
"clause_num":9
}
},
{
"table_id":1,
"table_name":"COMPILE_ALIAS",
"table_type":"compile",
"valid_column":8,
"custom": {
"compile_id":1,
"tags":6,
"clause_num":9
}
},
{
"table_id":2,
"table_name":"COMPILE_CONJUNCTION",
"db_tables":["COMPILE", "COMPILE_ALIAS"],
"default_compile_table":2, //key:indicate this is the default compile table, value:can be anything(not care)
"table_type":"compile",
"valid_column":8,
"custom": {
"compile_id":1,
"tags":6,
"clause_num":9
}
},
{
"table_id":3,
"table_name":"GROUP2COMPILE",
"table_type":"group2compile",
"associated_compile_table_id":2, //associate compile conjunction table
"valid_column":3,
"custom": {
"group_id":1,
"compile_id":2,
"not_flag":4,
"virtual_table_name":5,
"clause_index":6
}
},
{
"table_id":4,
"table_name":"GROUP2GROUP",
"table_type":"group2group",
"valid_column":4,
"custom": {
"group_id":1,
"super_group_id":2,
"is_exclude":3
}
},
{
"table_id":5,
"table_name":"HTTP_URL",
"table_type":"expr",
"valid_column":7,
"custom": {
"item_id":1,
"group_id":2,
"keywords":3,
"expr_type":4,
"match_method":5,
"is_hexbin":6
}
}
]
```
rule stored in maat_json.json
```json
{
"compile_table": "COMPILE",
"group2compile_table": "GROUP2COMPILE",
"group2group_table": "GROUP2GROUP",
"rules": [
{
"compile_id": 123,
"service": 1,
"action": 1,
"do_blacklist": 1,
"do_log": 1,
"user_region": "anything",
"is_valid": "yes",
"groups": [
{
"group_name": "Untitled",
"regions": [
{
"table_name": "HTTP_URL",
"table_type": "expr",
"table_content":
{
"keywords": "multiple disciplines",
"expr_type": "none",
"match_method": "exact",
"format": "uncase plain"
}
}
]
}
]
},
{
"compile_id": 124,
"service": 1,
"action": 1,
"do_blacklist": 1,
"do_log": 1,
"user_region": "anything",
"is_valid": "yes",
"groups": [
{
"group_name": "Untitled",
"regions": [
{
"table_name": "HTTP_URL",
"table_type": "expr",
"table_content":
{
"keywords": "baidu.com",
"expr_type": "none",
"match_method": "suffix",
"format": "uncase plain"
}
}
]
}
]
},
{
"compile_id": 125,
"service": 1,
"action": 1,
"do_blacklist": 1,
"do_log": 1,
"user_region": "anything",
"is_valid": "yes",
"groups": [
{
"group_name": "Untitled",
"regions": [
{
"table_name": "HTTP_URL",
"table_type": "expr",
"table_content":
{
"keywords": "www",
"expr_type": "none",
"match_method": "prefix",
"format": "uncase plain"
}
}
]
}
]
},
{
"compile_id": 126,
"service": 1,
"action": 1,
"do_blacklist": 1,
"do_log": 1,
"user_region": "anything",
"is_valid": "yes",
"groups": [
{
"group_name": "Untitled",
"regions": [
{
"table_name": "HTTP_URL",
"table_type": "expr",
"table_content":
{
"keywords": "abc&123",
"expr_type": "and",
"match_method": "sub",
"format": "uncase plain"
}
}
]
}
]
},
{
"compile_id": 127,
"service": 1,
"action": 1,
"do_blacklist": 1,
"do_log": 1,
"user_region": "anything",
"is_valid": "yes",
"groups": [
{
"group_name": "Untitled",
"regions": [
{
"table_name": "HTTP_URL",
"table_type": "expr",
"table_content":
{
"keywords": "action=search\\&query=(.*)",
"expr_type": "regex",
"match_method": "sub",
"format": "uncase plain"
}
}
]
}
]
},
{
"compile_id": 128,
"service": 1,
"action": 1,
"do_blacklist": 1,
"do_log": 1,
"user_region": "anything",
"is_valid": "yes",
"groups": [
{
"group_name": "Untitled",
"regions": [
{
"table_name": "HTTP_URL",
"table_type": "expr",
"table_content":
{
"keywords": "1-1:48&3-4:4C4C",
"expr_type": "offset",
"match_method": "sub",
"format": "uncase plain"
}
}
]
}
]
}
]
}
```
scanning
```c
#include <assert.h>
#include "maat.h"
#define ARRAY_SIZE 16
const char *json_filename = "./maat_json.json";
const char *table_info_path = "./table_info.conf";
int main()
{
// initialize maat options which will be used by maat_new()
struct maat_options *opts = maat_options_new();
maat_options_set_json_file(opts, json_filename);
maat_options_set_logger(opts, "./sample_test.log", LOG_LEVEL_INFO);
// create maat instance, rules in table_info.conf will be loaded.
struct maat *maat_instance = maat_new(opts, table_info_path);
assert(maat_instance != NULL);
maat_options_free(opts);
const char *table_name = "HTTP_URL"; //maat_json.json has HTTP_URL rule
int table_id = maat_get_table_id(maat_instance, table_name);
assert(table_id == 3); // defined in table_info.conf
int thread_id = 0;
long long results[ARRAY_SIZE] = {0};
size_t n_hit_result = 0;
struct maat_state *state = maat_state_new(maat_instance, thread_id);
assert(state != NULL);
const char *scan_data1 = "There are multiple disciplines";
int ret = maat_scan_string(maat_instance, table_id, scan_data1, strlen(scan_data1),
results, ARRAY_SIZE, &n_hit_result, state);
assert(ret == MAAT_SCAN_HIT);
assert(n_hit_result == 1);
assert(results[0] == 123);
maat_state_reset(state);
const char *scan_data2 = "www.baidu.com";
ret = maat_scan_string(maat_instance, table_id, scan_data2, strlen(scan_data2),
results, ARRAY_SIZE, &n_hit_result, state);
assert(ret == MAAT_SCAN_HIT);
assert(n_hit_result == 1);
assert(results[0] == 124);
maat_state_reset(state);
const char *scan_data3 = "www.google.com";
ret = maat_scan_string(maat_instance, table_id, scan_data3, strlen(scan_data3),
results, ARRAY_SIZE, &n_hit_result, state);
assert(ret == MAAT_SCAN_HIT);
assert(n_hit_result == 1);
assert(results[0] == 125);
maat_state_reset(state);
const char *scan_data4 = "alphabet abc, digit 123";
ret = maat_scan_string(maat_instance, table_id, scan_data4, strlen(scan_data4),
results, ARRAY_SIZE, &n_hit_result, state);
assert(ret == MAAT_SCAN_HIT);
assert(n_hit_result == 1);
assert(results[0] == 126);
maat_state_reset(state);
const char *scan_data5 = "http://www.cyberessays.com/search_results.php?action=search&query=username,abckkk,1234567";
ret = maat_scan_string(maat_instance, table_id, scan_data5, strlen(scan_data5),
results, ARRAY_SIZE, &n_hit_result, state);
assert(ret == MAAT_SCAN_HIT);
assert(n_hit_result == 1);
assert(results[0] == 127);
maat_state_reset(state);
const char *scan_data6 = "HELLO WORLD";
ret = maat_scan_string(maat_instance, table_id, scan_data6, strlen(scan_data6),
results, ARRAY_SIZE, &n_hit_result, state);
assert(ret == MAAT_SCAN_HIT);
assert(n_hit_result == 1);
assert(results[0] == 128);
maat_state_free(state);
return 0;
}
```
### 1.2 <a name='ExprPlusItemTable'></a> expr_plus item table
Describe extended matching rules for strings by adding the district column.
- table format
| **FieldName** | **type** | **NULL** | **constraint** |
| ---------------- | -------------- | -------- | ------- |
| **item_id** | LONG LONG | N | primary key |
| **group_id** | LONG LONG | N | group2group or group2compile table's group_id |
| **district** | VARCHAR2(1024) | N | describe the effective position of the keywords |
| **keywords** | VARCHAR2(1024) | N | field to match during scanning |
| **expr_type** | INT | N | 0(keywords), 1(AND expr), 2(regular expr), 3(substring with offset)
| **match_method** | INT | N | only useful when expr_type is 0 |
| **is_hexbin** | INT | N | 0(not HEX & case insensitive, this is default value) 1(HEX & case sensitive) 2(not HEX & case sensitive) |
| **is_valid** | INT | N | 0(invalid), 1(valid) |
For example, if the district is User-Agent and keywords is Chrome, scanning in the following way will hit.
```c
const char *scan_data = "Chrome is fast";
const char *district = "User-Agent";
maat_state_set_scan_district(..., district, ...);
maat_scan_string(..., scan_data, ...)
```
### 1.3 <a name='IPPlusItemTable'></a> ip_plus item table
Describe matching rules for IP address. Both the address and port are represented by string, IPv4 is dotted decimal and IPv6 is colon separated hexadecimal.
- table format
| **FieldName** | **type** | **NULL** | **constraint** |
| ------------- | ------------ | -------- | -------------- |
| item_id | LONG LONG | N | primary key |
| group_id | LONG LONG | N | group2group or group2compile table's group_id |
| addr_type | INT | N | Ipv4 = 4 Ipv6 = 6 |
| addr_format | VARCHAR2(40) | N | ip addr format, single/range/CIDR/mask |
| ip1 | VARCHAR2(40) | N | start ip |
| ip2 | VARCHAR2(40) | N | end ip |
| port_format | VARCHAR2(40) | N | port format, single/range |
| port1 | VARCHAR2(6) | N | start port number |
| port2 | VARCHAR2(6) | N | end port number |
| protocol | INT | N | default(-1) TCP(6) UDP(17), user define field |
| is_valid | INT | N | 0(invalid), 1(valid) |
### 1.4 <a name='NumericItemTable'></a> numeric range item table
Determine whether an integer is within a certain numerical range.
- table format
| **FieldName** | **type** | **NULL** | **constraint** |
| ------------- | -------- | -------- | -------------- |
| item_id | INT | N | primary key |
| group_id | INT | N | group2group or group2compile table's group_id |
| low_boundary | INT | N | lower bound of the numerical range(including lb), 0 ~ (2^32 - 1)|
| up_boundary | INT | N | upper bound of the numerical range(including ub), 0 ~ (2^32 - 1)|
| is_valid | INT | N | 0(invalid), 1(valid) |
### 1.5 <a name="FlagItemTable"></a> flag item table
- table format
| **FieldName** | **type** | **NULL** | **constraint** |
| ------------- | -------- | -------- | -------------- |
| item_id | INT | N | primary key |
| group_id | INT | N | group2group or group2compile table's group_id |
| flag | INT | N | flag, 0 ~ (2^32 - 1)|
| flag_mask | INT | N | flag_mask, 0 ~ (2^32 - 1)|
| is_valid | INT | N | 0(invalid), 1(valid) |
### 1.6 <a name="FlagPlusItemTable"></a> flag_plus item table
- table format
| **FieldName** | **type** | **NULL** | **constraint** |
| ------------- | -------- | -------- | -------------- |
| item_id | INT | N | primary key |
| group_id | INT | N | group2group or group2compile table's group_id |
| district | INT | N | describe the effective position of the flag |
| flag | INT | N | flag, 0 ~ (2^32 - 1)|
| flag_mask | INT | N | flag_mask, 0 ~ (2^32 - 1)|
| is_valid | INT | N | 0(invalid), 1(valid) |
### 2. <a name='group2grouptable'></a> group2group table
Describe the relationship between groups.
- table format
| **FieldName** | **type** | **NULL** | **constraint** |
| ----------------- | --------- | -------- | ---------------|
| group_id | LONG LONG | N | reference from xx_item table's group_id |
| superior_group_id | LONG LONG | N | group_id include or exclude specified super_group_id |
| is_exlude | Bool | N | 0(include) 1(exclude) |
| is_valid | Bool | N | 0(invalid), 1(valid) |
### 3. <a name='group2compiletable'></a> group2compile table
Describe the relationship between group and compile.
- table format
| **FieldName** | **type** | **NULL** | **constraint** |
| ------------- | ------------- | -------- | ------- |
| group_id | LONG LONG | N | reference from xx_item table's group_id|
| compile_id | LONG LONG | N | compile ID |
| is_valid | INT | N | 0(invalid), 1(valid) |
| not_flag | INT | N | logical 'NOT', identify a NOT clause, 0(no) 1(yes) |
| virtual_table | VARCHAR2(256) | N | virtual table name, default:”null” |
| Nth_clause | INT | N | the clause seq in (conjunctive normal form)CNF, from 0 to 7. groups with the same clause ID are logical 'OR' |
NOTE: If group_id is invalid in xx_item table, it must be marked as invalid in this table.
### 4. <a name='compiletable'></a> compile table
Describe the specific policy, One maat instance can has multiple compile tables with different names.
- table format
| **FieldName** | **type** | **NULL** | **constraint** |
| ---------------- | -------------- | -------- | --------------- |
| compile_id | LONG LONG | N | primary key, policy ID |
| service | INT | N | such as URL keywords or User Agent etc. |
| action | VARCHAR(1) | N | recommended definitions: 0(Blocking) 1(Monitoring) 2(whitelist) |
| do_blacklist | VARCHAR(1) | N | 0(no)，1(yes) transparent to maat |
| do_log | VARCHAR(1) | N | 0(no)，1(yes)，default 1 transparent to maat |
| tags | VARCHAR2(1024) | N | default 0，means no tag |
| user_region | VARCHAR2(8192) | N | default 0 transparent to maat |
| is_valid | INT | N | 0(invalid)，1(valid) |
| clause_num | INT | N | no more than 8 clauses |
| evaluation_order | DOUBLE | N | | default 0 |
### 5. <a name='plugintable'></a> plugin table
There is no fixed rule format of the plugin table, which is determined by business side. The plugin table supports two sets of callback functions, registered with **maat_table_callback_register** and **maat_plugin_table_ex_schema_register** respectively.
maat_table_callback_register
```c
/*
When the plugin table rules are updated, start will be called first and only once, then update will be called by each rule item, and finish will be called last and only once.
If rules have been loaded but maat_table_callback_register has not yet been called, maat will cache the loaded rules and perform the callbacks(start, update, finish) when registration is complete.
*/
typedef void maat_start_callback_t(int update_type, ...);
//table_line points to one complete rule line, such as: "1\tHeBei\tShijiazhuang\t1\t0"
typedef void maat_update_callback_t(..., const char *table_line, ...);
typedef void maat_finish_callback_t(...);
int maat_table_callback_register(...,
maat_start_callback_t *start,
maat_update_callback_t *update,
maat_finish_callback_t *finish,
...);
```
maat_plugin_table_ex_schema_register
```c
/*
*/
typedef void maat_ex_new_func_t(..., const char *key, const char *table_line, ...);
typedef void maat_ex_free_func_t(...);
typedef void maat_ex_dup_func_t(...);
int maat_plugin_table_ex_schema_register(...,
maat_ex_new_func_t *new_func,
maat_ex_free_func_t *free_func,
maat_ex_dup_func_t *dup_func,
...);
```
Plugin table supports three types of keys(pointer, integer and ip_addr) for ex_data callback.
**pointer key(compatible with maat3)**
(1) schema
```json
{
"table_id":1,
"table_name":"TEST_PLUGIN_POINTER_KEY_TYPE",
"table_type":"plugin",
"valid_column":4,
"custom": {
"key_type":"pointer",
"key":2,
"tag":5
}
}
```
(2) plugin table rules
```json
{
"table_name": "TEST_PLUGIN_POINTER_KEY_TYPE",
"table_content": [
"1\tHeBei\tShijiazhuang\t1\t0",
"2\tHeNan\tZhengzhou\t1\t0",
"3\tShanDong\tJinan\t1\t0",
"4\tShanXi\tTaiyuan\t1\t0"
]
}
```
(3) register callback
```c
void plugin_EX_new_cb(const char *table_name, int table_id, const char *key,
const char *table_line, void **ad, long argl, void *argp);
void plugin_EX_free_cb(int table_id, void **ad, long argl, void *argp);
void plugin_EX_dup_cb(int table_id, void **to, void **from, long argl, void *argp);
const char *table_name = "TEST_PLUGIN_POINTER_KEY_TYPE";
int table_id = maat_get_table_id(maat_inst, table_name);
int plugin_ex_data_counter = 0;
maat_plugin_table_ex_schema_register(maat_inst, table_name,
plugin_EX_new_cb,
plugin_EX_free_cb,
plugin_EX_dup_cb,
0, &plugin_ex_data_counter);
```
(4) get ex_data
```c
const char *key1 = "HeBei";
const char *table_name = "TEST_PLUGIN_POINTER_KEY_TYPE";
int table_id = maat_get_table_id(maat_instance, table_name);
maat_plugin_table_get_ex_data(maat_instance, table_id, key1, strlen(key1));
```
**integer key**
support integers of different lengths, such as int(4 bytes), long long(8 bytes).
(1) schema
```json
{
"table_id":1,
"table_name":"TEST_PLUGIN_INT_KEY_TYPE",
"table_type":"plugin",
"valid_column":4,
"custom": {
"key_type":"integer",
"key_len":4
"key":2,
"tag":5
}
}
{
"table_id":2,
"table_name":"TEST_PLUGIN_LONG_KEY_TYPE",
"table_type":"plugin",
"valid_column":4,
"custom": {
"key_type":"integer",
"key_len":8
"key":2,
"tag":5
}
}
```
(2) plugin table rules
```json
{
"table_name": "TEST_PLUGIN_INT_KEY_TYPE",
"table_content": [
"1\t101\tChina\t1\t0",
"2\t102\tAmerica\t1\t0",
"3\t103\tRussia\t1\t0",
"4\t104\tJapan\t1\t0"
]
}
{
"table_name": "TEST_PLUGIN_LONG_KEY_TYPE",
"table_content": [
"1\t11111111\tShijiazhuang\t1\t0",
"2\t22222222\tZhengzhou\t1\t0",
"3\t33333333\tJinan\t1\t0",
"4\t44444444\tTaiyuan\t1\t0"
]
}
```
(3) register callback
same as above
(4) get ex_data
```c
//int
int key1 = 101; //key length must be 4 bytes which specified in table_info.conf
const char *table_name = "TEST_PLUGIN_INT_KEY_TYPE";
int table_id = maat_get_table_id(maat_instance, table_name);
maat_plugin_table_get_ex_data(maat_instance, table_id, key1, sizeof(key1));
```
**ip_addr key**
support ip address(ipv4 or ipv6) as key.
(1) schema
```json
{
"table_id":1,
"table_name":"TEST_PLUGIN_IP_KEY_TYPE",
"table_type":"plugin",
"valid_column":4,
"custom": {
"key_type":"ip_addr",
"addr_type":1,
"key":2
}
}
```
The addr_type column indicates whether the key is a v4 or v6 address.
(2) plugin table rules
```json
{
"table_name": "TEST_PLUGIN_IP_KEY_TYPE",
"table_content": [
"4\t100.64.1.1\tXiZang\t1\t0",
"4\t100.64.1.2\tXinJiang\t1\t0",
"6\t2001:da8:205:1::101\tGuiZhou\t1\t0",
"6\t1001:da8:205:1::101\tSiChuan\t1\t0"
]
}
```
(3) register callback
same as above
(4) get ex_data
```c
uint32_t ipv4_addr;
inet_pton(AF_INET, "100.64.1.1", &ipv4_addr);
const char *table_name = "TEST_PLUGIN_IP_KEY_TYPE";
table_id = maat_get_table_id(maat_instance, table_name);
maat_plugin_table_get_ex_data(maat_instance, table_id, (char *)&ipv4_addr, sizeof(ipv4_addr));
```
### 6. <a name='ip_plugintable'></a> ip_plugin table
Similar to plugin table but the key of maat_ip_plugin_table_get_ex_data is ip address.
### 7. <a name='FQDNPlugintable'></a> fqdn_plugin table
Scan the input string according to the domain name hierarchy '.'
Return results order:
1. sort by decreasing the length of the hit rule
2.
For example:
1. example.com.cn
2. com.cn
3. example.com.cn
4. cn
5. ample.com.cn
If the input string is example.com.cn，则返回结果顺序为：3，1，2，4。规则5中的ample不是域名层级的一部分，不返回。
### 8. <a name='boolplugintable'></a> bool_plugin table
按照布尔表达式扫描输入的整数数组，如[100,1000,2,3]。
布尔表达式规则为“&”分隔的数字，例如“1&2&1000”。
### 9. <a name='virtualtable'></a> virtual table
虚拟一个配置表，其内容为特定物理域配置表的视图。实践中，通常采用网络流量的属性作为虚拟表名，如HTTP_HOST、SSL_SNI等。一个虚拟表可以建立在多个不同类型的物理表之上，但不允许建立在其它虚拟表上。
虚拟表以分组为单位引用实体表中的域配置，引用关系在分组关系表中描述。一个分组可被同一个编译配置的不同虚拟表引用。例如下表，一个关键字的分组keyword_group_1，被一条compile_1的Request Body和Response Body两个虚拟表引用。
| **分组ID** | **父ID** | **有效标志** | **非运算标志位** | **父节点类型** | **分组所属虚拟表** |
| ------------------- | --------- | ------------ | ---------------- | -------------- | ------------------ |
| **keyword_group_1** | compile_1 | 1 | 0 | 0 | REQUEST_BODY |
| **keyword_group_1** | compile_1 | 1 | 0 | 0 | RESPONSE_BODY |
### 10. <a name='conjunctiontable'></a> conjunction table
表名不同，但table id相同的表。旨在数据库表文件和MAAT API之间提供一个虚拟层，通过API调用一次扫描，即可扫描多张同类配置表。
使用方法：
1. 在配置表描述文件中，将需要连接的多个表共用一个table_id；
2. 通过Maat_table_register注册被连接表中的任意一个表名，使用该id进行扫描。
被连接的配置表的各项属性以在配置表描述文件（table_info.conf）中第一个出现的同ID描述行为准，同一table_id下最多支持8个配置表。
支持所有类型表的连接，包括各类域配置、回调类配置。配置分组和配置编译的连接没有意义。
### 11. <a name='ForeignFiles'></a>Foreign Files
回调类配置中，特定字段可以指向一个外部内容，目前支持指向Redis中的一个key。
回调表的外键列，必须具备”redis://”前缀。存放在Redis中的外键内容，其Key必须具备”__FILE_”前缀。当Key为“null”时，表示该文件为空。
例如，原始文件为./testdata/mesa_logo.jpg，计算其MD5值后，得到redis的外键__FILE_795700c2e31f7de71a01e8350cf18525，写入回调表后的格式如下：
```
14 ./testdata/digest_test.data redis://__FILE_795700c2e31f7de71a01e8350cf18525 1
```
​ 回调表中的一行最多允许8个外键，外键内容可以通过Maat_cmd_set_file函数设置。
​ Maat在通知回调表前会将外键拉取到本地文件，并将外键列替换为本地文件路径。
​ 内容外键的声明方法，参见本文档-配置表描述文件一节。
### 12. <a name='Tags'></a>Tags
通过将Maat接受标签与配置标签的匹配，实现有选择的配置加载。其中配置标签是一个标签数组的集合，记为”tag_sets”，Maat接受标签是标签数组，记为”tags”。
配置标签是指存放在编译配置或分组配置上的标签，标识着该配置在那些Maat实例中生效。由多个tag_set构成，1个set内的多个tag是与的关系，1个tag的多个值是或的关系，值内部用”/”表示层次结构。
格式为一个不含回车、空格的JSON，结构为:
若干tag集合（数组）->tag集合（数组）->若干tag（数组）->{tag名称，tag值（数组）}
例如：
```json
{"tag_sets":[[{"tag":"location","value":["北京/朝阳/华严北里","上海/浦东/陆家嘴"]},{"tag":"isp","value":["电信","移动"]}],[{"tag":"location","value":["北京"]},{"tag":"isp","value":["联通"]}]]}
```
上例有2个tag分组：
- 分组1：（"北京/朝阳/华严北里"∨"上海/浦东/陆家嘴"）∧("电信"∨"移动")
- 分组2：（"北京"∧"联通"）
- 分组1∨分组2
Maat实例初始化时，可以设置自身的标签信息，称为接受标签。格式为同样要求的JSON，内有多个标签，加载配置时匹配实例标签和配置的生效范围标签。例如：
```json
{"tags":[{"tag":"location","value":"北京/朝阳/华严北里/甲22号”},{"tag":"isp","value":"电信"}]}
```
该Maat实例在加载以下标签时：
1. {"tag_sets":[[{"tag":"location","value":["北京/朝阳"]},{"tag":"isp","value":["联通","移动"]}]}，不被接受，因为isp tag不匹配。
2. {"tag_sets":[[{"tag":"location","value":["北京"]}]]}，接受，空tag在任意tag上生效。
对于Maat实例接受标签和配置标签name不匹配的异常情况，Maat遵循不违背即接受的原则，全部接受。
- Maat实例的接受标签是配置标签的真子集时，即tags 属于tag_set，Maat会接受该配置。
- 例如：接受标签为：{"tags":[{"tag":"location","value":"北京”}]} ，配置标签为：{"tags":[{"tag":"location","value":"北京/朝阳”},{"tag":"isp","value":"电信"}]} ，Maat会接受该配置，因为实例仅要求”location”满足“北京”，未对“isp”标签的值作出要求。
- 配置标签是Maat实例接受标签的真子集时，即tag_sets属于tags，Maat会接受该配置。
- 例如：接受标签为：{"tags":[{"tag":"location","value":"北京/朝阳”},{"tag":"isp","value":"电信"}]}，配置标签为：{"tags":[{"tag":"location","value":"北京”}]}，Maat会接受该配置。配置没有“isp”标签，并未违背Maat接受条件。
- Maat实例的接受标签和配置标签的交集为空时，Maat会接受该配置。
当配置标签为“0”或“{}”时，无论Maat实例的接受标签是什么都会接受，这一特性用于向前兼容未设置标签的配置。

123
docs/terminology.md Normal file
View File

@@ -0,0 +1,123 @@
# Terminology
* [Item](#item)
* [Group(Object)](#groupobject)
* [Compile(Policy)](#compilepolicy)
* [Clause(Condition)](#clause)
* [Literal](#literal)
* [Physical table](#physical-table)
* [Virtual table](#virtual-table)
* [Table schema](#table-schema)
* [Table runtime](#table-runtime)
* [Table rule](#table-ruleconfiguration)
* [Conjunction](#conjunction)
* [Maat state](#conjunction)
* [Maat stream](#maat-stream)
* [Half/Full hit](#halffull-hit)
* [Hit path](#hit-path)
* [Redis](#redis)
## Item
As a filter for network attributes, the smallest unit of a rule
- Eg1: specify that the UserAgent field in the HTTP protocol contains substrings "Chrome" and "11.8.1",
&ensp;&ensp; HTTP UserAgent: Chrome & 11.8.1
- Eg2: specify that the domain name in the HTTP protocol ends with ".emodao.com"
&ensp;&ensp; HTTP HOST: *.emodao.com
- Eg3: specify client IP address belongs to the C segment of 202.118.101.*
&ensp;&ensp; Source IP: 202.11.101.0/24
There are multiple types of items stored in corresponding tables such as string, IP and numerical range, more details can be found in [Item table](./maat_table.md#11-item-table).
## Group(Object)
A group defines a set that can contain different types of items and can also reference other groups. The definition is made using items, which can be used to add to or exclude from the group definition. Groups can also have subordinate groups whose definitions are included in or excluded from the superior group.
- An item only belongs to one group, but one group can has multiple items. The multiple items under the same group are logical 'OR' relationships. e.g.(g1 = item1 | item2).
- A group can be included or excluded by other groups. For example, if group1 and group2 is included by group3, then the group3 is called the superior(super) group, and group1(group2) is called the subordinate(sub) group. There's a logical 'OR' relationship between the included sub groups under the same group, e.g.(g3 = incl-g1 | incl-g2). There's a logical 'AND' relationship between included group and excluded group under the same group, e.g.(g4 = incl-g1 & excl-g2).
- Group supports multi-level nesting, see [group hierarchy](./overview.md#groupobject-nesting-and-hierarchies)
- A Group can be referenced by different compiles.
The relationship between group and group is stored in the [group2group table](./maat_table.md#14-group2group-table).
## Compile(Policy)
A conjunctive normal form(CNF) consisting of multiple groups and virtual tables.
`Note`: A compile can contain up to 8 clauses and multiple clauses in the same compile can be logical 'AND' and logical 'NOT' relationships.
The relationship between group and compile is stored in the [group2compile table](./maat_table.md#13-group2compile-table).
<img src="./imgs/CNF.jpg" alt="exclude" style="zoom:80%" />
## Clause
A clause consists of several Literals and the relationship between them is a `logical 'OR'`.
Clauses are divided into two categories based on whether they contain the logical "NOT" operation: `clause` and `NOT-clause`. In Maat, the logical "NOT" only appears in the clause, which means that if you want to use the logical "NOT", you need to configure clauses for the rules.
## Literal
A Literal consists of `vtable_id(virtual table id)` and `group_id`. During the rules loading process, a unique clause_id will be generated based on the combination of virtual table_id and group_id in the same clause.
## Physical table
Different rules are stored in different tables in the actual database, including [item table](./maat_table.md#11-item-table), [compile table](./maat_table.md#12-compile-table), [group2compile table](./maat_table.md#13-group2compile-table), [group2group table](./maat_table.md#14-group2group-table), and [xx_plugin table](./maat_table.md#15-plugin-table), and so on.
## Virtual table
A virtual table references a physical table. In practice, network traffic attributes are commonly used as virtual table, such as HTTP_HOST, SSL_SNI, etc. The constraints of virtual tables are as follows:
* A virtual table can only reference one physical table. If it need to reference multiple physical tables of the same type, these physical tables can be first joined together into one table and then referenced.
* A physical table can be referenced by different virtual tables. For example, the keyword_table can be referenced by two virtual tables, http_request_body_virt and http_response_body_virt.
<img src="./imgs/virt-phy-mapping.png" width="300" height="150" >
## Table schema
Defines the type of table and the configuration format, determining the specific meaning of each column in the table. Maat parses the configuration according to the schema, more details can be found in [table schema](./maat_table.md#1-table-schema).
## Table runtime
The runtime generated by loading the configuration in the table into memory. Different tables have different runtimes. The group2compile table is merged with the corresponding compile table to generate a compile runtime, meaning there is no separate group2compile runtime.
Different scanning api use runtimes of different tables. For example, the HTTP_URL table is of type expr, and its corresponding scanning interface is maat_scan_string. Therefore, when calling this scanning interface, the API internally uses the runtime of the HTTP_URL table to perform the actual scanning task.
## Table rule(configuration)
Different types of rules are stored in tables of different types, such as IP addresses, numerical ranges, strings, and so on.
## Conjunction
By default, maat builds a separate runtime for each physical table, which can be used for rule matching by specifying the table ID during scanning. If the user wants to combine multiple physical tables of the same type into a single table for runtime build and scan, it means conjunction of multiple tables.
`Note`: Only physical tables support conjunction.
## Maat state
Maat state is used to record intermediate states of scans. For example, a set of scans includes an IP address scan and an IP geolocation scan. The configured rule can determine whether the scan is matched only after the two scans are complete.
In this case, the intermediate state of the first scan is stored in the maat state until the second scan is completed to provide the final hit result.
## Maat stream
Maat supports not only block-based scanning but also stream-based scanning. For more information on stream-based scanning, please refer [hyperscan](https://intel.github.io/hyperscan/dev-reference/runtime.html#runtime).
## Half/Full hit
From the diagram of [configuration relationship](./overview.md#12-configuration-relationship), it can be seen that if the group that is hit is not referenced by compile, or even if referenced by compile, but after logical operations no compile is hit, then this hit is called a half hit; if a compile is hit, then this hit is called a full hit.
## Hit path
From the relationship of item, group, and compile mentioned above, if a scan hits a certain compile, there must be a logical path composed of item_id -> group_id -> compile_id. Maat names this path the hit path. If a group has nested references, the hit path should be item_id -> sub_group_id -> group_id -> compile_id.
## Redis
In-memory data store，see https://redis.io/. It has a leader follower replication to ensure the high availability of rules

67
docs/thread_mode.md
View File

@@ -1,67 +0,0 @@
# Thread mode
Maat will create a monitor loop thread internally when calling maat_new to create maat instance. Scaning threads are created extenal maat caller. So all maat_scan_xx APIs are per-thread
![The subordinate object](./imgs/thread_mode.png)
Sample
```c
const char *table_info_path = "table_info.conf";
size_t thread_num = 5;
struct thread_param {
int thread_id;
struct maat *maat_inst;
const char *table_name;
};
void *string_scan_thread(void *arg)
{
struct thread_param *param = (struct thread_param *)arg;
struct maat *maat_inst = param->maat_inst;
const char *table_name = param->table_name;
const char *scan_data = "String TEST should hit";
long long results[ARRAY_SIZE] = {0};
size_t n_hit_result = 0;
struct maat_state *state = maat_state_new(maat_inst, param->thread_id);
int table_id = maat_get_table_id(maat_inst, table_name);
int ret = maat_scan_string(maat_inst, table_id, scan_data, strlen(scan_data),
results, ARRAY_SIZE, &n_hit_result, state);
EXPECT_EQ(ret, MAAT_SCAN_HIT);
EXPECT_EQ(n_hit_result, 1);
EXPECT_EQ(results[0], 123);
maat_state_free(state);
return NULL;
}
int main()
{
struct maat_options *opts = maat_options_new();
maat_options_set_caller_thread_number(opts, thread_num);
struct maat *maat_inst = maat_new(opts, table_info_path);
size_t i = 0;
pthread_t threads[thread_num];
struct thread_param thread_params[thread_num];
for (i = 0; i < thread_num; i++) {
thread_params[i].maat_inst = maat_inst;
thread_params[i].thread_id = i;
thread_params[i].table_name = table_name;
pthread_create(&threads[i], NULL, string_scan_thread, thread_params+i);
}
for (i = 0; i < thread_num; i++) {
pthread_join(threads[i], NULL);
}
return 0;
}
```

9
docs/thread_model.md Normal file
View File

@@ -0,0 +1,9 @@
# Thread model
When the maat_new interface is called to create a maat instance, maat will internally create a monitor_loop thread, which will periodically check if there are any configuration updates in Redis or JSON files. If there are updates, it will update the corresponding table's runtime. The thread that executes the actual scanning task is created by the caller, and maat supports multi-threaded scanning. The thread model is illustrated in the diagram below:
<img src="./imgs/thread-model.png" width="750" height="450" >
From the diagram, it can be seen that multiple threads call the scanning interface, which internally references the effective runtime. Since the scanning interface needs to ensure high performance, when a configuration update occurs, the runtime cannot be directly modified to avoid potential multi-threading safety issues. Therefore, the RCU mechanism is employed, where the monitor_loop thread clones an updating runtime and performs update operations on this runtime. When the updating runtime is ready, it is switched to a new effective runtime through a commit action, and subsequent scans will use this runtime.
One issue to consider is if a thread calls the scanning interface and the commit operation occurs immediately, resulting in the updating runtime becoming the new effective runtime. The original effective runtime cannot be destroyed immediately because the thread's scanning task is not yet complete. When can the original effective runtime be destroyed? It is necessary to evaluate the longest time for a single scanning task, as outlined in [performance](./overview.md#6-performance). It is evident that, in a known configuration scenario, the longest time is 30us. Therefore, the delay for destroying the runtime should not be less than 30us. In the current implementation, it is set to 100us to ensure that invalid runtime pointers are not accessed.

32
docs/tools.md
View File

@@ -1,32 +0,0 @@
# Tools
## maat_redis_tool
1. Dump the configuration in redis to a local IRIS format file.
2. Write the configuration in JSON format to redis. Note: Group multiplexing is not supported in this mode.
For specific usage, run ./maat_redis_tool -h
## compress JSON configuration using gzip command
```shell
$ gzip -9 < maat_json.json > maat_json.json.gz
```
maat_options_set_json_file_gzip_flag can indicate whether maat instance needs to decompress the json configuration before loading it. This API can be used in conjunction with the encryption command below, compressed first and then encrypted.
## using openssl enc to encrypt/decrypt configuration
maat_options_set_json_file_decrypt_key can specify the decryption key for the JSON file to be decrypted.
encrypt：
```shell
$ openssl enc -e -aes-128-cbc -k 123456 -p -nosalt -in inputfile.txt -out encryptfile.txt
```
decrypt：
```shell
$ openssl enc -d -aes-128-cbc -k 123456 -p -nosalt -in encryptfile.txt -out tmp.txt
```

0
examples/CMakeLists.txt Normal file
View File

0
examples/ip_plugin/ip_plugin.c Normal file
View File

0
examples/ip_scanning/ip_scanning.c Normal file
View File

6
include/maat.h
View File

@@ -165,9 +165,9 @@ void maat_free(struct maat *instance);
void maat_reload_log_level(struct maat *instance, enum log_level level); void maat_reload_log_level(struct maat *instance, enum log_level level);
/** /**
* Each thread can call this function initially, maat will maintain the * For xx_plugin table, each thread can call this function initially, maat will maintain the
* thread_id internally, So it's no need to pass thread_id by maat_scan_xx and * thread_id internally and the number of times the xx_plugin_table_get_ex_data interface
* xx_plugin_get_ex_data API * is called by different threads is counted.
*/ */
void maat_register_thread(struct maat *instance); void maat_register_thread(struct maat *instance);

233
readme.md
View File

@@ -1,226 +1,41 @@
<h1 align="left"> <h1 align="left">
<img src="./docs/imgs/maat_logo.png" height="40px" alt="swarmkv logo"/> <img src="./docs/imgs/logo.png" height="60px" alt="maat logo"/>
</h1> </h1>
**Unified description framework for network flow processing configuration** **Maat is a unified description framework for network flow processing configuration**
## Origin ## Name origin
Maat was the goddness of harmony, justice, and truth in ancient Egypt. Her feather served as the measure that determined whether the souls of the departed would successfully reach the paradise of the afterlife. We use this meaning metaphorically to indicate whether the scanning hits an effective rule or not, which is the core function of Maat.
Maat was the goddness of harmony, justice, and truth in ancient Egyptian. Her feather was the measure that determined whether the souls of the departed would reach the paradise of the afterlife successfully. We use this meaning to metaphorically indicate whether scannning has hit or not. ## Why should I use Maat?
Please imagine the following scenario:
* S1：The user adds 10 security deny policies and expects that if the traffic matches any of the 10 configured policies, the traffic should be denied. If the number of policies grows to 1,000, 10,000 or more, and at the same time different policies require different actions such as asllow, monitor, etc.
The Maat framework abstracts the configuration in network flow processing. It supports dynamic loading and multi-machine synchronization of configurations. The core function of Maat is to determine whether a loaded rule has been hit through scanning. * S2：The user has pre-stored a large amount of IP geolocation information in a table and expects to quickly retrieve the geolocation information of a given IP address.
Maat supports three configuration loading modes. The above are two typical usage scenarios of Maat, one is the scanning scenario, and the other is the callback scenario. In addition, Maat also provides detailed path information and statistical information for hit policies, and more advanced features are waiting for you to explore, try it!
* **Redis mode**(for production): the data source is usually a relational database, such as Oracle, MySQL. ## Getting Started
* **JSON File mode**(for production and debugging) Read [getting started](./docs/getting_started.md) for building steps and play with [examples](./examples/).
* **IRIS File mode**(for troubleshooting)
**Note**: Redis mode and JSON File mode support configuration dynamic loading
Maat is used as a dynamic library by applications and it's API is defined in the header file(maat.h).
## Building
```shell
mkdir build && cd build
cmake ..
make
make install
```
## Sample
A complete use case consists of three parts
* **table schema**: define how to parse rule in different table, that is specify what each column in a table represents.
* **rule**: different types of rules are stored in tables of the corresponding type.
* **scanning API**: used by application to find whether scan data has hit loaded rules.
### 1. table schema
Table schema is stored in a json file(such as table_info.conf), which is loaded when maat instance is created.
```json
[
{
"table_id":0,
"table_name":"COMPILE",
"table_type":"compile",
"valid_column":8,
"custom": {
"compile_id":1,
"tags":6,
"clause_num":9
}
},
{
"table_id":1,
"table_name":"COMPILE_ALIAS",
"table_type":"compile",
"valid_column":8,
"custom": {
"compile_id":1,
"tags":6,
"clause_num":9
}
},
{
"table_id":2,
"table_name":"COMPILE_CONJUNCTION",
"db_tables":["COMPILE", "COMPILE_ALIAS"],
"default_compile_table":2, //key:indicate this is the default compile table, value:can be anything(not care)
"table_type":"compile",
"valid_column":8,
"custom": {
"compile_id":1,
"tags":6,
"clause_num":9
}
},
{
"table_id":3,
"table_name":"GROUP2COMPILE",
"table_type":"group2compile",
"associated_compile_table_id":2, //associate compile conjunction table
"valid_column":3,
"custom": {
"group_id":1,
"compile_id":2,
"not_flag":4,
"virtual_table_name":5,
"clause_index":6
}
},
{
"table_id":4,
"table_name":"GROUP2GROUP",
"table_type":"group2group",
"valid_column":4,
"custom": {
"group_id":1,
"super_group_id":2,
"is_exclude":3
}
},
{
"table_id":5,
"table_name":"HTTP_URL",
"table_type":"expr",
"valid_column":7,
"custom": {
"item_id":1,
"group_id":2,
"keywords":3,
"expr_type":4,
"match_method":5,
"is_hexbin":6
}
}
]
```
### 2. rule
Rules are stored in a json file(such as maat_json.json), which is loaded when maat instance is created.
```json
{
"compile_table": "COMPILE",
"group2compile_table": "GROUP2COMPILE",
"group2group_table": "GROUP2GROUP",
"rules": [
{
"compile_id": 123,
"service": 1,
"action": 1,
"do_blacklist": 1,
"do_log": 1,
"user_region": "anything",
"is_valid": "yes",
"groups": [
{
"group_name": "Untitled",
"regions": [
{
"table_name": "HTTP_URL",
"table_type": "expr",
"table_content":
{
"keywords": "Hello Maat",
"expr_type": "none",
"match_method": "sub",
"format": "uncase plain"
}
}
]
}
]
}
]
}
```
### 3. scanning API
Given an example for how to use maat API (JSON File mode)
```C
#include <assert.h>
#include "maat.h"
#define ARRAY_SIZE 16
const char *json_filename = "./maat_json.json";
const char *table_info_path = "./table_info.conf";
int main()
{
// initialize maat options which will be used by maat_new()
struct maat_options *opts = maat_options_new();
maat_options_set_json_file(opts, json_filename);
maat_options_set_logger(opts, "./sample_test.log", LOG_LEVEL_INFO);
// create maat instance, rules in table_info.conf will be loaded.
struct maat *maat_instance = maat_new(opts, table_info_path);
assert(maat_instance != NULL);
maat_options_free(opts);
const char *table_name = "HTTP_URL"; //maat_json.json has HTTP_URL rule
int table_id = maat_get_table_id(maat_instance, table_name);
assert(table_id == 3); // defined in table_info.conf
int thread_id = 0;
long long results[ARRAY_SIZE] = {0};
size_t n_hit_result = 0;
struct maat_state *state = maat_state_new(maat_instance, thread_id);
assert(state != NULL);
const char *scan_data = "Hello Maat, nice to meet you";
/**
* Becase maat instance has loaded rule in table_info.conf which keywords is "Hello Maat",
so maat_scan_string should return hit flag and rule's compile_id stored in results array.
*/
int ret = maat_scan_string(maat_instance, table_id, scan_data, strlen(scan_data),
results, ARRAY_SIZE, &n_hit_result, state);
assert(ret == MAAT_SCAN_HIT);
assert(n_hit_result == 1);
assert(results[0] == 123);
maat_state_free(state);
return 0;
}
```
## More details ## More details
* [Introduction](./docs/introduction.md) * [Overview](./docs/overview.md)
* [Table schema](./docs/table_schema.md) * [Terminology](./docs/terminology.md)
* [Rules](./docs/table_data.md) * [Maat table](./docs/maat_table.md)
* [Logical operation](./docs/logical_operation.md) * [Configuration management](./docs/configuration_management.md)
* [Scan API](./docs/scan_api.md) * [Group hierarchy](./docs/group_hierarchy.md)
* [Thread mode](./docs/thread_mode.md) * [Logical combinations](./docs/logical_combinations.md)
* [Tools](./docs/tools.md) * [Thread model](./docs/thread_model.md)
* [Monitor tools](./docs/monitor_tools.md)
* [API reference](./docs/api_reference.md)
* [Performance](./docs/overview.md#6-performance)

4
test/CMakeLists.txt
View File

@@ -37,7 +37,7 @@ target_link_libraries(bool_matcher_gtest maat_frame_static gtest_static)
add_executable(maat_ex_data_gtest maat_ex_data_gtest.cpp) add_executable(maat_ex_data_gtest maat_ex_data_gtest.cpp)
target_link_libraries(maat_ex_data_gtest maat_frame_static gtest_static) target_link_libraries(maat_ex_data_gtest maat_frame_static gtest_static)
add_subdirectory(group_exclude) add_subdirectory(group_nesting)
add_subdirectory(ipport_plugin) add_subdirectory(ipport_plugin)
add_subdirectory(benchmark) add_subdirectory(benchmark)
@@ -52,7 +52,7 @@ file(COPY tsgrule DESTINATION ./)
file(COPY testdata DESTINATION ./) file(COPY testdata DESTINATION ./)
file(COPY test_streamfiles DESTINATION ./) file(COPY test_streamfiles DESTINATION ./)
file(COPY json_update DESTINATION ./) file(COPY json_update DESTINATION ./)
file(COPY group_exclude DESTINATION ./) file(COPY group_nesting DESTINATION ./)
file(COPY ipport_plugin DESTINATION ./) file(COPY ipport_plugin DESTINATION ./)
file(COPY benchmark DESTINATION ./) file(COPY benchmark DESTINATION ./)

4
test/group_exclude/CMakeLists.txt
View File

@@ -1,4 +0,0 @@
include_directories(${PROJECT_SOURCE_DIR}/src/inc_internal)
add_executable(group_exclude_gtest group_exclude_gtest.cpp)
target_link_libraries(group_exclude_gtest maat_frame_static gtest_static)

4
test/group_nesting/CMakeLists.txt Normal file
View File

@@ -0,0 +1,4 @@
include_directories(${PROJECT_SOURCE_DIR}/src/inc_internal)
add_executable(group_nesting_gtest group_nesting_gtest.cpp)
target_link_libraries(group_nesting_gtest maat_frame_static gtest_static)

0
test/group_exclude/group_exclude_L3.conf → test/group_nesting/group_nesting_L3.conf
View File

0
test/group_exclude/group_exclude_L3.png → test/group_nesting/group_nesting_L3.png
View File

Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 200 KiB

After

Width:  |  Height:  |  Size: 200 KiB

0
test/group_exclude/group_exclude_L4.conf → test/group_nesting/group_nesting_L4.conf
View File

0
test/group_exclude/group_exclude_L4.png → test/group_nesting/group_nesting_L4.png
View File

Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 74 KiB

After

Width:  |  Height:  |  Size: 74 KiB

0
test/group_exclude/group_exclude_exceed.conf → test/group_nesting/group_nesting_exceed.conf
View File

0
test/group_exclude/group_exclude_exceed.png → test/group_nesting/group_nesting_exceed.png
View File

Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 46 KiB

After

Width:  |  Height:  |  Size: 46 KiB

38
test/group_exclude/group_exclude_gtest.cpp → test/group_nesting/group_nesting_gtest.cpp
View File

@@ -8,7 +8,7 @@
#include "maat_group.h" #include "maat_group.h"
#include <gtest/gtest.h> #include <gtest/gtest.h>
#define MODULE_GROUP_EXCLUDE_GTEST module_name_str("maat.group_exclude_gtest") #define MODULE_GROUP_NESTING_GTEST module_name_str("maat.group_nesting_gtest")
#define MAX_IDS_STR_LEN 64 #define MAX_IDS_STR_LEN 64
#define MAX_ITEM_NUM 64 #define MAX_ITEM_NUM 64
@@ -129,13 +129,13 @@ protected:
int ret = load_file_to_memory(g_table_info_path, &json_buff, &json_buff_size); int ret = load_file_to_memory(g_table_info_path, &json_buff, &json_buff_size);
if (ret < 0) { if (ret < 0) {
log_fatal(logger, MODULE_GROUP_EXCLUDE_GTEST, "load_file_to_memory failed."); log_fatal(logger, MODULE_GROUP_NESTING_GTEST, "load_file_to_memory failed.");
assert(0); assert(0);
} }
cJSON *root = cJSON_Parse((const char *)json_buff); cJSON *root = cJSON_Parse((const char *)json_buff);
if (!root) { if (!root) {
log_fatal(logger, MODULE_GROUP_EXCLUDE_GTEST, "cJSON_Parse failed."); log_fatal(logger, MODULE_GROUP_NESTING_GTEST, "cJSON_Parse failed.");
assert(0); assert(0);
} }
@@ -144,7 +144,7 @@ protected:
cJSON_Delete(root); cJSON_Delete(root);
if (NULL == g2g_schema) { if (NULL == g2g_schema) {
log_fatal(logger, MODULE_GROUP_EXCLUDE_GTEST, "group2group_schema_new failed."); log_fatal(logger, MODULE_GROUP_NESTING_GTEST, "group2group_schema_new failed.");
assert(0); assert(0);
} }
} }
@@ -171,12 +171,12 @@ TEST_F(MaatGroupExclude, level_3_function) {
void *g2g_runtime = group2group_runtime_new(g2g_schema, 1, garbage_bin, logger); void *g2g_runtime = group2group_runtime_new(g2g_schema, 1, garbage_bin, logger);
if (NULL == g2g_runtime) { if (NULL == g2g_runtime) {
log_fatal(logger, MODULE_GROUP_EXCLUDE_GTEST, "group2group_runtime_new failed."); log_fatal(logger, MODULE_GROUP_NESTING_GTEST, "group2group_runtime_new failed.");
assert(0); assert(0);
} }
memset(&rules, 0, sizeof(rules)); memset(&rules, 0, sizeof(rules));
int ret = parse_config_file("group_exclude_L3.conf", &rules); int ret = parse_config_file("group_nesting_L3.conf", &rules);
EXPECT_EQ(ret, 0); EXPECT_EQ(ret, 0);
for (size_t i = 0; i < rules.n_add_item; i++) { for (size_t i = 0; i < rules.n_add_item; i++) {
@@ -236,12 +236,12 @@ TEST_F(MaatGroupExclude, level_3_perf) {
void *g2g_runtime = group2group_runtime_new(g2g_schema, 1, garbage_bin, logger); void *g2g_runtime = group2group_runtime_new(g2g_schema, 1, garbage_bin, logger);
if (NULL == g2g_runtime) { if (NULL == g2g_runtime) {
log_fatal(logger, MODULE_GROUP_EXCLUDE_GTEST, "group2group_runtime_new failed."); log_fatal(logger, MODULE_GROUP_NESTING_GTEST, "group2group_runtime_new failed.");
assert(0); assert(0);
} }
memset(&rules, 0, sizeof(rules)); memset(&rules, 0, sizeof(rules));
int ret = parse_config_file("group_exclude_L3.conf", &rules); int ret = parse_config_file("group_nesting_L3.conf", &rules);
EXPECT_EQ(ret, 0); EXPECT_EQ(ret, 0);
for (size_t i = 0; i < rules.n_add_item; i++) { for (size_t i = 0; i < rules.n_add_item; i++) {
@@ -266,7 +266,7 @@ TEST_F(MaatGroupExclude, level_3_perf) {
} }
clock_gettime(CLOCK_MONOTONIC, &end); clock_gettime(CLOCK_MONOTONIC, &end);
long long time_elapse_ms = (end.tv_sec - start.tv_sec) * 1000 + (end.tv_nsec - start.tv_nsec) / 1000000; long long time_elapse_ms = (end.tv_sec - start.tv_sec) * 1000 + (end.tv_nsec - start.tv_nsec) / 1000000;
log_info(logger, MODULE_GROUP_EXCLUDE_GTEST, "level_3_basic hit 2 super_groups scan consume time %lldms", log_info(logger, MODULE_GROUP_NESTING_GTEST, "level_3_basic hit 2 super_groups scan consume time %lldms",
time_elapse_ms); time_elapse_ms);
//delete group_id = 7, super_group_id = 6, is_exclude = 1 //delete group_id = 7, super_group_id = 6, is_exclude = 1
@@ -283,7 +283,7 @@ TEST_F(MaatGroupExclude, level_3_perf) {
} }
clock_gettime(CLOCK_MONOTONIC, &end); clock_gettime(CLOCK_MONOTONIC, &end);
time_elapse_ms = (end.tv_sec - start.tv_sec) * 1000 + (end.tv_nsec - start.tv_nsec) / 1000000; time_elapse_ms = (end.tv_sec - start.tv_sec) * 1000 + (end.tv_nsec - start.tv_nsec) / 1000000;
log_info(logger, MODULE_GROUP_EXCLUDE_GTEST, "level_3_basic hit 3 super_groups scan consume time %lldms", log_info(logger, MODULE_GROUP_NESTING_GTEST, "level_3_basic hit 3 super_groups scan consume time %lldms",
time_elapse_ms); time_elapse_ms);
//delete group_id = 13, super_group_id = 4, is_exclude = 1 //delete group_id = 13, super_group_id = 4, is_exclude = 1
@@ -300,7 +300,7 @@ TEST_F(MaatGroupExclude, level_3_perf) {
} }
clock_gettime(CLOCK_MONOTONIC, &end); clock_gettime(CLOCK_MONOTONIC, &end);
time_elapse_ms = (end.tv_sec - start.tv_sec) * 1000 + (end.tv_nsec - start.tv_nsec) / 1000000; time_elapse_ms = (end.tv_sec - start.tv_sec) * 1000 + (end.tv_nsec - start.tv_nsec) / 1000000;
log_info(logger, MODULE_GROUP_EXCLUDE_GTEST, "level_3_basic hit 4 super_groups scan consume time %lldms", log_info(logger, MODULE_GROUP_NESTING_GTEST, "level_3_basic hit 4 super_groups scan consume time %lldms",
time_elapse_ms); time_elapse_ms);
group2group_runtime_free(g2g_runtime); group2group_runtime_free(g2g_runtime);
@@ -313,12 +313,12 @@ TEST_F(MaatGroupExclude, level_4_function) {
void *g2g_runtime = group2group_runtime_new(g2g_schema, 1, garbage_bin, logger); void *g2g_runtime = group2group_runtime_new(g2g_schema, 1, garbage_bin, logger);
if (NULL == g2g_runtime) { if (NULL == g2g_runtime) {
log_fatal(logger, MODULE_GROUP_EXCLUDE_GTEST, "group2group_runtime_new failed."); log_fatal(logger, MODULE_GROUP_NESTING_GTEST, "group2group_runtime_new failed.");
assert(0); assert(0);
} }
memset(&rules, 0, sizeof(rules)); memset(&rules, 0, sizeof(rules));
int ret = parse_config_file("group_exclude_L4.conf", &rules); int ret = parse_config_file("group_nesting_L4.conf", &rules);
EXPECT_EQ(ret, 0); EXPECT_EQ(ret, 0);
for (size_t i = 0; i < rules.n_add_item; i++) { for (size_t i = 0; i < rules.n_add_item; i++) {
@@ -370,12 +370,12 @@ TEST_F(MaatGroupExclude, level_4_perf) {
void *g2g_runtime = group2group_runtime_new(g2g_schema, 1, garbage_bin, logger); void *g2g_runtime = group2group_runtime_new(g2g_schema, 1, garbage_bin, logger);
if (NULL == g2g_runtime) { if (NULL == g2g_runtime) {
log_fatal(logger, MODULE_GROUP_EXCLUDE_GTEST, "group2group_runtime_new failed."); log_fatal(logger, MODULE_GROUP_NESTING_GTEST, "group2group_runtime_new failed.");
assert(0); assert(0);
} }
memset(&rules, 0, sizeof(rules)); memset(&rules, 0, sizeof(rules));
int ret = parse_config_file("group_exclude_L4.conf", &rules); int ret = parse_config_file("group_nesting_L4.conf", &rules);
EXPECT_EQ(ret, 0); EXPECT_EQ(ret, 0);
for (size_t i = 0; i < rules.n_add_item; i++) { for (size_t i = 0; i < rules.n_add_item; i++) {
@@ -400,7 +400,7 @@ TEST_F(MaatGroupExclude, level_4_perf) {
} }
clock_gettime(CLOCK_MONOTONIC, &end); clock_gettime(CLOCK_MONOTONIC, &end);
long long time_elapse_ms = (end.tv_sec - start.tv_sec) * 1000 + (end.tv_nsec - start.tv_nsec) / 1000000; long long time_elapse_ms = (end.tv_sec - start.tv_sec) * 1000 + (end.tv_nsec - start.tv_nsec) / 1000000;
log_info(logger, MODULE_GROUP_EXCLUDE_GTEST, "level_4_basic hit 5 super_groups scan consume time %lldms", log_info(logger, MODULE_GROUP_NESTING_GTEST, "level_4_basic hit 5 super_groups scan consume time %lldms",
time_elapse_ms); time_elapse_ms);
//delete group_id = 10, super_group_id = 6, is_exclude = 1 //delete group_id = 10, super_group_id = 6, is_exclude = 1
@@ -417,7 +417,7 @@ TEST_F(MaatGroupExclude, level_4_perf) {
} }
clock_gettime(CLOCK_MONOTONIC, &end); clock_gettime(CLOCK_MONOTONIC, &end);
time_elapse_ms = (end.tv_sec - start.tv_sec) * 1000 + (end.tv_nsec - start.tv_nsec) / 1000000; time_elapse_ms = (end.tv_sec - start.tv_sec) * 1000 + (end.tv_nsec - start.tv_nsec) / 1000000;
log_info(logger, MODULE_GROUP_EXCLUDE_GTEST, "level_4_basic hit 7 super_groups scan consume time %lldms", log_info(logger, MODULE_GROUP_NESTING_GTEST, "level_4_basic hit 7 super_groups scan consume time %lldms",
time_elapse_ms); time_elapse_ms);
group2group_runtime_free(g2g_runtime); group2group_runtime_free(g2g_runtime);
@@ -430,12 +430,12 @@ TEST_F(MaatGroupExclude, level_exceed_function) {
void *g2g_runtime = group2group_runtime_new(g2g_schema, 1, garbage_bin, logger); void *g2g_runtime = group2group_runtime_new(g2g_schema, 1, garbage_bin, logger);
if (NULL == g2g_runtime) { if (NULL == g2g_runtime) {
log_fatal(logger, MODULE_GROUP_EXCLUDE_GTEST, "group2group_runtime_new failed."); log_fatal(logger, MODULE_GROUP_NESTING_GTEST, "group2group_runtime_new failed.");
assert(0); assert(0);
} }
memset(&rules, 0, sizeof(rules)); memset(&rules, 0, sizeof(rules));
int ret = parse_config_file("group_exclude_exceed.conf", &rules); int ret = parse_config_file("group_nesting_exceed.conf", &rules);
EXPECT_EQ(ret, 0); EXPECT_EQ(ret, 0);
for (size_t i = 0; i < rules.n_add_item; i++) { for (size_t i = 0; i < rules.n_add_item; i++) {

0
test/group_exclude/group_exclude_table_info.conf → test/group_nesting/group_nesting_table_info.conf
View File