gfwleak/galaxy-deployment-schema-updater-tool
Archived
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

原schema-upgrade项目更名，发布初版

Browse Source
This commit is contained in:
qidaijie
2023-09-26 14:48:35 +08:00
parent 28f935a8fc
commit ae9ea847dc
70 changed files with 30477 additions and 80 deletions
Download Patch File Download Diff File Expand all files Collapse all files

89
README.md
View File

@@ -2,91 +2,20 @@
## Getting started
## 简介
To make it easy for you to get started with GitLab, here's a list of recommended next steps.
Galaxy-qgw-service Schema版本更新工具包，用于不同版本之间的Schema更新，保留用户已配置的TTL和字段状态信息。
Already a pro? Just edit this README.md and make it your own. Want to make it easy? [Use the template at the bottom](#editing-this-readme)!
对ttl和visibility进行远程配置的拉取，进行对应规则的修改，并保存本地配置文件和推送修改后的文件到nacos
## Add your files
- [ ] [Create](https://docs.gitlab.com/ee/user/project/repository/web_editor.html#create-a-file) or [upload](https://docs.gitlab.com/ee/user/project/repository/web_editor.html#upload-a-file) files
- [ ] [Add files using the command line](https://docs.gitlab.com/ee/gitlab-basics/add-file.html#add-a-file-using-the-command-line) or push an existing Git repository with the following command:
## 启动方式
```
cd existing_repo
git remote add origin https://git.mesalab.cn/galaxy/deployment/schema-updater-tool.git
git branch -M main
git push -uf origin main
java -jar schema-updater-tool-{version}.jar {schema folder} {nacos address} {nacos namespcae}
```
## Integrate with your tools
样例：
```
java -jar schema-updater-tool-3.1.jar schema/ 192.168.44.12 test
```
- [ ] [Set up project integrations](https://git.mesalab.cn/galaxy/deployment/schema-updater-tool/-/settings/integrations)
## Collaborate with your team
- [ ] [Invite team members and collaborators](https://docs.gitlab.com/ee/user/project/members/)
- [ ] [Create a new merge request](https://docs.gitlab.com/ee/user/project/merge_requests/creating_merge_requests.html)
- [ ] [Automatically close issues from merge requests](https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically)
- [ ] [Enable merge request approvals](https://docs.gitlab.com/ee/user/project/merge_requests/approvals/)
- [ ] [Automatically merge when pipeline succeeds](https://docs.gitlab.com/ee/user/project/merge_requests/merge_when_pipeline_succeeds.html)
## Test and Deploy
Use the built-in continuous integration in GitLab.
- [ ] [Get started with GitLab CI/CD](https://docs.gitlab.com/ee/ci/quick_start/index.html)
- [ ] [Analyze your code for known vulnerabilities with Static Application Security Testing(SAST)](https://docs.gitlab.com/ee/user/application_security/sast/)
- [ ] [Deploy to Kubernetes, Amazon EC2, or Amazon ECS using Auto Deploy](https://docs.gitlab.com/ee/topics/autodevops/requirements.html)
- [ ] [Use pull-based deployments for improved Kubernetes management](https://docs.gitlab.com/ee/user/clusters/agent/)
- [ ] [Set up protected environments](https://docs.gitlab.com/ee/ci/environments/protected_environments.html)
***
# Editing this README
When you're ready to make this README your own, just edit this file and use the handy template below (or feel free to structure it however you want - this is just a starting point!). Thank you to [makeareadme.com](https://www.makeareadme.com/) for this template.
## Suggestions for a good README
Every project is different, so consider which of these sections apply to yours. The sections used in the template are suggestions for most open source projects. Also keep in mind that while a README can be too long and detailed, too long is better than too short. If you think your README is too long, consider utilizing another form of documentation rather than cutting out information.
## Name
Choose a self-explaining name for your project.
## Description
Let people know what your project can do specifically. Provide context and add a link to any reference visitors might be unfamiliar with. A list of Features or a Background subsection can also be added here. If there are alternatives to your project, this is a good place to list differentiating factors.
## Badges
On some READMEs, you may see small images that convey metadata, such as whether or not all the tests are passing for the project. You can use Shields to add some to your README. Many services also have instructions for adding a badge.
## Visuals
Depending on what you are making, it can be a good idea to include screenshots or even a video (you'll frequently see GIFs rather than actual videos). Tools like ttygif can help, but check out Asciinema for a more sophisticated method.
## Installation
Within a particular ecosystem, there may be a common way of installing things, such as using Yarn, NuGet, or Homebrew. However, consider the possibility that whoever is reading your README is a novice and would like more guidance. Listing specific steps helps remove ambiguity and gets people to using your project as quickly as possible. If it only runs in a specific context like a particular programming language version or operating system or has dependencies that have to be installed manually, also add a Requirements subsection.
## Usage
Use examples liberally, and show the expected output if you can. It's helpful to have inline the smallest example of usage that you can demonstrate, while providing links to more sophisticated examples if they are too long to reasonably include in the README.
## Support
Tell people where they can go to for help. It can be any combination of an issue tracker, a chat room, an email address, etc.
## Roadmap
If you have ideas for releases in the future, it is a good idea to list them in the README.
## Contributing
State if you are open to contributions and what your requirements are for accepting them.
For people who want to make changes to your project, it's helpful to have some documentation on how to get started. Perhaps there is a script that they should run or some environment variables that they need to set. Make these steps explicit. These instructions could also be useful to your future self.
You can also document commands to lint the code or run tests. These steps help to ensure high code quality and reduce the likelihood that the changes inadvertently break something. Having instructions for running tests is especially helpful if it requires external setup, such as starting a Selenium server for testing in a browser.
## Authors and acknowledgment
Show your appreciation to those who have contributed to the project.
## License
For open source projects, say how it is licensed.
## Project status
If you have run out of energy or time for your project, put a note at the top of the README saying that development has slowed down or stopped completely. Someone may choose to fork your project or volunteer to step in as a maintainer or owner, allowing your project to keep going. You can also make an explicit request for maintainers.

157
pom.xml Normal file
View File

@@ -0,0 +1,157 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.zdjizhi</groupId>
<artifactId>schema-updater-tool</artifactId>
<version>v3.1</version>
<name>schema-updater-tool</name>
<url>http://www.example.com</url>
<repositories>
<repository>
<id>nexus</id>
<name>Team Nexus Repository</name>
<url>http://192.168.40.153:8099/content/groups/public</url>
</repository>
<repository>
<id>maven-ali</id>
<url>http://maven.aliyun.com/nexus/content/groups/public/</url>
<releases>
<!--<enabled>true</enabled>-->
</releases>
<snapshots>
<!--<enabled>true</enabled>-->
<checksumPolicy>fail</checksumPolicy>
</snapshots>
</repository>
</repositories>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<nacos.version>1.4.1</nacos.version>
<galaxy.tools.version>1.2</galaxy.tools.version>
<zdjz.tools.version>1.0.8</zdjz.tools.version>
<scope.type>provided</scope.type>
<!--<scope.type>compile</scope.type>-->
</properties>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.4.2</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<transformers>
<transformer
implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>com.zdjizhi.topology.UpgradeSchema</mainClass>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>io.github.zlika</groupId>
<artifactId>reproducible-build-maven-plugin</artifactId>
<version>0.2</version>
<executions>
<execution>
<goals>
<goal>strip-jar</goal>
</goals>
<phase>package</phase>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>2.3.2</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
</plugins>
<resources>
<resource>
<directory>properties</directory>
<includes>
<include>**/*.properties</include>
<include>**/*.xml</include>
</includes>
<filtering>false</filtering>
</resource>
<resource>
<directory>src\main\java</directory>
<includes>
<include>log4j.properties</include>
</includes>
<filtering>false</filtering>
</resource>
</resources>
</build>
<dependencies>
<dependency>
<groupId>com.zdjizhi</groupId>
<artifactId>galaxy</artifactId>
<version>${zdjz.tools.version}</version>
<exclusions>
<exclusion>
<artifactId>slf4j-log4j12</artifactId>
<groupId>org.slf4j</groupId>
</exclusion>
<exclusion>
<artifactId>log4j-over-slf4j</artifactId>
<groupId>org.slf4j</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.12</version>
<scope>test</scope>
</dependency>
<!-- https://mvnrepository.com/artifact/com.alibaba.nacos/nacos-client -->
<dependency>
<groupId>com.alibaba.nacos</groupId>
<artifactId>nacos-client</artifactId>
<version>${nacos.version}</version>
</dependency>
<dependency>
<groupId>cn.hutool</groupId>
<artifactId>hutool-all</artifactId>
<version>5.7.17</version>
</dependency>
<dependency>
<groupId>com.jayway.jsonpath</groupId>
<artifactId>json-path</artifactId>
<version>2.4.0</version>
</dependency>
</dependencies>
</project>

12
properties/common_config.properties Normal file
View File

@@ -0,0 +1,12 @@
#nacos username
nacos.username=nacos
#nacos password
nacos.pin=nacos
#nacos group
nacos.group=Galaxy
#not Schema tables
non.schema.tables=ck-filter.json,druid-filter.json,es-filter.json,hbase-filter.json,engine-filter.json,public_code_info.json,public_schema_info.json,version.json,knowledge_base.json,liveChart_interim.json,liveChart_session.json

19
src/main/java/com/zdjizhi/common/SchemaConfig.java Normal file
View File

@@ -0,0 +1,19 @@
package com.zdjizhi.common;
import com.zdjizhi.utils.system.SchemaConfigurations;
/**
* @author Administrator
*/
public class SchemaConfig {
/**
* Nacos
*/
public static final String NACOS_PIN = SchemaConfigurations.getStringProperty(0, "nacos.pin");
public static final String NACOS_GROUP = SchemaConfigurations.getStringProperty(0, "nacos.group");
public static final String NACOS_USERNAME = SchemaConfigurations.getStringProperty(0, "nacos.username");
public static final String NON_SCHEMA_TABLES = SchemaConfigurations.getStringProperty(0, "non.schema.tables");
}

233
src/main/java/com/zdjizhi/topology/UpgradeSchema.java Normal file
View File

@@ -0,0 +1,233 @@
package com.zdjizhi.topology;
import cn.hutool.json.JSONArray;
import cn.hutool.json.JSONObject;
import cn.hutool.log.Log;
import cn.hutool.log.LogFactory;
import com.jayway.jsonpath.Configuration;
import com.jayway.jsonpath.JsonPath;
import com.zdjizhi.common.SchemaConfig;
import com.zdjizhi.utils.JsonUtil;
import com.zdjizhi.utils.StringUtil;
import com.zdjizhi.utils.nacos.ApiUtil;
import org.apache.commons.io.FileUtils;
import java.io.File;
import java.io.IOException;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.Map;
public class UpgradeSchema {
private static final Log logger = LogFactory.get();
@SuppressWarnings("unchecked")
public static void main(String[] args) {
try {
String path = args[0];
String nacos_server = args[1];
String namespace = args[2];
File file = new File(path);
File[] fileNames = file.listFiles();
if (fileNames != null) {
for (File fileName : fileNames) {
if (!fileName.isDirectory()) {
String dataId = fileName.getName();
if (!SchemaConfig.NON_SCHEMA_TABLES.contains(dataId)) {
String oldSchema = ApiUtil.getConfiguration(nacos_server, dataId, namespace);
if (StringUtil.isNotBlank(oldSchema)) {
String tmpSchema = FileUtils.readFileToString(fileName, "UTF-8");
if ("json".equals(JsonUtil.lastName(dataId)) && documentTypeJudgment(dataId, tmpSchema) && StringUtil.isNotBlank(tmpSchema)) {
logger.info("================开始对{}表进行更新操作================", dataId);
Map<String, Integer> ttlMap = new HashMap<>(16);
ttlMap.putAll(getOldTTL(oldSchema));
Map<String, String> visibilityMap = new HashMap<>(16);
visibilityMap.putAll(getOldVisibility(oldSchema));
JSONObject json = new JSONObject(tmpSchema, false, true);
changeTableTTL(oldSchema, json);
String newSchema = upgradeSchema(json, ttlMap, visibilityMap);
boolean status = ApiUtil.pushConfiguration(nacos_server, dataId, newSchema, namespace);
outMessage(status, dataId);
} else {
boolean status = ApiUtil.pushConfiguration(nacos_server, dataId, tmpSchema, namespace);
outMessage(status, dataId);
}
} else {
logger.warn("Nacos未查询到{}，判断为新文件，直接推送至Nacos", dataId);
boolean status = ApiUtil.pushConfiguration(nacos_server, dataId, FileUtils.readFileToString(fileName, "UTF-8"), namespace);
outMessage(status, dataId);
}
} else {
logger.warn("{}非Schema表，直接推送至Nacos", dataId);
boolean status = ApiUtil.pushConfiguration(nacos_server, dataId, FileUtils.readFileToString(fileName, "UTF-8"), namespace);
outMessage(status, dataId);
}
}
}
}
} catch (ArrayIndexOutOfBoundsException ae) {
logger.error("Please enter parameters!");
logger.error("example: java -jar schema-updater-tool-{version}.jar {schema folder} {nacos address} {nacos namespcae}");
logger.error("example: java -jar schema-updater-tool-3.1.jar schema/ 192.168.44.12 test");
ae.printStackTrace();
} catch (RuntimeException | IOException e) {
e.printStackTrace();
}
}
/**
* 获取nacos内已有schema的Visibility信息
*
* @param schema schema内容
* @return Visibility合集{name:Visibility}
*/
private static Map<String, String> getOldVisibility(String schema) {
Map<String, String> tmpMap = new HashMap<>(16);
Object document = Configuration.defaultConfiguration().jsonProvider().parse(schema);
ArrayList<Object> oldFields = JsonPath.read(document, "$.fields");
for (Object oldField : oldFields) {
try {
if (oldField.toString().contains("visibility")) {
String fieldName = JsonPath.read(oldField, "$.name");
String fieldVisbility = JsonPath.read(oldField, "$.doc.visibility");
tmpMap.put(fieldName, fieldVisbility);
}
} catch (Exception e) {
logger.error("解析visibility信息异常" + e.getMessage());
}
}
return tmpMap;
}
/**
* 获取nacos内已有schema的TTL信息
*
* @param schema schema内容
* @return TTL合集{name:ttl}
*/
private static Map<String, Integer> getOldTTL(String schema) {
Map<String, Integer> tmpMap = new HashMap<>(16);
try {
JSONObject json = new JSONObject(schema, false, true);
if (json.containsKey("fields")) {
JSONArray oldFields = json.getJSONArray("fields");
for (Object oldField : oldFields) {
JSONObject jsonTmp = new JSONObject(oldField.toString(), false, true);
String fieldName = jsonTmp.getStr("name");
JSONObject doc = new JSONObject(jsonTmp.getStr("doc"), false, true);
if (doc.containsKey("ttl")) {
tmpMap.put(fieldName, doc.getInt("ttl"));
}
}
}
} catch (Exception e) {
logger.error("解析ttl信息异常" + e.getMessage());
}
return tmpMap;
}
/**
* 获取nacos内已有schema表TTL，修改新Schema表TTL
*
* @param schema 旧schema内容
* @param json 新schema-json
*/
private static void changeTableTTL(String schema, JSONObject json) {
Object document = Configuration.defaultConfiguration().jsonProvider().parse(schema);
try {
Integer oldTTL = JsonPath.read(document, "$.doc.ttl");
if (oldTTL != null) {
json.getJSONObject("doc").set("ttl", oldTTL);
}
} catch (RuntimeException e) {
logger.warn("该表不包含表TTL信息!");
}
}
/**
* 更新schema
*
* @param json schema
* @param ttlMap ttl数据
* @return 同步ttl后的schema
*/
private static String upgradeSchema(JSONObject json, Map<String, Integer> ttlMap, Map<String, String> visibilityMap) {
JSONArray newFields = json.getJSONArray("fields");
JSONArray tmpFields = new JSONArray();
for (Object fields : newFields) {
JSONObject jsonTmp = new JSONObject(fields.toString(), false, true);
String name = jsonTmp.getStr("name");
JSONObject doc = new JSONObject(jsonTmp.getStr("doc"), false, true);
if (ttlMap.containsKey(name)) {
Integer ttlValue = ttlMap.get(name);
if (ttlValue != null) {
doc.set("ttl", ttlMap.get(name));
logger.info("字段:{}的TTL值变更为：{}", name, ttlMap.get(name));
}
}
if (visibilityMap.containsKey(name)) {
String nowType = doc.get("visibility").toString();
String oldType = visibilityMap.get(name);
if (!oldType.equals(nowType)) {
if (!"hidden".equals(oldType)) {
if ("enabled".equals(nowType) || "disabled".equals(nowType)) {
doc.set("visibility", visibilityMap.get(name));
logger.info("字段:{}的visibility值变更为：{}", name, visibilityMap.get(name));
}
}
}
}
jsonTmp.set("doc", doc);
tmpFields.add(jsonTmp);
}
json.set("fields", tmpFields);
return json.toStringPretty();
}
/**
* 读取nacos内schema中内容，判断是否有需要处理的ttl字段和visibility字段
*
* @param schema schema内容
* @return 是否需要处理当前表
*/
private static boolean documentTypeJudgment(String filesName, String schema) {
try {
JSONObject json = new JSONObject(schema, false, true);
if (json.containsKey("fields")) {
JSONArray fields = json.getJSONArray("fields");
for (Object field : fields) {
JSONObject jsonTmp = new JSONObject(field.toString(), false, true);
if (jsonTmp.containsKey("doc")) {
JSONObject doc = new JSONObject(jsonTmp.getStr("doc"), false, true);
if (doc.containsKey("ttl") || doc.containsKey("visibility")) {
return true;
}
}
}
}
} catch (Exception e) {
logger.error(filesName + "解析JSON异常！异常信息为：" + e.getMessage());
}
return false;
}
private static void outMessage(boolean status, String dataId) {
if (status) {
logger.info("配置{}已成功推送至Nacos！", dataId);
} else {
logger.error("配置{}推送Nacos失败！", dataId);
}
}
}

29
src/main/java/com/zdjizhi/utils/JsonUtil.java Normal file
View File

@@ -0,0 +1,29 @@
package com.zdjizhi.utils;
/**
* @author qidaijie
* @Package com.zdjizhi.utils
* @Description:
* @date 2022/7/2615:32
*/
public class JsonUtil {
/**
* 截取文件的后缀名，判断文件类型是否处理
*
* @param fileName 要截取的文件
* @return 文件拓展名
*/
public static String lastName(String fileName) {
if (StringUtil.isNotBlank(fileName)) {
String[] split = fileName.split("\\.");
if (split.length > 1) {
return split[split.length - 1];
} else {
return "";
}
}
return "";
}
}

68
src/main/java/com/zdjizhi/utils/nacos/ApiUtil.java Normal file
View File

@@ -0,0 +1,68 @@
package com.zdjizhi.utils.nacos;
import cn.hutool.log.Log;
import cn.hutool.log.LogFactory;
import com.alibaba.nacos.api.NacosFactory;
import com.alibaba.nacos.api.PropertyKeyConst;
import com.alibaba.nacos.api.config.ConfigService;
import com.alibaba.nacos.api.exception.NacosException;
import com.zdjizhi.common.SchemaConfig;
import java.util.Properties;
public class ApiUtil {
private static final Log logger = LogFactory.get();
/**
* 获取nacos上的配置文件
*
* @param dataId 配置文件名称
* @return 配置内容
*/
public static String getConfiguration(String nacos_server, String dataId, String namespace) {
String content = null;
Properties properties = new Properties();
properties.setProperty(PropertyKeyConst.SERVER_ADDR, nacos_server);
properties.setProperty(PropertyKeyConst.NAMESPACE, namespace);
properties.setProperty(PropertyKeyConst.USERNAME, SchemaConfig.NACOS_USERNAME);
properties.setProperty(PropertyKeyConst.PASSWORD, SchemaConfig.NACOS_PIN);
try {
ConfigService configService = NacosFactory.createConfigService(properties);
content = configService.getConfig(dataId, SchemaConfig.NACOS_GROUP, 5000);
} catch (NacosException e) {
logger.error("Failed to get configuration content from NACOS! The exception message is:" + e.getMessage());
e.printStackTrace();
}
return content;
}
/**
* 推送配置文件到nacos
*
* @param dataId 配置文件名称
* @param schema 配置内容
* @return 推送状态
*/
public static boolean pushConfiguration(String nacos_server, String dataId, String schema, String namespace) {
boolean pushStatus = false;
Properties properties = new Properties();
properties.setProperty(PropertyKeyConst.SERVER_ADDR, nacos_server);
properties.setProperty(PropertyKeyConst.NAMESPACE, namespace);
properties.setProperty(PropertyKeyConst.USERNAME, SchemaConfig.NACOS_USERNAME);
properties.setProperty(PropertyKeyConst.PASSWORD, SchemaConfig.NACOS_PIN);
try {
String[] split = dataId.split("\\.");
String type = split[split.length - 1];
ConfigService configService = NacosFactory.createConfigService(properties);
pushStatus = configService.publishConfig(dataId, SchemaConfig.NACOS_GROUP, schema, type);
} catch (NacosException e) {
logger.error("Failed to push configuration to NACOS! The exception message is:" + e.getMessage());
e.printStackTrace();
}
return pushStatus;
}
}

58
src/main/java/com/zdjizhi/utils/system/SchemaConfigurations.java Normal file
View File

@@ -0,0 +1,58 @@
package com.zdjizhi.utils.system;
import com.zdjizhi.utils.StringUtil;
import java.io.IOException;
import java.util.Locale;
import java.util.Properties;
/**
* @author Administrator
*/
public final class SchemaConfigurations {
private static Properties propService = new Properties();
public static String getStringProperty(Integer type, String key) {
if (type == 0) {
return propService.getProperty(key);
} else {
return null;
}
}
public static Integer getIntProperty(Integer type, String key) {
if (type == 0) {
return Integer.parseInt(propService.getProperty(key));
} else {
return null;
}
}
public static Long getLongProperty(Integer type, String key) {
if (type == 0) {
return Long.parseLong(propService.getProperty(key));
} else {
return null;
}
}
public static Boolean getBooleanProperty(Integer type, String key) {
if (type == 0) {
return StringUtil.equals(propService.getProperty(key).toLowerCase().trim().toUpperCase(Locale.ENGLISH), "true");
} else {
return null;
}
}
static {
try {
propService.load(SchemaConfigurations.class.getClassLoader().getResourceAsStream("common_config.properties"));
} catch (IOException | RuntimeException e) {
propService = null;
}
}
}

25
src/main/log4j.properties Normal file
View File

@@ -0,0 +1,25 @@
#Log4j
log4j.rootLogger=console,file
# 控制台日志设置
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.Threshold=INFO
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=[%d{yyyy-MM-dd HH\:mm\:ss}] [%-5p] [Thread\:%t] %l %x - <%m>%n
# 文件日志设置
log4j.appender.file=org.apache.log4j.DailyRollingFileAppender
log4j.appender.file.Threshold=info
log4j.appender.file.encoding=UTF-8
log4j.appender.file.Append=true
#路径请用相对路径，做好相关测试输出到应用目下
log4j.appender.file.file=${nis.root}/log/galaxy-name.log
log4j.appender.file.DatePattern='.'yyyy-MM-dd
log4j.appender.file.layout=org.apache.log4j.PatternLayout
#log4j.appender.file.layout.ConversionPattern=%d{HH:mm:ss} %X{ip} [%t] %5p %c{1} %m%n
log4j.appender.file.layout.ConversionPattern=[%d{yyyy-MM-dd HH\:mm\:ss}] [%-5p] %X{ip} [Thread\:%t] %l %x - %m%n
##MyBatis 配置，com.nis.web.dao是mybatis接口所在包
#log4j.logger.com.nis.web.dao=debug
##bonecp数据源配置
#log4j.category.com.jolbox=debug,console

368
testSchemaFiles/active_defence_event.json Normal file
View File

@@ -0,0 +1,368 @@
{
"type": "record",
"name": "active_defence_event",
"namespace": "tsg_galaxy_v3",
"doc": {
"primary_key": "common_log_id",
"partition_key": "common_recv_time",
"index_key": [
"common_log_id",
"common_recv_time",
"common_policy_id"
],
"schema_query": {
"dimensions": [
"common_policy_id",
"ad_target_ip",
"ad_cc_target_url"
],
"metrics": [
"ad_target_ip",
"ad_sent_byte_num",
"ad_sent_pkt_num",
"ad_cc_initiate_connection_num",
"ad_cc_established_connection_num",
"ad_cc_rejected_connection_num"
],
"filters": [
"common_policy_id",
"ad_target_ip",
"ad_target_port",
"ad_protocol",
"common_address_type",
"ad_sent_byte_num",
"ad_sent_pkt_num",
"ad_cc_initiate_connection_num",
"ad_cc_established_connection_num",
"ad_cc_rejected_connection_num"
]
},
"schema_type": {
"REFLECTION": {
"columns": [
"common_recv_time",
"common_log_id",
"common_policy_id",
"common_address_type",
"common_device_id",
"common_egress_link_id",
"common_ingress_link_id",
"common_entrance_id",
"common_user_region",
"ad_method",
"ad_protocol",
"ad_target_ip",
"ad_target_port",
"ad_target_ip_location",
"ad_target_ip_asn",
"ad_reflector_profile_id",
"ad_sent_pkt_num",
"ad_sent_byte_num",
"ad_generate_time"
],
"default_columns": [
"common_recv_time",
"common_log_id",
"common_policy_id",
"ad_target_ip",
"ad_target_port",
"ad_reflector_profile_id",
"ad_sent_pkt_num",
"ad_sent_byte_num"
]
},
"FLOOD": {
"columns": [
"common_recv_time",
"common_log_id",
"common_policy_id",
"common_address_type",
"common_device_id",
"common_egress_link_id",
"common_ingress_link_id",
"common_entrance_id",
"common_user_region",
"ad_method",
"ad_protocol",
"ad_target_ip",
"ad_target_port",
"ad_target_ip_location",
"ad_target_ip_asn",
"ad_claimed_src_ip_profile_id",
"ad_sent_pkt_num",
"ad_sent_byte_num",
"ad_generate_time"
],
"default_columns": [
"common_recv_time",
"common_log_id",
"common_policy_id",
"ad_target_ip",
"ad_target_port",
"ad_claimed_src_ip_profile_id",
"ad_protocol"
]
},
"CC": {
"columns": [
"common_recv_time",
"common_log_id",
"common_policy_id",
"common_address_type",
"common_device_id",
"common_egress_link_id",
"common_ingress_link_id",
"common_entrance_id",
"common_user_region",
"ad_method",
"ad_protocol",
"ad_cc_target_url",
"ad_claimed_src_ip_profile_id",
"ad_cc_initiate_connection_num",
"ad_cc_established_connection_num",
"ad_cc_rejected_connection_num",
"ad_generate_time"
],
"default_columns": [
"common_recv_time",
"common_log_id",
"common_policy_id",
"ad_cc_target_url",
"ad_claimed_src_ip_profile_id",
"ad_protocol"
]
}
},
"default_columns": [
"common_recv_time",
"common_log_id",
"common_policy_id",
"ad_target_ip",
"ad_target_port",
"ad_cc_target_url"
]
},
"fields": [
{
"name": "common_recv_time",
"label": "Receive Time",
"doc": {
"constraints": {
"type": "timestamp"
},
"format": {
"functions": "current_timestamp"
},
"visibility": "enabled"
},
"type": "long"
},
{
"name": "common_log_id",
"label": "Log ID",
"doc": {
"format": {
"functions": "snowflake_id"
},
"visibility": "enabled"
},
"type": "long"
},
{
"name": "common_policy_id",
"label": "Policy ID",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "common_address_type",
"label": "Address Type",
"doc": {
"data": [
{
"code": "4",
"value": "ipv4"
},
{
"code": "6",
"value": "ipv6"
}
],
"visibility": "enabled"
},
"type": "int"
},
{
"name": "common_entrance_id",
"label": "Entrance ID",
"doc": {
"visibility": "disabled"
},
"type": "int"
},
{
"name": "common_device_id",
"label": "Device ID",
"doc": {
"visibility": "enabled"
},
"type": "string"
},
{
"name": "common_egress_link_id",
"label": "Egress Link ID",
"doc": {
"visibility": "hidden"
},
"type": "int"
},
{
"name": "common_ingress_link_id",
"label": "Ingress Link ID",
"doc": {
"visibility": "hidden"
},
"type": "int"
},
{
"name": "common_user_region",
"label": "User Region",
"doc": {
"visibility": "hidden"
},
"type": "string"
},
{
"name": "ad_target_ip",
"label": "Target IP",
"doc": {
"constraints": {
"type": "ip"
},
"format": {
"functions": "geo_ip_country,geo_asn",
"appendTo": "ad_target_ip_location,ad_target_ip_asn"
},
"visibility": "enabled"
},
"type": "string"
},
{
"name": "ad_target_port",
"label": "Target Port",
"doc": {
"visibility": "enabled"
},
"type": "int"
},
{
"name": "ad_cc_target_url",
"label": "Target URL",
"doc": {
"visibility": "enabled"
},
"type": "string"
},
{
"name": "ad_target_ip_location",
"label": "Target Location",
"doc": {
"visibility": "enabled"
},
"type": "string"
},
{
"name": "ad_target_ip_asn",
"label": "Target ASN",
"doc": {
"visibility": "enabled"
},
"type": "string"
},
{
"name": "ad_protocol",
"label": "Protocol",
"doc": {
"visibility": "enabled"
},
"type": "string"
},
{
"name": "ad_method",
"label": "Method",
"doc": {
"visibility": "enabled"
},
"type": "string"
},
{
"name": "ad_claimed_src_ip_profile_id",
"label": "Claimed Profile ID",
"doc": {
"visibility": "enabled"
},
"type": "int"
},
{
"name": "ad_reflector_profile_id",
"label": "Reflector Profile ID",
"doc": {
"visibility": "enabled"
},
"type": "int"
},
{
"name": "ad_sent_pkt_num",
"label": "Packets Sent",
"doc": {
"visibility": "enabled"
},
"type": "int"
},
{
"name": "ad_sent_byte_num",
"label": "Bytes Sent",
"doc": {
"visibility": "enabled"
},
"type": "int"
},
{
"name": "ad_cc_initiate_connection_num",
"label": "Initiate Numbers",
"doc": {
"visibility": "enabled"
},
"type": "int"
},
{
"name": "ad_cc_established_connection_num",
"label": "Established Numbers",
"doc": {
"visibility": "enabled"
},
"type": "int"
},
{
"name": "ad_cc_rejected_connection_num",
"label": "Rejected Numbers",
"doc": {
"visibility": "enabled"
},
"type": "int"
},
{
"name": "ad_generate_time",
"label": "Generate Time",
"doc": {
"constraints": {
"type": "timestamp"
},
"visibility": "enabled"
},
"type": "int"
}
]
}

110
testSchemaFiles/assessment_event.json Normal file
View File

@@ -0,0 +1,110 @@
{
"type": "record",
"name": "assessment_event",
"namespace": "tsg_galaxy_v3",
"doc": {
"primary_key": "common_log_id",
"partition_key": "common_recv_time",
"index_key": [
"common_log_id",
"common_recv_time"
],
"functions": {
"$ref": "public_schema_info.json#/functions"
}
},
"fields": [
{
"name": "common_recv_time",
"label": "Receive Time",
"doc": {
"constraints": {
"type": "timestamp"
},
"visibility": "enabled"
},
"type": "long"
},
{
"name": "common_log_id",
"label": "Log ID",
"doc": {
"format": {
"functions": "snowflake_id"
},
"visibility": "enabled"
},
"type": "long"
},
{
"name": "lot_number",
"label": "Lot Number",
"doc": {
"visibility": "enabled"
},
"type": "string"
},
{
"name": "file_name",
"label": "File Name",
"doc": {
"visibility": "enabled"
},
"type": "string"
},
{
"name": "features",
"label": "Features",
"doc": {
"visibility": "hidden"
},
"type": "string"
},
{
"name": "assessment_type",
"label": "Assessment Type",
"doc": {
"visibility": "enabled"
},
"type": "string"
},
{
"name": "size",
"label": "Size",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "file_checksum_sha",
"label": "SHA256",
"doc": {
"visibility": "enabled"
},
"type": "string"
},
{
"name": "assessment_date",
"label": "Assessment Date",
"doc": {
"constraints": {
"type": "timestamp"
},
"visibility": "enabled"
},
"type": "long"
},
{
"name": "assessment_file",
"label": "Assessment File",
"doc": {
"constraints": {
"type": "file"
},
"visibility": "enabled"
},
"type": "string"
}
]
}

99
testSchemaFiles/ck-filter.json Normal file
View File

@@ -0,0 +1,99 @@
{
"version": "1.0",
"name": "ClickHouse-Raw",
"namespace": "ClickHouse",
"filters": [
{
"name":"@start",
"value": "'2021-10-19 10:00:00'"
},
{
"name":"@end",
"value": "'2021-10-20 11:00:00'"
},
{
"name":"@common_filter",
"value": [
"common_log_id=1153021139190754263",
"common_client_ip='118.180.48.74'",
"common_client_ip='120.242.132.200'",
"common_internal_ip='223.116.37.192'",
"common_server_ip='8.8.8.8'",
"common_server_ip='114.114.114.114'",
"common_server_ip!='114.114.114.114'",
"common_server_ip='120.239.72.226'",
"common_external_ip='111.10.53.14'",
"common_client_port=52607",
"common_server_port=443",
"common_c2s_pkt_num>5",
"common_s2c_pkt_num>5",
"common_c2s_byte_num>100",
"common_s2c_byte_num<200",
"common_schema_type='DNS'",
"common_establish_latency_ms>200",
"common_con_duration_ms>10000",
"common_stream_trace_id=1153021139190754263",
"common_tcp_client_isn=2857077935",
"common_tcp_server_isn=0",
"http_domain='qq.com'",
"http_domain!='qq.com'",
"http_domain='yunser.com'",
"mail_account='abc@xx.com'",
"mail_subject='test'",
"dns_qname='qbwup.imtt.qq.com'",
"ssl_sni='mmbiz.qpic.cn'",
"ssl_sni='openai.qq.com'",
"ssl_con_latency_ms>100",
"ssl_ja3_hash='a0e9f5d64349fb13191bc781f81f42e1'",
"common_client_ip='36.189.226.21' and common_server_ip='8.8.8.8'",
"common_server_ip='111.10.53.14' and common_server_port=443",
"common_server_ip like '120.239%'",
"common_server_ip not like '120.239%'",
"common_server_ip like '%114.114%'",
"mail_account like 'abc@%'",
"http_domain like '%baidu.com%'",
"ssl_sni like '%google.com'",
"http_domain like 'baidu%'",
"http_domain like '%baidu.com%'",
"common_client_ip in ('120.239.72.226','114.114.114.114')",
"common_client_ip not in ('120.239.72.226','114.114.114.114')",
"common_server_ip='116.177.248.126' and notEmpty(http_domain)",
"common_server_ip='116.177.248.126' and common_client_ip='120.242.132.200'",
"common_server_ip='116.177.248.126' and common_stream_trace_id=1153021139190754263",
"common_client_ip='120.242.132.200' and common_server_ip='116.177.248.126'",
"http_domain='qq.com' or common_server_ip='120.239.72.226'",
"common_server_port not in (80,443)",
"http_domain not like '%qq.com'"
]
},
{
"name":"@index_filter",
"value": [
"common_log_id=1153021139190754263",
"common_client_ip='118.180.48.74'",
"common_client_ip='120.242.132.200'",
"common_server_ip='114.114.114.114'",
"common_server_ip!='114.114.114.114'",
"common_server_ip='120.239.72.226'",
"http_domain='qq.com'",
"http_domain!='qq.com'",
"http_domain='yunser.com'",
"ssl_sni='mmbiz.qpic.cn'",
"ssl_sni='openai.qq.com'",
"common_server_ip like '120.239%'",
"common_server_ip not like '120.239%'",
"common_server_ip like '%114.114%'",
"common_subscriber_id='%test%'",
"http_domain like 'baidu%'",
"http_domain like '%baidu.com%'",
"common_client_ip in ('120.239.72.226','114.114.114.114')",
"common_client_ip not in ('120.239.72.226','114.114.114.114')",
"common_server_ip='116.177.248.126' and notEmpty(http_domain)",
"common_server_ip='116.177.248.126' and common_client_ip='120.242.132.200'",
"common_server_ip='116.177.248.126' and common_stream_trace_id=1153021139190754263",
"common_client_ip='120.242.132.200' and common_server_ip='116.177.248.126'",
"http_domain='qq.com' or common_server_ip='120.239.72.226'"
]
}
]
}

118
testSchemaFiles/ck-queries-template.sql Normal file
View File

@@ -0,0 +1,118 @@
--Q01.Count(1)
select count(1) FROM tsg_galaxy_v3.session_record WHERE common_recv_time >= toUnixTimestamp(@start) AND common_recv_time < toUnixTimestamp(@end)
--Q02.All Fields Query (default)
SELECT * FROM tsg_galaxy_v3.session_record AS session_record WHERE common_recv_time >= toUnixTimestamp(@start) AND common_recv_time < toUnixTimestamp(@end) LIMIT 30
--Q03.All Fields Query order by Time desc
SELECT * FROM tsg_galaxy_v3.session_record AS session_record WHERE common_recv_time >= toUnixTimestamp(@start) AND common_recv_time < toUnixTimestamp(@end) ORDER BY common_recv_time DESC LIMIT 30
--Q04.All Fields Query order by Time asc
SELECT * FROM tsg_galaxy_v3.session_record AS session_record WHERE common_recv_time >= toUnixTimestamp(@start) AND common_recv_time < toUnixTimestamp(@end) ORDER BY common_recv_time asc LIMIT 30
--Q05.All Fields Query by Filter
SELECT * FROM tsg_galaxy_v3.session_record AS session_record WHERE common_recv_time >= toUnixTimestamp(@start) AND common_recv_time < toUnixTimestamp(@end) AND @common_filter ORDER BY common_recv_time DESC LIMIT 30
--Q06.Default Fields Query by Filter
SELECT toDateTime(common_recv_time) AS common_recv_time , common_log_id , common_client_ip , common_client_port , common_server_ip , common_server_port FROM tsg_galaxy_v3.session_record AS session_record WHERE common_recv_time >= toUnixTimestamp(@start) AND common_recv_time < toUnixTimestamp(@end) AND @common_filter ORDER BY common_recv_time DESC LIMIT 30
--Q07.All Fields Query (sub query by time)
SELECT * FROM tsg_galaxy_v3.session_record AS session_record WHERE toDateTime(common_recv_time) IN ( SELECT toDateTime(common_recv_time) FROM tsg_galaxy_v3.session_record AS session_record WHERE common_recv_time >= toUnixTimestamp(@start) AND common_recv_time < toUnixTimestamp(@end) ORDER BY common_recv_time DESC LIMIT 30 ) AND common_recv_time >= toUnixTimestamp(@start) AND common_recv_time < toUnixTimestamp(@end) ORDER BY common_recv_time DESC LIMIT 30
--Q08.All Fields Query (sub query by log id)
SELECT * FROM tsg_galaxy_v3.session_record AS session_record WHERE common_log_id IN ( SELECT common_log_id FROM tsg_galaxy_v3.session_record AS session_record WHERE common_recv_time >= toUnixTimestamp(@start) AND common_recv_time < toUnixTimestamp(@end) ORDER BY common_recv_time DESC LIMIT 30 ) AND common_recv_time >= toUnixTimestamp(@start) AND common_recv_time < toUnixTimestamp(@end) ORDER BY common_recv_time DESC LIMIT 30
--Q09.Default Field Query (sub query by time)
SELECT toDateTime(common_recv_time) AS common_recv_time_str , common_log_id , common_client_ip , common_client_port , common_server_ip , common_server_port FROM tsg_galaxy_v3.session_record AS session_record WHERE toDateTime(common_recv_time) IN ( SELECT toDateTime(common_recv_time) FROM tsg_galaxy_v3.session_record AS session_record WHERE common_recv_time >= toUnixTimestamp(@start) AND common_recv_time < toUnixTimestamp(@end) ORDER BY toDateTime(common_recv_time) DESC LIMIT 30 ) AND common_recv_time >= toUnixTimestamp(@start) AND common_recv_time < toUnixTimestamp(@end) ORDER BY common_recv_time DESC LIMIT 30
--Q10.Default Field Query (sub query by log id)
SELECT toDateTime(common_recv_time) AS common_recv_time_str , common_log_id , common_client_ip , common_client_port , common_server_ip , common_server_port FROM tsg_galaxy_v3.session_record AS session_record WHERE common_log_id IN ( select common_log_id FROM tsg_galaxy_v3.session_record AS session_record WHERE common_recv_time >= toUnixTimestamp(@start) AND common_recv_time < toUnixTimestamp(@end) ORDER BY toDateTime(common_recv_time) DESC LIMIT 30 ) AND ( common_recv_time >= toUnixTimestamp(@start) AND common_recv_time < toUnixTimestamp(@end)) ORDER BY common_recv_time DESC LIMIT 30
--Q11.Default Field Query by Server IP (sub query by log id with Index Table)
SELECT toDateTime(common_recv_time) AS common_recv_time_str , common_log_id , common_client_ip , common_client_port , common_server_ip , common_server_port FROM tsg_galaxy_v3.session_record AS session_record WHERE common_log_id IN ( Select common_log_id FROM tsg_galaxy_v3.session_record_common_server_ip AS session_record_common_server_ip WHERE common_recv_time >= toUnixTimestamp(@start) AND common_recv_time < toUnixTimestamp(@end) AND @index_filter ORDER BY toDateTime(common_recv_time) DESC LIMIT 30 ) AND ( common_recv_time >= toUnixTimestamp(@start) AND common_recv_time < toUnixTimestamp(@end) AND @index_filter ) ORDER BY common_recv_time DESC LIMIT 30
--Q12.Default Field Query by Client IP (sub query by log id with Index Table)
SELECT toDateTime(common_recv_time) AS common_recv_time_str , common_log_id , common_client_ip , common_client_port , common_server_ip , common_server_port FROM tsg_galaxy_v3.session_record AS session_record WHERE common_log_id IN ( Select common_log_id FROM tsg_galaxy_v3.session_record_common_client_ip AS session_record_common_client_ip WHERE common_recv_time >= toUnixTimestamp(@start) AND common_recv_time < toUnixTimestamp(@end) AND @index_filter ORDER BY toDateTime(common_recv_time) DESC LIMIT 30 ) AND ( common_recv_time >= toUnixTimestamp(@start) AND common_recv_time < toUnixTimestamp(@end) AND @index_filter ) ORDER BY common_recv_time DESC LIMIT 30
--Q13.Default Field Query by Domain (sub query by log id with Index Table)
SELECT toDateTime(common_recv_time) AS common_recv_time_str , common_log_id , common_client_ip , common_client_port , common_server_ip , common_server_port FROM tsg_galaxy_v3.session_record AS session_record WHERE common_log_id IN ( Select common_log_id FROM tsg_galaxy_v3.session_record_http_domain AS session_record_http_domain WHERE common_recv_time >= toUnixTimestamp(@start) AND common_recv_time < toUnixTimestamp(@end) AND @index_filter ORDER BY toDateTime(common_recv_time) DESC LIMIT 30 ) AND ( common_recv_time >= toUnixTimestamp(@start) AND common_recv_time < toUnixTimestamp(@end) AND @index_filter ) ORDER BY common_recv_time DESC LIMIT 30
--Q14.All Fields Query by Client IP (sub query by log id with index Table)
SELECT * FROM tsg_galaxy_v3.session_record AS session_record WHERE common_log_id IN ( SELECT common_log_id FROM tsg_galaxy_v3.session_record_common_client_ip AS session_record_common_client_ip WHERE common_recv_time >= toUnixTimestamp(@start) AND common_recv_time < toUnixTimestamp(@end) AND @index_filter ORDER BY toDateTime(common_recv_time) DESC LIMIT 30 ) AND ( common_recv_time >= toUnixTimestamp(@start) AND common_recv_time < toUnixTimestamp(@end) AND @index_filter ) ORDER BY common_recv_time desc LIMIT 30
--Q15.All Fields Query by Server IP(sub query by log id with index Table)
SELECT * FROM tsg_galaxy_v3.session_record AS session_record WHERE common_log_id IN ( SELECT common_log_id FROM tsg_galaxy_v3.session_record_common_server_ip AS session_record_common_server_ip WHERE common_recv_time >= toUnixTimestamp(@start) AND common_recv_time < toUnixTimestamp(@end) AND @index_filter ORDER BY common_recv_time LIMIT 30 ) AND ( common_recv_time >= toUnixTimestamp(@start) AND common_recv_time < toUnixTimestamp(@end) AND @index_filter ) ORDER BY common_recv_time desc LIMIT 30
--Q16.All Fields Query by Domain(sub query by log id with index Table)
SELECT * FROM tsg_galaxy_v3.session_record AS session_record WHERE common_log_id IN ( SELECT common_log_id FROM tsg_galaxy_v3.session_record_http_domain AS session_record_http_domain WHERE common_recv_time >= toUnixTimestamp(@start) AND common_recv_time < toUnixTimestamp(@end) AND @index_filter ORDER BY common_recv_time LIMIT 30 ) AND ( common_recv_time >= toUnixTimestamp(@start) AND common_recv_time < toUnixTimestamp(@end) AND @index_filter ) ORDER BY common_recv_time desc LIMIT 30
--Q17.Session Logs Sent to Database Trend(Time Grain 5 minute)
SELECT toUnixTimestamp(toDateTime(toStartOfInterval(toDateTime(common_recv_time),INTERVAL 5 MINUTE))) AS "Receive Time", count(common_log_id) AS "logs" FROM tsg_galaxy_v3.session_record AS session_record WHERE ( ( common_recv_time >= toUnixTimestamp(@start) AND common_recv_time < toUnixTimestamp(@end) ) ) GROUP BY "Receive Time" LIMIT 10000
--Q18.Traffic Bandwidth Trend(Time Grain 30 second)
SELECT toDateTime(toUnixTimestamp(toDateTime(toStartOfInterval(toDateTime(common_recv_time),INTERVAL 30 SECOND)))) AS stat_time, sum(common_c2s_byte_num) AS bytes_sent, sum(common_s2c_byte_num) AS bytes_received, sum(common_c2s_byte_num + common_s2c_byte_num) AS bytes, sum(common_c2s_pkt_num + common_s2c_pkt_num) AS packets, sum(common_sessions) AS sessions FROM tsg_galaxy_v3.session_record AS session_record WHERE common_recv_time >= toDateTime(@start) AND common_recv_time < toDateTime(@end) GROUP BY stat_time ORDER BY stat_time ASC LIMIT 10000
--Q19.Log Tend by Type (Time Grain 5 minute)
SELECT toDateTime(toUnixTimestamp(toDateTime(toStartOfInterval(toDateTime(common_recv_time),INTERVAL 5 MINUTE)))) AS stat_time, common_schema_type AS type, sum(common_sessions) AS sessions, sum(common_c2s_byte_num + common_s2c_byte_num) AS bytes, sum(common_c2s_pkt_num + common_s2c_pkt_num) AS packets FROM tsg_galaxy_v3.session_record AS session_record WHERE common_recv_time >= toUnixTimestamp(@start) AND common_recv_time < toUnixTimestamp(@end) GROUP BY stat_time, common_schema_type ORDER BY stat_time ASC LIMIT 10000
--Q20.Traffic Metrics Analytic
SELECT round(sum(common_s2c_byte_num) * 8 / 300,2) AS trafficInBits, round(sum(common_c2s_byte_num) * 8 / 300,2) AS trafficOutBits, round(sum(common_s2c_byte_num + common_c2s_byte_num) * 8 / 300,2) AS trafficTotalBits, round(sum(common_s2c_pkt_num) / 300,2) AS trafficInPackets, round(sum(common_c2s_pkt_num) / 300,2) AS trafficOutPackets, round(sum(common_s2c_pkt_num + common_c2s_pkt_num) / 300,2) AS trafficTotalPackets, round(sum(common_sessions) / 300,2) AS sessions FROM tsg_galaxy_v3.session_record AS session_record WHERE common_recv_time >= toUnixTimestamp(@start) AND common_recv_time < toUnixTimestamp(@end)
--Q21.Traffic Endpoints Metrics Trend(Time Grain 5 minute)
SELECT toUnixTimestamp(toDateTime(toStartOfInterval(toDateTime(common_recv_time),INTERVAL 5 MINUTE))) AS "Receive Time", uniq(common_internal_ip) AS "Unique Internal IP", uniq(common_external_ip) AS "Unique External IP", uniq(common_subscriber_id) AS "Unique Subscriber ID", sum(coalesce(common_c2s_byte_num + common_s2c_byte_num, 0)) AS "Bytes", sum(coalesce(common_sessions, 0)) AS "Sessions" FROM tsg_galaxy_v3.session_record AS session_record WHERE ( ( common_recv_time >= toDateTime(@start) AND common_recv_time < toDateTime(@end) ) ) GROUP BY "Receive Time" LIMIT 10000
--Q22.Endpoint Unique Num by L4 Protocol
SELECT 'all' AS type, uniq(common_client_ip) AS client_ips, uniq(common_internal_ip) AS internal_ips, uniq(common_server_ip) AS server_ips, uniq(common_external_ip) AS external_ips, uniq(common_subscriber_id) as subscriber_ids FROM tsg_galaxy_v3.session_record AS session_record WHERE common_recv_time >= toDateTime(@start) AND common_recv_time < toDateTime(@end) UNION ALL SELECT 'tcp' AS type, uniq(common_client_ip) AS client_ips, uniq(common_internal_ip) AS internal_ips, uniq(common_server_ip) AS server_ips, uniq(common_external_ip) AS external_ips, uniq(common_subscriber_id) as subscriber_ids FROM tsg_galaxy_v3.session_record AS session_record WHERE common_recv_time >= toDateTime(@start) AND common_recv_time < toDateTime(@end) AND common_l4_protocol IN ( 'IPv4_TCP', 'IPv6_TCP' ) UNION ALL SELECT 'UDP' AS type, uniq(common_client_ip) AS client_ips, uniq(common_internal_ip) AS internal_ips, uniq(common_server_ip) AS server_ips, uniq(common_external_ip) AS external_ips, uniq(common_subscriber_id) as subscriber_ids FROM tsg_galaxy_v3.session_record AS session_record WHERE common_recv_time >= toDateTime(@start) AND common_recv_time < toDateTime(@end) AND common_l4_protocol IN ( 'IPv4_UDP', 'IPv6_UDP' )
--Q23.One-sided Connection Trend(Time Grain 5 minute)
SELECT toDateTime(toUnixTimestamp(toDateTime(toStartOfInterval(toDateTime(common_recv_time),INTERVAL 5 MINUTE)))) AS stat_time, (CASE WHEN common_stream_dir = 1 THEN 'c2s' WHEN common_stream_dir = 2 THEN 's2c' WHEN common_stream_dir = 3 THEN 'double' ELSE 'None' END) AS type, sum(common_sessions) AS sessions FROM tsg_galaxy_v3.session_record AS session_record WHERE common_recv_time >= toDateTime(@start) AND common_recv_time < toDateTime(@end) GROUP BY stat_time, common_stream_dir ORDER BY stat_time ASC LIMIT 10000
--Q24. Estimated One-sided Sessions with Bandwidth
SELECT toUnixTimestamp(toDateTime(toStartOfInterval(toDateTime(common_recv_time),INTERVAL 5 MINUTE))) AS "Receive Time", sum(common_sessions) AS "sessions", sum(if(common_stream_dir <> 3, common_sessions, 0)) AS "one_side_sessions", sum(coalesce(common_c2s_byte_num + common_s2c_byte_num, 0)) AS "Bytes", round(one_side_sessions / sessions, 2) AS one_side_percent FROM tsg_galaxy_v3.session_record AS session_record WHERE ( ( common_recv_time >= toDateTime(@start) AND common_recv_time < toDateTime(@end) ) ) GROUP BY "Receive Time" LIMIT 10000
--Q25.Estimated TCP Sequence Gap Loss
SELECT toUnixTimestamp(toDateTime(toStartOfInterval(toDateTime(common_recv_time),INTERVAL 5 MINUTE))) AS "Receive Time", sum(common_c2s_byte_num + common_s2c_byte_num) AS "bytes", sum(common_c2s_tcp_lostlen + common_s2c_tcp_lostlen) AS "gap_loss_bytes", round(gap_loss_bytes / bytes, 2) AS gap_loss_percent FROM tsg_galaxy_v3.session_record AS session_record WHERE ( ( common_recv_time >= toDateTime(@start) AND common_recv_time < toDateTime(@end) ) ) AND ( common_l4_protocol IN ( 'IPv4_TCP', 'IPv6_TCP' ) ) GROUP BY "Receive Time" LIMIT 10000
--Q26.Top30 Server IP by Bytes
SELECT "server_ip" AS "server_ip" , SUM(coalesce("bytes",0)) AS "bytes" , SUM(coalesce("bytes_sent",0)) AS "Sent" , SUM(coalesce("bytes_received",0)) AS "Received" , SUM(coalesce("sessions",0)) AS "sessions" FROM ( SELECT SUM(coalesce(common_c2s_byte_num,0)) AS "bytes_sent" , SUM(coalesce(common_s2c_byte_num,0)) AS "bytes_received" , SUM(common_c2s_byte_num+common_s2c_byte_num) AS "bytes" , SUM(coalesce(common_sessions,0)) AS "sessions" , common_server_ip AS "server_ip" FROM tsg_galaxy_v3.session_record WHERE ( ( common_recv_time >= toDateTime(@start) AND common_recv_time < toDateTime(@end) ) ) AND ( notEmpty( common_server_ip) ) GROUP BY "server_ip" ORDER BY "bytes" desc ) GROUP BY "server_ip" ORDER BY "bytes" desc LIMIT 30
--Q27.Top30 Client IP by Sessions
SELECT common_client_ip , COUNT(*) AS sessions FROM tsg_galaxy_v3.session_record WHERE common_recv_time >= toDateTime(@start) AND common_recv_time < toDateTime(@end) GROUP BY common_client_ip ORDER BY sessions desc LIMIT 0,30
--Q28.Top30 TCP Server Ports by Sessions
SELECT "Server Port" AS "Server Port", sum(coalesce("Sessions", 0)) AS "Sessions" FROM (SELECT common_server_port AS "Server Port", sum(coalesce(common_sessions, 0)) AS "Sessions" FROM tsg_galaxy_v3.session_record AS session_record WHERE ( ( common_recv_time >= toDateTime(@start) AND common_recv_time < toDateTime(@end) ) ) AND ( common_l4_protocol IN ( 'IPv4_TCP', 'IPv6_TCP' ) ) GROUP BY "Server Port" LIMIT 1048576) GROUP BY "Server Port" ORDER BY "Sessions" DESC LIMIT 30
--Q29.Top30 Domian by Bytes
SELECT "domain" AS "Website Domain" , SUM(coalesce("bytes",0)) AS "Throughput" FROM ( SELECT SUM(coalesce(common_c2s_byte_num,0)) AS "bytes_sent" , SUM(coalesce(common_s2c_byte_num,0)) AS "bytes_received" , SUM(coalesce(common_c2s_byte_num+common_s2c_byte_num,0)) AS "bytes" , http_domain AS "domain" FROM tsg_galaxy_v3.session_record WHERE ( ( common_recv_time >= toDateTime(@start) AND common_recv_time < toDateTime(@end) ) ) AND ( notEmpty( http_domain) ) GROUP BY "domain" ORDER BY "bytes" desc ) GROUP BY "domain" ORDER BY "Throughput" desc LIMIT 30
--Q30.Top30 Endpoint Devices by Bandwidth
SELECT "device_id" AS "device_id", sum(coalesce("bytes", 0)) AS "bytes", sum(coalesce("bytes_sent", 0)) AS "Sent", sum(coalesce("bytes_received", 0)) AS "Received" FROM (SELECT sum(coalesce(common_c2s_byte_num, 0)) AS "bytes_sent", sum(coalesce(common_s2c_byte_num, 0)) AS "bytes_received", sum(common_c2s_byte_num + common_s2c_byte_num) AS bytes, common_device_id AS "device_id" FROM tsg_galaxy_v3.session_record AS session_record WHERE ( ( common_recv_time >= toDateTime(@start) AND common_recv_time < toDateTime(@end) ) ) GROUP BY "device_id" ORDER BY "bytes" DESC LIMIT 1048576) GROUP BY "device_id" ORDER BY "bytes" DESC LIMIT 30
--Q31.Top30 Domain by Unique Client IP
SELECT "Http.Domain" AS "Http.Domain", sum(coalesce("Client IP", 0)) AS "Client IP" FROM (SELECT http_domain AS "Http.Domain", uniq(common_client_ip) AS "Client IP" FROM tsg_galaxy_v3.session_record AS session_record WHERE ( ( common_recv_time >= toDateTime(@start) AND common_recv_time < toDateTime(@end) ) ) AND ( notEmpty(http_domain) ) GROUP BY "Http.Domain" ORDER BY "Client IP" DESC LIMIT 1048576) GROUP BY "Http.Domain" ORDER BY "Client IP" DESC LIMIT 30
--Q32.Top100 Most Time Consuming Domains
SELECT "Domain" AS "Domain", avg(coalesce("Avg Establish Latency(ms)", 0)) AS "Avg Establish Latency(ms)" FROM (SELECT http_domain AS "Domain", avg(coalesce(common_establish_latency_ms, 0)) AS "Avg Establish Latency(ms)" FROM tsg_galaxy_v3.session_record AS session_record WHERE ( ( common_recv_time >= toDateTime(@start) AND common_recv_time < toDateTime(@end) ) ) AND ( notEmpty(http_domain) ) GROUP BY "Domain" LIMIT 1048576) GROUP BY "Domain" ORDER BY "Avg Establish Latency(ms)" DESC LIMIT 100
--Q33.Top30 Sources by Sessions
SELECT "source" AS "source", sum(coalesce("sessions", 0)) AS "sessions" FROM (SELECT coalesce(nullif(common_subscriber_id, ''), nullif(common_client_ip, '')) AS "source", sum(coalesce(common_sessions, 0)) AS "sessions" FROM tsg_galaxy_v3.session_record AS session_record WHERE ( ( common_recv_time >= toDateTime(@start) AND common_recv_time < toDateTime(@end) ) ) GROUP BY "source" ORDER BY "sessions" DESC LIMIT 1048576) GROUP BY "source" ORDER BY "sessions" DESC LIMIT 30
--Q34.Top30 Destinations by Sessions
SELECT "destination" AS "destination", sum(coalesce("sessions", 0)) AS "sessions" FROM (SELECT coalesce(nullif(http_domain, ''), nullif(common_server_ip, '')) AS "destination", sum(coalesce(common_sessions, 0)) AS "sessions" FROM tsg_galaxy_v3.session_record AS session_record WHERE ( ( common_recv_time >= toDateTime(@start) AND common_recv_time < toDateTime(@end) ) ) GROUP BY "destination" ORDER BY "sessions" DESC LIMIT 1048576) GROUP BY "destination" ORDER BY "sessions" DESC LIMIT 30
--Q35.Top30 Destination Regions by Bandwidth
SELECT "server_location" AS "server_location", sum(coalesce("bytes", 0)) AS "bytes", sum(coalesce("bytes_sent", 0)) AS "Sent", sum(coalesce("bytes_received", 0)) AS "Received" FROM (SELECT arrayElement(splitByString(',', common_server_location), length(splitByString(',', common_server_location))) AS "server_location", sum(coalesce(common_c2s_byte_num, 0)) AS "bytes_sent", sum(coalesce(common_s2c_byte_num, 0)) AS "bytes_received", sum(coalesce(common_c2s_byte_num + common_s2c_byte_num, 0)) AS "bytes", sum(coalesce(common_sessions, 0)) AS "sessions" FROM tsg_galaxy_v3.session_record AS session_record WHERE ( ( common_recv_time >= toDateTime(@start) AND common_recv_time < toDateTime(@end) ) ) GROUP BY "server_location" ORDER BY "bytes" DESC LIMIT 1048576) GROUP BY "server_location" ORDER BY "bytes" DESC LIMIT 30
--Q36.Top30 URLS by Sessions
SELECT "Http URL" AS "Http URL", sum(coalesce("Sessions", 0)) AS "Sessions" FROM (SELECT http_url AS "Http URL", sum(coalesce(common_sessions, 0)) AS "Sessions" FROM tsg_galaxy_v3.session_record AS session_record WHERE ( ( common_recv_time >= toDateTime(@start) AND common_recv_time < toDateTime(@end) ) ) GROUP BY "Http URL" LIMIT 1048576) GROUP BY "Http URL" ORDER BY "Sessions" DESC LIMIT 30
--Q37.Top30 Destination Transmission APP by Bandwidth
SELECT "server_ip" AS "server_ip", groupUniqArray(coalesce("trans_app", 0)) AS "trans_app", sum(coalesce("bytes", 0)) AS "bytes", sum(coalesce("bytes_sent", 0)) AS "Sent", sum(coalesce("bytes_received", 0)) AS "Received" FROM (SELECT sum(coalesce(common_c2s_byte_num, 0)) AS "bytes_sent", sum(coalesce(common_s2c_byte_num, 0)) AS "bytes_received", sum(common_c2s_byte_num + common_s2c_byte_num) AS "bytes", groupUniqArray(concat(common_l4_protocol, '/', toString(common_server_port))) AS "trans_app", common_server_ip AS "server_ip" FROM tsg_galaxy_v3.session_record AS session_record WHERE ( ( common_recv_time >= toDateTime(@start) AND common_recv_time < toDateTime(@end) ) ) AND ( notEmpty(common_server_ip) ) GROUP BY "server_ip" ORDER BY "bytes" DESC LIMIT 1048576) GROUP BY "server_ip" ORDER BY "bytes" DESC LIMIT 30
--Q38.Browsing Users by Website domains and Sessions
SELECT "Subscriber ID" AS "Subscriber ID", "Http.Domain" AS "Http.Domain", sum(coalesce("sessions", 0)) AS "sessions" FROM (SELECT http_domain AS "Http.Domain", common_subscriber_id AS "Subscriber ID", sum(coalesce(common_sessions, 0)) AS "sessions" FROM tsg_galaxy_v3.session_record AS session_record WHERE ( ( common_recv_time >= toDateTime(@start) AND common_recv_time < toDateTime(@end) ) ) AND ( notEmpty(http_domain) AND notEmpty(common_subscriber_id) ) GROUP BY "Http.Domain", "Subscriber ID" ORDER BY "sessions" DESC LIMIT 1048576) GROUP BY "Subscriber ID", "Http.Domain" ORDER BY "sessions" DESC LIMIT 10000
--Q39.Top Domain and Server IP by Bytes Sent
SELECT "Http.Domain" AS "Http.Domain" , "Server IP" AS "Server IP" , SUM(coalesce("Bytes Sent",0)) AS "Bytes Sent" FROM ( SELECT common_server_ip AS "Server IP" , http_domain AS "Http.Domain" , SUM(coalesce(common_c2s_byte_num+common_s2c_byte_num,0)) AS "Bytes" , SUM(coalesce(common_c2s_byte_num,0)) AS "Bytes Sent" , SUM(coalesce(common_s2c_byte_num,0)) AS "Bytes Received" FROM tsg_galaxy_v3.session_record WHERE ( ( common_recv_time >= toDateTime(@start) AND common_recv_time < toDateTime(@end) ) ) AND ( notEmpty( http_domain) ) GROUP BY "Server IP" , "Http.Domain" ORDER BY "Bytes" desc LIMIT 1048576 ) GROUP BY "Http.Domain" , "Server IP" ORDER BY "Bytes Sent" desc LIMIT 10000
--Q40.Top30 Website Domains by Client IP and Sessions
SELECT "Http.Domain" AS "Http.Domain", "Client IP" AS "Client IP", sum(coalesce("sessions", 0)) AS "sessions" FROM (SELECT common_client_ip AS "Client IP", http_domain AS "Http.Domain", sum(coalesce(common_sessions, 0)) AS "sessions" FROM tsg_galaxy_v3.session_record AS session_record WHERE ( ( common_recv_time >= toDateTime(@start) AND common_recv_time < toDateTime(@end) ) ) AND ( notEmpty(http_domain) ) GROUP BY "Client IP", "Http.Domain" ORDER BY "sessions" DESC LIMIT 1048576) GROUP BY "Http.Domain", "Client IP" ORDER BY "sessions" DESC LIMIT 10000
--Q41.Domain is Accessed by Unique Client IP Trend(bytes Time Grain 5 minute)
SELECT toDateTime(intDiv(toUInt32(toDateTime(toDateTime(common_recv_time))),300)*300) AS _time , http_domain AS Domain, COUNT(DISTINCT(common_client_ip)) AS nums FROM tsg_galaxy_v3.session_record AS session_record WHERE common_recv_time >= toDateTime(@start) AND common_recv_time < toDateTime(@end) AND notEmpty(http_domain) AND http_domain IN ( SELECT http_domain FROM tsg_galaxy_v3.session_record AS session_record WHERE common_recv_time >= toDateTime(@start) AND common_recv_time < toDateTime(@end) AND notEmpty(http_domain) GROUP BY http_domain ORDER BY SUM(common_s2c_byte_num+common_c2s_byte_num) DESC LIMIT 5 ) GROUP BY toDateTime(intDiv(toUInt32(toDateTime(toDateTime(common_recv_time))),300)*300) , http_domain ORDER BY toDateTime(intDiv(toUInt32(toDateTime(toDateTime(common_recv_time))),300)*300) DESC LIMIT 10000
--Q42. Domain is Accessed by Unique Client IP Trend(sessions,Time Grain 5 minute)
SELECT toDateTime(intDiv(toUInt32(toDateTime(toDateTime(common_recv_time))),3600)*3600) AS stat_time , http_domain , uniq (common_client_ip) AS nums FROM tsg_galaxy_v3.session_record AS session_record WHERE common_recv_time >= toDateTime(@start)-604800 AND common_recv_time < toDateTime(@end) AND http_domain IN ( SELECT http_domain FROM tsg_galaxy_v3.session_record AS session_record WHERE common_recv_time >= toDateTime(@start) AND common_recv_time < toDateTime(@end) AND notEmpty(http_domain) GROUP BY http_domain ORDER BY COUNT(*) desc LIMIT 5 ) group by toDateTime(intDiv(toUInt32(toDateTime(toDateTime(common_recv_time))), 3600)*3600), http_domain ORDER BY stat_time desc LIMIT 10000
--Q43.Bandwidth Trend with Device ID(Time Grain 5 minute)
SELECT toUnixTimestamp(toDateTime(toStartOfInterval(toDateTime(common_recv_time),INTERVAL 5 MINUTE))) AS "Receive Time", common_device_id AS "Device ID", sum(coalesce(common_c2s_byte_num + common_s2c_byte_num, 0)) AS "Bytes" FROM tsg_galaxy_v3.session_record AS session_record WHERE ( ( common_recv_time >= toDateTime(@start) AND common_recv_time < toDateTime(@end) ) ) GROUP BY "Receive Time", "Device ID" LIMIT 10000
--Q44.Internal IP by Sled IP and Sessions
SELECT "Internal IP" AS "Internal IP", "Sled IP" AS "Sled IP", sum(coalesce("Sessions", 0)) AS "Sessions" FROM (SELECT common_sled_ip AS "Sled IP", common_internal_ip AS "Internal IP", sum(coalesce(common_sessions, 0)) AS "Sessions" FROM tsg_galaxy_v3.session_record AS session_record WHERE ( ( common_recv_time >= toDateTime(@start) AND common_recv_time < toDateTime(@end) ) ) GROUP BY "Sled IP", "Internal IP" LIMIT 1048576) GROUP BY "Internal IP", "Sled IP" ORDER BY "Sessions" DESC LIMIT 10000
--Q45.Bandwidth Trend with Internal IP (Time Grain 5 minute)
SELECT toUnixTimestamp(toDateTime(toStartOfInterval(toDateTime(common_recv_time),INTERVAL 5 MINUTE))) AS "Receive Time", sum(coalesce(common_c2s_byte_num + common_s2c_byte_num, 0)) AS "Bytes", sum(coalesce(common_c2s_pkt_num + common_s2c_pkt_num, 0)) AS "Packets", sum(coalesce(common_sessions, 0)) AS "New Sessions", sum(coalesce(common_c2s_byte_num, 0)) AS "Bytes Sent", sum(coalesce(common_s2c_byte_num, 0)) AS "Bytes Received", sum(coalesce(common_c2s_pkt_num, 0)) AS "Packets Sent", sum(coalesce(common_s2c_pkt_num, 0)) AS "Packets Received" FROM tsg_galaxy_v3.session_record AS session_record WHERE ( ( common_recv_time >= toDateTime(@start) AND common_recv_time < toDateTime(@end) ) AND @common_filter ) GROUP BY "Receive Time" LIMIT 10000
--Q46.Top30 Domains Detail with Internal IP
SELECT "Domain" AS "Domain", sum(coalesce("Sessions", 0)) AS "Sessions" FROM (SELECT http_domain AS "Domain", sum(coalesce(common_sessions, 0)) AS "Sessions" FROM tsg_galaxy_v3.session_record AS session_record WHERE ( ( common_recv_time >= toDateTime(@start) AND common_recv_time < toDateTime(@end) ) AND @common_filter ) AND ( notEmpty(http_domain) ) GROUP BY "Domain" LIMIT 1048576) GROUP BY "Domain" ORDER BY "Sessions" DESC LIMIT 30
--Q47.Top30 URLS Detail with Internal IP
SELECT "URL" AS "URL", sum(coalesce("Sessions", 0)) AS "Sessions" FROM (SELECT http_url AS "URL", sum(coalesce(common_sessions, 0)) AS "Sessions" FROM tsg_galaxy_v3.session_record AS session_record WHERE ( ( common_recv_time >= toDateTime(@start) AND common_recv_time < toDateTime(@end) ) AND @common_filter ) AND ( notEmpty(http_url) ) GROUP BY "URL" LIMIT 1048576) GROUP BY "URL" ORDER BY "Sessions" DESC LIMIT 30
--Q48.Top Domains with Unique Client IP and Subscriber ID
SELECT "Http.Domain" AS "Http.Domain", sum(coalesce("Unique Client IP", 0)) AS "Unique Client IP", sum(coalesce("Unique Subscriber ID", 0)) AS "Unique Subscriber ID" FROM (SELECT http_domain AS "Http.Domain", uniq(common_client_ip) AS "Unique Client IP", uniq(common_subscriber_id) AS "Unique Subscriber ID" FROM tsg_galaxy_v3.session_record AS session_record WHERE ( ( common_recv_time >= toDateTime(@start) AND common_recv_time < toDateTime(@end) ) ) AND ( notEmpty(http_domain) ) GROUP BY "Http.Domain" LIMIT 1048576) GROUP BY "Http.Domain" ORDER BY "Unique Client IP" DESC LIMIT 100
--Q49.Top100 Domains by Packets sent
SELECT "Http.Domain" AS "Http.Domain", sum(coalesce("Packets Sent", 0)) AS "Packets Sent" FROM (SELECT http_domain AS "Http.Domain", sum(coalesce(common_c2s_pkt_num, 0)) AS "Packets Sent" FROM tsg_galaxy_v3.session_record AS session_record WHERE ( ( common_recv_time >= toDateTime(@start) AND common_recv_time < toDateTime(@end) ) ) GROUP BY "Http.Domain" LIMIT 1048576) GROUP BY "Http.Domain" ORDER BY "Packets Sent" DESC LIMIT 100
--Q50.Internal and External asymmetric traffic
SELECT "Internal IP" AS "Internal IP", "External IP" AS "External IP", "Sled IP" AS "Sled IP", sum(coalesce("Sessions", 0)) AS "Sessions" FROM (SELECT common_sled_ip AS "Sled IP", common_external_ip AS "External IP", common_internal_ip AS "Internal IP", sum(coalesce(common_c2s_byte_num + common_s2c_byte_num, 0)) AS "Bytes Sent+Bytes Received", sum(coalesce(common_sessions, 0)) AS "Sessions" FROM tsg_galaxy_v3.session_record AS session_record WHERE ( ( common_recv_time >= toDateTime(@start) AND common_recv_time < toDateTime(@end) ) ) AND ( common_stream_dir != 3 ) GROUP BY "Sled IP", "External IP", "Internal IP" LIMIT 1048576) GROUP BY "Internal IP", "External IP", "Sled IP" ORDER BY "Sessions" DESC LIMIT 500
--Q51.Client and Server ASN asymmetric traffic
SELECT "Client ASN" AS "Client ASN", "Server ASN" AS "Server ASN", sum(coalesce("Sessions", 0)) AS "Sessions" FROM (SELECT common_server_asn AS "Server ASN", common_client_asn AS "Client ASN", sum(coalesce(common_sessions, 0)) AS "Sessions" FROM tsg_galaxy_v3.session_record AS session_record WHERE ( ( common_recv_time >= toDateTime(@start) AND common_recv_time < toDateTime(@end) ) ) AND ( common_stream_dir != 3 ) GROUP BY "Server ASN", "Client ASN" LIMIT 1048576) GROUP BY "Client ASN", "Server ASN" ORDER BY "Sessions" DESC LIMIT 500
--Q52.Top handshake latency by Website and Client IPs
SELECT "SSL.SNI" AS "SSL.SNI", "Client IP" AS "Client IP", avg(coalesce("Establish Latency(ms)", 0)) AS "Establish Latency(ms)" FROM (SELECT common_client_ip AS "Client IP", ssl_sni AS "SSL.SNI", avg(coalesce(common_establish_latency_ms, 0)) AS "Establish Latency(ms)" FROM tsg_galaxy_v3.session_record AS session_record WHERE ( ( common_recv_time >= toDateTime(@start) AND common_recv_time < toDateTime(@end) ) ) GROUP BY "Client IP", "SSL.SNI" LIMIT 1048576) GROUP BY "SSL.SNI", "Client IP" ORDER BY "Establish Latency(ms)" DESC LIMIT 500
--Q53.Domain baidu.com Drill down Client IP
select common_client_ip as "Client IP" , avg(common_establish_latency_ms) as "Establishing Time Mean(ms)", count(1) as Responses,any(common_client_location) as Location FROM tsg_galaxy_v3.session_record where common_recv_time >= toDateTime(@start) AND common_recv_time < toDateTime(@end) and http_domain='baidu.com' group by "Client IP" order by Responses desc limit 100
--Q54.Domain baidu.com Drill down Server IP
select common_server_ip as "Server IP" , avg(http_response_latency_ms) as "Server Processing Time Mean(ms)", count(1) as Responses,any(common_server_location) as Location FROM tsg_galaxy_v3.session_record where common_recv_time >= toDateTime(@start) AND common_recv_time < toDateTime(@end) and http_domain='baidu.com' group by "Server IP" order by Responses desc limit 100
--Q55.Domain baidu.com Drill down URI
select http_url as "URI" , avg(http_response_latency_ms) as "Server Processing Time Mean(ms)", count(1) as Responses FROM tsg_galaxy_v3.session_record where common_recv_time >= toDateTime(@start) AND common_recv_time < toDateTime(@end) and http_domain='baidu.com' group by "URI" order by Responses desc limit 100
--Q56.L7 Protocol Metrics
select common_l7_protocol as "Protocol" , uniq(common_client_ip) as "Clients" , uniq(common_server_ip) as "Servers", count(1) as Sessions,sum(common_c2s_byte_num+common_s2c_byte_num) as bytes FROM tsg_galaxy_v3.session_record where common_recv_time >= toDateTime(@start) AND common_recv_time < toDateTime(@end) and notEmpty(common_l7_protocol) group by common_l7_protocol order by bytes desc
--Q57.L7 Protocol SIP Drill down Client IP
select common_client_ip as "Client IP" , count(1) as Sessions,sum(common_c2s_byte_num) as "Bytes Out", sum(common_s2c_byte_num) as "Bytes In",any(common_client_location) as Location FROM tsg_galaxy_v3.session_record where common_recv_time >= toDateTime(@start) AND common_recv_time < toDateTime(@end) and common_l7_protocol='SIP' group by "Client IP" order by Sessions desc limit 100
--Q58.L7 Protocol SIP Drill down Server IP
select common_server_ip as "Server IP" , count(1) as Sessions,sum(common_c2s_byte_num) as "Bytes Out", sum(common_s2c_byte_num) as "Bytes In",any(common_server_location) as Location FROM tsg_galaxy_v3.session_record where common_recv_time >= toDateTime(@start) AND common_recv_time < toDateTime(@end) and common_l7_protocol='SIP' group by "Server IP" order by Sessions desc limit 100
--Q59.Top5 Server IP keys with Unique Client IPs Trend (Grain 5 minute)
SELECT toDateTime(intDiv(toUInt32(toDateTime(toDateTime(common_recv_time))),300)*300) AS _time , common_server_ip AS server_ip, COUNT(DISTINCT(common_client_ip)) AS nums FROM tsg_galaxy_v3.session_record AS session_record WHERE common_recv_time >= toDateTime(@start) AND common_recv_time < toDateTime(@end) AND common_server_ip IN ( SELECT common_server_ip FROM tsg_galaxy_v3.session_record AS session_record WHERE common_recv_time >= toDateTime(@start) AND common_recv_time < toDateTime(@end) GROUP BY common_server_ip ORDER BY count(*) DESC LIMIT 5 ) GROUP BY toDateTime(intDiv(toUInt32(toDateTime(toDateTime(common_recv_time))),300)*300) , server_ip ORDER BY toDateTime(intDiv(toUInt32(toDateTime(toDateTime(common_recv_time))),300)*300) DESC LIMIT 10000

11
testSchemaFiles/clusters.json Normal file
View File

@@ -0,0 +1,11 @@
{
"namespace": "system",
"type": "record",
"name": "clusters",
"fields": [
{
"name": "host_address",
"type": "string"
}
]
}

11
testSchemaFiles/columns_cluster.json Normal file
View File

@@ -0,0 +1,11 @@
{
"namespace": "system",
"type": "record",
"name": "columns_cluster",
"fields": [
{
"name": "database",
"type": "string"
}
]
}

11
testSchemaFiles/disks_cluster.json Normal file
View File

@@ -0,0 +1,11 @@
{
"namespace": "system",
"type": "record",
"name": "disks_cluster",
"fields": [
{
"name": "name",
"type": "string"
}
]
}

11
testSchemaFiles/distributed_ddl_queue.json Normal file
View File

@@ -0,0 +1,11 @@
{
"namespace": "system",
"type": "record",
"name": "distributed_ddl_queue",
"fields": [
{
"name": "name",
"type": "string"
}
]
}

434
testSchemaFiles/dos_event.json Normal file
View File

@@ -0,0 +1,434 @@
{
"type":"record",
"name":"dos_event",
"namespace":"tsg_galaxy_v3",
"doc":
{
"primary_key":"log_id",
"partition_key":"start_time",
"ttl":null,
"default_ttl":2592000,
"index_key":
[
"log_id",
"start_time",
"destination_ip"
],
"functions":
{
"aggregation":
[
{
"name":"COUNT",
"label":"COUNT",
"function":"count(expr)"
},
{
"name":"COUNT_DISTINCT",
"label":"COUNT_DISTINCT",
"function":"count(distinct expr)"
},
{
"name":"AVG",
"label":"AVG",
"function":"avg(expr)"
},
{
"name":"SUM",
"label":"SUM",
"function":"sum(expr)"
},
{
"name":"MAX",
"label":"MAX",
"function":"max(expr)"
},
{
"name":"MIN",
"label":"MIN",
"function":"min(expr)"
}
],
"operator":
[
{
"name":"=",
"label":"=",
"function":"expr = value"
},
{
"name":"!=",
"label":"!=",
"function":"expr != value"
},
{
"name":">",
"label":">",
"function":"expr > value"
},
{
"name":"<",
"label":"<",
"function":"expr < value"
},
{
"name":">=",
"label":">=",
"function":"expr >= value"
},
{
"name":"<=",
"label":"<=",
"function":"expr <= value"
},
{
"name":"has",
"label":"HAS",
"function":"has(expr, value)"
},
{
"name":"in",
"label":"IN",
"function":"expr in (values)"
},
{
"name":"not in",
"label":"NOT IN",
"function":"expr not in (values)"
},
{
"name":"like",
"label":"LIKE",
"function":"expr like value"
},
{
"name":"not like",
"label":"NOT LIKE",
"function":"expr not like value"
},
{
"name":"notEmpty",
"label":"NOT EMPTY",
"function":"notEmpty(expr)"
},
{
"name":"empty",
"label":"EMPTY",
"function":"empty(expr)"
}
]
},
"schema_query":
{
"references":
{
"aggregation":
[
{
"type":"int",
"functions":"COUNT,COUNT_DISTINCT,AVG,SUM,MAX,MIN"
},
{
"type":"long",
"functions":"COUNT,COUNT_DISTINCT,AVG,SUM,MAX,MIN"
},
{
"type":"float",
"functions":"COUNT,COUNT_DISTINCT,AVG,SUM,MAX,MIN"
},
{
"type":"double",
"functions":"COUNT,COUNT_DISTINCT,AVG,SUM,MAX,MIN"
},
{
"type":"string",
"functions":"COUNT,COUNT_DISTINCT"
},
{
"type":"date",
"functions":"COUNT,COUNT_DISTINCT,MAX,MIN"
},
{
"type":"timestamp",
"functions":"COUNT,COUNT_DISTINCT,MAX,MIN"
}
],
"operator":
[
{
"type":"int",
"functions":"=,!=,>,<,>=,<=,in,not in"
},
{
"type":"long",
"functions":"=,!=,>,<,>=,<=,in,not in"
},
{
"type":"float",
"functions":"=,!=,>,<,>=,<="
},
{
"type":"double",
"functions":"=,!=,>,<,>=,<="
},
{
"type":"string",
"functions":"=,!=,in,not in,like,not like,notEmpty,empty"
},
{
"type":"date",
"functions":"=,!=,>,<,>=,<="
},
{
"type":"timestamp",
"functions":"=,!=,>,<,>=,<="
},
{
"type":"array",
"functions":"has"
}
]
}
},
"default_columns":
[
"log_id",
"attack_type",
"source_ip_list",
"destination_ip",
"severity",
"start_time",
"end_time",
"packet_rate",
"bit_rate",
"session_rate"
],
"internal_columns":
[
"start_time",
"log_id",
"end_time"
]
},
"fields":
[
{
"name":"start_time",
"label":"Start Time",
"doc":
{
"allow_query":"false",
"constraints":
{
"type":"timestamp"
},
"visibility":"enabled",
"ttl":null
},
"type":"long"
},
{
"name":"end_time",
"label":"End Time",
"doc":
{
"allow_query":"false",
"constraints":
{
"type":"timestamp"
},
"visibility":"enabled",
"ttl":null
},
"type":"long"
},
{
"name":"log_id",
"label":"Log ID",
"doc":
{
"format":
{
"functions":"snowflake_id"
},
"visibility":"enabled",
"ttl":null
},
"type":"long"
},
{
"name":"attack_type",
"label":"Attack Type",
"doc":
{
"constraints":
{
"operator_functions":"=,!="
},
"data":
[
{
"code":"TCP SYN Flood",
"value":"TCP SYN Flood"
},
{
"code":"UDP Flood",
"value":"UDP Flood"
},
{
"code":"ICMP Flood",
"value":"ICMP Flood"
},
{
"code":"DNS Flood",
"value":"DNS Flood"
}
],
"visibility":"enabled",
"ttl":null
},
"type":"string"
},
{
"name":"severity",
"label":"Severity",
"doc":
{
"constraints":
{
"operator_functions":"=,!="
},
"data":
[
{
"code":"Critical",
"value":"Critical"
},
{
"code":"Severe",
"value":"Severe"
},
{
"code":"Major",
"value":"Major"
},
{
"code":"Warning",
"value":"Warning"
},
{
"code":"Minor",
"value":"Minor"
}
],
"visibility":"enabled",
"ttl":null
},
"type":"string"
},
{
"name":"conditions",
"label":"Conditions",
"doc":
{
"visibility":"enabled",
"ttl":null
},
"type":"string"
},
{
"name":"destination_ip",
"label":"Destination IP",
"doc":
{
"visibility":"enabled",
"ttl":null
},
"type":"string"
},
{
"name":"destination_country",
"label":"Destination Country",
"doc":
{
"visibility":"enabled",
"ttl":null
},
"type":"string"
},
{
"name":"source_ip_list",
"label":"Source IPs",
"doc":
{
"visibility":"enabled",
"ttl":null
},
"type":"string"
},
{
"name":"source_country_list",
"label":"Source Countries",
"doc":
{
"visibility":"enabled",
"ttl":null
},
"type":"string"
},
{
"name":"session_rate",
"label":"Sessions/s",
"doc":
{
"constraints":
{
"type":"sessions/sec"
},
"visibility":"enabled",
"ttl":null
},
"type":"long"
},
{
"name":"packet_rate",
"label":"Packets/s",
"doc":
{
"constraints":
{
"type":"packets/sec"
},
"visibility":"enabled",
"ttl":null
},
"type":"long"
},
{
"name":"bit_rate",
"label":"Bits/s",
"doc":
{
"constraints":
{
"type":"bits/sec"
},
"visibility":"enabled",
"ttl":null
},
"type":"long"
}
]
}

21
testSchemaFiles/druid-filter.json Normal file
View File

@@ -0,0 +1,21 @@
{
"version": "1.0",
"name": "druid-Raw",
"namespace": "druid",
"filters": [
{
"name":"@start",
"value": "'2021-10-19 10:00:00'"
},
{
"name":"@end",
"value": "'2021-10-20 11:00:00'"
},
{
"name":"@common_filter",
"value": [
"common_client_ip='192.168.44.21'and common_server_port=443"
]
}
]
}

92
testSchemaFiles/druid-queries-template.sql Normal file
View File

@@ -0,0 +1,92 @@
--Q01.All Security Event Hits
select policy_id, sum(hits) as hits from security_event_hits_log where __time >@start and __time <@end group by policy_id
--Q02.Security Event Hits with Policy ID 0
select policy_id, sum(hits) as hits from security_event_hits_log where __time >@start and __time <@end and policy_id in (0) group by policy_id
--Q03.All Security Event Hits Trend by 5min A
select TIME_FORMAT(time_floor(__time,'PT5M'),'yyyy-MM-dd HH:mm:ss') as start_time, sum(hits) as hits from security_event_hits_log where __time >= TIMESTAMP @start and __time < TIMESTAMP @end group by TIME_FORMAT(time_floor(__time,'PT5M'),'yyyy-MM-dd HH:mm:ss') limit 10000
--Q04.Security Event Hit Time（first and last time) A
select policy_id,TIME_FORMAT(min(__time) ,'yyyy-MM-dd HH:mm:ss') as first_used, TIME_FORMAT(max(__time) ,'yyyy-MM-dd HH:mm:ss') as last_used from security_event_hits_log where policy_id in (0) group by policy_id
--Q05.Top 200 Security Policies
select policy_id, sum(hits) as hits from security_event_hits_log where __time >=TIMESTAMP @start and __time <TIMESTAMP @end group by policy_id order by hits desc limit 200
--Q06.Top 200 Security Policies with Action
select policy_id, action, sum(hits) as hits from security_event_hits_log where __time >=@start and __time <@end group by policy_id, action order by hits desc limit 200
--Q07.All Proxy Event Hits
select policy_id, sum(hits) as hits from proxy_event_hits_log where __time >=@start and __time <@end group by policy_id
--Q08.Proxy Event Hits with Policy ID 0
select policy_id, sum(hits) as hits from proxy_event_hits_log where __time >=@start and __time <@end and policy_id=0 group by policy_id
--Q09.All Proxy Event Hits Trend by 5min A
select TIME_FORMAT(time_floor(__time,'PT5M'),'yyyy-MM-dd HH:mm:ss') as start_time, sum(hits) as hits from proxy_event_hits_log where __time >= TIMESTAMP @start and __time <TIMESTAMP @end group by TIME_FORMAT(time_floor(__time,'PT5M'),'yyyy-MM-dd HH:mm:ss') limit 10000
--Q10.Proxy Event Hit Time（first and last time) A
select policy_id,TIME_FORMAT(min(__time) ,'yyyy-MM-dd HH:mm:ss') as first_used, TIME_FORMAT(max(__time) ,'yyyy-MM-dd HH:mm:ss') as last_used from proxy_event_hits_log where policy_id in (0) group by policy_id
--Q11.Top 200 Proxy Policies
select policy_id, sum(hits) as hits from proxy_event_hits_log where __time >=TIMESTAMP @start and __time <TIMESTAMP @end group by policy_id order by hits desc limit 200
--Q12.Top 200 Proxy Policies with sub Action
select policy_id, sub_action as action, sum(hits) as hits from proxy_event_hits_log where __time >=@start and __time <@end group by policy_id, sub_action order by hits desc limit 200
--Q13.Proxy Action Hits
select sub_action as action, sum(hits) as hits from proxy_event_hits_log where __time >= TIMESTAMP @start and __time < TIMESTAMP @end group by sub_action
--Q14.Proxy Action Hits Trend by 5min
select TIME_FORMAT(time_floor(__time,'PT5M'),'yyyy-MM-dd HH:mm:ss') as start_time, sub_action as action, sum(hits) as hits from proxy_event_hits_log where __time >=TIMESTAMP @start and __time <TIMESTAMP @end group by TIME_FORMAT(time_floor(__time,'PT5M'),'yyyy-MM-dd HH:mm:ss') , sub_action limit 10000
--Q15.Traffic Metrics Pinning Hits
SELECT sum(not_pinning_num) AS sessions, 'notPinningNum' AS type FROM traffic_metrics_log WHERE __time >= @start AND __time < @end UNION ALL SELECT sum(pinning_num) AS sessions, 'pinningNum' AS type FROM traffic_metrics_log WHERE __time >= @start AND __time < @end UNION ALL SELECT sum(maybe_pinning_num) AS sessions, 'maybePinningNum' AS type FROM traffic_metrics_log WHERE __time >= @start AND __time < @end
--Q16.Traffic Metrics Pinning Trend by 5Min
SELECT TIME_FORMAT( MILLIS_TO_TIMESTAMP( 1000 * (TIMESTAMP_TO_MILLIS(time_floor(0.001 * TIMESTAMP_TO_MILLIS( __time) * 1000,'PT300S'))/1000)),'YYYY-MM-dd HH:mm:ss') AS statisticTime, sum(pinning_num) AS sessions FROM traffic_metrics_log WHERE __time >= @start AND __time < @end GROUP BY TIME_FORMAT( MILLIS_TO_TIMESTAMP( 1000 * (TIMESTAMP_TO_MILLIS(time_floor(0.001 * TIMESTAMP_TO_MILLIS( __time) * 1000,'PT300S'))/1000)),'YYYY-MM-dd HH:mm:ss') LIMIT 10000
--Q17.Traffic Metrics Not Pinning Trend by 5Min
SELECT TIME_FORMAT( MILLIS_TO_TIMESTAMP( 1000 * (TIMESTAMP_TO_MILLIS(time_floor(0.001 * TIMESTAMP_TO_MILLIS( __time) * 1000,'PT300S'))/1000)),'YYYY-MM-dd HH:mm:ss') AS statisticTime, sum(not_pinning_num) AS sessions FROM traffic_metrics_log WHERE __time>= @start AND __time < @end GROUP BY TIME_FORMAT( MILLIS_TO_TIMESTAMP( 1000 * (TIMESTAMP_TO_MILLIS(time_floor(0.001 * TIMESTAMP_TO_MILLIS( __time) * 1000,'PT300S'))/1000)),'YYYY-MM-dd HH:mm:ss') LIMIT 10000
--Q18.Traffic Metrics Maybe Pinning Trend by 5Min
SELECT TIME_FORMAT( MILLIS_TO_TIMESTAMP( 1000 * (TIMESTAMP_TO_MILLIS(time_floor(0.001 * TIMESTAMP_TO_MILLIS( __time) * 1000,'PT300S'))/1000)),'YYYY-MM-dd HH:mm:ss') AS statisticTime, sum(maybe_pinning_num) AS sessions FROM traffic_metrics_log WHERE __time >= @start AND __time < @end GROUP BY TIME_FORMAT( MILLIS_TO_TIMESTAMP( 1000 * (TIMESTAMP_TO_MILLIS(time_floor(0.001 * TIMESTAMP_TO_MILLIS( __time) * 1000,'PT300S'))/1000)),'YYYY-MM-dd HH:mm:ss') LIMIT 10000
--Q19.Traffic Metrics Throughput Bytes IN/OUT
select sum(total_in_bytes) as traffic_in_bytes, sum(total_out_bytes) as traffic_out_bytes from traffic_metrics_log where __time >=TIMESTAMP @start and __time <TIMESTAMP @end
--Q20. Traffic Metrics Throughput Packets IN/OUT
select sum(total_in_packets) as traffic_in_packets, sum(total_out_packets) as traffic_out_packets from traffic_metrics_log where __time >=TIMESTAMP @start and __time <TIMESTAMP @end
--Q21.Traffic Metrics New Sessions
select sum(new_conn_num) as sessions from traffic_metrics_log where __time >=TIMESTAMP @start and __time <TIMESTAMP @end
--Q22.Traffic Metrics Bandwidth Bytes IN/OUT
select TIME_FORMAT(time_floor(__time,'PT30S'),'yyyy-MM-dd HH:mm:ss') as stat_time, 'traffic_in_bytes' as type, sum(total_in_bytes) as bytes from traffic_metrics_log where __time >= @start and __time < @end group by TIME_FORMAT(time_floor(__time,'PT30S'),'yyyy-MM-dd HH:mm:ss') union all select TIME_FORMAT(time_floor(__time,'PT30S'),'yyyy-MM-dd HH:mm:ss') as stat_time, 'traffic_out_bytes' as type, sum(total_out_bytes) as bytes from traffic_metrics_log where __time >= @start and __time < @end group by TIME_FORMAT(time_floor(__time,'PT30S'),'yyyy-MM-dd HH:mm:ss')
--Q23.Traffic Metrics Bandwidth Packets IN/OUT
select TIME_FORMAT(time_floor(__time,'PT30S'),'yyyy-MM-dd HH:mm:ss') as stat_time, 'traffic_in_packets' as type, sum(total_in_packets) as packets from traffic_metrics_log where __time >=TIMESTAMP @start and __time <TIMESTAMP @end group by TIME_FORMAT(time_floor(__time,'PT30S'),'yyyy-MM-dd HH:mm:ss') union all select TIME_FORMAT(time_floor(__time,'PT30S'),'yyyy-MM-dd HH:mm:ss') as stat_time, 'traffic_out_packets' as type, sum(total_out_packets) as packets from traffic_metrics_log where __time >=TIMESTAMP @start and __time <TIMESTAMP @end group by TIME_FORMAT(time_floor(__time,'PT30S'),'yyyy-MM-dd HH:mm:ss')
--Q24.Traffic Metrics New Sessions Trend by 5Min
select TIME_FORMAT(time_floor(__time,'PT30S'),'yyyy-MM-dd HH:mm:ss') as stat_time, 'new_conn_num' as type, sum(new_conn_num) as sessions from traffic_metrics_log where __time >= @start and __time < @end group by TIME_FORMAT(time_floor(__time,'PT30S'),'yyyy-MM-dd HH:mm:ss')
--Q25.Traffic Metrics New and Live Sessions
select sum(new_conn_num) as new_conn_num, sum(established_conn_num) as established_conn_num from traffic_metrics_log where __time >=TIMESTAMP @start and __time <TIMESTAMP @end
--Q26.Traffic Metrics New and Live Sessions Trend by 5Min
select TIME_FORMAT(time_floor(__time,'PT30S'),'yyyy-MM-dd HH:mm:ss') as stat_time, 'new_conn_num' as type, sum(new_conn_num) as sessions from traffic_metrics_log where __time >=TIMESTAMP @start and __time < TIMESTAMP @end group by TIME_FORMAT(time_floor(__time,'PT30S'),'yyyy-MM-dd HH:mm:ss') union all select TIME_FORMAT(time_floor(__time,'PT30S'),'yyyy-MM-dd HH:mm:ss') as stat_time, 'established_conn_num' as type, sum(established_conn_num) as sessions from traffic_metrics_log where __time >= TIMESTAMP @start and __time < TIMESTAMP @end group by TIME_FORMAT(time_floor(__time,'PT30S'),'yyyy-MM-dd HH:mm:ss')
--Q27.Traffic Metrics Security Throughput Bytes
select sum(default_in_bytes+default_out_bytes) as default_bytes, sum(allow_in_bytes+allow_out_bytes) as allow_bytes, sum(deny_in_bytes+deny_out_bytes) as deny_bytes, sum(monitor_in_bytes+monitor_out_bytes) as monitor_bytes, sum(intercept_in_bytes+intercept_out_bytes) as intercept_bytes from traffic_metrics_log where __time >=TIMESTAMP @start and __time < TIMESTAMP @end
--Q28.Traffic Metrics Security Throughput Packets
select sum(default_in_packets+default_out_packets) as default_packets, sum(allow_in_packets+allow_in_packets) as allow_packets, sum(deny_in_packets+deny_out_packets) as deny_packets, sum(monitor_in_packets+monitor_out_packets) as monitor_packets, sum(intercept_in_packets+intercept_out_packets) as intercept_packets from traffic_metrics_log where __time >=TIMESTAMP @start and __time <TIMESTAMP @end
--Q29.Traffic Metrics Security Throughput Sessions
select sum(default_conn_num) as default_sessions, sum(allow_conn_num) as allow_sessions, sum(deny_conn_num) as deny_sessions, sum(monitor_conn_num) as monitor_sessions, sum(intercept_conn_num) as intercept_sessions from traffic_metrics_log where __time >=TIMESTAMP @start and __time <TIMESTAMP @end
--Q30.Traffic Metrics Security Bandwidth Bytes by 5Min
select TIME_FORMAT(time_floor(__time,'PT5M'),'yyyy-MM-dd HH:mm:ss') as stat_time, 'default_bytes' as type, sum(default_in_bytes+default_out_bytes) as bytes from traffic_metrics_log where __time >=TIMESTAMP @start and __time <TIMESTAMP @end group by TIME_FORMAT(time_floor(__time,'PT5M'),'yyyy-MM-dd HH:mm:ss') union all select TIME_FORMAT(time_floor(__time,'PT5M'),'yyyy-MM-dd HH:mm:ss') as stat_time, 'allow_bytes' as type, sum(allow_in_bytes+allow_out_bytes) as bytes from traffic_metrics_log where __time >=TIMESTAMP @start and __time <TIMESTAMP @end group by TIME_FORMAT(time_floor(__time,'PT5M'),'yyyy-MM-dd HH:mm:ss') union all select TIME_FORMAT(time_floor(__time,'PT5M'),'yyyy-MM-dd HH:mm:ss') as stat_time, 'deny_bytes' as type, sum(deny_in_bytes+deny_out_bytes) as bytes from traffic_metrics_log where __time >= TIMESTAMP @start and __time <TIMESTAMP @end group by TIME_FORMAT(time_floor(__time,'PT5M'),'yyyy-MM-dd HH:mm:ss') union all select TIME_FORMAT(time_floor(__time,'PT5M'),'yyyy-MM-dd HH:mm:ss') as stat_time, 'monitor_bytes' as type, sum(monitor_in_bytes+monitor_out_bytes) as bytes from traffic_metrics_log where __time >=TIMESTAMP @start and __time <TIMESTAMP @end group by TIME_FORMAT(time_floor(__time,'PT5M'),'yyyy-MM-dd HH:mm:ss') union all select TIME_FORMAT(time_floor(__time,'PT5M'),'yyyy-MM-dd HH:mm:ss') as stat_time, 'intercept_bytes' as type, sum(intercept_in_bytes+intercept_out_bytes) as bytes from traffic_metrics_log where __time >= @start and __time < @end group by TIME_FORMAT(time_floor(__time,'PT5M'),'yyyy-MM-dd HH:mm:ss')
--Q31.Traffic Metrics Security Bandwidth Packets by 5Min
select TIME_FORMAT(time_floor(__time,'PT5M'),'yyyy-MM-dd HH:mm:ss') as stat_time, 'default_packets' as type, sum(default_in_packets+default_out_packets) as packets from traffic_metrics_log where __time >=TIMESTAMP @start and __time <TIMESTAMP @end group by TIME_FORMAT(time_floor(__time,'PT5M'),'yyyy-MM-dd HH:mm:ss') union all select TIME_FORMAT(time_floor(__time,'PT5M'),'yyyy-MM-dd HH:mm:ss') as stat_time, 'allow_packets' as type, sum(allow_in_packets+allow_out_packets) as packets from traffic_metrics_log where __time >=TIMESTAMP @start and __time <TIMESTAMP @end group by TIME_FORMAT(time_floor(__time,'PT5M'),'yyyy-MM-dd HH:mm:ss') union all select TIME_FORMAT(time_floor(__time,'PT5M'),'yyyy-MM-dd HH:mm:ss') as stat_time, 'deny_packets' as type, sum(deny_in_packets+deny_out_packets) as packets from traffic_metrics_log where __time >=TIMESTAMP @start and __time <TIMESTAMP @end group by TIME_FORMAT(time_floor(__time,'PT5M'),'yyyy-MM-dd HH:mm:ss') union all select TIME_FORMAT(time_floor(__time,'PT5M'),'yyyy-MM-dd HH:mm:ss') as stat_time, 'monitor_packets' as type, sum(monitor_in_packets+monitor_out_packets) as packets from traffic_metrics_log where __time >=TIMESTAMP @start and __time <TIMESTAMP @end group by TIME_FORMAT(time_floor(__time,'PT5M'),'yyyy-MM-dd HH:mm:ss') union all select TIME_FORMAT(time_floor(__time,'PT5M'),'yyyy-MM-dd HH:mm:ss') as stat_time, 'intercept_packets' as type, sum(intercept_in_packets+intercept_out_packets) as packets from traffic_metrics_log where __time >=TIMESTAMP @start and __time <TIMESTAMP @end group by TIME_FORMAT(time_floor(__time,'PT5M'),'yyyy-MM-dd HH:mm:ss')
--Q32.Traffic Metrics Security Sessions Trend by 5Min
select TIME_FORMAT(time_floor(__time,'PT5M'),'yyyy-MM-dd HH:mm:ss') as stat_time, 'default_conn_num' as type, sum(default_conn_num) as sessions from traffic_metrics_log where __time >=TIMESTAMP @start and __time <TIMESTAMP @end group by TIME_FORMAT(time_floor(__time,'PT5M'),'yyyy-MM-dd HH:mm:ss') union all select TIME_FORMAT(time_floor(__time,'PT5M'),'yyyy-MM-dd HH:mm:ss') as stat_time, 'allow_conn_num' as type, sum(allow_conn_num) as sessions from traffic_metrics_log where __time >=TIMESTAMP @start and __time <TIMESTAMP @end group by TIME_FORMAT(time_floor(__time,'PT5M'),'yyyy-MM-dd HH:mm:ss') union all select TIME_FORMAT(time_floor(__time,'PT5M'),'yyyy-MM-dd HH:mm:ss') as stat_time, 'deny_conn_num' as type, sum(deny_conn_num) as sessions from traffic_metrics_log where __time >=TIMESTAMP @start and __time <TIMESTAMP @end group by TIME_FORMAT(time_floor(__time,'PT5M'),'yyyy-MM-dd HH:mm:ss') union all select TIME_FORMAT(time_floor(__time,'PT5M'),'yyyy-MM-dd HH:mm:ss') as stat_time, 'monitor_conn_num' as type, sum(monitor_conn_num) as sessions from traffic_metrics_log where __time >=TIMESTAMP @start and __time <TIMESTAMP @end group by TIME_FORMAT(time_floor(__time,'PT5M'),'yyyy-MM-dd HH:mm:ss') union all select TIME_FORMAT(time_floor(__time,'PT5M'),'yyyy-MM-dd HH:mm:ss') as stat_time, 'intercept_conn_num' as type, sum(intercept_conn_num) as sessions from traffic_metrics_log where __time >=TIMESTAMP @start and __time <TIMESTAMP @end group by TIME_FORMAT(time_floor(__time,'PT5M'),'yyyy-MM-dd HH:mm:ss')
--Q33.Top 100 Client IP by Sessions
select source as client_ip, sum(session_num) as sessions, sum(c2s_byte_num) as sent_bytes, sum(s2c_byte_num) as received_bytes, sum(c2s_byte_num + s2c_byte_num) as bytes, sum(c2s_pkt_num) as sent_packets ,sum(s2c_pkt_num) as received_packets, sum(c2s_pkt_num+s2c_pkt_num) as packets from top_client_ip_log where __time >=TIMESTAMP @start and __time <TIMESTAMP @end and order_by='sessions' group by source order by sessions desc limit 100
--Q34.Top 100 Server IP by Sessions
select destination as server_ip, sum(session_num) as sessions, sum(c2s_byte_num) as sent_bytes, sum(s2c_byte_num) as received_bytes, sum(c2s_byte_num + s2c_byte_num) as bytes, sum(c2s_pkt_num) as sent_packets ,sum(s2c_pkt_num) as received_packets, sum(c2s_pkt_num+s2c_pkt_num) as packets from top_server_ip_log where __time >= @start and __time < @end and order_by='sessions' group by destination order by sessions desc limit 100
--Q35.Top 100 Internal IP by Sessions
select source as internal_ip, sum(session_num) as sessions, sum(c2s_byte_num) as sent_bytes, sum(s2c_byte_num) as received_bytes, sum(c2s_byte_num + s2c_byte_num) as bytes, sum(c2s_pkt_num) as sent_packets ,sum(s2c_pkt_num) as received_packets, sum(c2s_pkt_num+s2c_pkt_num) as packets from top_internal_host_log where __time >=TIMESTAMP @start and __time <TIMESTAMP @end and order_by='sessions' group by source order by sessions desc limit 100
--Q36.Top 100 External IP by Sessions
select destination as external_ip, sum(session_num) as sessions, sum(c2s_byte_num) as sent_bytes, sum(s2c_byte_num) as received_bytes, sum(c2s_byte_num + s2c_byte_num) as bytes, sum(c2s_pkt_num) as sent_packets ,sum(s2c_pkt_num) as received_packets, sum(c2s_pkt_num+s2c_pkt_num) as packets from top_external_host_log where __time >= @start and __time < @end and order_by='sessions' group by destination order by sessions desc limit 100
--Q37.Top 100 Domain by Bytes
select domain, sum(session_num) as sessions, sum(c2s_byte_num) as sent_bytes, sum(s2c_byte_num) as received_bytes, sum(c2s_byte_num + s2c_byte_num) as bytes, sum(c2s_pkt_num) as sent_packets ,sum(s2c_pkt_num) as received_packets, sum(c2s_pkt_num+s2c_pkt_num) as packets from top_website_domain_log where __time >=TIMESTAMP @start and __time <TIMESTAMP @end and order_by='bytes' group by domain order by bytes desc limit 100
--Q38.Top 100 Subscriber ID by Sessions
select subscriber_id, sum(session_num) as sessions, sum(c2s_byte_num) as sent_bytes, sum(s2c_byte_num) as received_bytes, sum(c2s_byte_num + s2c_byte_num) as bytes, sum(c2s_pkt_num) as sent_packets ,sum(s2c_pkt_num) as received_packets, sum(c2s_pkt_num+s2c_pkt_num) as packets from top_user_log where __time >=TIMESTAMP @start and __time <TIMESTAMP @end and order_by='sessions' group by subscriber_id order by sessions desc limit 100
--Q39.Top 100 Hit URLS by hits
select url,sum(session_num) as hits from top_urls_log where __time >=TIMESTAMP @start and __time <TIMESTAMP @end group by url order by hits desc limit 100
--Q40.Proxy Event Unique ISP
SELECT policy_id, APPROX_COUNT_DISTINCT_DS_HLL(isp) as num FROM proxy_event_hits_log where __time >= @start and __time < @end group by policy_id
--Q41.Traffic Composition Metrics
SELECT APPROX_COUNT_DISTINCT_DS_HLL(ip_object) AS uniq_client_ip, SUM(one_sided_connections) AS one_sided_connections, SUM(uncategorized_bytes) AS total_uncategorized_bytes, SUM(fragmentation_packets) AS fragmentation_packets, SUM(sequence_gap_loss) AS sequence_gap_loss_bytes, SUM(s2c_byte_num+c2s_byte_num) AS summaryTotalBytes, SUM(s2c_pkt_num+c2s_pkt_num) AS summaryTotalPackets, SUM(sessions) AS summarySessions FROM traffic_summary_log WHERE __time >= TIMESTAMP @start AND __time < TIMESTAMP @end LIMIT 1
--Q42.Traffic Composition Throughput
(SELECT SUM(c2s_byte_num + s2c_byte_num) as total_bytes, SUM(sessions) as total_sessions, (SUM(c2s_byte_num + s2c_byte_num) * 8)/((TIMESTAMP_TO_MILLIS(TIMESTAMP @end )-TIMESTAMP_TO_MILLIS(TIMESTAMP @start ))/1000) AS data_rate FROM traffic_protocol_stat_log WHERE __time >= TIMESTAMP @start AND __time < TIMESTAMP @end AND protocol_id = 'ETHERNET' LIMIT 1) UNION ALL ( SELECT SUM(sessions), 0, 0 FROM traffic_protocol_stat_log WHERE __time >= TIMESTAMP @start AND __time < TIMESTAMP @end AND protocol_id = 'ETHERNET' GROUP BY __time ORDER BY __time DESC LIMIT 1 )
--Q43.Traffic Composition Protocol Tree
SELECT protocol_id, SUM(sessions) as sessions,SUM(c2s_byte_num) as c2s_byte_num, SUM(c2s_pkt_num) as c2s_pkt_num, SUM(s2c_byte_num) as s2c_byte_num, SUM(s2c_pkt_num) as s2c_pkt_num FROM traffic_protocol_stat_log WHERE __time >= TIMESTAMP @start AND __time < TIMESTAMP @end GROUP BY protocol_id
--Q44.System Quota
SELECT log_type, SUM(used_size) as used_size, SUM(max_size) * 7/10 as max_size, TIME_FORMAT(LATEST(last_storage) * 1000,'YYYY-MM-dd') as first_storage FROM ( SELECT log_type, LATEST(used_size) as used_size, LATEST(max_size) as max_size, LATEST(last_storage) as last_storage FROM sys_storage_log WHERE __time >= CURRENT_TIMESTAMP - INTERVAL '1' HOUR AND data_center != '' GROUP BY data_center,log_type ) GROUP BY log_type
--Q45.System Quota Daily Trend
select TIME_FORMAT(__time,'YYYY-MM-dd') as stat_time,log_type as type, sum(aggregate_size) as used_size from sys_storage_log where __time >= @start and __time < @end group by TIME_FORMAT(__time,'YYYY-MM-dd'), log_type
--Q46.Traffic Statistics(Metrics01)
select sum(total_hit_sessions) as total_hit_sessions, sum(total_bytes_transferred) as total_bytes_transferred, sum(total_packets_transferred) as total_packets_transferred, sum(total_new_sessions) as total_new_sessions , sum(total_close_sessions) as total_close_sessions, sum(average_new_sessions_per_second) as average_new_sessions_per_second , sum(average_bytes_per_second) as average_bytes_per_second , sum(average_packets_per_second) as average_packets_per_second , COUNT(DISTINCT(device_id)) as device_num, sum(live_sessions) as average_live_sessions from ( select device_id, sum(intercept_conn_num + monitor_conn_num + deny_conn_num + allow_conn_num) as total_hit_sessions, sum(total_in_bytes + total_out_bytes) as total_bytes_transferred, sum(total_in_packets + total_out_packets) as total_packets_transferred, sum(new_conn_num) as total_new_sessions, sum(close_conn_num) as total_close_sessions, avg(nullif(new_conn_num, 0))/ 5 as average_new_sessions_per_second, avg(nullif(total_in_bytes + total_out_bytes, 0))* 8 / 5 as average_bytes_per_second, avg(nullif(total_in_packets + total_out_packets, 0))/ 5 as average_packets_per_second, avg(nullif(established_conn_num, 0)) as live_sessions from traffic_metrics_log where __time >= @start and __time < @end group by device_id)

53
testSchemaFiles/engine-filter.json Normal file
View File

@@ -0,0 +1,53 @@
{
"version": "1.0",
"name": "Engine-Raw",
"namespace": "Engine",
"filters": [
{
"name":"@start",
"value": "'2021-10-19 10:00:00'"
},
{
"name":"@end",
"value": "'2021-10-20 11:00:00'"
},
{
"name":"@common_filter",
"value": [
"common_log_id=1153021139190754263",
"common_client_ip='36.189.226.21'",
"common_internal_ip='223.116.37.192'",
"common_server_ip='8.8.8.8'",
"common_external_ip='111.10.53.14'",
"common_client_port=52607",
"common_server_port=443",
"common_c2s_pkt_num>5",
"common_s2c_pkt_num>5",
"common_c2s_byte_num>100",
"common_s2c_byte_num<200",
"common_schema_type='DNS'",
"common_establish_latency_ms>200",
"common_con_duration_ms>10000",
"common_stream_trace_id=1153021139190754263",
"common_tcp_client_isn=2857077935",
"common_tcp_server_isn=0",
"http_domain='microsoft.com'",
"mail_account='abc@xx.com'",
"mail_subject='test'",
"dns_qname='qbwup.imtt.qq.com'",
"ssl_sni='note.youdao.com'",
"ssl_con_latency_ms>100",
"ssl_ja3_hash='a0e9f5d64349fb13191bc781f81f42e1'",
"common_client_ip='36.189.226.21' and common_server_ip='8.8.8.8'",
"common_server_ip='111.10.53.14' and common_server_port=443",
"mail_account like 'abc@%'",
"http_domain like '%baidu.com%'",
"ssl_sni like '%youdao.com'",
"common_client_ip in ('36.189.226.21','111.10.53.14')",
"common_server_port not in (80,443)",
"notEmpty(http_domain)",
"http_domain not like '%microsoft.com'"
]
}
]
}

126
testSchemaFiles/engine-queries-template.sql Normal file
View File

@@ -0,0 +1,126 @@
--Q01.CK DateTime
select toDateTime(common_recv_time) as common_recv_time from session_record where common_recv_time >= toDateTime(@start) and common_recv_time< toDateTime(@end) limit 20
--Q02.Standard DateTime
select FROM_UNIXTIME(common_recv_time) as common_recv_time from session_record where common_recv_time >= UNIX_TIMESTAMP(@start) and common_recv_time< UNIX_TIMESTAMP(@end) limit 20
--Q03.count(1)
select count(1) from session_record where common_recv_time >= toDateTime(@start) and common_recv_time< toDateTime(@end)
--Q04.count(*)
select count(*) from session_record where common_recv_time >= toDateTime(@start) and common_recv_time< toDateTime(@end)
--Q05.UDF APPROX_COUNT_DISTINCT_DS_HLL
SELECT policy_id, APPROX_COUNT_DISTINCT_DS_HLL(isp) as num FROM proxy_event_hits_log where __time >= @start and __time < @end and policy_id=0 group by policy_id
--Q06.UDF TIME_FLOOR_WITH_FILL
select TIME_FLOOR_WITH_FILL(common_recv_time,'PT5M','previous') as stat_time from session_record where common_recv_time > toDateTime(@start) and common_recv_time < toDateTime(@end) group by stat_time
--Q07.UDF GEO IP
select IP_TO_GEO(common_client_ip) as geo,IP_TO_CITY(common_server_ip) as city,IP_TO_COUNTRY(common_server_ip) as country from session_record limit 10
--Q08.Special characters
select * from session_record where (common_protocol_label ='/$' or common_client_ip like'%') limit 10
--Q09.Federation Query
select * from (select FROM_UNIXTIME(TIME_FLOOR_WITH_FILL(common_recv_time,'PT5M','zero')) as stat_time from session_record where common_recv_time >= toDateTime(@start) and common_recv_time< toDateTime(@end) group by stat_time order by stat_time asc)
--Q10.Closed session Record Logs
select * from session_record where common_recv_time >= toDateTime(@start) and common_recv_time< toDateTime(@end) AND @common_filter order by common_recv_time desc limit 20
--Q11.Interim Session Record Logs
select * from interim_session_record where common_recv_time >= toDateTime(@start) and common_recv_time< toDateTime(@end) AND @common_filter order by common_recv_time desc limit 20
--Q12.Transaction Record Logs
select * from transaction_record where common_recv_time >= toDateTime(@start) and common_recv_time< toDateTime(@end) order by common_recv_time desc limit 20
--Q13.Security Event Logs
select * from security_event where common_recv_time >= UNIX_TIMESTAMP(@start) and common_recv_time< UNIX_TIMESTAMP(@end) AND @common_filter order by common_recv_time desc limit 0,20
--Q14.Proxy Event Logs
select * from proxy_event where common_recv_time >= UNIX_TIMESTAMP(@start) and common_recv_time< UNIX_TIMESTAMP(@end) order by common_recv_time desc limit 0,20
--Q15.Radius Record Logs
select * from radius_record where common_recv_time >= UNIX_TIMESTAMP(@start) and common_recv_time< UNIX_TIMESTAMP(@end) order by common_recv_time desc limit 0,20
--Q16.GTPC Record Logs
select * from gtpc_record where common_recv_time >= UNIX_TIMESTAMP(@start) and common_recv_time< UNIX_TIMESTAMP(@end) order by common_recv_time desc limit 0,20
--Q17.Closed session record with fields
select toDateTime(common_recv_time) AS common_recv_time, common_log_id, common_subscriber_id, common_imei, common_imsi, common_phone_number, common_client_ip, common_internal_ip, common_client_port, common_l4_protocol, common_address_type, common_server_ip, common_server_port, common_external_ip, common_direction, common_sled_ip, common_client_location, common_client_asn, common_server_location, common_server_asn, common_sessions, common_c2s_pkt_num, common_s2c_pkt_num, common_c2s_byte_num, common_s2c_byte_num, common_schema_type, common_device_id, common_device_group, common_app_behavior, common_app_label, common_tunnels, common_protocol_label, common_userdefine_app_name, common_l7_protocol, common_service_category, toDateTime(common_start_time) AS common_start_time, toDateTime(common_end_time) AS common_end_time, common_establish_latency_ms, common_con_duration_ms, common_stream_dir, common_stream_trace_id, common_c2s_ipfrag_num, common_s2c_ipfrag_num, common_c2s_tcp_lostlen, common_s2c_tcp_lostlen, common_c2s_tcp_unorder_num, common_s2c_tcp_unorder_num, common_c2s_pkt_retrans, common_s2c_pkt_retrans, common_c2s_byte_retrans, common_s2c_byte_retrans, common_tcp_client_isn, common_tcp_server_isn, toDateTime(common_processing_time) AS common_processing_time, toDateTime(common_ingestion_time) AS common_ingestion_time, http_url, http_host, http_domain, http_request_line, http_response_line, http_request_content_length, http_request_content_type, http_response_content_length, http_response_content_type, http_request_body, http_response_body, http_cookie, http_referer, http_user_agent, http_set_cookie, http_version, http_response_latency_ms, http_session_duration_ms, http_action_file_size, mail_protocol_type, mail_account, mail_from_cmd, mail_to_cmd, mail_from, mail_to, mail_cc, mail_bcc, mail_subject, mail_attachment_name, mail_eml_file, dns_message_id, dns_qr, dns_opcode, dns_aa, dns_tc, dns_rd, dns_ra, dns_rcode, dns_qdcount, dns_ancount, dns_nscount, dns_arcount, dns_qname, dns_qtype, dns_qclass, dns_sub, dns_response_latency_ms, ssl_sni, ssl_cn, ssl_pinningst, ssl_intercept_state, ssl_server_side_latency, ssl_client_side_latency, ssl_server_side_version, ssl_client_side_version, ssl_cert_verify, ssl_error, ssl_con_latency_ms, ssl_ja3_hash, ssl_cert_issuer, ssl_cert_subject, quic_version, quic_sni, quic_user_agent, ftp_account, ftp_url, ftp_content, ftp_link_type, app_extra_info, sip_call_id, sip_originator_description, sip_responder_description, sip_user_agent, sip_server, sip_originator_sdp_connect_ip, sip_originator_sdp_media_port, sip_originator_sdp_media_type, sip_originator_sdp_content, sip_responder_sdp_connect_ip, sip_responder_sdp_media_port, sip_responder_sdp_media_type, sip_responder_sdp_content, sip_duration_s, sip_bye, rtp_payload_type_c2s, rtp_payload_type_s2c, rtp_pcap_path, ssh_version, ssh_auth_success, ssh_client_version, ssh_server_version, ssh_cipher_alg, ssh_mac_alg, ssh_compression_alg, ssh_kex_alg, ssh_host_key_alg, ssh_host_key, ssh_hassh, stratum_cryptocurrency, stratum_mining_pools, stratum_mining_program from session_record where common_recv_time >= toDateTime(@start) and common_recv_time< toDateTime(@end) limit 20
--Q18.Interim session record with fields
SELECT toDateTime(common_recv_time) AS common_recv_time, common_log_id, common_subscriber_id, common_imei, common_imsi, common_phone_number, common_client_ip, common_internal_ip, common_client_port, common_l4_protocol, common_address_type, common_server_ip, common_server_port, common_external_ip, common_direction, common_sled_ip, common_client_location, common_client_asn, common_server_location, common_server_asn, common_sessions, common_c2s_pkt_num, common_s2c_pkt_num, common_c2s_byte_num, common_s2c_byte_num, common_c2s_pkt_diff, common_s2c_pkt_diff, common_c2s_byte_diff, common_s2c_byte_diff, common_schema_type, common_device_id, common_device_group, common_app_behavior, common_app_label, common_tunnels, common_protocol_label, common_l7_protocol, common_service_category, toDateTime(common_start_time) AS common_start_time, toDateTime(common_end_time) AS common_end_time, common_establish_latency_ms, common_con_duration_ms, common_stream_dir, common_stream_trace_id, common_c2s_ipfrag_num, common_s2c_ipfrag_num, common_c2s_tcp_lostlen, common_s2c_tcp_lostlen, common_c2s_tcp_unorder_num, common_s2c_tcp_unorder_num, common_c2s_pkt_retrans, common_s2c_pkt_retrans, common_c2s_byte_retrans, common_s2c_byte_retrans, common_tcp_client_isn, common_tcp_server_isn, toDateTime(common_processing_time) AS common_processing_time, toDateTime(common_ingestion_time) AS common_ingestion_time, http_url, http_host, http_domain, http_request_line, http_response_line, http_request_content_length, http_request_content_type, http_response_content_length, http_response_content_type, http_request_body, http_response_body, http_cookie, http_referer, http_user_agent, http_set_cookie, http_version, http_response_latency_ms, http_session_duration_ms, http_action_file_size, mail_protocol_type, mail_account, mail_from_cmd, mail_to_cmd, mail_from, mail_to, mail_cc, mail_bcc, mail_subject, mail_attachment_name, mail_eml_file, dns_message_id, dns_qr, dns_opcode, dns_aa, dns_tc, dns_rd, dns_ra, dns_rcode, dns_qdcount, dns_ancount, dns_nscount, dns_arcount, dns_qname, dns_qtype, dns_qclass, dns_sub, dns_response_latency_ms, ssl_sni, ssl_cn, ssl_pinningst, ssl_intercept_state, ssl_server_side_latency, ssl_client_side_latency, ssl_server_side_version, ssl_client_side_version, ssl_cert_verify, ssl_error, ssl_con_latency_ms, ssl_ja3_hash, ssl_cert_issuer, ssl_cert_subject, quic_version, quic_sni, quic_user_agent, ftp_account, ftp_url, ftp_content, ftp_link_type, app_extra_info, sip_call_id, sip_originator_description, sip_responder_description, sip_user_agent, sip_server, sip_originator_sdp_connect_ip, sip_originator_sdp_media_port, sip_originator_sdp_media_type, sip_originator_sdp_content, sip_responder_sdp_connect_ip, sip_responder_sdp_media_port, sip_responder_sdp_media_type, sip_responder_sdp_content, sip_duration_s, sip_bye, rtp_payload_type_c2s, rtp_payload_type_s2c, rtp_pcap_path, rtp_originator_dir, ssh_version, ssh_auth_success, ssh_client_version, ssh_server_version, ssh_cipher_alg, ssh_mac_alg, ssh_compression_alg, ssh_kex_alg, ssh_host_key_alg, ssh_host_key, ssh_hassh, stratum_cryptocurrency, stratum_mining_pools, stratum_mining_program FROM interim_session_record where common_recv_time >= @start and common_recv_time < @end order by common_recv_time desc limit 100000
--Q19.Security Event Logs with fields
SELECT toDateTime(common_recv_time) AS common_recv_time, common_log_id, common_policy_id, common_subscriber_id, common_imei, common_imsi, common_phone_number, common_client_ip, common_internal_ip, common_client_port, common_l4_protocol, common_address_type, common_server_ip, common_server_port, common_external_ip, common_action, common_direction, common_sled_ip, common_client_location, common_client_asn, common_server_location, common_server_asn, common_sessions, common_c2s_pkt_num, common_s2c_pkt_num, common_c2s_byte_num, common_s2c_byte_num, common_schema_type, common_device_id, common_device_group, common_app_behavior, common_app_label, common_tunnels, common_protocol_label, common_userdefine_app_name, common_l7_protocol, common_service_category, toDateTime(common_start_time) AS common_start_time, toDateTime(common_end_time) AS common_end_time, common_establish_latency_ms, common_con_duration_ms, common_stream_dir, common_stream_error, common_stream_trace_id, common_packet_capture_file, common_tcp_client_isn, common_tcp_server_isn, toDateTime(common_processing_time) AS common_processing_time, toDateTime(common_ingestion_time) AS common_ingestion_time, common_mirrored_pkts, common_mirrored_bytes, http_url, http_host, http_domain, http_request_line, http_response_line, http_request_content_length, http_request_content_type, http_response_content_length, http_response_content_type, http_request_body, http_response_body, http_cookie, http_referer, http_user_agent, http_set_cookie, http_version, http_response_latency_ms, http_action_file_size, http_session_duration_ms, mail_protocol_type, mail_account, mail_from_cmd, mail_to_cmd, mail_from, mail_to, mail_cc, mail_bcc, mail_subject, mail_attachment_name, dns_message_id, dns_qr, dns_opcode, dns_aa, dns_tc, dns_rd, dns_ra, dns_rcode, dns_qdcount, dns_ancount, dns_nscount, dns_arcount, dns_qname, dns_qtype, dns_qclass, dns_sub, ssl_sni, ssl_san, ssl_cn, ssl_pinningst, ssl_intercept_state, ssl_passthrough_reason, ssl_server_side_latency, ssl_client_side_latency, ssl_server_side_version, ssl_client_side_version, ssl_cert_verify, ssl_error, ssl_con_latency_ms, ssl_ja3_hash, ssl_cert_issuer, ssl_cert_subject, quic_version, quic_sni, quic_user_agent, ftp_account, ftp_url, ftp_content, ftp_link_type, app_extra_info, sip_call_id, sip_originator_description, sip_responder_description, sip_user_agent, sip_server, sip_originator_sdp_connect_ip, sip_originator_sdp_media_port, sip_originator_sdp_media_type, sip_originator_sdp_content, sip_responder_sdp_connect_ip, sip_responder_sdp_media_port, sip_responder_sdp_media_type, sip_responder_sdp_content, sip_duration_s, sip_bye, rtp_payload_type_c2s, rtp_payload_type_s2c, rtp_pcap_path, ssh_version, ssh_auth_success, ssh_client_version, ssh_server_version, ssh_cipher_alg, ssh_mac_alg, ssh_compression_alg, ssh_kex_alg, ssh_host_key_alg, ssh_host_key, ssh_hassh, stratum_cryptocurrency, stratum_mining_pools, stratum_mining_program from security_event where common_recv_time >= @start and common_recv_time < @end order by common_recv_time desc limit 100000
--Q20.Radius ON/OFF Logs For Frame IP
select framed_ip, arraySlice(groupUniqArray(concat(toString(event_timestamp),':', if(acct_status_type=1,'start','stop'))),1,100000) as timeseries from radius_onff_log where event_timestamp >=toDateTime(@start) and event_timestamp <toDateTime(@end) group by framed_ip limit 20
--Q21.Radius ON/OFF Logs For Account
select account, arraySlice(groupUniqArray(concat(toString(event_timestamp),':', if(acct_status_type=1,'start','stop'))),1,100000) as timeseries from radius_onff_log where event_timestamp >= toDateTime(@start) and event_timestamp < toDateTime(@end) group by account
--Q22.Radius ON/OFF Logs total Account number
select count(distinct(framed_ip)) as active_ip_num , sum(acct_session_time) as online_duration from (select any(framed_ip) as framed_ip ,max(acct_session_time) as acct_session_time from radius_onff_log where account='000jS' and event_timestamp >= toDateTime(@start) and event_timestamp < toDateTime(@end) group by acct_session_id)
--Q23.Radius ON/OFF Logs Account Access Detail
select max(if(acct_status_type=1,event_timestamp,0)) as start_time,max(if(acct_status_type=2,event_timestamp,0)) as end_time, any(framed_ip) as ip,max(acct_session_time) as online_duration from radius_onff_log where event_timestamp >= toDateTime(@start) and event_timestamp < toDateTime(@end) group by acct_session_id order by start_time desc limit 200
--Q24.Report for Client IP
select common_client_ip, count(*) as sessions from session_record where common_recv_time>= toStartOfDay(toDateTime(@start))-604800 and common_recv_time< toStartOfDay(toDateTime(@end)) group by common_client_ip order by sessions desc limit 0,100
--Q25.Report for Server IP
select common_server_ip, count(*) as sessions from session_record where common_recv_time>= toStartOfDay(toDateTime(@start))-604800 and common_recv_time< toStartOfDay(toDateTime(@start)) group by common_server_ip order by sessions desc limit 0,100
--Q26.Report for SSL SNI
select ssl_sni, count(*) as sessions from session_record where common_recv_time>= toStartOfDay(toDateTime(@start))-604800 and common_recv_time< toStartOfDay(toDateTime(@start)) group by ssl_sni order by sessions desc limit 0,100
--Q27.Report for SSL APP
select common_app_label as applicaiton, count(*) as sessions from session_record where common_recv_time>= toStartOfDay(toDateTime(@start))-604800 and common_recv_time< toStartOfDay(toDateTime(@start)) group by applicaiton order by sessions desc limit 0,100
--Q28.Report for Domains
select http_domain AS domain,SUM(coalesce(common_c2s_byte_num, 0)) AS sent_bytes,SUM(coalesce(common_s2c_byte_num, 0)) AS received_bytes,SUM(coalesce(common_c2s_byte_num, 0)+coalesce(common_s2c_byte_num, 0)) AS bytes FROM session_record WHERE common_recv_time >= toStartOfDay(toDateTime(@start))-86400 AND common_recv_time < toStartOfDay(toDateTime(@start)) and notEmpty(domain) GROUP BY domain ORDER BY bytes DESC LIMIT 100
--Q29.Report for Domains with unique Client IP
select toDateTime(intDiv(toUInt32(toDateTime(toDateTime(common_recv_time))), 300)*300) as stat_time, http_domain, uniq (common_client_ip) as nums from session_record where common_recv_time >= toStartOfDay(toDateTime(@start))-86400 AND common_recv_time < toStartOfDay(toDateTime(@start)) and http_domain in (select http_domain from session_record where common_recv_time >= toStartOfDay(toDateTime(@start))-86400 AND common_recv_time < toStartOfDay(toDateTime(@start)) and notEmpty(http_domain) group by http_domain order by SUM(coalesce(common_c2s_byte_num, 0)+coalesce(common_s2c_byte_num, 0)) desc limit 10 ) group by toDateTime(intDiv(toUInt32(toDateTime(toDateTime(common_recv_time))), 300)*300), http_domain order by stat_time asc limit 500
--Q30. Report for HTTP Host
SELECT http_host as host, SUM(coalesce(common_c2s_byte_num, 0)) AS sent_bytes,SUM(coalesce(common_s2c_byte_num, 0)) AS received_bytes,SUM(coalesce(common_c2s_byte_num, 0)+coalesce(common_s2c_byte_num, 0)) AS bytes FROM session_record WHERE common_recv_time>= toStartOfDay(toDateTime(@start))-604800 and common_recv_time< toStartOfDay(toDateTime(@start)) and notEmpty(http_host) GROUP BY host ORDER BY bytes DESC limit 100 union all SELECT 'totals' as host, SUM(coalesce(common_c2s_byte_num, 0)) AS sent_bytes, SUM(coalesce(common_s2c_byte_num, 0)) AS received_bytes, SUM(coalesce(common_c2s_byte_num, 0)+coalesce(common_s2c_byte_num, 0)) AS bytes from session_record where common_recv_time>= toStartOfDay(toDateTime(@start))-604800 and common_recv_time< toStartOfDay(toDateTime(@start)) and notEmpty(http_host)
--Q31.Report for HTTP/HTTPS URLS with Sessions
SELECT http_url AS url,count(*) AS sessions FROM proxy_event WHERE common_recv_time >= toStartOfDay(toDateTime(@start))-86400 AND common_recv_time < toStartOfDay(toDateTime(@start)) and notEmpty(http_url) GROUP BY url ORDER BY sessions DESC LIMIT 100
--Q32.Report for HTTP/HTTPS URLS with UNIQUE Client IP
select toDateTime(intDiv(toUInt32(toDateTime(toDateTime(common_recv_time))), 300)*300) as stat_time, http_url, count(distinct(common_client_ip)) as nums from proxy_event where common_recv_time >= toStartOfDay(toDateTime(@start))-86400 AND common_recv_time < toStartOfDay(toDateTime(@start)) and http_url IN (select http_url from proxy_event where common_recv_time >= toStartOfDay(toDateTime(@start))-86400 AND common_recv_time < toStartOfDay(toDateTime(@start)) and notEmpty(http_url) group by http_url order by count(*) desc limit 10 ) group by toDateTime(intDiv(toUInt32(toDateTime(toDateTime(common_recv_time))), 300)*300), http_url order by stat_time asc limit 500
--Q33.Report for Subscriber ID with Sessions
select common_subscriber_id as user, count(*) as sessions from session_record where common_recv_time>= toStartOfDay(toDateTime(@start))-604800 and common_recv_time< toStartOfDay(toDateTime(@start)) and notEmpty(user) group by common_subscriber_id order by sessions desc limit 0,100
--Q34.Report for Subscriber ID with Bandwidth
SELECT common_subscriber_id as user,SUM(coalesce(common_c2s_byte_num, 0)) AS sent_bytes,SUM(coalesce(common_s2c_byte_num, 0)) AS received_bytes,SUM(coalesce(common_c2s_byte_num, 0)+coalesce(common_s2c_byte_num, 0)) AS bytes FROM session_record WHERE common_recv_time>= toStartOfDay(toDateTime(@start))-604800 and common_recv_time< toStartOfDay(toDateTime(@start)) and notEmpty(user) GROUP BY user ORDER BY bytes DESC LIMIT 100
--Q35.Report Unique Endpoints
select uniq(common_client_ip) as "Client IP",uniq(common_server_ip) as "Server IP",uniq(common_internal_ip) as "Internal IP",uniq(common_external_ip) as "External IP",uniq(http_domain) as "Domain",uniq(ssl_sni) as "SNI" from session_record where common_recv_time>= toStartOfDay(toDateTime(@start))-604800 and common_recv_time< toStartOfDay(toDateTime(@start))
--Q36.TopN Optimizer
SELECT http_url AS url, SUM(common_sessions) AS sessions FROM session_record WHERE common_recv_time >= toDateTime(@start) AND common_recv_time < toDateTime(@end) AND notEmpty(http_url) GROUP BY http_url ORDER BY sessions DESC limit 10
--Q37.All Security Event Hits Trend by 5min B
select DATE_FORMAT(FROM_UNIXTIME(FLOOR(UNIX_TIMESTAMP(__time)/300)*300),'%Y-%m-%d %H:%i:%s') as start_time, sum(hits) as hits from security_event_hits_log where __time >= @start and __time < @end group by DATE_FORMAT(FROM_UNIXTIME(FLOOR(UNIX_TIMESTAMP(__time)/300)*300),'%Y-%m-%d %H:%i:%s') limit 10000
--Q38.Security Event Hit Time（first and last time) B
select policy_id, DATE_FORMAT(min(__time) ,'%Y-%m-%d %H:%i:%s') as first_used, DATE_FORMAT(max(__time) ,'%Y-%m-%d %H:%i:%s') as last_used from security_event_hits_log where policy_id in (0) group by policy_id
--Q39.All Proxy Event Hits Trend by 5min B
select FROM_UNIXTIME(FLOOR(UNIX_TIMESTAMP(__time)/300)*300) as start_time, sum(hits) as hits from proxy_event_hits_log where __time >= @start and __time < @end group by FROM_UNIXTIME(FLOOR(UNIX_TIMESTAMP(__time)/300)*300) limit 10000
--Q40.Proxy Event Hit Time（first and last time) B
select policy_id, DATE_FORMAT(min(__time) ,'%Y-%m-%d %H:%i:%s') as first_used, DATE_FORMAT(max(__time) ,'%Y-%m-%d %H:%i:%s') as last_used from proxy_event_hits_log where policy_id in (0) group by policy_id
--Q41.Traffic Composition Protocol Tree Trend
(SELECT TIME_FORMAT(MILLIS_TO_TIMESTAMP( 1000 * TIME_FLOOR_WITH_FILL(TIMESTAMP_TO_MILLIS(__time)/1000, 'PT30S', 'zero')), 'yyyy-MM-dd HH:mm:ss') as stat_time, protocol_id as type, sum(c2s_byte_num + s2c_byte_num) as bytes from traffic_protocol_stat_log where __time >= @start AND __time < @end and protocol_id = 'ETHERNET' group by TIME_FORMAT(MILLIS_TO_TIMESTAMP( 1000 * TIME_FLOOR_WITH_FILL(TIMESTAMP_TO_MILLIS(__time)/1000, 'PT30S', 'zero')), 'yyyy-MM-dd HH:mm:ss'), protocol_id order by stat_time asc) union all (SELECT TIME_FORMAT(MILLIS_TO_TIMESTAMP( 1000 * TIME_FLOOR_WITH_FILL(TIMESTAMP_TO_MILLIS(__time)/1000, 'PT30S', 'zero')), 'yyyy-MM-dd HH:mm:ss') as stat_time, protocol_id as type, sum(c2s_byte_num + s2c_byte_num) as bytes from traffic_protocol_stat_log where __time >= @start AND __time < @end and protocol_id like CONCAT('ETHERNET','.%') and LENGTH(protocol_id) = LENGTH(REPLACE(protocol_id,'.','')) + 1 + 0 group by TIME_FORMAT(MILLIS_TO_TIMESTAMP( 1000 * TIME_FLOOR_WITH_FILL(TIMESTAMP_TO_MILLIS(__time)/1000, 'PT30S', 'zero')), 'yyyy-MM-dd HH:mm:ss'), protocol_id order by stat_time asc)
--Q42.Traffic Metrics Security Action Hits Trend
select FROM_UNIXTIME(TIME_FLOOR_WITH_FILL(UNIX_TIMESTAMP(__time),'PT1800S','zero')) as statisticTime, sum(default_in_bytes + default_out_bytes) as default_bytes, sum(default_in_packets + default_out_packets) as default_packets, sum(default_conn_num) as default_sessions, sum(allow_in_bytes + allow_out_bytes) as allow_bytes, sum(allow_in_packets + allow_out_packets) as allow_packets, sum(allow_conn_num) as allow_sessions, sum(deny_in_bytes + deny_out_bytes) as deny_bytes, sum(deny_in_packets + deny_out_packets) as deny_packets, sum(deny_conn_num) as deny_sessions, sum(monitor_in_bytes + monitor_out_bytes) as monitor_bytes, sum(monitor_in_packets + monitor_out_packets) as monitor_packets, sum(monitor_conn_num) as monitor_sessions, sum(intercept_in_bytes + intercept_out_bytes) as intercept_bytes, sum(intercept_in_packets + intercept_out_packets) as intercept_packets, sum(intercept_conn_num) as intercept_sessions from traffic_metrics_log where __time >= @start and __time < @end group by FROM_UNIXTIME(TIME_FLOOR_WITH_FILL(UNIX_TIMESTAMP(__time),'PT1800S','zero')) limit 100000
--Q43.Traffic Metrics Proxy Action Hits Trend
SELECT FROM_UNIXTIME(TIME_FLOOR_WITH_FILL(UNIX_TIMESTAMP(__time),'PT1800S','zero')) AS statisticTime,SUM(intcp_allow_num) AS intercept_allow_conn_num,SUM(intcp_mon_num) AS intercept_monitor_conn_num,SUM(intcp_deny_num) AS intercept_deny_conn_num,SUM(intcp_rdirt_num) AS intercept_redirect_conn_num,SUM(intcp_repl_num) AS intercept_replace_conn_num,SUM(intcp_hijk_num) AS intercept_hijack_conn_num,SUM(intcp_ins_num) AS intercept_insert_conn_num FROM traffic_metrics_log WHERE __time >= @start AND __time < @end GROUP BY FROM_UNIXTIME(TIME_FLOOR_WITH_FILL(UNIX_TIMESTAMP(__time), 'PT1800S', 'zero')) LIMIT 100000
--Q44.Traffic Statistics(Metrics02)
select FROM_UNIXTIME(stat_time) as max_active_date_by_sessions, total_live_sessions as max_live_sessions from ( select stat_time, sum(live_sessions) as total_live_sessions from ( select TIME_FLOOR_WITH_FILL(UNIX_TIMESTAMP(__time), 'P1D') as stat_time, device_id, avg(established_conn_num) as live_sessions from traffic_metrics_log where __time >= @start and __time<@end group by TIME_FLOOR_WITH_FILL(UNIX_TIMESTAMP(__time), 'P1D'), device_id) group by stat_time order by total_live_sessions desc limit 1 )
--Q45.Traffic Summary(Bandwidth Trend)
select * from ( select DATE_FORMAT(FROM_UNIXTIME(TIME_FLOOR_WITH_FILL(UNIX_TIMESTAMP(__time),'PT1h','zero')),'%Y-%m-%d %H:%i:%s') as stat_time,'traffic_in_bytes' as type, sum(total_in_bytes) as bytes from traffic_metrics_log where __time >= @start and __time < @end group by DATE_FORMAT(FROM_UNIXTIME(TIME_FLOOR_WITH_FILL(UNIX_TIMESTAMP(__time),'PT1h','zero')),'%Y-%m-%d %H:%i:%s'), 'traffic_in_bytes' union all select DATE_FORMAT(FROM_UNIXTIME(TIME_FLOOR_WITH_FILL(UNIX_TIMESTAMP(__time),'PT1h','zero')),'%Y-%m-%d %H:%i:%s') as stat_time,'traffic_out_bytes' as type,sum(total_out_bytes) as bytes from traffic_metrics_log where __time >= @start and __time < @end group by DATE_FORMAT(FROM_UNIXTIME(TIME_FLOOR_WITH_FILL(UNIX_TIMESTAMP(__time),'PT1h','zero')),'%Y-%m-%d %H:%i:%s'),'traffic_out_bytes' ) order by stat_time asc limit 100000
--Q46.Traffic Summary(Sessions Trend)
select DATE_FORMAT(FROM_UNIXTIME(TIME_FLOOR_WITH_FILL(UNIX_TIMESTAMP(__time),'PT1h','zero')),'%Y-%m-%d %H:%i:%s') as stat_time, 'total_conn_num' as type, sum(new_conn_num) as sessions from traffic_metrics_log where __time >= @start and __time < @end group by DATE_FORMAT(FROM_UNIXTIME(TIME_FLOOR_WITH_FILL(UNIX_TIMESTAMP(__time),'PT1h','zero')),'%Y-%m-%d %H:%i:%s'), 'total_conn_num' order by stat_time asc limit 10000
--Q47.Domain Baidu.com Metrics
select FROM_UNIXTIME(min(common_recv_time)) as "First Seen" , FROM_UNIXTIME(max(common_recv_time)) as "Last Seen" , median(http_response_latency_ms) as "Server Processing Time Median(ms)", count(1) as Responses,any(common_server_location) as Location from session_record WHERE common_recv_time >= toDateTime(@start) AND common_recv_time < toDateTime(@end) AND http_domain='baidu.com'
--Q48.TIME_FLOOR_WITH_FILL 01
select "Device Group" as "Device Group" ,"Data Center" as "Data Center" ,FROM_UNIXTIME("End Time") as "End Time" , sum("counter") as "counter" from (select common_device_group as "Device Group" ,common_data_center as "Data Center" ,TIME_FLOOR_WITH_FILL (common_end_time,'PT1H','zero') as "End Time" ,count(common_log_id) as "counter" from session_record where common_recv_time >= toDateTime(@start) and common_recv_time< toDateTime(@end) group by "Device Group","Data Center","End Time") group by "Device Group" ,"Data Center" ,"End Time" order by "End Time" asc limit 5
--Q49.TIME_FLOOR_WITH_FILL 02
select FROM_UNIXTIME("End Time") as "End Time" , sum("counter") as "counter" from (select common_device_group as "Device Group" ,common_data_center as "Data Center" ,TIME_FLOOR_WITH_FILL (common_end_time,'PT1H','zero') as "End Time" ,count(common_log_id) as "counter" ,count(http_domain) as "HTTP.Domain" from security_event where ((common_recv_time >= toDateTime('2021-10-19 00:00:00') and common_recv_time < toDateTime('2021-10-20 00:00:00')) ) AND ( ( common_action = 2 ) ) group by "Device Group","Data Center","End Time") group by "End Time" order by "End Time" asc
--Q50.CONVERT_TZ (Druid) 01
SELECT CONVERT_TZ('2019-09-09 09:09:09','GMT','MET') as test_time from proxy_event_hits_log limit 1
--Q51.CONVERT_TZ (Druid) 02
SELECT CONVERT_TZ('2019-09-09 09:09:09','Europe/London','America/New_York') as test_time from proxy_event_hits_log limit 1
--Q52.CONVERT_TZ (Druid) 03
SELECT CONVERT_TZ(now(),'GMT','America/New_York') as test_time from proxy_event_hits_log limit 1
--Q53.CONVERT_TZ (clickhouse) 01
SELECT CONVERT_TZ('2019-09-09 09:09:09','GMT','MET') as test_time from session_record limit 1
--Q54.CONVERT_TZ (clickhouse) 02
SELECT CONVERT_TZ('2019-09-09 09:09:09','Europe/London','America/New_York') as test_time from session_record limit 1
--Q55.CONVERT_TZ (clickhouse) 03
SELECT CONVERT_TZ(now(),'GMT','America/New_York') as test_time from session_record limit 1
--Q56.CONVERT_TZ (hbase) 01
SELECT CONVERT_TZ('2019-09-09 09:09:09','GMT','MET') as test_time from report_result limit 1
--Q57.CONVERT_TZ (hbase) 02
SELECT CONVERT_TZ('2019-09-09 09:09:09','Europe/London','America/New_York') as test_time from report_result limit 1
--Q58.CONVERT_TZ (hbase) 03
SELECT CONVERT_TZ(now(),'GMT','America/New_York') as test_time from report_result limit 1
--Q59.CONVERT_TZ (elasticsearch)
SELECT CONVERT_TZ('2019-09-09 09:09:09','Europe/London','America/New_York') as time from report_result limit 1
--Q60.Authentication failed(code 516)
SELECT toDateTime(common_recv_time) AS common_recv_time, common_log_id, common_subscriber_id, common_imei, common_imsi, common_phone_number, common_client_ip, common_internal_ip, common_client_port, common_l4_protocol, common_address_type, common_server_ip, common_server_port, common_external_ip, common_direction, common_sled_ip, common_client_location, common_client_asn, common_server_location, common_server_asn, common_sessions, common_c2s_pkt_num, common_s2c_pkt_num, common_c2s_byte_num, common_s2c_byte_num, common_c2s_pkt_diff, common_s2c_pkt_diff, common_c2s_byte_diff, common_s2c_byte_diff, common_schema_type, common_device_id, common_device_group, common_app_behavior, common_app_label, common_tunnels, common_protocol_label, common_l7_protocol, common_service_category, toDateTime(common_start_time) AS common_start_time, toDateTime(common_end_time) AS common_end_time, common_establish_latency_ms, common_con_duration_ms, common_stream_dir, common_stream_trace_id, common_c2s_ipfrag_num, common_s2c_ipfrag_num, common_c2s_tcp_lostlen, common_s2c_tcp_lostlen, common_c2s_tcp_unorder_num, common_s2c_tcp_unorder_num, common_c2s_pkt_retrans, common_s2c_pkt_retrans, common_c2s_byte_retrans, common_s2c_byte_retrans, common_tcp_client_isn, common_tcp_server_isn, toDateTime(common_processing_time) AS common_processing_time, http_url, http_host, http_domain, http_request_content_length, http_request_content_type, http_response_content_length, http_response_content_type, http_request_body, http_response_body, http_cookie, http_referer, http_user_agent, http_set_cookie, http_version, http_response_latency_ms, http_session_duration_ms, http_action_file_size, mail_protocol_type, mail_account, mail_from_cmd, mail_to_cmd, mail_from, mail_to, mail_cc, mail_bcc, mail_subject, mail_attachment_name, mail_eml_file, dns_message_id, dns_qr, dns_opcode, dns_aa, dns_tc, dns_rd, dns_ra, dns_rcode, dns_qdcount, dns_ancount, dns_nscount, dns_arcount, dns_qname, dns_qtype, dns_qclass, dns_sub, ssl_sni, ssl_cn, ssl_pinningst, ssl_intercept_state, ssl_server_side_latency, ssl_client_side_latency, ssl_server_side_version, ssl_client_side_version, ssl_cert_verify, ssl_error, ssl_con_latency_ms, ssl_ja3_hash, ssl_cert_issuer, ssl_cert_subject, quic_version, quic_sni, quic_user_agent, ftp_account, ftp_url, ftp_content, ftp_link_type, app_extra_info, sip_call_id, sip_originator_description, sip_responder_description, sip_user_agent, sip_server, sip_originator_sdp_connect_ip, sip_originator_sdp_media_port, sip_originator_sdp_media_type, sip_originator_sdp_content, sip_responder_sdp_connect_ip, sip_responder_sdp_media_port, sip_responder_sdp_media_type, sip_responder_sdp_content, sip_duration_s, sip_bye, rtp_payload_type_c2s, rtp_payload_type_s2c, rtp_pcap_path, rtp_originator_dir, ssh_version, ssh_auth_success, ssh_client_version, ssh_server_version, ssh_cipher_alg, ssh_mac_alg, ssh_compression_alg, ssh_kex_alg, ssh_host_key_alg, ssh_host_key, ssh_hassh, stratum_cryptocurrency, stratum_mining_pools, stratum_mining_program FROM interim_session_record AS interim_session_record WHERE common_recv_time >= toUnixTimestamp(@start) AND common_recv_time < toUnixTimestamp(@end) ORDER BY common_recv_time DESC LIMIT 43233, 20
--Q61.Function MAX_DURATION
SELECT destination_ip, IP_TO_GEO(destination_ip) AS destination_geo, MAX_DURATION(end_time,600) AS max_duration, any(destination_country) AS destination_country, groupUniqArray(arrayJoin(splitByString(',',source_country_list))) AS source_coutries,max(bit_rate) AS max_bit_rate,max(packet_rate) AS max_packet_rate,max(session_rate) AS max_session_rate,min(start_time) AS first_active_time,max(end_time) AS last_active_time,groupUniqArray(attack_type) AS attack_type,count(*) AS count from dos_event where start_time >= toUnixTimestamp(@start) AND start_time < toUnixTimestamp(@end) GROUP BY destination_ip ORDER BY count desc
--Q62.notEmpty(druid)
SELECT device_id from traffic_metrics_log where __time >= @start and __time < @end AND notEmpty(device_id) limit 10
--Q63.empty(druid)
SELECT device_id from traffic_metrics_log where __time >= @start and __time < @end AND empty(device_id) limit 10

15
testSchemaFiles/es-filter.json Normal file
View File

@@ -0,0 +1,15 @@
{
"version": "1.0",
"name": "es-Raw",
"namespace": "tsg",
"filters": [
{
"name":"@start",
"value": "cast(now() as long)/1000 -3600"
},
{
"name":"@end",
"value": "cast(now() as long)/1000"
}
]
}

1
testSchemaFiles/es-queries-template.sql Normal file
View File

@@ -0,0 +1 @@
--Q01.empty

1613
testSchemaFiles/gtpc_record.json Normal file
View File

File diff suppressed because it is too large Load Diff

15
testSchemaFiles/hbase-filter.json Normal file
View File

@@ -0,0 +1,15 @@
{
"version": "1.0",
"name": "hbase-Raw",
"namespace": "tsg",
"filters": [
{
"name":"@start",
"value": "'2021-10-19 10:00:00'"
},
{
"name":"@end",
"value": "'2021-10-20 11:00:00'"
}
]
}

4
testSchemaFiles/hbase-queries-template.sql Normal file
View File

@@ -0,0 +1,4 @@
--Q01. 范围查询
SELECT last_update_time FROM relation_account_framedip WHERE last_update_time>=CAST(TO_TIMESTAMP (@start,'yyyy-MM-dd HH:mm:ss','Asia/Shanghai') AS UNSIGNED_LONG) AND last_update_time<CAST(TO_TIMESTAMP (@end,'yyyy-MM-dd HH:mm:ss','Asia/Shanghai') AS UNSIGNED_LONG) LIMIT 30
--Q02. KV查询
select * from relation_account_framedip where ROWKEY = '0a771a381088e7d72ded13e998c06cbe' limit 1

3796
testSchemaFiles/interim_session_record.json Normal file
View File

File diff suppressed because it is too large Load Diff

42
testSchemaFiles/job_result.json Normal file
View File

@@ -0,0 +1,42 @@
{
"type": "record",
"name": "job_result",
"namespace": "tsg_galaxy",
"fields": [
{
"name": "ROWKEY",
"label": "Row Key",
"type": "string"
},
{
"name": "is_done",
"label": "Done",
"type": "boolean"
},
{
"name": "is_canceled",
"label": "Canceled",
"type": "boolean"
},
{
"name": "done_progress",
"label": "Progress",
"type": "double"
},
{
"name": "last_query_time",
"label": "Last Query Time",
"type": "long"
},
{
"name": "duration_time",
"label": "Duration Time",
"type": "long"
},
{
"name": "discovery_field",
"label": "Discovery Field",
"type": "string"
}
]
}

163
testSchemaFiles/liveChart_interim.json Normal file
View File

@@ -0,0 +1,163 @@
{
"type": "record",
"name": "liveChart_interim",
"in": "INTERIM-SESSION-RECORD",
"out": "TRAFFIC-PROTOCOL-STAT",
"task": "Protocol-Distribution",
"doc": {
"timestamp": {
"name": "stat_time",
"type": "long"
},
"dimensions": [
{
"name": "protocol_id",
"fieldName": "common_protocol_label",
"type": "string"
},
{
"name": "entrance_id",
"fieldName": "common_entrance_id",
"type": "string"
},
{
"name": "isp",
"fieldName": "common_isp",
"type": "string"
},
{
"name": "data_center",
"fieldName": "common_data_center",
"type": "string"
},
{
"name": "device_group",
"fieldName": "common_device_group",
"type": "string"
}
],
"metrics": [
{
"function": "sum",
"name": "sessions",
"fieldName": "common_sessions",
"type": "long"
},
{
"function": "sum",
"name": "c2s_byte_num",
"fieldName": "common_c2s_byte_diff",
"type": "long"
},
{
"function": "sum",
"name": "s2c_byte_num",
"fieldName": "common_s2c_byte_diff",
"type": "long"
},
{
"function": "sum",
"name": "c2s_pkt_num",
"fieldName": "common_c2s_pkt_diff",
"type": "long"
},
{
"function": "sum",
"name": "s2c_pkt_num",
"fieldName": "common_s2c_pkt_diff",
"type": "long"
},
{
"function": "sum",
"name": "c2s_ipfrag_num",
"fieldName": "common_c2s_ipfrag_num",
"type": "long"
},
{
"function": "sum",
"name": "s2c_ipfrag_num",
"fieldName": "common_s2c_ipfrag_num",
"type": "long"
},
{
"function": "sum",
"name": "c2s_tcp_lostlen",
"fieldName": "common_c2s_tcp_lostlen",
"type": "long"
},
{
"function": "sum",
"name": "s2c_tcp_lostlen",
"fieldName": "common_s2c_tcp_lostlen",
"type": "long"
},
{
"function": "sum",
"name": "c2s_tcp_unorder_num",
"fieldName": "common_c2s_tcp_unorder_num",
"type": "long"
},
{
"function": "sum",
"name": "s2c_tcp_unorder_num",
"fieldName": "common_s2c_tcp_unorder_num",
"type": "long"
},
{
"function": "disCount",
"name": "unique_sip_num",
"fieldName": "common_server_ip",
"type": "long"
},
{
"function": "disCount",
"name": "unique_cip_num",
"fieldName": "common_client_ip",
"type": "long"
}
],
"filters": [
{
"fieldName": "common_protocol_label",
"type": "notempty"
}
],
"transforms": [
{
"function": "combination",
"name": "protocol_id",
"fieldName": "common_protocol_label",
"parameters": "common_l7_protocol,."
},
{
"function": "combination",
"name": "protocol_id",
"fieldName": "common_protocol_label",
"parameters": "common_app_label,."
},
{
"function": "flattenSpec",
"name": "device_group",
"fieldName": "common_device_tag",
"parameters": "$.tags[?(@.tag=='device_group')].value"
},
{
"function": "hierarchy",
"name": "protocol_id",
"fieldName": "common_l7_protocol",
"parameters": "."
}
],
"action": [
{
"label": "Default",
"metrics": "c2s_byte_num,s2c_byte_num,c2s_pkt_num,s2c_pkt_num"
}
],
"granularity": {
"type": "period",
"period": "15S"
}
},
"fields": []
}

163
testSchemaFiles/liveChart_session.json Normal file
View File

@@ -0,0 +1,163 @@
{
"type": "record",
"name": "liveChart_session",
"in": "SESSION-RECORD",
"out": "TRAFFIC-PROTOCOL-STAT",
"task": "Protocol-Distribution",
"doc": {
"timestamp": {
"name": "stat_time",
"type": "long"
},
"dimensions": [
{
"name": "protocol_id",
"fieldName": "common_protocol_label",
"type": "string"
},
{
"name": "entrance_id",
"fieldName": "common_entrance_id",
"type": "string"
},
{
"name": "isp",
"fieldName": "common_isp",
"type": "string"
},
{
"name": "data_center",
"fieldName": "common_data_center",
"type": "string"
},
{
"name": "device_group",
"fieldName": "common_device_group",
"type": "string"
}
],
"metrics": [
{
"function": "sum",
"name": "sessions",
"fieldName": "common_sessions",
"type": "long"
},
{
"function": "sum",
"name": "c2s_byte_num",
"fieldName": "common_c2s_byte_diff",
"type": "long"
},
{
"function": "sum",
"name": "s2c_byte_num",
"fieldName": "common_s2c_byte_diff",
"type": "long"
},
{
"function": "sum",
"name": "c2s_pkt_num",
"fieldName": "common_c2s_pkt_diff",
"type": "long"
},
{
"function": "sum",
"name": "s2c_pkt_num",
"fieldName": "common_s2c_pkt_diff",
"type": "long"
},
{
"function": "sum",
"name": "c2s_ipfrag_num",
"fieldName": "common_c2s_ipfrag_num",
"type": "long"
},
{
"function": "sum",
"name": "s2c_ipfrag_num",
"fieldName": "common_s2c_ipfrag_num",
"type": "long"
},
{
"function": "sum",
"name": "c2s_tcp_lostlen",
"fieldName": "common_c2s_tcp_lostlen",
"type": "long"
},
{
"function": "sum",
"name": "s2c_tcp_lostlen",
"fieldName": "common_s2c_tcp_lostlen",
"type": "long"
},
{
"function": "sum",
"name": "c2s_tcp_unorder_num",
"fieldName": "common_c2s_tcp_unorder_num",
"type": "long"
},
{
"function": "sum",
"name": "s2c_tcp_unorder_num",
"fieldName": "common_s2c_tcp_unorder_num",
"type": "long"
},
{
"function": "disCount",
"name": "unique_sip_num",
"fieldName": "common_server_ip",
"type": "long"
},
{
"function": "disCount",
"name": "unique_cip_num",
"fieldName": "common_client_ip",
"type": "long"
}
],
"filters": [
{
"fieldName": "common_protocol_label",
"type": "notempty"
}
],
"transforms": [
{
"function": "combination",
"name": "protocol_id",
"fieldName": "common_protocol_label",
"parameters": "common_l7_protocol,."
},
{
"function": "combination",
"name": "protocol_id",
"fieldName": "common_protocol_label",
"parameters": "common_app_label,."
},
{
"function": "flattenSpec",
"name": "device_group",
"fieldName": "common_device_tag",
"parameters": "$.tags[?(@.tag=='device_group')].value"
},
{
"function": "hierarchy",
"name": "protocol_id",
"fieldName": "common_l7_protocol",
"parameters": "."
}
],
"action": [
{
"label": "Default",
"metrics": "sessions,c2s_byte_num,s2c_byte_num,c2s_pkt_num,s2c_pkt_num,c2s_ipfrag_num,s2c_ipfrag_num,c2s_tcp_lostlen,s2c_tcp_lostlen,c2s_tcp_unorder_num,s2c_tcp_unorder_num"
}
],
"granularity": {
"type": "period",
"period": "15S"
}
},
"fields": []
}

87
testSchemaFiles/meta_data.json Normal file
View File

@@ -0,0 +1,87 @@
{
"metadata": [
{
"namespace": "tsg_galaxy_v3",
"group": "CLICKHOUSE_GROUP",
"tables": [
"radius_onff_log",
"session_record",
"session_record_common_client_ip",
"session_record_common_server_ip",
"session_record_http_domain",
"interim_session_record",
"transaction_record",
"radius_record",
"voip_record",
"gtpc_record",
"security_event",
"proxy_event",
"dos_event",
"active_defence_event",
"sys_packet_capture_event",
"assessment_event"
]
},
{
"namespace": "system",
"group": "CLICKHOUSE_GROUP",
"tables": [
"query_log_cluster",
"tables_cluster",
"columns_cluster",
"disks_cluster",
"parts_cluster",
"processes",
"query_log",
"tables",
"clusters",
"distributed_ddl_queue"
]
},
{
"namespace": "druid",
"group": "DRUID_GROUP",
"tables": [
"top_internal_host_log",
"top_client_ip_log",
"top_external_host_log",
"top_server_ip_log",
"top_website_domain_log",
"top_user_log",
"top_urls_log",
"proxy_event_hits_log",
"security_event_hits_log",
"traffic_summary_log",
"traffic_protocol_stat_log",
"traffic_metrics_log",
"traffic_app_stat_log",
"traffic_top_destination_ip_metrics_log",
"sys_storage_log"
]
},
{
"namespace": "etl",
"group": "ETL_GROUP",
"tables": [
"liveChart_session",
"liveChart_interim"
]
},
{
"namespace":"tsg",
"group":"HBASE_GROUP",
"tables":[
"report_result"
]
},
{
"namespace": "tsg_galaxy",
"group": "HBASE_GROUP",
"tables": [
"relation_account_framedip",
"recommendation_app_cip",
"job_result"
]
}
]
}

11
testSchemaFiles/parts_cluster.json Normal file
View File

@@ -0,0 +1,11 @@
{
"namespace": "system",
"type": "record",
"name": "parts_cluster",
"fields": [
{
"name": "name",
"type": "string"
}
]
}

11
testSchemaFiles/processes.json Normal file
View File

@@ -0,0 +1,11 @@
{
"namespace": "system",
"type": "record",
"name": "processes",
"fields": [
{
"name": "query_id",
"type": "string"
}
]
}

2271
testSchemaFiles/proxy_event.json Normal file
View File

File diff suppressed because it is too large Load Diff

157
testSchemaFiles/proxy_event_hits_log.json Normal file
View File

@@ -0,0 +1,157 @@
{
"type": "record",
"name": "proxy_event_hits_log",
"namespace": "druid",
"doc": {
"partition_key": "__time",
"functions": {
"$ref": "public_schema_info.json#/functions"
},
"schema_query": {
"references": {
"$ref": "public_schema_info.json#/schema_query/references"
}
}
},
"fields": [
{
"name": "__time",
"label": "Time",
"type": "string",
"doc": {
"constraints": {
"type": "timestamp"
},
"visibility": "enabled"
}
},
{
"name": "isp",
"label": "ISP",
"type": "string",
"doc": {
"visibility": "disabled"
}
},
{
"name": "entrance_id",
"label": "Entrance ID",
"type": "long",
"doc": {
"visibility": "disabled"
}
},
{
"name": "hits",
"label": "Hits",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "policy_id",
"label": "Policy ID",
"type": "long",
"doc": {
"constraints": {
"operator_functions": "=,in"
},
"visibility": "enabled"
}
},
{
"name": "action",
"label": "Action",
"doc": {
"visibility": "hidden"
},
"type": "long"
},
{
"name": "sub_action",
"label": "Action",
"type": "string",
"doc": {
"constraints": {
"operator_functions": "=,in"
},
"data": [
{
"code": "allow",
"value": "Allow"
},
{
"code": "deny",
"value": "Deny"
},
{
"code": "monitor",
"value": "Monitor"
},
{
"code": "replace",
"value": "Replace"
},
{
"code": "redirect",
"value": "Redirect"
},
{
"code": "insert",
"value": "Insert"
},
{
"code": "hijack",
"value": "Hijack"
},
{
"code": "edit_element",
"value": "Edit Element"
}
],
"visibility": "enabled"
}
},
{
"name": "ip_object",
"label": "IP Object",
"doc": {
"visibility": "enabled"
},
"type": "string"
},
{
"name": "country",
"label": "Country",
"doc": {
"visibility": "enabled"
},
"type": "string"
},
{
"name": "location",
"label": "Location",
"doc": {
"visibility": "enabled"
},
"type": "string"
},
{
"name": "c2s_byte_num",
"label": "Bytes Sent",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "s2c_byte_num",
"label": "Bytes Received",
"doc": {
"visibility": "enabled"
},
"type": "long"
}
]
}

167
testSchemaFiles/public_code_info.json Normal file
View File

@@ -0,0 +1,167 @@
{
"CDN": {
"Akamai": [
"akadns.net",
"akagtm.org",
"akahost.net",
"akamai.com",
"akamaiedge.net",
"akamaiedge-staging.net",
"akamaientrypoint.net",
"akamaihd.net",
"akamai.net",
"akamaistream.net",
"akamaitech.net",
"akamaitechnologies.com",
"akamaitechnologies.fr",
"akamaized.net",
"akam.net",
"akasecure.net",
"edgekey.net",
"edgesuite.net"
],
"Cloudflare": [
"cloudflareaccess.com",
"cloudflareclient.com",
"cloudflare.com",
"cloudflare-dm-cmpimg.com",
"cloudflareinsights.com",
"cloudflare-ipfs.com",
"cloudflare.net",
"cloudflareok.com",
"cloudflareportal.com",
"cloudflare-quic.com",
"cloudflareresolve.com",
"cloudflaressl.com",
"cloudflarestatus.com",
"cloudflare-terms-of-service-abuse.com",
"sn-cloudflare.com"
],
"Google": [
"cache.google.com",
"googlevideo.com"
],
"Amazon CloudFront": [
"cloudfront.net"
],
"Fastly": [
"astly-analytics.com",
"fastly.com",
"fastly-debug.com",
"fastlydns.net",
"fastly-insights.com",
"fastly.io",
"fastlylabs.com",
"fastlylb.net",
"fastly.net",
"fastly-status.com",
"secretcdn-stg.net"
],
"Bunny": [
"b-cdn.net",
"bunnyinfra.net",
"bunny.net"
],
"G-Core": [
"gcdn.co",
"gcorelabs.com"
],
"KeyCDN": [
"keycdn.com",
"kxcdn.com"
],
"Alibaba": [
"alicdn.com"
],
"Edgecast": [
"edgecastcdn.net",
"edgecast.com",
"edgecastdns.net",
"phicdn.net",
"verizondigitalmedia.com",
"verizonmedia.com"
],
"Huawei": [
"cdnhwc1.com",
"cdnhwc2.com",
"cdnhwc3.com",
"cdnhwc5.com",
"cdnhwc6.com",
"cdnhwc7.com",
"cdnhwc8.com",
"livehwc3.cn"
],
"Azure Front Door": [
"a-msedge.net",
"au-msedge.net",
"b-msedge.net",
"c-msedge.net",
"cn-msedge.net",
"dc-msedge.net",
"e-msedge.net",
"exo-msedge.net",
"fbs1-t-msedge.net",
"fbs2-a-msedge.net",
"fbs2-e-msedge.net",
"fb-t-msedge.net",
"f-msedge.net",
"k-msedge.net",
"l-msedge.net",
"m1-msedge.net",
"msedge.net",
"o-msedge.net",
"q-msedge.net",
"q-t-msedge.net",
"segment2-s-msedge.net",
"s-msedge.net",
"t-msedge.net"
],
"BaishanCloud": [
"baishancloud.com"
],
"CDN77": [
"cdn77.com",
"cdn77.org"
],
"Limelight Networks": [
"delvenetworks.com",
"limelight.com",
"lldns.net",
"llnw.com",
"llnwd.net",
"llnwi.net",
"llnw.net",
"llnw-trials.com"
],
"Lumen": [
"footprintdns.com",
"footprint.net"
],
"Meta": [
"fbcdn.net"
],
"StackPath": [
"highwinds.com",
"hwcdn.net",
"stackpath.com",
"stackpathedge.net"
],
"Wangsu": [
"cdn20.com",
"cdn30.com",
"cdnetworks.com",
"cdnetworks.net",
"chinanetcenter.com",
"lxdns.com",
"quantil.com",
"wangsu.com",
"wscdns.com",
"wscloudcdn.com",
"wsdvs.com",
"wsglb0.com",
"wswebcdn.com",
"wswebpic.com",
"wtxcdn.com"
]
}
}

2247
testSchemaFiles/public_schema_info.json Normal file
View File

File diff suppressed because it is too large Load Diff

11
testSchemaFiles/query_log.json Normal file
View File

@@ -0,0 +1,11 @@
{
"namespace": "system",
"type": "record",
"name": "query_log",
"fields": [
{
"name": "query_id",
"type": "string"
}
]
}

11
testSchemaFiles/query_log_cluster.json Normal file
View File

@@ -0,0 +1,11 @@
{
"namespace": "system",
"type": "record",
"name": "query_log_cluster",
"fields": [
{
"name": "type",
"type": "string"
}
]
}

62
testSchemaFiles/radius_onff_log.json Normal file
View File

@@ -0,0 +1,62 @@
{
"type": "record",
"name": "radius_onff_log",
"namespace": "tsg_galaxy_v3",
"doc": {
"partition_key": "event_timestamp",
"index_key": [
"account",
"event_timestamp"
]
},
"fields": [
{
"name": "event_timestamp",
"label": "Event Time",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "account",
"label": "Account",
"doc": {
"visibility": "enabled"
},
"type": "string"
},
{
"name": "framed_ip",
"label": "Framed IP",
"doc": {
"visibility": "enabled"
},
"type": "string"
},
{
"name": "acct_session_id",
"label": "Acct Session ID",
"doc": {
"visibility": "enabled"
},
"type": "string"
},
{
"name": "acct_status_type",
"label": "Acct Status Type",
"doc": {
"visibility": "enabled"
},
"type": "int"
},
{
"name": "acct_session_time",
"label": "Acct Session Time",
"doc": {
"visibility": "enabled"
},
"type": "int"
}
]
}

1725
testSchemaFiles/radius_record.json Normal file
View File

File diff suppressed because it is too large Load Diff

27
testSchemaFiles/recommendation_app_cip.json Normal file
View File

@@ -0,0 +1,27 @@
{
"type": "record",
"name": "recommendation_app_cip",
"namespace": "tsg_galaxy",
"fields": [
{
"name": "ROWKEY",
"label": "Row Key",
"type": "string"
},
{
"name": "app_label",
"label": "APP Label",
"type": "string"
},
{
"name": "last_update_time",
"label": "Last Update Time",
"type": "long"
},
{
"name": "client_ip_list",
"label": "Client IP List",
"type": "string"
}
]
}

37
testSchemaFiles/relation_account_framedip.json Normal file
View File

@@ -0,0 +1,37 @@
{
"type": "record",
"name": "relation_account_framedip",
"namespace": "tsg_galaxy",
"fields": [
{
"name":"ROWKEY",
"label":"Row Key",
"type":"string"
},
{
"name":"acct_status_type",
"label":"Acct Status Type",
"type":"string"
},
{
"name":"first_found_time",
"label":"First Found Time",
"type":"long"
},
{
"name":"last_update_time",
"label":"Last Update Time",
"type":"long"
},
{
"name":"framed_ip",
"label":"Framed IP",
"type":"string"
},
{
"name":"account",
"label":"Account",
"type":"string"
}
]
}

32
testSchemaFiles/report_result.json Normal file
View File

@@ -0,0 +1,32 @@
{
"type": "record",
"name": "report_result",
"namespace": "tsg",
"fields": [
{
"name":"ROWKEY",
"label":"Row Key",
"type":"string"
},
{
"name":"excute_sql",
"label":"Excute SQL",
"type":"string"
},
{
"name":"read_rows",
"label":"Read Rows",
"type":"long"
},
{
"name":"result_id",
"label":"Result ID",
"type":"int"
},
{
"name":"result",
"label":"Result",
"type":"string"
}
]
}

3853
testSchemaFiles/security_event.json Normal file
View File

File diff suppressed because it is too large Load Diff

109
testSchemaFiles/security_event_hits_log.json Normal file
View File

@@ -0,0 +1,109 @@
{
"type": "record",
"name": "security_event_hits_log",
"namespace": "druid",
"doc": {
"partition_key": "__time",
"functions": {
"$ref": "public_schema_info.json#/functions"
},
"schema_query": {
"references": {
"$ref": "public_schema_info.json#/schema_query/references"
}
}
},
"fields": [
{
"name": "__time",
"label": "Time",
"type": "string",
"doc": {
"constraints": {
"type": "timestamp"
},
"visibility": "enabled"
}
},
{
"name": "isp",
"label": "ISP",
"type": "string",
"doc": {
"visibility": "disabled"
}
},
{
"name": "entrance_id",
"label": "Entrance ID",
"type": "long",
"doc": {
"visibility": "disabled"
}
},
{
"name": "policy_id",
"label": "Policy ID",
"type": "long",
"doc": {
"constraints": {
"operator_functions": "=,in"
},
"visibility": "enabled"
}
},
{
"name": "action",
"label": "Action",
"type": "long",
"doc": {
"constraints": {
"operator_functions": "=,in"
},
"data": [
{
"code": "1",
"value": "Monitor"
},
{
"code": "2",
"value": "Intercept"
},
{
"code": "16",
"value": "Deny"
},
{
"code": "128",
"value": "Allow"
}
],
"visibility": "enabled"
}
},
{
"name": "hits",
"label": "Hits",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "c2s_byte_num",
"label": "Bytes Sent",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "s2c_byte_num",
"label": "Bytes Received",
"doc": {
"visibility": "enabled"
},
"type": "long"
}
]
}

3813
testSchemaFiles/session_record.json Normal file
View File

File diff suppressed because it is too large Load Diff

174
testSchemaFiles/session_record_common_client_ip.json Normal file
View File

@@ -0,0 +1,174 @@
{
"type":"record",
"name":"session_record_common_client_ip",
"namespace":"tsg_galaxy_v3",
"doc":
{
"primary_key":"common_log_id",
"partition_key":"common_recv_time",
"ttl":null,
"default_ttl":2592000,
"index_key":
[
"common_client_ip",
"common_server_ip",
"common_recv_time"
]
},
"fields":
[
{
"name":"common_log_id",
"type":"long",
"doc":
{
"visibility":"enabled",
"ttl":null
}
},
{
"name":"common_recv_time",
"type":"long",
"doc":
{
"visibility":"enabled",
"ttl":null
}
},
{
"name":"common_server_ip",
"type":"string",
"doc":
{
"visibility":"enabled",
"ttl":null
}
},
{
"name":"common_client_ip",
"type":"string",
"doc":
{
"visibility":"enabled",
"ttl":null
}
},
{
"name":"common_sled_ip",
"type":"string",
"doc":
{
"visibility":"enabled",
"ttl":null
}
},
{
"name":"common_entrance_id",
"type":"int",
"doc":
{
"visibility":"disabled",
"ttl":null
}
},
{
"name":"common_subscriber_id",
"type":"string",
"doc":
{
"visibility":"enabled",
"ttl":null
}
},
{
"name":"common_stream_trace_id",
"type":"long",
"doc":
{
"visibility":"enabled",
"ttl":null
}
},
{
"name":"common_schema_type",
"type":"string",
"doc":
{
"visibility":"enabled",
"ttl":null
}
},
{
"name":"common_client_port",
"type":"int",
"doc":
{
"visibility":"enabled",
"ttl":null
}
},
{
"name":"common_server_port",
"type":"int",
"doc":
{
"visibility":"enabled",
"ttl":null
}
},
{
"name":"common_app_label",
"type":"string",
"doc":
{
"visibility":"enabled",
"ttl":null
}
},
{
"name":"common_direction",
"type":"int",
"doc":
{
"visibility":"enabled",
"ttl":null
}
},
{
"name":"http_domain",
"type":"string",
"doc":
{
"visibility":"enabled",
"ttl":null
}
},
{
"name":"ssl_sni",
"type":"string",
"doc":
{
"visibility":"enabled",
"ttl":null
}
}
]
}

174
testSchemaFiles/session_record_common_server_ip.json Normal file
View File

@@ -0,0 +1,174 @@
{
"type":"record",
"name":"session_record_common_server_ip",
"namespace":"tsg_galaxy_v3",
"doc":
{
"primary_key":"common_log_id",
"partition_key":"common_recv_time",
"ttl":null,
"default_ttl":2592000,
"index_key":
[
"common_server_ip",
"common_client_ip",
"common_recv_time"
]
},
"fields":
[
{
"name":"common_log_id",
"type":"long",
"doc":
{
"visibility":"enabled",
"ttl":null
}
},
{
"name":"common_recv_time",
"type":"long",
"doc":
{
"visibility":"enabled",
"ttl":null
}
},
{
"name":"common_server_ip",
"type":"string",
"doc":
{
"visibility":"enabled",
"ttl":null
}
},
{
"name":"common_client_ip",
"type":"string",
"doc":
{
"visibility":"enabled",
"ttl":null
}
},
{
"name":"common_sled_ip",
"type":"string",
"doc":
{
"visibility":"enabled",
"ttl":null
}
},
{
"name":"common_entrance_id",
"type":"int",
"doc":
{
"visibility":"disabled",
"ttl":null
}
},
{
"name":"common_subscriber_id",
"type":"string",
"doc":
{
"visibility":"enabled",
"ttl":null
}
},
{
"name":"common_stream_trace_id",
"type":"long",
"doc":
{
"visibility":"enabled",
"ttl":null
}
},
{
"name":"common_schema_type",
"type":"string",
"doc":
{
"visibility":"enabled",
"ttl":null
}
},
{
"name":"common_client_port",
"type":"int",
"doc":
{
"visibility":"enabled",
"ttl":null
}
},
{
"name":"common_server_port",
"type":"int",
"doc":
{
"visibility":"enabled",
"ttl":null
}
},
{
"name":"common_app_label",
"type":"string",
"doc":
{
"visibility":"enabled",
"ttl":null
}
},
{
"name":"common_direction",
"type":"int",
"doc":
{
"visibility":"enabled",
"ttl":null
}
},
{
"name":"http_domain",
"type":"string",
"doc":
{
"visibility":"enabled",
"ttl":null
}
},
{
"name":"ssl_sni",
"type":"string",
"doc":
{
"visibility":"enabled",
"ttl":null
}
}
]
}

173
testSchemaFiles/session_record_http_domain.json Normal file
View File

@@ -0,0 +1,173 @@
{
"type":"record",
"name":"session_record_http_domain",
"namespace":"tsg_galaxy_v3",
"doc":
{
"primary_key":"common_log_id",
"partition_key":"common_recv_time",
"ttl":null,
"default_ttl":2592000,
"index_key":
[
"http_domain",
"common_recv_time"
]
},
"fields":
[
{
"name":"common_log_id",
"type":"long",
"doc":
{
"visibility":"enabled",
"ttl":null
}
},
{
"name":"common_recv_time",
"type":"long",
"doc":
{
"visibility":"enabled",
"ttl":null
}
},
{
"name":"common_server_ip",
"type":"string",
"doc":
{
"visibility":"enabled",
"ttl":null
}
},
{
"name":"common_client_ip",
"type":"string",
"doc":
{
"visibility":"enabled",
"ttl":null
}
},
{
"name":"common_sled_ip",
"type":"string",
"doc":
{
"visibility":"enabled",
"ttl":null
}
},
{
"name":"common_entrance_id",
"type":"int",
"doc":
{
"visibility":"disabled",
"ttl":null
}
},
{
"name":"common_subscriber_id",
"type":"string",
"doc":
{
"visibility":"enabled",
"ttl":null
}
},
{
"name":"common_stream_trace_id",
"type":"long",
"doc":
{
"visibility":"enabled",
"ttl":null
}
},
{
"name":"common_schema_type",
"type":"string",
"doc":
{
"visibility":"enabled",
"ttl":null
}
},
{
"name":"common_client_port",
"type":"int",
"doc":
{
"visibility":"enabled",
"ttl":null
}
},
{
"name":"common_server_port",
"type":"int",
"doc":
{
"visibility":"enabled",
"ttl":null
}
},
{
"name":"common_app_label",
"type":"string",
"doc":
{
"visibility":"enabled",
"ttl":null
}
},
{
"name":"common_direction",
"type":"int",
"doc":
{
"visibility":"enabled",
"ttl":null
}
},
{
"name":"http_domain",
"type":"string",
"doc":
{
"visibility":"enabled",
"ttl":null
}
},
{
"name":"ssl_sni",
"type":"string",
"doc":
{
"visibility":"enabled",
"ttl":null
}
}
]
}

941
testSchemaFiles/sys_packet_capture_event.json Normal file
View File

@@ -0,0 +1,941 @@
{
"type": "record",
"name": "sys_packet_capture_event",
"namespace": "tsg_galaxy_v3",
"doc": {
"primary_key": "common_log_id",
"partition_key": "common_recv_time",
"index_key": [
"common_log_id",
"common_recv_time",
"common_policy_id"
]
},
"fields": [
{
"name": "common_recv_time",
"type": "long",
"doc": {
"constraints": {
"type": "timestamp"
},
"format": {
"functions": "current_timestamp"
},
"visibility": "enabled"
},
"label": "Receive Time"
},
{
"name": "common_log_id",
"type": "long",
"doc": {
"format": {
"functions": "snowflake_id"
},
"visibility": "enabled"
},
"label": "Log ID"
},
{
"name": "common_policy_id",
"type": "long",
"doc": {
"visibility": "hidden"
},
"label": "Policy ID"
},
{
"name": "common_subscriber_id",
"type": "string",
"doc": {
"visibility": "enabled"
},
"label": "Subscriber ID"
},
{
"name": "common_imei",
"type": "string",
"doc": {
"visibility": "disabled"
},
"label": "IMEI"
},
{
"name": "common_imsi",
"type": "string",
"doc": {
"visibility": "disabled"
},
"label": "IMSI"
},
{
"name": "common_phone_number",
"type": "string",
"doc": {
"visibility": "disabled"
},
"label": "Phone Number"
},
{
"name": "common_client_ip",
"type": "string",
"doc": {
"constraints": {
"type": "ip"
},
"visibility": "enabled"
},
"label": "Client IP"
},
{
"name": "common_internal_ip",
"type": "string",
"doc": {
"constraints": {
"type": "ip"
},
"visibility": "enabled"
},
"label": "Internal IP"
},
{
"name": "common_client_port",
"type": "int",
"doc": {
"visibility": "enabled"
},
"label": "Client Port"
},
{
"name": "common_l4_protocol",
"type": "string",
"doc": {
"visibility": "enabled"
},
"label": "L4 Protocol"
},
{
"name": "common_address_type",
"type": "int",
"doc": {
"data": [
{
"code": "4",
"value": "ipv4"
},
{
"code": "6",
"value": "ipv6"
}
],
"visibility": "enabled"
},
"label": "Address Type"
},
{
"name": "common_server_ip",
"type": "string",
"doc": {
"constraints": {
"type": "ip"
},
"visibility": "enabled"
},
"label": "Server IP"
},
{
"name": "common_server_port",
"type": "int",
"doc": {
"visibility": "enabled"
},
"label": "Server Port"
},
{
"name": "common_external_ip",
"type": "string",
"doc": {
"constraints": {
"type": "ip"
},
"visibility": "enabled"
},
"label": "External IP"
},
{
"name": "common_action",
"type": "int",
"doc": {
"data": [
{
"code": "0",
"value": "None"
},
{
"code": "1",
"value": "Monitor"
},
{
"code": "2",
"value": "Intercept"
},
{
"code": "16",
"value": "Deny"
},
{
"code": "128",
"value": "Allow"
}
],
"visibility": "enabled"
},
"label": "Action"
},
{
"name": "common_direction",
"type": "int",
"doc": {
"data": [
{
"code": "69",
"value": "outbound"
},
{
"code": "73",
"value": "inbound"
}
],
"visibility": "enabled"
},
"label": "Direction"
},
{
"name": "common_entrance_id",
"type": "int",
"doc": {
"visibility": "disabled"
},
"label": "Entrance ID"
},
{
"name": "common_sled_ip",
"type": "string",
"doc": {
"constraints": {
"type": "ip"
},
"visibility": "enabled"
},
"label": "Sled IP"
},
{
"name": "common_client_location",
"type": "string",
"doc": {
"visibility": "enabled"
},
"label": "Client Location"
},
{
"name": "common_client_asn",
"type": "string",
"doc": {
"visibility": "enabled"
},
"label": "Client ASN"
},
{
"name": "common_server_location",
"type": "string",
"doc": {
"visibility": "enabled"
},
"label": "Server Location"
},
{
"name": "common_server_asn",
"type": "string",
"doc": {
"visibility": "enabled"
},
"label": "Server ASN"
},
{
"name": "common_sessions",
"type": "long",
"doc": {
"visibility": "enabled"
},
"label": "Sessions"
},
{
"name": "common_c2s_pkt_num",
"type": "long",
"doc": {
"visibility": "enabled"
},
"label": "Packets Sent"
},
{
"name": "common_s2c_pkt_num",
"type": "long",
"doc": {
"visibility": "enabled"
},
"label": "Packets Received"
},
{
"name": "common_c2s_byte_num",
"type": "long",
"doc": {
"visibility": "enabled"
},
"label": "Bytes Sent"
},
{
"name": "common_s2c_byte_num",
"type": "long",
"doc": {
"visibility": "enabled"
},
"label": "Bytes Received"
},
{
"name": "common_c2s_pkt_diff",
"label": "Packets Sent (Delta)",
"doc": {
"visibility": "hidden"
},
"type": "long"
},
{
"name": "common_s2c_pkt_diff",
"label": "Packets Received (Delta)",
"doc": {
"visibility": "hidden"
},
"type": "long"
},
{
"name": "common_c2s_byte_diff",
"label": "Bytes Sent (Delta)",
"doc": {
"visibility": "hidden"
},
"type": "long"
},
{
"name": "common_s2c_byte_diff",
"label": "Bytes Received (Delta)",
"doc": {
"visibility": "hidden"
},
"type": "long"
},
{
"name": "common_service",
"type": "int",
"doc": {
"visibility": "disabled"
},
"label": "Service"
},
{
"name": "common_schema_type",
"type": "string",
"doc": {
"data": [
{
"code": "BASE",
"value": "BASE"
},
{
"code": "HTTP",
"value": "HTTP"
},
{
"code": "MAIL",
"value": "MAIL"
},
{
"code": "DNS",
"value": "DNS"
},
{
"code": "SSL",
"value": "SSL"
},
{
"code": "FTP",
"value": "FTP"
}
],
"visibility": "hidden"
},
"label": "Schema Type"
},
{
"name": "common_user_tags",
"type": "string",
"doc": {
"visibility": "disabled"
},
"label": "User Tags"
},
{
"name": "common_sub_action",
"type": "string",
"doc": {
"data": [
{
"code": "allow",
"value": "Allow"
},
{
"code": "deny",
"value": "Deny"
},
{
"code": "monitor",
"value": "Monitor"
},
{
"code": "replace",
"value": "Replace"
},
{
"code": "redirect",
"value": "Redirect"
},
{
"code": "insert",
"value": "Insert"
},
{
"code": "hijack",
"value": "Hijack"
}
],
"visibility": "hidden"
},
"label": "Sub Action"
},
{
"name": "common_user_region",
"type": "string",
"doc": {
"visibility": "hidden"
},
"label": "User Region"
},
{
"name": "common_device_id",
"type": "string",
"doc": {
"visibility": "enabled"
},
"label": "Device ID"
},
{
"name": "common_egress_link_id",
"label": "Egress Link ID",
"doc": {
"visibility": "hidden"
},
"type": "int"
},
{
"name": "common_ingress_link_id",
"label": "Ingress Link ID",
"doc": {
"visibility": "hidden"
},
"type": "int"
},
{
"name": "common_isp",
"type": "string",
"doc": {
"visibility": "disabled"
},
"label": "ISP"
},
{
"name": "common_device_tag",
"type": "string",
"doc": {
"visibility": "hidden",
"format": {
"functions": "flattenSpec,flattenSpec",
"appendTo": "common_data_center,common_device_group",
"param": "$.tags[?(@.tag=='data_center')].value,$.tags[?(@.tag=='device_group')].value"
}
},
"label": "Device Tag"
},
{
"name": "common_data_center",
"label": "Data Center",
"doc": {
"constraints": {
"operator_functions": "=,!="
},
"data": {
"$ref": "device_tag.json#",
"key": "$[?(@.tagType=='data_center')].subTags.[?(@.tagType=='data_center')]['tagValue']",
"value": "$[?(@.tagType=='data_center')].subTags.[?(@.tagType=='data_center')]['tagName']"
},
"visibility": "enabled"
},
"type": "string"
},
{
"name": "common_device_group",
"label": "Device Group",
"doc": {
"constraints": {
"operator_functions": "=,!="
},
"data": {
"$ref": "device_tag.json#",
"key": "$[?(@.tagType=='device_group')].subTags.[?(@.tagType=='device_group')]['tagValue']",
"value": "$[?(@.tagType=='device_group')].subTags.[?(@.tagType=='device_group')]['tagName']"
},
"visibility": "enabled"
},
"type": "string"
},
{
"name": "common_app_behavior",
"label": "Application Behavior",
"doc": {
"visibility": "hidden"
},
"type": "string"
},
{
"name": "common_encapsulation",
"type": "int",
"doc": {
"data": [
{
"code": "0",
"value": "Ethernet"
},
{
"code": "8",
"value": "PPP"
},
{
"code": "12",
"value": "CiscoHDLC"
}
],
"visibility": "enabled"
},
"label": "Encapsulation"
},
{
"name": "common_app_label",
"type": "string",
"doc": {
"visibility": "disabled"
},
"label": "Application Label"
},
{
"name": "common_tunnels",
"type": "string",
"doc": {
"visibility": "hidden"
},
"label": "Tunnels"
},
{
"name": "common_protocol_label",
"type": "string",
"doc": {
"visibility": "hidden"
},
"label": "Protocol Label"
},
{
"name": "common_app_id",
"type": "string",
"label": "Application ID",
"doc": {
"visibility": "hidden"
}
},
{
"name": "common_userdefine_app_name",
"label": "User Define App Name",
"type": "string",
"doc": {
"visibility": "hidden"
}
},
{
"name": "common_app_identify_info",
"label": "App Identity Info",
"doc": {
"visibility": "hidden"
},
"type": "string"
},
{
"name": "common_app_surrogate_id",
"type": "string",
"label": "Surrogate ID",
"doc": {
"visibility": "hidden"
}
},
{
"name": "common_l7_protocol",
"type": "string",
"doc": {
"visibility": "enabled"
},
"label": "L7 Protocol"
},
{
"name": "common_service_category",
"label": "FQDN Category",
"doc": {
"visibility": "enabled"
},
"type": {
"type": "array",
"items": "int"
}
},
{
"name": "common_start_time",
"type": "long",
"doc": {
"constraints": {
"type": "timestamp"
},
"visibility": "hidden"
},
"label": "Start Time"
},
{
"name": "common_end_time",
"type": "long",
"doc": {
"constraints": {
"type": "timestamp"
},
"visibility": "hidden"
},
"label": "End Time"
},
{
"name": "common_establish_latency_ms",
"type": "long",
"doc": {
"visibility": "hidden"
},
"label": "TCP Handshake Latency (ms)"
},
{
"name": "common_con_duration_ms",
"type": "long",
"doc": {
"visibility": "hidden"
},
"label": "Duration (ms)"
},
{
"name": "common_stream_dir",
"type": "int",
"doc": {
"data": [
{
"code": "1",
"value": "c2s"
},
{
"code": "2",
"value": "s2c"
},
{
"code": "3",
"value": "double"
}
],
"visibility": "enabled"
},
"label": "Stream Direction"
},
{
"name": "common_address_list",
"type": "string",
"doc": {
"visibility": "disabled"
},
"label": "Address List"
},
{
"name": "common_has_dup_traffic",
"type": "int",
"doc": {
"data": [
{
"code": "0",
"value": "No"
},
{
"code": "1",
"value": "Yes"
}
],
"visibility": "hidden"
},
"label": "Duplication Traffic"
},
{
"name": "common_stream_error",
"type": "string",
"doc": {
"visibility": "hidden"
},
"label": "Stream Error"
},
{
"name": "common_stream_trace_id",
"type": "long",
"doc": {
"visibility": "enabled"
},
"label": "Session ID"
},
{
"name": "common_link_info_c2s",
"type": "string",
"doc": {
"visibility": "hidden"
},
"label": "Link Info (c2s)"
},
{
"name": "common_link_info_s2c",
"type": "string",
"doc": {
"visibility": "hidden"
},
"label": "Link Info (s2c)"
},
{
"name": "common_packet_capture_file",
"label": "Packet Capture File",
"doc": {
"visibility": "hidden",
"constraints": {
"type": "file"
}
},
"type": "string"
},
{
"name": "common_c2s_ipfrag_num",
"type": "long",
"doc": {
"visibility": "hidden"
},
"label": "Fragmentation Packets (c2s)"
},
{
"name": "common_s2c_ipfrag_num",
"type": "long",
"doc": {
"visibility": "hidden"
},
"label": "Fragmentation Packets (s2c)"
},
{
"name": "common_c2s_tcp_lostlen",
"type": "long",
"doc": {
"visibility": "hidden"
},
"label": "Sequence Gap Loss (c2s)"
},
{
"name": "common_s2c_tcp_lostlen",
"type": "long",
"doc": {
"visibility": "hidden"
},
"label": "Sequence Gap Loss (s2c)"
},
{
"name": "common_c2s_tcp_unorder_num",
"type": "long",
"doc": {
"visibility": "hidden"
},
"label": "Unordered Packets (c2s)"
},
{
"name": "common_s2c_tcp_unorder_num",
"type": "long",
"doc": {
"visibility": "hidden"
},
"label": "Unordered Packets (s2c)"
},
{
"name": "common_c2s_pkt_retrans",
"type": "long",
"doc": {
"visibility": "enabled"
},
"label": "Packet Retransmission (c2s)"
},
{
"name": "common_s2c_pkt_retrans",
"type": "long",
"doc": {
"visibility": "enabled"
},
"label": "Packet Retransmission (s2c)"
},
{
"name": "common_c2s_byte_retrans",
"type": "long",
"doc": {
"visibility": "enabled"
},
"label": "Byte Retransmission (c2s)"
},
{
"name": "common_s2c_byte_retrans",
"type": "long",
"doc": {
"visibility": "enabled"
},
"label": "Byte Retransmission (s2c)"
},
{
"name": "common_tcp_client_isn",
"label": "TCP Client ISN",
"doc": {
"visibility": "disabled"
},
"type": "long"
},
{
"name": "common_tcp_server_isn",
"label": "TCP Server ISN",
"doc": {
"visibility": "disabled"
},
"type": "long"
},
{
"name": "common_first_ttl",
"type": "int",
"doc": {
"visibility": "hidden"
},
"label": "First TTL"
},
{
"name": "common_processing_time",
"type": "long",
"doc": {
"constraints": {
"type": "timestamp"
},
"format": {
"functions": "current_timestamp"
},
"visibility": "enabled"
},
"label": "Processing Time"
},
{
"name": "common_ingestion_time",
"label": "Ingestion Time",
"doc": {
"constraints": {
"type": "timestamp"
},
"format": {
"functions": "ingestion_time"
},
"visibility": "enabled"
},
"type": "long"
},
{
"name": "common_mirrored_pkts",
"label": "Mirrored Packets",
"type": "long",
"doc": {
"visibility": "hidden"
}
},
{
"name": "common_mirrored_bytes",
"label": "Mirrored Bytes",
"type": "long",
"doc": {
"visibility": "hidden"
}
},
{
"name": "nic_name",
"type": "string",
"doc": {
"visibility": "enabled"
},
"label": "Nic Name"
},
{
"name": "origin_source_mac",
"type": "string",
"doc": {
"visibility": "enabled"
},
"label": "Origin Source Mac"
},
{
"name": "origin_dest_mac",
"type": "string",
"doc": {
"visibility": "enabled"
},
"label": "Origin Dest Mac"
},
{
"name": "packet_url",
"type": "string",
"doc": {
"visibility": "enabled"
},
"label": "Packet URL"
},
{
"name": "pcap_storage_task_id",
"type": "int",
"doc": {
"visibility": "enabled"
},
"label": "Task ID"
},
{
"name": "pcap_storage_duration",
"type": "int",
"doc": {
"visibility": "enabled"
},
"label": "Duration"
}
]
}

88
testSchemaFiles/sys_storage_log.json Normal file
View File

@@ -0,0 +1,88 @@
{
"type": "record",
"name": "sys_storage_log",
"namespace": "druid",
"doc": {
"partition_key": "__time",
"functions": {
"$ref": "public_schema_info.json#/functions"
},
"schema_query": {
"filters": [
"data_center"
],
"references": {
"$ref": "public_schema_info.json#/schema_query/references"
}
}
},
"fields": [
{
"name": "__time",
"label": "Time",
"type": "string",
"doc": {
"constraints": {
"type": "timestamp"
},
"visibility": "enabled"
}
},
{
"name": "data_center",
"label": "Data Center",
"type": "string",
"doc": {
"constraints": {
"operator_functions": "=,in"
},
"data": {
"$ref": "device_tag.json#",
"key": "$[?(@.tagType=='data_center')].subTags.[?(@.tagType=='data_center')]['tagValue']",
"value": "$[?(@.tagType=='data_center')].subTags.[?(@.tagType=='data_center')]['tagName']"
},
"visibility": "enabled"
}
},
{
"name": "log_type",
"label": "Log Type",
"doc": {
"visibility": "enabled"
},
"type": "string"
},
{
"name": "max_size",
"label": "Max Size",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "used_size",
"label": "Used Size",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "aggregate_size",
"label": "Aggregate Size",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "last_storage",
"label": "Last Storage",
"doc": {
"visibility": "enabled"
},
"type": "long"
}
]
}

11
testSchemaFiles/tables.json Normal file
View File

@@ -0,0 +1,11 @@
{
"namespace": "system",
"type": "record",
"name": "tables",
"fields": [
{
"name": "name",
"type": "string"
}
]
}

11
testSchemaFiles/tables_cluster.json Normal file
View File

@@ -0,0 +1,11 @@
{
"namespace": "system",
"type": "record",
"name": "tables_cluster",
"fields": [
{
"name": "database",
"type": "string"
}
]
}

117
testSchemaFiles/top_client_ip_log.json Normal file
View File

@@ -0,0 +1,117 @@
{
"type": "record",
"name": "top_client_ip_log",
"namespace": "druid",
"doc": {
"partition_key": "__time",
"functions": {
"$ref": "public_schema_info.json#/functions"
},
"schema_query": {
"references": {
"$ref": "public_schema_info.json#/schema_query/references"
}
}
},
"fields": [
{
"name": "__time",
"label": "Time",
"type": "string",
"doc": {
"constraints": {
"type": "timestamp"
},
"visibility": "enabled"
}
},
{
"name": "data_center",
"label": "Data Center",
"type": "string",
"doc": {
"constraints": {
"operator_functions": "=,in"
},
"data": {
"$ref": "device_tag.json#",
"key": "$[?(@.tagType=='data_center')].subTags.[?(@.tagType=='data_center')]['tagValue']",
"value": "$[?(@.tagType=='data_center')].subTags.[?(@.tagType=='data_center')]['tagName']"
},
"visibility": "enabled"
}
},
{
"name": "device_group",
"label": "Device Group",
"type": "string",
"doc": {
"constraints": {
"operator_functions": "=,in"
},
"data": {
"$ref": "device_tag.json#",
"key": "$[?(@.tagType=='device_group')].subTags.[?(@.tagType=='device_group')]['tagValue']",
"value": "$[?(@.tagType=='device_group')].subTags.[?(@.tagType=='device_group')]['tagName']"
},
"visibility": "enabled"
}
},
{
"name": "source",
"label": "Client IP",
"doc": {
"visibility": "enabled"
},
"type": "string"
},
{
"name": "session_num",
"label": "Sessions",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "c2s_byte_num",
"label": "Bytes Sent",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "s2c_byte_num",
"label": "Bytes Received",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "c2s_pkt_num",
"label": "Packets Sent",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "s2c_pkt_num",
"label": "Packets Received",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "order_by",
"label": "Order By",
"doc": {
"visibility": "enabled"
},
"type": "string"
}
]
}

117
testSchemaFiles/top_external_host_log.json Normal file
View File

@@ -0,0 +1,117 @@
{
"type": "record",
"name": "top_external_host_log",
"namespace": "druid",
"doc": {
"partition_key": "__time",
"functions": {
"$ref": "public_schema_info.json#/functions"
},
"schema_query": {
"references": {
"$ref": "public_schema_info.json#/schema_query/references"
}
}
},
"fields": [
{
"name": "__time",
"label": "Time",
"type": "string",
"doc": {
"constraints": {
"type": "timestamp"
},
"visibility": "enabled"
}
},
{
"name": "data_center",
"label": "Data Center",
"type": "string",
"doc": {
"constraints": {
"operator_functions": "=,in"
},
"data": {
"$ref": "device_tag.json#",
"key": "$[?(@.tagType=='data_center')].subTags.[?(@.tagType=='data_center')]['tagValue']",
"value": "$[?(@.tagType=='data_center')].subTags.[?(@.tagType=='data_center')]['tagName']"
},
"visibility": "enabled"
}
},
{
"name": "device_group",
"label": "Device Group",
"type": "string",
"doc": {
"constraints": {
"operator_functions": "=,in"
},
"data": {
"$ref": "device_tag.json#",
"key": "$[?(@.tagType=='device_group')].subTags.[?(@.tagType=='device_group')]['tagValue']",
"value": "$[?(@.tagType=='device_group')].subTags.[?(@.tagType=='device_group')]['tagName']"
},
"visibility": "enabled"
}
},
{
"name": "destination",
"label": "External IP",
"doc": {
"visibility": "enabled"
},
"type": "string"
},
{
"name": "session_num",
"label": "Sessions",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "c2s_byte_num",
"label": "Bytes Sent",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "s2c_byte_num",
"label": "Bytes Received",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "c2s_pkt_num",
"label": "Packets Sent",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "s2c_pkt_num",
"label": "Packets Received",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "order_by",
"label": "Order By",
"doc": {
"visibility": "enabled"
},
"type": "string"
}
]
}

117
testSchemaFiles/top_internal_host_log.json Normal file
View File

@@ -0,0 +1,117 @@
{
"type": "record",
"name": "top_internal_host_log",
"namespace": "druid",
"doc": {
"partition_key": "__time",
"functions": {
"$ref": "public_schema_info.json#/functions"
},
"schema_query": {
"references": {
"$ref": "public_schema_info.json#/schema_query/references"
}
}
},
"fields": [
{
"name": "__time",
"label": "Time",
"type": "string",
"doc": {
"constraints": {
"type": "timestamp"
},
"visibility": "enabled"
}
},
{
"name": "data_center",
"label": "Data Center",
"type": "string",
"doc": {
"constraints": {
"operator_functions": "=,in"
},
"data": {
"$ref": "device_tag.json#",
"key": "$[?(@.tagType=='data_center')].subTags.[?(@.tagType=='data_center')]['tagValue']",
"value": "$[?(@.tagType=='data_center')].subTags.[?(@.tagType=='data_center')]['tagName']"
},
"visibility": "enabled"
}
},
{
"name": "device_group",
"label": "Device Group",
"type": "string",
"doc": {
"constraints": {
"operator_functions": "=,in"
},
"data": {
"$ref": "device_tag.json#",
"key": "$[?(@.tagType=='device_group')].subTags.[?(@.tagType=='device_group')]['tagValue']",
"value": "$[?(@.tagType=='device_group')].subTags.[?(@.tagType=='device_group')]['tagName']"
},
"visibility": "enabled"
}
},
{
"name": "source",
"label": "Internal IP",
"doc": {
"visibility": "enabled"
},
"type": "string"
},
{
"name": "session_num",
"label": "Sessions",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "c2s_byte_num",
"label": "Bytes Sent",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "s2c_byte_num",
"label": "Bytes Received",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "c2s_pkt_num",
"label": "Packets Sent",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "s2c_pkt_num",
"label": "Packets Received",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "order_by",
"label": "Order By",
"doc": {
"visibility": "enabled"
},
"type": "string"
}
]
}

117
testSchemaFiles/top_server_ip_log.json Normal file
View File

@@ -0,0 +1,117 @@
{
"type": "record",
"name": "top_server_ip_log",
"namespace": "druid",
"doc": {
"partition_key": "__time",
"functions": {
"$ref": "public_schema_info.json#/functions"
},
"schema_query": {
"references": {
"$ref": "public_schema_info.json#/schema_query/references"
}
}
},
"fields": [
{
"name": "__time",
"label": "Time",
"type": "string",
"doc": {
"constraints": {
"type": "timestamp"
},
"visibility": "enabled"
}
},
{
"name": "data_center",
"label": "Data Center",
"type": "string",
"doc": {
"constraints": {
"operator_functions": "=,in"
},
"data": {
"$ref": "device_tag.json#",
"key": "$[?(@.tagType=='data_center')].subTags.[?(@.tagType=='data_center')]['tagValue']",
"value": "$[?(@.tagType=='data_center')].subTags.[?(@.tagType=='data_center')]['tagName']"
},
"visibility": "enabled"
}
},
{
"name": "device_group",
"label": "Device Group",
"type": "string",
"doc": {
"constraints": {
"operator_functions": "=,in"
},
"data": {
"$ref": "device_tag.json#",
"key": "$[?(@.tagType=='device_group')].subTags.[?(@.tagType=='device_group')]['tagValue']",
"value": "$[?(@.tagType=='device_group')].subTags.[?(@.tagType=='device_group')]['tagName']"
},
"visibility": "enabled"
}
},
{
"name": "destination",
"label": "Server IP",
"doc": {
"visibility": "enabled"
},
"type": "string"
},
{
"name": "session_num",
"label": "Sessions",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "c2s_byte_num",
"label": "Bytes Sent",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "s2c_byte_num",
"label": "Bytes Received",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "c2s_pkt_num",
"label": "Packets Sent",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "s2c_pkt_num",
"label": "Packets Received",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "order_by",
"label": "Order By",
"doc": {
"visibility": "enabled"
},
"type": "string"
}
]
}

37
testSchemaFiles/top_urls_log.json Normal file
View File

@@ -0,0 +1,37 @@
{
"type": "record",
"name": "top_urls_log",
"namespace": "druid",
"doc": {
"partition_key": "__time"
},
"fields": [
{
"name": "__time",
"label": "Time",
"type": "string",
"doc": {
"constraints": {
"type": "timestamp"
},
"visibility": "enabled"
}
},
{
"name": "url",
"label": "URL",
"doc": {
"visibility": "enabled"
},
"type": "string"
},
{
"name": "session_num",
"label": "Sessions",
"doc": {
"visibility": "enabled"
},
"type": "long"
}
]
}

117
testSchemaFiles/top_user_log.json Normal file
View File

@@ -0,0 +1,117 @@
{
"type": "record",
"name": "top_user_log",
"namespace": "druid",
"doc": {
"partition_key": "__time",
"functions": {
"$ref": "public_schema_info.json#/functions"
},
"schema_query": {
"references": {
"$ref": "public_schema_info.json#/schema_query/references"
}
}
},
"fields": [
{
"name": "__time",
"label": "Time",
"type": "string",
"doc": {
"constraints": {
"type": "timestamp"
},
"visibility": "enabled"
}
},
{
"name": "data_center",
"label": "Data Center",
"type": "string",
"doc": {
"constraints": {
"operator_functions": "=,in"
},
"data": {
"$ref": "device_tag.json#",
"key": "$[?(@.tagType=='data_center')].subTags.[?(@.tagType=='data_center')]['tagValue']",
"value": "$[?(@.tagType=='data_center')].subTags.[?(@.tagType=='data_center')]['tagName']"
},
"visibility": "enabled"
}
},
{
"name": "device_group",
"label": "Device Group",
"type": "string",
"doc": {
"constraints": {
"operator_functions": "=,in"
},
"data": {
"$ref": "device_tag.json#",
"key": "$[?(@.tagType=='device_group')].subTags.[?(@.tagType=='device_group')]['tagValue']",
"value": "$[?(@.tagType=='device_group')].subTags.[?(@.tagType=='device_group')]['tagName']"
},
"visibility": "enabled"
}
},
{
"name": "subscriber_id",
"label": "Subscriber ID",
"doc": {
"visibility": "enabled"
},
"type": "string"
},
{
"name": "session_num",
"label": "Sessions",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "c2s_byte_num",
"label": "Bytes Sent",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "s2c_byte_num",
"label": "Bytes Received",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "c2s_pkt_num",
"label": "Packets Sent",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "s2c_pkt_num",
"label": "Packets Received",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "order_by",
"label": "Order By",
"doc": {
"visibility": "enabled"
},
"type": "string"
}
]
}

117
testSchemaFiles/top_website_domain_log.json Normal file
View File

@@ -0,0 +1,117 @@
{
"type": "record",
"name": "top_website_domain_log",
"namespace": "druid",
"doc": {
"partition_key": "__time",
"functions": {
"$ref": "public_schema_info.json#/functions"
},
"schema_query": {
"references": {
"$ref": "public_schema_info.json#/schema_query/references"
}
}
},
"fields": [
{
"name": "__time",
"label": "Time",
"type": "string",
"doc": {
"constraints": {
"type": "timestamp"
},
"visibility": "enabled"
}
},
{
"name": "data_center",
"label": "Data Center",
"type": "string",
"doc": {
"constraints": {
"operator_functions": "=,in"
},
"data": {
"$ref": "device_tag.json#",
"key": "$[?(@.tagType=='data_center')].subTags.[?(@.tagType=='data_center')]['tagValue']",
"value": "$[?(@.tagType=='data_center')].subTags.[?(@.tagType=='data_center')]['tagName']"
},
"visibility": "enabled"
}
},
{
"name": "device_group",
"label": "Device Group",
"type": "string",
"doc": {
"constraints": {
"operator_functions": "=,in"
},
"data": {
"$ref": "device_tag.json#",
"key": "$[?(@.tagType=='device_group')].subTags.[?(@.tagType=='device_group')]['tagValue']",
"value": "$[?(@.tagType=='device_group')].subTags.[?(@.tagType=='device_group')]['tagName']"
},
"visibility": "enabled"
}
},
{
"name": "domain",
"label": "Domain",
"doc": {
"visibility": "enabled"
},
"type": "string"
},
{
"name": "session_num",
"label": "Sessions",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "c2s_byte_num",
"label": "Bytes Sent",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "s2c_byte_num",
"label": "Bytes Received",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "c2s_pkt_num",
"label": "Packets Sent",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "s2c_pkt_num",
"label": "Packets Received",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "order_by",
"label": "Order By",
"doc": {
"visibility": "enabled"
},
"type": "string"
}
]
}

112
testSchemaFiles/traffic_app_stat_log.json Normal file
View File

@@ -0,0 +1,112 @@
{
"type": "record",
"name": "traffic_app_stat_log",
"namespace": "druid",
"doc": {
"partition_key": "__time",
"functions": {
"$ref": "public_schema_info.json#/functions"
},
"schema_query": {
"references": {
"$ref": "public_schema_info.json#/schema_query/references"
}
}
},
"fields": [
{
"name": "__time",
"label": "Time",
"type": "string",
"doc": {
"constraints": {
"type": "timestamp"
},
"visibility": "enabled"
}
},
{
"name": "data_center",
"label": "Data Center",
"type": "string",
"doc": {
"constraints": {
"operator_functions": "=,in"
},
"data": {
"$ref": "device_tag.json#",
"key": "$[?(@.tagType=='data_center')].subTags.[?(@.tagType=='data_center')]['tagValue']",
"value": "$[?(@.tagType=='data_center')].subTags.[?(@.tagType=='data_center')]['tagName']"
},
"visibility": "enabled"
}
},
{
"name": "device_group",
"label": "Device Group",
"type": "string",
"doc": {
"constraints": {
"operator_functions": "=,in"
},
"data": {
"$ref": "device_tag.json#",
"key": "$[?(@.tagType=='device_group')].subTags.[?(@.tagType=='device_group')]['tagValue']",
"value": "$[?(@.tagType=='device_group')].subTags.[?(@.tagType=='device_group')]['tagName']"
},
"visibility": "enabled"
}
},
{
"name": "app_name",
"label": "APP Name",
"type": "string",
"doc": {
"constraints": {
"operator_functions": "=,in"
},
"visibility": "enabled"
}
},
{
"name": "session_num",
"label": "Sessions",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "c2s_byte_num",
"label": "Bytes Sent",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "s2c_byte_num",
"label": "Bytes Received",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "c2s_pkt_num",
"label": "Packets Sent",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "s2c_pkt_num",
"label": "Packets Received",
"doc": {
"visibility": "enabled"
},
"type": "long"
}
]
}

437
testSchemaFiles/traffic_metrics_log.json Normal file
View File

@@ -0,0 +1,437 @@
{
"type": "record",
"name": "traffic_metrics_log",
"namespace": "druid",
"doc": {
"partition_key": "__time",
"functions": {
"$ref": "public_schema_info.json#/functions"
},
"schema_query": {
"references": {
"$ref": "public_schema_info.json#/schema_query/references"
}
}
},
"fields": [
{
"name": "__time",
"label": "Time",
"type": "string",
"doc": {
"constraints": {
"type": "timestamp"
},
"visibility": "enabled"
}
},
{
"name": "device_id",
"label": "Device ID",
"doc": {
"visibility": "enabled"
},
"type": "string"
},
{
"name": "entrance_id",
"label": "Entrance ID",
"type": "long",
"doc": {
"visibility": "disabled"
}
},
{
"name": "allow_conn_num",
"label": "Allow Sessions",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "allow_in_bytes",
"label": "Allow Bytes (Ingress)",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "allow_in_packets",
"label": "Allow Packets (Ingress)",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "allow_out_bytes",
"label": "Allow Bytes (Egress)",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "allow_out_packets",
"label": "Allow Packets (Egress)",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "close_conn_num",
"label": "Closed Sessions",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "default_conn_num",
"label": "Default Sessions",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "default_in_bytes",
"label": "Default Bytes (Ingress)",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "default_in_packets",
"label": "Default Packets (Ingress)",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "default_out_bytes",
"label": "Default Bytes (Egress)",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "default_out_packets",
"label": "Default Packets (Egress)",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "deny_conn_num",
"label": "Deny Sessions",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "deny_in_bytes",
"label": "Deny Bytes (Ingress)",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "deny_in_packets",
"label": "Deny Packets (Ingress)",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "deny_out_bytes",
"label": "Deny Bytes (Egress)",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "deny_out_packets",
"label": "Deny Packets (Egress)",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "intercept_conn_num",
"label": "Intercept Sessions",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "intercept_in_bytes",
"label": "Intercept Bytes (Ingress)",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "intercept_in_packets",
"label": "Intercept Packets (Ingress)",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "intercept_out_bytes",
"label": "Intercept Bytes (Egress)",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "intercept_out_packets",
"label": "Intercept Packets (Egress)",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "established_conn_num",
"label": "Established Sessions",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "monitor_conn_num",
"label": "Monitor Sessions",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "monitor_in_bytes",
"label": "Monitor Bytes (Ingress)",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "monitor_in_packets",
"label": "Monitor Packets (Ingress)",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "monitor_out_bytes",
"label": "Monitor Bytes (Egress)",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "monitor_out_packets",
"label": "Monitor Packets (Egress)",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "new_conn_num",
"label": "New Sessions",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "total_in_bytes",
"label": "Total Bytes (Ingress)",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "total_in_packets",
"label": "Total Packets (Ingress)",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "total_out_bytes",
"label": "Total Bytes (Egress)",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "total_out_packets",
"label": "Total Packets (Egress)",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "alert_bytes",
"label": "Alert Bytes",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "hijk_bytes",
"label": "Hijack Bytes",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "ins_bytes",
"label": "Insert Bytes",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "intcp_allow_num",
"label": "Intercept Allow Sessions",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "intcp_deny_num",
"label": "Intercept Deny Sessions",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "intcp_hijk_num",
"label": "Intercept Hijack Sessions",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "intcp_ins_num",
"label": "Intercept Insert Sessions",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "intcp_mon_num",
"label": "Intercept Monitor Sessions",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "intcp_rdirt_num",
"label": "Intercept Redirect Sessions",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "intcp_repl_num",
"label": "Intercept Replace Sessions",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "maybe_pinning_num",
"label": "Maybe Pinning Sessions",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "not_pinning_num",
"label": "Not Pinning Sessions",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "pinning_num",
"label": "Pinning Sessions",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "ad_cc_bytes",
"label": "AD CC Bytes",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "ad_flood_bytes",
"label": "AD Flood Bytes",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "ad_reflection_bytes",
"label": "AD Reflection Bytes",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "intcp_edit_elem_num",
"label": "Intercept Edit Element Sessions",
"doc": {
"visibility": "enabled"
},
"type": "long"
}
]
}

177
testSchemaFiles/traffic_protocol_stat_log.json Normal file
View File

@@ -0,0 +1,177 @@
{
"type": "record",
"name": "traffic_protocol_stat_log",
"namespace": "druid",
"doc": {
"partition_key": "__time",
"functions": {
"$ref": "public_schema_info.json#/functions"
},
"schema_query": {
"filters": [
"data_center",
"device_group"
],
"references": {
"$ref": "public_schema_info.json#/schema_query/references"
}
}
},
"fields": [
{
"name": "__time",
"label": "Time",
"type": "string",
"doc": {
"constraints": {
"type": "timestamp"
},
"visibility": "enabled"
}
},
{
"name": "protocol_id",
"label": "Protocol ID",
"doc": {
"visibility": "enabled"
},
"type": "string"
},
{
"name": "isp",
"label": "ISP",
"doc": {
"visibility": "enabled"
},
"type": "string"
},
{
"name": "entrance_id",
"label": "Entrance ID",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "data_center",
"label": "Data Center",
"type": "string",
"doc": {
"constraints": {
"operator_functions": "=,!="
},
"data": {
"$ref": "device_tag.json#",
"key": "$[?(@.tagType=='data_center')].subTags.[?(@.tagType=='data_center')]['tagValue']",
"value": "$[?(@.tagType=='data_center')].subTags.[?(@.tagType=='data_center')]['tagName']"
},
"visibility": "enabled"
}
},
{
"name": "device_group",
"label": "Device Group",
"type": "string",
"doc": {
"constraints": {
"operator_functions": "=,!="
},
"data": {
"$ref": "device_tag.json#",
"key": "$[?(@.tagType=='device_group')].subTags.[?(@.tagType=='device_group')]['tagValue']",
"value": "$[?(@.tagType=='device_group')].subTags.[?(@.tagType=='device_group')]['tagName']"
},
"visibility": "enabled"
}
},
{
"name": "sessions",
"label": "Sessions",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "c2s_pkt_num",
"label": "Packets Sent",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "s2c_pkt_num",
"label": "Packets Received",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "c2s_byte_num",
"label": "Bytes Sent",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "s2c_byte_num",
"label": "Bytes Received",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "c2s_ipfrag_num",
"label": "Fragmentation Packets (c2s)",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "s2c_ipfrag_num",
"label": "Fragmentation Packets (s2c)",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "c2s_tcp_lostlen",
"label": "Sequence Gap Loss (c2s)",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "s2c_tcp_lostlen",
"label": "Sequence Gap Loss (s2c)",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "c2s_tcp_unorder_num",
"label": "Unordered Packets (c2s)",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "s2c_tcp_unorder_num",
"label": "Unordered Packets (s2c)",
"doc": {
"visibility": "enabled"
},
"type": "long"
}
]
}

211
testSchemaFiles/traffic_summary_log.json Normal file
View File

@@ -0,0 +1,211 @@
{
"type": "record",
"name": "traffic_summary_log",
"namespace": "druid",
"doc": {
"partition_key": "__time",
"functions": {
"$ref": "public_schema_info.json#/functions"
},
"schema_query": {
"references": {
"$ref": "public_schema_info.json#/schema_query/references"
}
}
},
"fields": [
{
"name": "__time",
"label": "Time",
"type": "string",
"doc": {
"constraints": {
"type": "timestamp"
},
"visibility": "enabled"
}
},
{
"name": "data_center",
"label": "Data Center",
"type": "string",
"doc": {
"constraints": {
"operator_functions": "=,in"
},
"data": {
"$ref": "device_tag.json#",
"key": "$[?(@.tagType=='data_center')].subTags.[?(@.tagType=='data_center')]['tagValue']",
"value": "$[?(@.tagType=='data_center')].subTags.[?(@.tagType=='data_center')]['tagName']"
},
"visibility": "enabled"
}
},
{
"name": "device_group",
"label": "Device Group",
"type": "string",
"doc": {
"constraints": {
"operator_functions": "=,in"
},
"data": {
"$ref": "device_tag.json#",
"key": "$[?(@.tagType=='device_group')].subTags.[?(@.tagType=='device_group')]['tagValue']",
"value": "$[?(@.tagType=='device_group')].subTags.[?(@.tagType=='device_group')]['tagName']"
},
"visibility": "enabled"
}
},
{
"name": "isp",
"label": "ISP",
"type": "string",
"doc": {
"visibility": "disabled"
}
},
{
"name": "entrance_id",
"label": "Entrance ID",
"type": "long",
"doc": {
"visibility": "disabled"
}
},
{
"name": "schema_type",
"label": "Schema Type",
"type": "string",
"doc": {
"data": [
{
"code": "BASE",
"value": "BASE"
},
{
"code": "MAIL",
"value": "MAIL"
},
{
"code": "DNS",
"value": "DNS"
},
{
"code": "HTTP",
"value": "HTTP"
},
{
"code": "SSL",
"value": "SSL"
},
{
"code": "QUIC",
"value": "QUIC"
},
{
"code": "FTP",
"value": "FTP"
},
{
"code": "SSH",
"value": "SSH"
},
{
"code": "Stratum",
"value": "Stratum"
}
],
"visibility": "enabled"
}
},
{
"name": "ip_object",
"label": "IP Object",
"doc": {
"visibility": "enabled"
},
"type": "string"
},
{
"name": "sessions",
"label": "Sessions",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "c2s_byte_num",
"label": "Bytes Sent",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "s2c_byte_num",
"label": "Bytes Received",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "c2s_pkt_num",
"label": "Packets Sent",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "s2c_pkt_num",
"label": "Packets Received",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "one_sided_connections",
"label": "One Sided Connections",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "uncategorized_bytes",
"label": "Uncategorized Bytes",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "fragmentation_packets",
"label": "Fragmentation Packets",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "sequence_gap_loss",
"label": "Sequence Gap Loss",
"doc": {
"visibility": "enabled"
},
"type": "long"
},
{
"name": "unorder_packets",
"label": "Unorder Packets",
"doc": {
"visibility": "enabled"
},
"type": "long"
}
]
}

113
testSchemaFiles/traffic_top_destination_ip_metrics_log.json Normal file
View File

@@ -0,0 +1,113 @@
{
"type": "record",
"name": "traffic_top_destination_ip_metrics_log",
"namespace": "druid",
"doc": {
"partition_key": "__time",
"functions": {
"$ref": "public_schema_info.json#/functions"
},
"schema_query": {
"filters": [
"common_data_center"
],
"references": {
"$ref": "public_schema_info.json#/schema_query/references"
}
}
},
"fields": [
{
"name": "__time",
"label": "Time",
"type": "string",
"doc": {
"constraints": {
"type": "timestamp"
},
"visibility": "enabled"
}
},
{
"name": "common_data_center",
"label": "Data Center",
"type": "string",
"doc": {
"constraints": {
"operator_functions": "=,in"
},
"data": {
"$ref": "device_tag.json#",
"key": "$[?(@.tagType=='data_center')].subTags.[?(@.tagType=='data_center')]['tagValue']",
"value": "$[?(@.tagType=='data_center')].subTags.[?(@.tagType=='data_center')]['tagName']"
},
"visibility": "enabled"
}
},
{
"name": "common_sled_ip",
"label": "Sled IP",
"doc": {
"visibility": "enabled"
},
"type": "string"
},
{
"name": "destination_ip",
"label": "Destination IP",
"doc": {
"visibility": "enabled"
},
"type": "string"
},
{
"name": "attack_type",
"label": "Attack type",
"doc": {
"visibility": "enabled"
},
"type": "string"
},
{
"name": "session_rate",
"label": "Sessions/s",
"type": "long",
"doc": {
"constraints": {
"type": "sessions/sec"
},
"visibility": "enabled"
}
},
{
"name": "packet_rate",
"label": "Packets/s",
"type": "long",
"doc": {
"constraints": {
"type": "packets/sec"
},
"visibility": "enabled"
}
},
{
"name": "bit_rate",
"label": "Bits/s",
"type": "long",
"doc": {
"constraints": {
"type": "bits/sec"
},
"visibility": "enabled"
}
},
{
"name": "partition_num",
"label": "Partition Num",
"doc": {
"visibility": "enabled"
},
"type": "long"
}
]
}

2551
testSchemaFiles/transaction_record.json Normal file
View File

File diff suppressed because it is too large Load Diff

186
testSchemaFiles/version.json Normal file
View File

@@ -0,0 +1,186 @@
{
"product": "Galaxy Cluster install package",
"version": "22.06",
"registered": "Geedge",
"updated": "2022-06-30 12:00:00",
"components": {
"oss": [
{
"name": "zookeeper",
"version": "3.4.10",
"licenseType": "Apache License 2.0",
"description": "分布式应用程序协调服务"
},
{
"name": "kafka",
"version": "1.0.0",
"licenseType": "Apache License 2.0",
"description": "消息队列"
},
{
"name": "habse",
"version": "2.2.3",
"licenseType": "Apache License 2.0",
"description": "用于文件系统和存储Radius数据"
},
{
"name": "flink",
"version": "1.13.1",
"licenseType": "Apache License 2.0",
"description": "流数据计算框架用于日志预处理及部分统计"
},
{
"name": "clickhouse",
"version": "21.8.13.1.altinitystable",
"licenseType": "Apache License 2.0",
"description": "原始日志数据库"
},
{
"name": "druid",
"version": "0.18.1",
"licenseType": "Apache License 2.0",
"description": "分析实时数据并提供低延迟查询的OLAP应用程序"
},
{
"name": "gohangout",
"version": "1.15.2.20220117",
"description": "动态获取原始日志表schema入库程序"
},
{
"name": "nacos",
"version": "2.0.2",
"licenseType": "Apache License 2.0",
"description": "分布式配置中心"
},
{
"name": "mariadb",
"version": "10.5.3",
"licenseType": "Apache License 2.0",
"description": "传统数据库用于nacos/druid/galaxy-job-service数据存储"
},
{
"name": "arangodb",
"version": "3.6.4",
"licenseType": "Apache License 2.0",
"description": "图数据库用于存储IPlearning统计结果"
}
],
"apps": [
{
"name": "galaxy-qgw-service",
"version": "356-rc1",
"description": "数据平台对外统一查询网关"
},
{
"name": "galaxy-report-service",
"version": "22.04.11",
"description": "自定义报表查询服务"
},
{
"name": "galaxy-hos-service",
"version": "22.06.23",
"description": "对象存储服务"
},
{
"name": "galaxy-job-admin",
"version": "v1.3.220308",
"description": "分布式任务调度平台"
},
{
"name": "galaxy-job-executor",
"version": "v1.3.220623",
"description": "分布式任务调度平台-执行器"
},
{
"name": "galaxy-gateway-nginx",
"version": "1.17.0",
"description": "查询网管负载均衡器"
},
{
"name": "node-exporter",
"version": "1.2.2",
"description": "暴露服务器prometheus指标插件"
},
{
"name": "packet_dump",
"version": "v2.3.1",
"description": "DPI补包插件"
}
],
"tasks": [
{
"name": "flink",
"topology": [
{
"name": "radius-relation-22-04-01.jar",
"md5": "d66faa3aeab2ba7abe382e27928b8f17",
"description": "Radius subscriber关系更新HBase程序"
},
{
"name": "log-completion-schema-220318-Nacos.jar",
"md5": "70a6fcde9c350519ea4d92c1fa853a83",
"description": "ETL程序 用于原始日志补全及汇聚程序"
},
{
"name": "flink-dos-detection.jar",
"md5": "0aef189f1e2c4a4e014655449df714e2",
"description": "ddos威胁检测程序"
},
{
"name": "flink-sql-submit.jar",
"md5": "d6432fd6a29253c23931562d72b46ef1",
"description": "TOPN计算程序"
},
{
"name": "log-olap-analysis-schema-220323-Nacos.jar",
"md5": "51779b623cd7aa2c3e4ff322549857d6",
"description": "Livecharts计算程序"
},
{
"name": "radius-account-knowledge-220413-sink.jar",
"md5": "f47d7f490484d33d797c16d47d02d90d",
"description": "Radius上下线记录程序"
},
{
"name": "log-stream-voip-relation-220418-Nacos.jar",
"md5": "a4a12ec7c46940a3e89da4420351354f",
"description": "VOIP融合程序"
},
{
"name": "flink-app-recommend-22-01-07.jar",
"md5": "0d88ad0b3f668248009c407999bb5f32",
"description": "APP白名单学习程序"
}
]
},
{
"name": "druid",
"topology": "proxy_event_hits_log.json,security_event_hits_log.json,sys_storage_log.json,top_client_ip_log.json,top_external_host_log.json,top_internal_host_log.json,top_server_ip_log.json,top_urls_log.json,top_user_log.json,top_website_domain_log.json,traffic_app_stat_log.json,traffic_metrics_log.json ,traffic_protocol_stat_log.json,traffic_summary_log.json ,traffic_top_destination_ip_metrics_log.json,urls_proxy_hot.json,urls_security_hot.json",
"segments": [
{
"name": "segments.zip",
"md5": "0a3c607226daaf35a53d302b968bf7f7",
"description": "内置segments用于生成对应的基础表结构"
},
{
"name": "druid_segments-tsg3.0.sql",
"md5": "03ccd14160de7af90973df5bd3893033",
"description":"内置segments元数据信息sql数据"
}
]
},
{
"name": "gohangout",
"topology": "k2ck_active_defence_event_tsgv3 ,k2ck_dos_event_tsgv3 ,k2ck_gtpc_record_tsgv3 ,k2ck_interim_session_record_tsgv3 ,k2ck_proxy_event_tsgv3 ,k2ck_radius_onff_log_tsgv3 ,k2ck_radius_record_tsgv3 ,k2ck_security_event_tsgv3 ,k2ck_session_record_tsgv3 ,k2ck_sys_packet_capture_event_tsgv3 ,k2ck_transaction_record_tsgv3 ,k2ck_voip_record_tsgv3",
"description": "原始/补全/统计日志入库"
},
{
"name": "clickhouse",
"topology": "create_ck_table.sql",
"md5": "7cc9775d22403fd09c14cdb744487428",
"description": "Clickhouse 全量建表语句"
}
]
}
}

1861
testSchemaFiles/voip_record.json Normal file
View File

File diff suppressed because it is too large Load Diff