All LogQL queries contain a log stream selector.
Optionally, the log stream selector can be followed by a log pipeline. A log pipeline is a set of stage expressions that are chained together and applied to the selected log streams. Each expression can filter out, parse, or mutate log lines and their respective labels.
+Optionally, the log stream selector can be followed by a log pipeline. A log pipeline is a set of stage expressions that are chained together and applied to the selected log streams. Each expression can filter out, parse, or mutate log lines and their respective labels.
The following example shows a full log query in action:
{container="query-frontend",namespace="loki-dev"} |= "metrics.go" | logfmt | duration > 10s and throughput_mb < 500
The query is composed of:
@@ -315,12 +315,12 @@ instance_cpu_time_ns{app="fox", proc="widget", rev="4d3a513", env="prod", job="cThe stream selector determines which log streams to include in a query’s results. A log stream is a unique source of log content, such as a file. A more granular log stream selector then reduces the number of searched streams to a manageable volume. This means that the labels passed to the log stream selector will affect the relative performance of the query’s execution.
-The log stream selector is specified by one or more comma-separated key-value pairs. Each key is a log label and each value is that label’s value. Curly braces ({ and }) delimit the stream selector.
+The log stream selector is specified by one or more comma-separated key-value pairs. Each key is a log label and each value is that label’s value. Curly braces ({ and }) delimit the stream selector.
Consider this stream selector:
{app="mysql",name="mysql-backup"}
All log streams that have both a label of app whose value is mysql and a label of name whose value is mysql-backup will be included in the query results. A stream may contain other pairs of labels and values, but only the specified pairs within the stream selector are used to determine which streams will be included within the query results.
The same rules that apply for Prometheus Label Selectors apply for Grafana Loki log stream selectors.
-The = operator after the label name is a label matching operator. The following label matching operators are supported:
The = operator after the label name is a label matching operator. The following label matching operators are supported:
=: exactly equal!=: not equalThe line filter expression does a distributed grep over the aggregated logs from the matching log streams. It searches the contents of the log line, discarding those lines that do not match the case sensitive expression.
+The line filter expression does a distributed grep over the aggregated logs from the matching log streams. It searches the contents of the log line, discarding those lines that do not match the case sensitive expression.
Each line filter expression has a filter operator followed by text or a regular expression. These filter operators are supported:
|=: Log line contains stringThe | label_format expression can rename, modify or add labels. It takes as parameter a comma separated list of equality operations, enabling multiple operations at once.
When both side are label identifiers, for example dst=src, the operation will rename the src label into dst.
The right side can alternatively be a template string (double quoted or backtick), for example dst="{{.status}} {{.query}}", in which case the dst label value is replaced by the result of the text/template evaluation. This is the same template engine as the | line_format expression, which means labels are available as variables and you can use the same list of functions.
The right side can alternatively be a template string (double quoted or backtick), for example dst="{{.status}} {{.query}}", in which case the dst label value is replaced by the result of the text/template evaluation. This is the same template engine as the | line_format expression, which means labels are available as variables and you can use the same list of functions.
In both cases, if the destination label doesn’t exist, then a new one is created.
-The renaming form dst=src will drop the src label after remapping it to the dst label. However, the template form will preserve the referenced labels, such that dst="{{.src}}" results in both dst and src having the same value.
The renaming form dst=src will drop the src label after remapping it to the dst label. However, the template form will preserve the referenced labels, such that dst="{{.src}}" results in both dst and src having the same value.
A single label name can only appear once per expression. This means| label_format foo=bar,foo="new"is not allowed but you can use two expressions for the desired effect:| label_format foo=bar | label_format foo="new"
level=debug ts=2020-10-02T10:10:42.092268913Z caller=logging.go:66 traceID=a9d4d8a928d8db1 msg="POST /api/prom/api/v1/query_range (200) 1.5s"
You can use multiple parsers (logfmt and regexp) like this.
{job="loki-ops/query-frontend"} | logfmt | line_format "{{.msg}}" | regexp "(?P<method>\\w+) (?P<path>[\\w|/]+) \\((?P<status>\\d+?)\\) (?P<duration>.*)"
This is possible because the | line_format reformats the log line to become POST /api/prom/api/v1/query_range (200) 1.5s which can then be parsed with the | regexp ... parser.
This is possible because the | line_format reformats the log line to become POST /api/prom/api/v1/query_range (200) 1.5s which can then be parsed with the | regexp ... parser.
Supported function for operating over unwrapped ranges are:
rate(unwrapped-range): calculates the number of entries per second rate(unwrapped-range): calculates per second rate of the sum of all values in the specified interval. rate_counter(unwrapped-range): calculates per second rate of the values in the specified interval and treating them as “counter metric” sum_over_time(unwrapped-range): the sum of all values in the specified interval. avg_over_time(unwrapped-range): the average value of all points in the specified interval. bottomk: Select smallest k elements by sample valueThe aggregation operators can either be used to aggregate over all label values or a set of distinct label values by including a without or a by clause:
<aggr-op>([parameter,] <vector expression>) [without|by (<label list>)]
<aggr-op>([parameter,] <vector expression>) [without|by (<label list>)]
parameter is required when using topk and bottomk. topk and bottomk are different from other aggregators in that a subset of the input samples, including the original labels, are returned in the result vector.
by and without are only used to group the input vector. The without clause removes the listed labels from the resulting vector, keeping all others. The by clause does the opposite, dropping labels that are not listed in the clause, even if their label values are identical between all elements of the vector.