promql
Querying prometheus
Prometheus provides a functional query language called PromQL (Prometheus Query Language) that lets the user select and aggregate time series data in real time. The result of an expression can either be shown as a graph, viewed as tabular data in Prometheus's expression browser, or consumed by external systems via the HTTP API.
Examples
This document is meant as a reference. For learning, it might be easier to start with a couple of examples.
Expression language data types
In Prometheus's expression language, an expression or sub-expression can evaluate to one of four types:
- Instant vector - a set of time series containing a single sample for each time series, all sharing the same timestamp
- Range vector - a set of time series containing a range of data points over time for each time series
- Scalar - a simple numeric floating point value
- String - a simple string value; currently unused
Depending on the use-case (e.g. when graphing vs. displaying the output of an expression), only some of these types are legal as the result from a user-specified expression. For example, an expression that returns an instant vector is the only type that can be directly graphed.
Literals
String literals
Strings may be specified as literals in single quotes, double quotes or backticks.
PromQL follows the same escaping rules as Go. In single or double quotes a backslash begins an escape sequence, which may be followed by a, b, f, n, r, t, v or \. Specific characters can be provided using octal (\nnn) or hexadecimal (\xnn, \unnnn and \Unnnnnnnn).
No escaping is processed inside backticks. Unlike Go, Prometheus does not discard newlines inside backticks.
Example:
"this is a string" 'these are unescaped: \n \\ \t' `these are not unescaped: \n ' " \t`
Float literals
Scalar float values can be written as literal integer or floating-point numbers in the format (whitespace only included for better readability):
[-+]?(
[0-9]*\.?[0-9]+([eE][-+]?[0-9]+)?
| 0[xX][0-9a-fA-F]+
| [nN][aA][nN]
| [iI][nN][fF]
)
Examples:
23 -2.43 3.4e-9 0x8f -Inf NaN
Time series Selectors
Instant vector selectors
Instant vector selectors allow the selection of a set of time series and a single sample value for each at a given timestamp (instant): in the simplest form, only a metric name is specified. This results in an instant vector containing elements for all time series that have this metric name.
This example selects all time series that have the http_requests_total metric name:
http_requests_total
It is possible to filter these time series further by appending a comma separated list of label matchers in curly braces ({}).
This example selects only those time series with the http_requests_total metric name that also have the job label set to prometheus and their group label set to canary:
http_requests_total{job="prometheus",group="canary"}
It is also possible to negatively match a label value, or to match label values against regular expressions. The following label matching operators exist:
=: Select labels that are exactly equal to the provided string.!=: Select labels that are not equal to the provided string.=~: Select labels that regex-match the provided string.!~: Select labels that do not regex-match the provided string.
Regex matches are fully anchored. A match of env=~"foo" is treated as env=~"^foo$".
For example, this selects all http_requests_total time series for staging, testing, and development environments and HTTP methods other than GET.
http_requests_total{environment=~"staging|testing|development",method!="GET"}
Label matchers that match empty label values also select all time series that do not have the specific label set at all. It is possible to have multiple matchers for the same label name.
Vector selectors must either specify a name or at least one label matcher that does not match the empty string. The following expression is illegal:
{job=~".*"} # Bad!
In contrast, these expressions are valid as they both have a selector that does not match empty label values.
{job=~".+"} # Good!
{job=~".*",method="get"} # Good!
Label matchers can also be applied to metric names by matching against the internal __name__ label. For example, the expression http_requests_total is equivalent to {__name__="http_requests_total"}. Matchers other than = (!=, =~, !~) may also be used. The following expression selects all metrics that have a name starting with job::
{__name__=~"job:.*"}
The metric name must not be one of the keywords bool, on, ignoring, group_left and group_right. The following expression is illegal:
on{} # Bad!
A workaround for this restriction is to use the __name__ label:
{__name__="on"} # Good!
All regular expressions in Prometheus use RE2 syntax.
Range Vector Selectors
Range vector literals work like instant vector literals, except that they select a range of samples back from the current instant. Syntactically, a time duration is appended in square brackets ([]) at the end of a vector selector to specify how far back in time values should be fetched for each resulting range vector element.
In this example, we select all the values we have recorded within the last 5 minutes for all time series that have the metric name http_requests_total and a job label set to prometheus:
http_requests_total{job="prometheus"}[5m]
Time Durations
Time durations are specified as a number, followed immediately by one of the following units:
ms- millisecondss- secondsm- minutesh- hoursd- days - assuming a day has always 24hw- weeks - assuming a week has always 7dy- years - assuming a year has always 365d
Time durations can be combined, by concatenation. Units must be ordered from the longest to the shortest. A given unit must only appear once in a time duration.
Here are some examples of valid time durations:
5h 1h30m 5m 10s
Offset modifier
The offset modifier allows changing the time offset for individual instant and range vectors in a query.
For example, the following expression returns the value of http_requests_total 5 minutes in the past relative to the current query evaluation time:
http_requests_total offset 5m
Note that the offset modifier always needs to follow the selector immediately, i.e. the following would be correct:
sum(http_requests_total{method="GET"} offset 5m) // GOOD.
While the following would be incorrect:
sum(http_requests_total{method="GET"}) offset 5m // INVALID.
The same works for range vectors. This returns the 5-minute rate that http_requests_total had a week ago:
rate(http_requests_total[5m] offset 1w)
For comparisons with temporal shifts forward in time, a negative offset can be specified:
rate(http_requests_total[5m] offset -1w)
Note that this allows a query to look ahead of its evaluation time.
@ modifier
The @ modifier allows changing the evaluation time for individual instant and range vectors in a query. The time supplied to the @ modifier is a unix timestamp and described with a float literal.
For example, the following expression returns the value of http_requests_total at 2021-01-04T07:40:00+00:00:
http_requests_total @ 1609746000
Note that the @ modifier always needs to follow the selector immediately, i.e. the following would be correct:
sum(http_requests_total{method="GET"} @ 1609746000) // GOOD.
While the following would be incorrect:
sum(http_requests_total{method="GET"}) @ 1609746000 // INVALID.
The same works for range vectors. This returns the 5-minute rate that http_requests_total had at 2021-01-04T07:40:00+00:00:
rate(http_requests_total[5m] @ 1609746000)
The @ modifier supports all representation of float literals described above within the limits of int64. It can also be used along with the offset modifier where the offset is applied relative to the @ modifier time irrespective of which modifier is written first. These 2 queries will produce the same result.
# offset after @ http_requests_total @ 1609746000 offset 5m # offset before @ http_requests_total offset 5m @ 1609746000
Additionally, start() and end() can also be used as values for the @ modifier as special values.
For a range query, they resolve to the start and end of the range query respectively and remain the same for all steps.
For an instant query, start() and end() both resolve to the evaluation time.
http_requests_total @ start() rate(http_requests_total[5m] @ end())
Note that the @ modifier allows a query to look ahead of its evaluation time.
Subquery
Subquery allows you to run an instant query for a given range and resolution. The result of a subquery is a range vector.
Syntax: <instant_query> '[' <range> ':' [<resolution>] ']' [ @ <float_literal> ] [ offset <duration> ]
<resolution>is optional. Default is the global evaluation interval.
Operators
Prometheus supports many binary and aggregation operators. These are described in detail in the expression language operators page.
Functions
Prometheus supports several functions to operate on data. These are described in detail in the expression language functions page.
Comments
PromQL supports line comments that start with #. Example:
# This is a comment
Gotchas
Staleness
When queries are run, timestamps at which to sample data are selected independently of the actual present time series data. This is mainly to support cases like aggregation (sum, avg, and so on), where multiple aggregated time series do not exactly align in time. Because of their independence, Prometheus needs to assign a value at those timestamps for each relevant time series. It does so by simply taking the newest sample before this timestamp.
If a target scrape or rule evaluation no longer returns a sample for a time series that was previously present, that time series will be marked as stale. If a target is removed, its previously returned time series will be marked as stale soon afterwards.
If a query is evaluated at a sampling timestamp after a time series is marked stale, then no value is returned for that time series. If new samples are subsequently ingested for that time series, they will be returned as normal.
If no sample is found (by default) 5 minutes before a sampling timestamp, no value is returned for that time series at this point in time. This effectively means that time series "disappear" from graphs at times where their latest collected sample is older than 5 minutes or after they are marked stale.
Staleness will not be marked for time series that have timestamps included in their scrapes. Only the 5 minute threshold will be applied in that case.
Avoiding slow queries and overloads
If a query needs to operate on a very large amount of data, graphing it might time out or overload the server or browser. Thus, when constructing queries over unknown data, always start building the query in the tabular view of Prometheus's expression browser until the result set seems reasonable (hundreds, not thousands, of time series at most). Only when you have filtered or aggregated your data sufficiently, switch to graph mode. If the expression still takes too long to graph ad-hoc, pre-record it via a recording rule.
This is especially relevant for Prometheus's query language, where a bare metric name selector like api_http_requests_total could expand to thousands of time series with different labels. Also keep in mind that expressions which aggregate over many time series will generate load on the server even if the output is only a small number of time series. This is similar to how it would be slow to sum all values of a column in a relational database, even if the output value is only a single number.
Operators
Binary operators
Prometheus's query language supports basic logical and arithmetic operators. For operations between two instant vectors, the matching behavior can be modified.
Arithmetic binary operators
The following binary arithmetic operators exist in Prometheus:
+(addition)-(subtraction)*(multiplication)/(division)%(modulo)^(power/exponentiation)
Binary arithmetic operators are defined between scalar/scalar, vector/scalar, and vector/vector value pairs.
Between two scalars, the behavior is obvious: they evaluate to another scalar that is the result of the operator applied to both scalar operands.
Between an instant vector and a scalar, the operator is applied to the value of every data sample in the vector. E.g. if a time series instant vector is multiplied by 2, the result is another vector in which every sample value of the original vector is multiplied by 2. The metric name is dropped.
Between two instant vectors, a binary arithmetic operator is applied to each entry in the left-hand side vector and its matching element in the right-hand vector. The result is propagated into the result vector with the grouping labels becoming the output label set. The metric name is dropped. Entries for which no matching entry in the right-hand vector can be found are not part of the result.
Trigonometric binary operators
The following trigonometric binary operators, which work in radians, exist in Prometheus:
atan2(based on https://pkg.go.dev/math#Atan2)
Trigonometric operators allow trigonometric functions to be executed on two vectors using vector matching, which isn't available with normal functions. They act in the same manner as arithmetic operators.
Comparison binary operators
The following binary comparison operators exist in Prometheus:
==(equal)!=(not-equal)>(greater-than)<(less-than)>=(greater-or-equal)<=(less-or-equal)
Comparison operators are defined between scalar/scalar, vector/scalar, and vector/vector value pairs. By default they filter. Their behavior can be modified by providing bool after the operator, which will return 0 or 1 for the value rather than filtering.
Between two scalars, the bool modifier must be provided and these operators result in another scalar that is either 0 (false) or 1 (true), depending on the comparison result.
Between an instant vector and a scalar, these operators are applied to the value of every data sample in the vector, and vector elements between which the comparison result is false get dropped from the result vector. If the bool modifier is provided, vector elements that would be dropped instead have the value 0 and vector elements that would be kept have the value 1. The metric name is dropped if the bool modifier is provided.
Between two instant vectors, these operators behave as a filter by default, applied to matching entries. Vector elements for which the expression is not true or which do not find a match on the other side of the expression get dropped from the result, while the others are propagated into a result vector with the grouping labels becoming the output label set. If the bool modifier is provided, vector elements that would have been dropped instead have the value 0 and vector elements that would be kept have the value 1, with the grouping labels again becoming the output label set. The metric name is dropped if the bool modifier is provided.
Logical/set binary operators
These logical/set binary operators are only defined between instant vectors:
and(intersection)or(union)unless(complement)
vector1 and vector2 results in a vector consisting of the elements of vector1 for which there are elements in vector2 with exactly matching label sets. Other elements are dropped. The metric name and values are carried over from the left-hand side vector.
vector1 or vector2 results in a vector that contains all original elements (label sets + values) of vector1 and additionally all elements of vector2 which do not have matching label sets in vector1.
vector1 unless vector2 results in a vector consisting of the elements of vector1 for which there are no elements in vector2 with exactly matching label sets. All matching elements in both vectors are dropped.
Vector matching
Operations between vectors attempt to find a matching element in the right-hand side vector for each entry in the left-hand side. There are two basic types of matching behavior: One-to-one and many-to-one/one-to-many.
Vector matching keywords
These vector matching keywords allow for matching between series with different label sets providing:
onignoring
Label lists provided to matching keywords will determine how vectors are combined. Examples can be found in One-to-one vector matches and in Many-to-one and one-to-many vector matches
Group modifiers
These group modifiers enable many-to-one/one-to-many vector matching:
group_leftgroup_right
Label lists can be provided to the group modifier which contain labels from the "one"-side to be included in the result metrics.
Many-to-one and one-to-many matching are advanced use cases that should be carefully considered. Often a proper use of ignoring(<labels>) provides the desired outcome.
Grouping modifiers can only be used for comparison and arithmetic. Operations as and, unless and or operations match with all possible entries in the right vector by default.
One-to-one vector matches
One-to-one finds a unique pair of entries from each side of the operation. In the default case, that is an operation following the format vector1 <operator> vector2. Two entries match if they have the exact same set of labels and corresponding values. The ignoring keyword allows ignoring certain labels when matching, while the on keyword allows reducing the set of considered labels to a provided list:
<vector expr> <bin-op> ignoring(<label list>) <vector expr> <vector expr> <bin-op> on(<label list>) <vector expr>
Example input:
method_code:http_errors:rate5m{method="get", code="500"} 24
method_code:http_errors:rate5m{method="get", code="404"} 30
method_code:http_errors:rate5m{method="put", code="501"} 3
method_code:http_errors:rate5m{method="post", code="500"} 6
method_code:http_errors:rate5m{method="post", code="404"} 21
method:http_requests:rate5m{method="get"} 600
method:http_requests:rate5m{method="del"} 34
method:http_requests:rate5m{method="post"} 120
Example query:
method_code:http_errors:rate5m{code="500"} / ignoring(code) method:http_requests:rate5m
This returns a result vector containing the fraction of HTTP requests with status code of 500 for each method, as measured over the last 5 minutes. Without ignoring(code) there would have been no match as the metrics do not share the same set of labels. The entries with methods put and del have no match and will not show up in the result:
{method="get"} 0.04 // 24 / 600
{method="post"} 0.05 // 6 / 120
Many-to-one and one-to-many vector matches
Many-to-one and one-to-many matchings refer to the case where each vector element on the "one"-side can match with multiple elements on the "many"-side. This has to be explicitly requested using the group_left or group_right modifiers, where left/right determines which vector has the higher cardinality.
<vector expr> <bin-op> ignoring(<label list>) group_left(<label list>) <vector expr> <vector expr> <bin-op> ignoring(<label list>) group_right(<label list>) <vector expr> <vector expr> <bin-op> on(<label list>) group_left(<label list>) <vector expr> <vector expr> <bin-op> on(<label list>) group_right(<label list>) <vector expr>
The label list provided with the group modifier contains additional labels from the "one"-side to be included in the result metrics. For on a label can only appear in one of the lists. Every time series of the result vector must be uniquely identifiable.
Example query:
method_code:http_errors:rate5m / ignoring(code) group_left method:http_requests:rate5m
In this case the left vector contains more than one entry per method label value. Thus, we indicate this using group_left. The elements from the right side are now matched with multiple elements with the same method label on the left:
{method="get", code="500"} 0.04 // 24 / 600
{method="get", code="404"} 0.05 // 30 / 600
{method="post", code="500"} 0.05 // 6 / 120
{method="post", code="404"} 0.175 // 21 / 120
Aggregation operators
Prometheus supports the following built-in aggregation operators that can be used to aggregate the elements of a single instant vector, resulting in a new vector of fewer elements with aggregated values:
sum(calculate sum over dimensions)min(select minimum over dimensions)max(select maximum over dimensions)avg(calculate the average over dimensions)group(all values in the resulting vector are 1)stddev(calculate population standard deviation over dimensions)stdvar(calculate population standard variance over dimensions)count(count number of elements in the vector)count_values(count number of elements with the same value)bottomk(smallest k elements by sample value)topk(largest k elements by sample value)quantile(calculate φ-quantile (0 ≤ φ ≤ 1) over dimensions)
These operators can either be used to aggregate over all label dimensions or preserve distinct dimensions by including a without or by clause. These clauses may be used before or after the expression.
<aggr-op> [without|by (<label list>)] ([parameter,] <vector expression>)
or
<aggr-op>([parameter,] <vector expression>) [without|by (<label list>)]
label list is a list of unquoted labels that may include a trailing comma, i.e. both (label1, label2) and (label1, label2,) are valid syntax.
without removes the listed labels from the result vector, while all other labels are preserved in the output. by does the opposite and drops labels that are not listed in the by clause, even if their label values are identical between all elements of the vector.
parameter is only required for count_values, quantile, topk and bottomk.
count_values outputs one time series per unique sample value. Each series has an additional label. The name of that label is given by the aggregation parameter, and the label value is the unique sample value. The value of each time series is the number of times that sample value was present.
topk and bottomk are different from other aggregators in that a subset of the input samples, including the original labels, are returned in the result vector. by and without are only used to bucket the input vector.
quantile calculates the φ-quantile, the value that ranks at number φ*N among the N metric values of the dimensions aggregated over. φ is provided as the aggregation parameter. For example, quantile(0.5, ...) calculates the median, quantile(0.95, ...) the 95th percentile. For φ = NaN, NaN is returned. For φ < 0, -Inf is returned. For φ > 1, +Inf is returned.
Example:
If the metric http_requests_total had time series that fan out by application, instance, and group labels, we could calculate the total number of seen HTTP requests per application and group over all instances via:
sum without (instance) (http_requests_total)
Which is equivalent to:
sum by (application, group) (http_requests_total)
If we are just interested in the total of HTTP requests we have seen in all applications, we could simply write:
sum(http_requests_total)
To count the number of binaries running each build version we could write:
count_values("version", build_version)
To get the 5 largest HTTP requests counts across all instances we could write:
topk(5, http_requests_total)
Binary operator precedence
The following list shows the precedence of binary operators in Prometheus, from highest to lowest.
^*,/,%,atan2+,-==,!=,<=,<,>=,>and,unlessor
Operators on the same precedence level are left-associative. For example, 2 * 3 % 2 is equivalent to (2 * 3) % 2. However ^ is right associative, so 2 ^ 3 ^ 2 is equivalent to 2 ^ (3 ^ 2).
Functions
Some functions have default arguments, e.g. year(v=vector(time()) instant-vector). This means that there is one argument v which is an instant vector, which if not provided it will default to the value of the expression vector(time()).
abs()
abs(v instant-vector) returns the input vector with all sample values converted to their absolute value.
absent()
absent(v instant-vector) returns an empty vector if the vector passed to it has any elements and a 1-element vector with the value 1 if the vector passed to it has no elements.
This is useful for alerting on when no time series exist for a given metric name and label combination.
absent(nonexistent{job="myjob"})
# => {job="myjob"}
absent(nonexistent{job="myjob",instance=~".*"})
# => {job="myjob"}
absent(sum(nonexistent{job="myjob"}))
# => {}
In the first two examples, absent() tries to be smart about deriving labels of the 1-element output vector from the input vector.
absent_over_time()
absent_over_time(v range-vector) returns an empty vector if the range vector passed to it has any elements and a 1-element vector with the value 1 if the range vector passed to it has no elements.
This is useful for alerting on when no time series exist for a given metric name and label combination for a certain amount of time.
absent_over_time(nonexistent{job="myjob"}[1h])
# => {job="myjob"}
absent_over_time(nonexistent{job="myjob",instance=~".*"}[1h])
# => {job="myjob"}
absent_over_time(sum(nonexistent{job="myjob"})[1h:])
# => {}
In the first two examples, absent_over_time() tries to be smart about deriving labels of the 1-element output vector from the input vector.
ceil()
ceil(v instant-vector) rounds the sample values of all elements in v up to the nearest integer.
changes()
For each input time series, changes(v range-vector) returns the number of times its value has changed within the provided time range as an instant vector.
clamp()
clamp(v instant-vector, min scalar, max scalar) clamps the sample values of all elements in v to have a lower limit of min and an upper limit of max.
Special cases: - Return an empty vector if min > max - Return NaN if min or max is NaN
clamp_max()
clamp_max(v instant-vector, max scalar) clamps the sample values of all elements in v to have an upper limit of max.
clamp_min()
clamp_min(v instant-vector, min scalar) clamps the sample values of all elements in v to have a lower limit of min.
day_of_month()
day_of_month(v=vector(time()) instant-vector) returns the day of the month for each of the given times in UTC. Returned values are from 1 to 31.
day_of_week()
day_of_week(v=vector(time()) instant-vector) returns the day of the week for each of the given times in UTC. Returned values are from 0 to 6, where 0 means Sunday etc.
day_of_year()
day_of_year(v=vector(time()) instant-vector) returns the day of the year for each of the given times in UTC. Returned values are from 1 to 365 for non-leap years, and 1 to 366 in leap years.
days_in_month()
days_in_month(v=vector(time()) instant-vector) returns number of days in the month for each of the given times in UTC. Returned values are from 28 to 31.
delta()
delta(v range-vector) calculates the difference between the first and last value of each time series element in a range vector v, returning an instant vector with the given deltas and equivalent labels. The delta is extrapolated to cover the full time range as specified in the range vector selector, so that it is possible to get a non-integer result even if the sample values are all integers.
The following example expression returns the difference in CPU temperature between now and 2 hours ago:
delta(cpu_temp_celsius{host="zeus"}[2h])
delta should only be used with gauges.
deriv()
deriv(v range-vector) calculates the per-second derivative of the time series in a range vector v, using simple linear regression. The range vector must have at least two samples in order to perform the calculation. When +Inf or -Inf are found in the range vector, the slope and offset value calculated will be NaN.
deriv should only be used with gauges.
exp()
exp(v instant-vector) calculates the exponential function for all elements in v. Special cases are:
Exp(+Inf) = +InfExp(NaN) = NaN
floor()
floor(v instant-vector) rounds the sample values of all elements in v down to the nearest integer.
histogram_quantile()
histogram_quantile(φ scalar, b instant-vector) calculates the φ-quantile (0 ≤ φ ≤ 1) from the buckets b of a histogram. (See histograms and summaries for a detailed explanation of φ-quantiles and the usage of the histogram metric type in general.) The samples in b are the counts of observations in each bucket. Each sample must have a label le where the label value denotes the inclusive upper bound of the bucket. (Samples without such a label are silently ignored.) The histogram metric type automatically provides time series with the _bucket suffix and the appropriate labels.
Use the rate() function to specify the time window for the quantile calculation.
Example: A histogram metric is called http_request_duration_seconds. To calculate the 90th percentile of request durations over the last 10m, use the following expression:
histogram_quantile(0.9, rate(http_request_duration_seconds_bucket[10m]))
The quantile is calculated for each label combination in http_request_duration_seconds. To aggregate, use the sum() aggregator around the rate() function. Since the le label is required by histogram_quantile(), it has to be included in the by clause. The following expression aggregates the 90th percentile by job:
histogram_quantile(0.9, sum by (job, le) (rate(http_request_duration_seconds_bucket[10m])))
To aggregate everything, specify only the le label:
histogram_quantile(0.9, sum by (le) (rate(http_request_duration_seconds_bucket[10m])))
The histogram_quantile() function interpolates quantile values by assuming a linear distribution within a bucket. The highest bucket must have an upper bound of +Inf. (Otherwise, NaN is returned.) If a quantile is located in the highest bucket, the upper bound of the second highest bucket is returned. A lower limit of the lowest bucket is assumed to be 0 if the upper bound of that bucket is greater than 0. In that case, the usual linear interpolation is applied within that bucket. Otherwise, the upper bound of the lowest bucket is returned for quantiles located in the lowest bucket.
If b has 0 observations, NaN is returned. If b contains fewer than two buckets, NaN is returned. For φ < 0, -Inf is returned. For φ > 1, +Inf is returned. For φ = NaN, NaN is returned.
holt_winters()
holt_winters(v range-vector, sf scalar, tf scalar) produces a smoothed value for time series based on the range in v. The lower the smoothing factor sf, the more importance is given to old data. The higher the trend factor tf, the more trends in the data is considered. Both sf and tf must be between 0 and 1.
holt_winters should only be used with gauges.
hour()
hour(v=vector(time()) instant-vector) returns the hour of the day for each of the given times in UTC. Returned values are from 0 to 23.
idelta()
idelta(v range-vector) calculates the difference between the last two samples in the range vector v, returning an instant vector with the given deltas and equivalent labels.
idelta should only be used with gauges.
increase()
increase(v range-vector) calculates the increase in the time series in the range vector. Breaks in monotonicity (such as counter resets due to target restarts) are automatically adjusted for. The increase is extrapolated to cover the full time range as specified in the range vector selector, so that it is possible to get a non-integer result even if a counter increases only by integer increments.
The following example expression returns the number of HTTP requests as measured over the last 5 minutes, per time series in the range vector:
increase(http_requests_total{job="api-server"}[5m])
increase should only be used with counters. It is syntactic sugar for rate(v) multiplied by the number of seconds under the specified time range window, and should be used primarily for human readability. Use rate in recording rules so that increases are tracked consistently on a per-second basis.
irate()
irate(v range-vector) calculates the per-second instant rate of increase of the time series in the range vector. This is based on the last two data points. Breaks in monotonicity (such as counter resets due to target restarts) are automatically adjusted for.
The following example expression returns the per-second rate of HTTP requests looking up to 5 minutes back for the two most recent data points, per time series in the range vector:
irate(http_requests_total{job="api-server"}[5m])
irate should only be used when graphing volatile, fast-moving counters. Use rate for alerts and slow-moving counters, as brief changes in the rate can reset the FOR clause and graphs consisting entirely of rare spikes are hard to read.
Note that when combining irate() with an aggregation operator (e.g. sum()) or a function aggregating over time (any function ending in _over_time), always take a irate() first, then aggregate. Otherwise irate() cannot detect counter resets when your target restarts.
label_join()
For each timeseries in v, label_join(v instant-vector, dst_label string, separator string, src_label_1 string, src_label_2 string, ...) joins all the values of all the src_labels using separator and returns the timeseries with the label dst_label containing the joined value. There can be any number of src_labels in this function.
This example will return a vector with each time series having a foo label with the value a,b,c added to it:
label_join(up{job="api-server",src1="a",src2="b",src3="c"}, "foo", ",", "src1", "src2", "src3")
label_replace()
For each timeseries in v, label_replace(v instant-vector, dst_label string, replacement string, src_label string, regex string) matches the regular expression regex against the value of the label src_label. If it matches, the value of the label dst_label in the returned timeseries will be the expansion of replacement, together with the original labels in the input. Capturing groups in the regular expression can be referenced with $1, $2, etc. If the regular expression doesn't match then the timeseries is returned unchanged.
This example will return timeseries with the values a:c at label service and a at label foo:
label_replace(up{job="api-server",service="a:c"}, "foo", "$1", "service", "(.*):.*")
ln()
ln(v instant-vector) calculates the natural logarithm for all elements in v. Special cases are:
ln(+Inf) = +Infln(0) = -Infln(x < 0) = NaNln(NaN) = NaN
log2()
log2(v instant-vector) calculates the binary logarithm for all elements in v. The special cases are equivalent to those in ln.
log10()
log10(v instant-vector) calculates the decimal logarithm for all elements in v. The special cases are equivalent to those in ln.
minute()
minute(v=vector(time()) instant-vector) returns the minute of the hour for each of the given times in UTC. Returned values are from 0 to 59.
month()
month(v=vector(time()) instant-vector) returns the month of the year for each of the given times in UTC. Returned values are from 1 to 12, where 1 means January etc.
predict_linear()
predict_linear(v range-vector, t scalar) predicts the value of time series t seconds from now, based on the range vector v, using simple linear regression. The range vector must have at least two samples in order to perform the calculation. When +Inf or -Inf are found in the range vector, the slope and offset value calculated will be NaN.
predict_linear should only be used with gauges.
rate()
rate(v range-vector) calculates the per-second average rate of increase of the time series in the range vector. Breaks in monotonicity (such as counter resets due to target restarts) are automatically adjusted for. Also, the calculation extrapolates to the ends of the time range, allowing for missed scrapes or imperfect alignment of scrape cycles with the range's time period.
The following example expression returns the per-second rate of HTTP requests as measured over the last 5 minutes, per time series in the range vector:
rate(http_requests_total{job="api-server"}[5m])
rate should only be used with counters. It is best suited for alerting, and for graphing of slow-moving counters.
Note that when combining rate() with an aggregation operator (e.g. sum()) or a function aggregating over time (any function ending in _over_time), always take a rate() first, then aggregate. Otherwise rate() cannot detect counter resets when your target restarts.
resets()
For each input time series, resets(v range-vector) returns the number of counter resets within the provided time range as an instant vector. Any decrease in the value between two consecutive samples is interpreted as a counter reset.
resets should only be used with counters.
round()
round(v instant-vector, to_nearest=1 scalar) rounds the sample values of all elements in v to the nearest integer. Ties are resolved by rounding up. The optional to_nearest argument allows specifying the nearest multiple to which the sample values should be rounded. This multiple may also be a fraction.
scalar()
Given a single-element input vector, scalar(v instant-vector) returns the sample value of that single element as a scalar. If the input vector does not have exactly one element, scalar will return NaN.
sgn()
sgn(v instant-vector) returns a vector with all sample values converted to their sign, defined as this: 1 if v is positive, -1 if v is negative and 0 if v is equal to zero.
sort()
sort(v instant-vector) returns vector elements sorted by their sample values, in ascending order.
sort_desc()
Same as sort, but sorts in descending order.
sqrt()
sqrt(v instant-vector) calculates the square root of all elements in v.
time()
time() returns the number of seconds since January 1, 1970 UTC. Note that this does not actually return the current time, but the time at which the expression is to be evaluated.
timestamp()
timestamp(v instant-vector) returns the timestamp of each of the samples of the given vector as the number of seconds since January 1, 1970 UTC.
vector()
vector(s scalar) returns the scalar s as a vector with no labels.
year()
year(v=vector(time()) instant-vector) returns the year for each of the given times in UTC.
<aggregation>_over_time()
The following functions allow aggregating each series of a given range vector over time and return an instant vector with per-series aggregation results:
avg_over_time(range-vector):the average value of all points in the specified interval.min_over_time(range-vector):the minimum value of all points in the specified interval.max_over_time(range-vector):the maximum value of all points in the specified interval.sum_over_time(range-vector):the sum of all values in the specified interval.count_over_time(range-vector):the count of all values in the specified interval.quantile_over_time(scalar, range-vector):the φ-quantile (0 ≤ φ ≤ 1) of the values in the specified interval.stddev_over_time(range-vector):the population standard deviation of the values in the specified interval.stdvar_over_time(range-vector):the population standard variance of the values in the specified interval.last_over_time(range-vector):the most recent point value in specified interval.present_over_time(range-vector):the value 1 for any series in the specified interval.
Note that all values in the specified interval have the same weight in the aggregation even if the values are not equally spaced throughout the interval.
Trigonometric Functions
The trigonometric functions work in radians:
acos(v instant-vector):calculates the arccosine of all elements in v (special cases).acosh(v instant-vector):calculates the inverse hyperbolic cosine of all elements in v (special cases).asin(v instant-vector):calculates the arcsine of all elements in v (special cases).asinh(v instant-vector):calculates the inverse hyperbolic sine of all elements in v (special cases).atan(v instant-vector):calculates the arctangent of all elements in v (special cases).atanh(v instant-vector):calculates the inverse hyperbolic tangent of all elements in v (special cases).cos(v instant-vector):calculates the cosine of all elements in v (special cases).cosh(v instant-vector):calculates the hyperbolic cosine of all elements in v (special cases).sin(v instant-vector):calculates the sine of all elements in v (special cases).sinh(v instant-vector):calculates the hyperbolic sine of all elements in v (special cases).tan(v instant-vector):calculates the tangent of all elements in v (special cases).tanh(v instant-vector):calculates the hyperbolic tangent of all elements in v (special cases).
The following are useful for converting between degrees and radians:
deg(v instant-vector): converts radians to degrees for all elements inv.pi(): returns pi.rad(v instant-vector): converts degrees to radians for all elements inv.
Query examples
Simple time series selection
Return all time series with the metric http_requests_total:
http_requests_total
Return all time series with the metric http_requests_total and the given job and handler labels:
http_requests_total{job="apiserver", handler="/api/comments"}
Return a whole range of time (in this case 5 minutes up to the query time) for the same vector, making it a range vector:
http_requests_total{job="apiserver", handler="/api/comments"}[5m]
Note that an expression resulting in a range vector cannot be graphed directly, but viewed in the tabular ("Console") view of the expression browser.
Using regular expressions, you could select time series only for jobs whose name match a certain pattern, in this case, all jobs that end with server:
http_requests_total{job=~".*server"}
All regular expressions in Prometheus use RE2 syntax.
To select all HTTP status codes except 4xx ones, you could run:
http_requests_total{status!~"4.."}
Subquery
Return the 5-minute rate of the http_requests_total metric for the past 30 minutes, with a resolution of 1 minute.
rate(http_requests_total[5m])[30m:1m]
This is an example of a nested subquery. The subquery for the deriv function uses the default resolution. Note that using subqueries unnecessarily is unwise.
max_over_time(deriv(rate(distance_covered_total[5s])[30s:5s])[10m:])
Using functions, operators, etc.
Return the per-second rate for all time series with the http_requests_total metric name, as measured over the last 5 minutes:
rate(http_requests_total[5m])
Assuming that the http_requests_total time series all have the labels job (fanout by job name) and instance (fanout by instance of the job), we might want to sum over the rate of all instances, so we get fewer output time series, but still preserve the job dimension:
sum by (job) ( rate(http_requests_total[5m]) )
If we have two different metrics with the same dimensional labels, we can apply binary operators to them and elements on both sides with the same label set will get matched and propagated to the output. For example, this expression returns the unused memory in MiB for every instance (on a fictional cluster scheduler exposing these metrics about the instances it runs):
(instance_memory_limit_bytes - instance_memory_usage_bytes) / 1024 / 1024
The same expression, but summed by application, could be written like this:
sum by (app, proc) ( instance_memory_limit_bytes - instance_memory_usage_bytes ) / 1024 / 1024
If the same fictional cluster scheduler exposed CPU usage metrics like the following for every instance:
instance_cpu_time_ns{app="lion", proc="web", rev="34d0f99", env="prod", job="cluster-manager"}
instance_cpu_time_ns{app="elephant", proc="worker", rev="34d0f99", env="prod", job="cluster-manager"}
instance_cpu_time_ns{app="turtle", proc="api", rev="4d3a513", env="prod", job="cluster-manager"}
instance_cpu_time_ns{app="fox", proc="widget", rev="4d3a513", env="prod", job="cluster-manager"}
...
...we could get the top 3 CPU users grouped by application (app) and process type (proc) like this:
topk(3, sum by (app, proc) (rate(instance_cpu_time_ns[5m])))
Assuming this metric contains one time series per running instance, you could count the number of running instances per application like this:
count by (app) (instance_cpu_time_ns)
-
Querying prometheus
- Examples
- Expression language data types
-
Literals
- String literals
- Float literals
-
Time series Selectors
- Instant vector selectors
- Range Vector Selectors
- Time Durations
- Offset modifier
- @ modifier
- Subquery
- Operators
- Functions
- Comments
-
Gotchas
- Staleness
- Avoiding slow queries and overloads
-
Operators
-
Binary operators
- Arithmetic binary operators
- Trigonometric binary operators
- Comparison binary operators
- Logical/set binary operators
-
Vector matching
- Vector matching keywords
- Group modifiers
- One-to-one vector matches
- Many-to-one and one-to-many vector matches
- Aggregation operators
- Binary operator precedence
-
-
Functions
- abs()
- absent()
- absent_over_time()
- ceil()
- changes()
- clamp()
- clamp_max()
- clamp_min()
- day_of_month()
- day_of_week()
- day_of_year()
- days_in_month()
- delta()
- deriv()
- exp()
- floor()
- histogram_quantile()
- holt_winters()
- hour()
- idelta()
- increase()
- irate()
- label_join()
- label_replace()
- ln()
- log2()
- log10()
- minute()
- month()
- predict_linear()
- rate()
- resets()
- round()
- scalar()
- sgn()
- sort()
- sort_desc()
- sqrt()
- time()
- timestamp()
- vector()
- year()
- <aggregation>_over_time()
- Trigonometric Functions
-
Query examples
- Simple time series selection
- Subquery
- Using functions, operators, etc.
LogQL: Log query language
LogQL is Grafana Loki’s PromQL-inspired query language. Queries act as if they are a distributed grep to aggregate log sources. LogQL uses labels and operators for filtering.
There are two types of LogQL queries:
- Log queries return the contents of log lines.
- Metric queries extend log queries to calculate values based on query results.
Log queries
All LogQL queries contain a log stream selector.
Optionally, the log stream selector can be followed by a log pipeline. A log pipeline is a set of stage expressions that are chained together and applied to the selected log streams. Each expression can filter out, parse, or mutate log lines and their respective labels.
The following example shows a full log query in action:
{container="query-frontend",namespace="loki-dev"} |= "metrics.go" | logfmt | duration > 10s and throughput_mb < 500
The query is composed of:
- a log stream selector
{container="query-frontend",namespace="loki-dev"}which targets thequery-frontendcontainer in theloki-devnamespace. - a log pipeline
|= "metrics.go" | logfmt | duration > 10s and throughput_mb < 500which will filter out log that contains the wordmetrics.go, then parses each log line to extract more labels and filter with them.
To avoid escaping special characters you can use the`(backtick) instead of"when quoting strings. For example`\w+`is the same as"\\w+". This is specially useful when writing a regular expression which contains multiple
backslashes that require escaping.
Log stream selector
The stream selector determines which log streams to include in a query’s results. A log stream is a unique source of log content, such as a file. A more granular log stream selector then reduces the number of searched streams to a manageable volume. This means that the labels passed to the log stream selector will affect the relative performance of the query’s execution.
The log stream selector is specified by one or more comma-separated key-value pairs. Each key is a log label and each value is that label’s value. Curly braces ({ and }) delimit the stream selector.
Consider this stream selector:
{app="mysql",name="mysql-backup"}
All log streams that have both a label of app whose value is mysql and a label of name whose value is mysql-backup will be included in the query results. A stream may contain other pairs of labels and values, but only the specified pairs within the stream selector are used to determine which streams will be included within the query results.
The same rules that apply for Prometheus Label Selectors apply for Grafana Loki log stream selectors.
The = operator after the label name is a label matching operator. The following label matching operators are supported:
=: exactly equal!=: not equal=~: regex matches!~: regex does not match
Regex log stream examples:
{name =~ "mysql.+"}{name !~ "mysql.+"}{name !~ `mysql-\d+`}
Note: The =~ regex operator is fully anchored, meaning regex must match against the entire string, including newlines. The regex . character does not match newlines by default. If you want the regex dot character to match newlines you can use the single-line flag, like so: (?s)search_term.+ matches search_term\n.
Log pipeline
A log pipeline can be appended to a log stream selector to further process and filter log streams. It is composed of a set of expressions. Each expression is executed in left to right sequence for each log line. If an expression filters out a log line, the pipeline will stop processing the current log line and start processing the next log line.
Some expressions can mutate the log content and respective labels, which will be then be available for further filtering and processing in subsequent expressions. An example that mutates is the expression
| line_format "{{.status_code}}"
Log pipeline expressions fall into one of three categories:
- Filtering expressions: line filter expressions and label filter expressions
- Parsing expressions
- Formatting expressions: line format expressions and label format expressions
Line filter expression
The line filter expression does a distributed grep over the aggregated logs from the matching log streams. It searches the contents of the log line, discarding those lines that do not match the case sensitive expression.
Each line filter expression has a filter operator followed by text or a regular expression. These filter operators are supported:
|=: Log line contains string!=: Log line does not contain string|~: Log line contains a match to the regular expression!~: Log line does not contain a match to the regular expression
Line filter expression examples:
-
Keep log lines that have the substring “error”:
|= "error"A complete query using this example:
{job="mysql"} |= "error" -
Discard log lines that have the substring “kafka.server:type=ReplicaManager”:
!= "kafka.server:type=ReplicaManager"A complete query using this example:
{instance=~"kafka-[23]",name="kafka"} != "kafka.server:type=ReplicaManager" -
Keep log lines that contain a substring that starts with
tsdb-opsand ends withio:2003. A complete query with a regular expression:{name="kafka"} |~ "tsdb-ops.*io:2003" -
Keep log lines that contain a substring that starts with
error=, and is followed by 1 or more word characters. A complete query with a regular expression:{name="cassandra"} |~ `error=\w+`
Filter operators can be chained. Filters are applied sequentially. Query results will have satisfied every filter. This complete query example will give results that include the string error, and do not include the string timeout.
{job="mysql"} |= "error" != "timeout"
When using |~ and !~, Go (as in Golang) RE2 syntax regex may be used. The matching is case-sensitive by default. Switch to case-insensitive matching by prefixing the regular expression with (?i).
While line filter expressions could be placed anywhere within a log pipeline, it is almost always better to have them at the beginning. Placing them at the beginning improves the performance of the query, as it only does further processing when a line matches. For example, while the results will be the same, the query specified with
{job="mysql"} |= "error" | json | line_format "{{.err}}"
will always run faster than
{job="mysql"} | json | line_format "{{.message}}" |= "error"
Line filter expressions are the fastest way to filter logs once the log stream selectors have been applied.
Line filter expressions have support matching IP addresses. See Matching IP addresses for details.
Label filter expression
Label filter expression allows filtering log line using their original and extracted labels. It can contain multiple predicates.
A predicate contains a label identifier, an operation and a value to compare the label with.
For example with cluster="namespace"the cluster is the label identifier, the operation is = and the value is “namespace”. The label identifier is always on the right side of the operation.
We support multiple value types which are automatically inferred from the query input.
- String is double quoted or backticked such as
"200"or `us-central1`. - Duration is a sequence of decimal numbers, each with optional fraction and a unit suffix, such as “300ms”, “1.5h” or “2h45m”. Valid time units are “ns”, “us” (or “µs”), “ms”, “s”, “m”, “h”.
- Number are floating-point number (64bits), such as
250,89.923. - Bytes is a sequence of decimal numbers, each with optional fraction and a unit suffix, such as “42MB”, “1.5Kib” or “20b”. Valid bytes units are “b”, “kib”, “kb”, “mib”, “mb”, “gib”, “gb”, “tib”, “tb”, “pib”, “pb”, “eib”, “eb”.
String type work exactly like Prometheus label matchers use in log stream selector. This means you can use the same operations (=,!=,=~,!~).
The string type is the only one that can filter out a log line with a label __error__.
Using Duration, Number and Bytes will convert the label value prior to comparision and support the following comparators:
==or=for equality.!=for inequality.>and>=for greater than and greater than or equal.<and<=for lesser than and lesser than or equal.
For instance, logfmt | duration > 1m and bytes_consumed > 20MB
If the conversion of the label value fails, the log line is not filtered and an __error__ label is added. To filters those errors see the pipeline errors section.
You can chain multiple predicates using and and or which respectively express the and and or binary operations. and can be equivalently expressed by a comma, a space or another pipe. Label filters can be place anywhere in a log pipeline.
This means that all the following expressions are equivalent:
| duration >= 20ms or size == 20kb and method!~"2.."
| duration >= 20ms or size == 20kb | method!~"2.."
| duration >= 20ms or size == 20kb , method!~"2.."
| duration >= 20ms or size == 20kb method!~"2.."
By default the precedence of multiple predicates is right to left. You can wrap predicates with parenthesis to force a different precedence left to right.
For example the following are equivalent.
| duration >= 20ms or method="GET" and size <= 20KB
| ((duration >= 20ms or method="GET") and size <= 20KB)
It will evaluate first duration >= 20ms or method="GET". To evaluate first method="GET" and size <= 20KB, make sure to use proper parenthesis as shown below.
| duration >= 20ms or (method="GET" and size <= 20KB)
Label filter expressions are the only expression allowed after the unwrap expression. This is mainly to allow filtering errors from the metric extraction.
Label filter expressions have support matching IP addresses. See Matching IP addresses for details.
Parser expression
Parser expression can parse and extract labels from the log content. Those extracted labels can then be used for filtering using label filter expressions or for metric aggregations.
Extracted label keys are automatically sanitized by all parsers, to follow Prometheus metric name convention.(They can only contain ASCII letters and digits, as well as underscores and colons. They cannot start with a digit.)
For instance, the pipeline | json will produce the following mapping:
{ "a.b": {c: "d"}, e: "f" }
->
{a_b_c="d", e="f"}
In case of errors, for instance if the line is not in the expected format, the log line won’t be filtered but instead will get a new __error__ label added.
If an extracted label key name already exists in the original log stream, the extracted label key will be suffixed with the _extracted keyword to make the distinction between the two labels. You can forcefully override the original label using a label formatter expression. However if an extracted key appears twice, only the latest label value will be kept.
Loki supports JSON, logfmt, pattern, regexp and unpack parsers.
It’s easier to use the predefined parsers json and logfmt when you can. If you can’t, the pattern and regexp parsers can be used for log lines with an unusual structure. The pattern parser is easier and faster to write; it also outperforms the regexp parser. Multiple parsers can be used by a single log pipeline. This is useful for parsing complex logs. There are examples in Multiple parsers.
JSON
The json parser operates in two modes:
1. without parameters:
Adding | json to your pipeline will extract all json properties as labels if the log line is a valid json document. Nested properties are flattened into label keys using the _ separator.
Note: Arrays are skipped.
For example the json parsers will extract from the following document:
{
"protocol": "HTTP/2.0",
"servers": ["129.0.1.1","10.2.1.3"],
"request": {
"time": "6.032",
"method": "GET",
"host": "foo.grafana.net",
"size": "55",
"headers": {
"Accept": "*/*",
"User-Agent": "curl/7.68.0"
}
},
"response": {
"status": 401,
"size": "228",
"latency_seconds": "6.031"
}
}
The following list of labels:
"protocol" => "HTTP/2.0" "request_time" => "6.032" "request_method" => "GET" "request_host" => "foo.grafana.net" "request_size" => "55" "response_status" => "401" "response_size" => "228" "response_latency_seconds" => "6.031"
2. with parameters:
Using | json label="expression", another="expression" in your pipeline will extract only the specified json fields to labels. You can specify one or more expressions in this way, the same as label_format; all expressions must be quoted.
Currently, we only support field access (my.field, my["field"]) and array access (list[0]), and any combination of these in any level of nesting (my.list[0]["field"]).
For example, | json first_server="servers[0]", ua="request.headers[\"User-Agent\"] will extract from the following document:
{
"protocol": "HTTP/2.0",
"servers": ["129.0.1.1","10.2.1.3"],
"request": {
"time": "6.032",
"method": "GET",
"host": "foo.grafana.net",
"size": "55",
"headers": {
"Accept": "*/*",
"User-Agent": "curl/7.68.0"
}
},
"response": {
"status": 401,
"size": "228",
"latency_seconds": "6.031"
}
}
The following list of labels:
"first_server" => "129.0.1.1" "ua" => "curl/7.68.0"
If an array or an object returned by an expression, it will be assigned to the label in json format.
For example, | json server_list="servers", headers="request.headers" will extract:
"server_list" => `["129.0.1.1","10.2.1.3"]`
"headers" => `{"Accept": "*/*", "User-Agent": "curl/7.68.0"}`
If the label to be extracted is same as the original JSON field, expression can be written as just | json <label>
For example, to extract servers fields as label, expression can be written as following
| json servers will extract:
"servers" => `["129.0.1.1","10.2.1.3"]`
Note that | json servers is same as | json servers="servers"
logfmt
The logfmt parser can be added using the | logfmt and will extract all keys and values from the logfmt formatted log line.
For example the following log line:
at=info method=GET path=/ host=grafana.net fwd="124.133.124.161" service=8ms status=200
will get those labels extracted:
"at" => "info" "method" => "GET" "path" => "/" "host" => "grafana.net" "fwd" => "124.133.124.161" "service" => "8ms" "status" => "200"
Pattern
The pattern parser allows the explicit extraction of fields from log lines by defining a pattern expression (| pattern "<pattern-expression>"). The expression matches the structure of a log line.
Consider this NGINX log line.
0.191.12.2 - - [10/Jun/2021:09:14:29 +0000] "GET /api/plugins/versioncheck HTTP/1.1" 200 2 "-" "Go-http-client/2.0" "13.76.247.102, 34.120.177.193" "TLSv1.2" "US" ""
This log line can be parsed with the expression
<ip> - - <_> "<method> <uri> <_>" <status> <size> <_> "<agent>" <_>
to extract these fields:
"ip" => "0.191.12.2" "method" => "GET" "uri" => "/api/plugins/versioncheck" "status" => "200" "size" => "2" "agent" => "Go-http-client/2.0"
A pattern expression is composed of captures and literals.
A capture is a field name delimited by the < and > characters. <example> defines the field name example. An unnamed capture appears as <_>.
The unnamed capture skips matched content.
Captures are matched from the line beginning or the previous set of literals, to the line end or the next set of literals. If a capture is not matched, the pattern parser will stop.
Literals can be any sequence of UTF-8 characters, including whitespace characters.
By default, a pattern expression is anchored at the start of the log line. If the expression starts with literals, then the log line must also start with the same set of literals. Use <_> at the beginning of the expression if you don’t want to anchor the expression at the start.
Consider the log line
level=debug ts=2021-06-10T09:24:13.472094048Z caller=logging.go:66 traceID=0568b66ad2d9294c msg="POST /loki/api/v1/push (204) 16.652862ms"
To match msg=", use the expression:
<_> msg="<method> <path> (<status>) <latency>"
A pattern expression is invalid if
- It does not contain any named capture.
- It contains two consecutive captures not separated by whitespace characters.
Regular expression
Unlike the logfmt and json, which extract implicitly all values and takes no parameters, the regexp parser takes a single parameter | regexp "<re>" which is the regular expression using the Golang RE2 syntax.
The regular expression must contain a least one named sub-match (e.g (?P<name>re)), each sub-match will extract a different label.
For example the parser | regexp "(?P<method>\\w+) (?P<path>[\\w|/]+) \\((?P<status>\\d+?)\\) (?P<duration>.*)" will extract from the following line:
POST /api/prom/api/v1/query_range (200) 1.5s
those labels:
"method" => "POST" "path" => "/api/prom/api/v1/query_range" "status" => "200" "duration" => "1.5s"
unpack
The unpack parser parses a JSON log line, unpacking all embedded labels from Promtail’s pack stage. A special property _entry will also be used to replace the original log line.
For example, using | unpack with the log line:
{
"container": "myapp",
"pod": "pod-3223f",
"_entry": "original log message"
}
extracts the container and pod labels; it sets original log message as the new log line.
You can combine the unpack and json parsers (or any other parsers) if the original embedded log line is of a specific format.
Line format expression
The line format expression can rewrite the log line content by using the text/template format. It takes a single string parameter | line_format "{{.label_name}}", which is the template format. All labels are injected variables into the template and are available to use with the {{.label_name}} notation.
For example the following expression:
{container="frontend"} | logfmt | line_format "{{.query}} {{.duration}}"
Will extract and rewrite the log line to only contains the query and the duration of a request.
You can use double quoted string for the template or backticks `{{.label_name}}` to avoid the need to escape special characters.
line_format also supports math functions. Example:
If we have the following labels ip=1.1.1.1, status=200 and duration=3000(ms), we can divide the duration by 1000 to get the value in seconds.
{container="frontend"} | logfmt | line_format "{{.ip}} {{.status}} {{div .duration 1000}}"
The above query will give us the line as 1.1.1.1 200 3
See template functions to learn about available functions in the template format.
Labels format expression
The | label_format expression can rename, modify or add labels. It takes as parameter a comma separated list of equality operations, enabling multiple operations at once.
When both side are label identifiers, for example dst=src, the operation will rename the src label into dst.
The right side can alternatively be a template string (double quoted or backtick), for example dst="{{.status}} {{.query}}", in which case the dst label value is replaced by the result of the text/template evaluation. This is the same template engine as the | line_format expression, which means labels are available as variables and you can use the same list of functions.
In both cases, if the destination label doesn’t exist, then a new one is created.
The renaming form dst=src will drop the src label after remapping it to the dst label. However, the template form will preserve the referenced labels, such that dst="{{.src}}" results in both dst and src having the same value.
A single label name can only appear once per expression. This means| label_format foo=bar,foo="new"is not allowed but you can use two expressions for the desired effect:| label_format foo=bar | label_format foo="new"
Log queries examples
Multiple filtering
Filtering should be done first using label matchers, then line filters (when possible) and finally using label filters. The following query demonstrate this.
{cluster="ops-tools1", namespace="loki-dev", job="loki-dev/query-frontend"} |= "metrics.go" !="out of order" | logfmt | duration > 30s or status_code!="200"
Multiple parsers
To extract the method and the path of the following logfmt log line:
level=debug ts=2020-10-02T10:10:42.092268913Z caller=logging.go:66 traceID=a9d4d8a928d8db1 msg="POST /api/prom/api/v1/query_range (200) 1.5s"
You can use multiple parsers (logfmt and regexp) like this.
{job="loki-ops/query-frontend"} | logfmt | line_format "{{.msg}}" | regexp "(?P<method>\\w+) (?P<path>[\\w|/]+) \\((?P<status>\\d+?)\\) (?P<duration>.*)"
This is possible because the | line_format reformats the log line to become POST /api/prom/api/v1/query_range (200) 1.5s which can then be parsed with the | regexp ... parser.
Formatting
The following query shows how you can reformat a log line to make it easier to read on screen.
{cluster="ops-tools1", name="querier", namespace="loki-dev"}
|= "metrics.go" != "loki-canary"
| logfmt
| query != ""
| label_format query="{{ Replace .query \"\\n\" \"\" -1 }}"
| line_format "{{ .ts}}\t{{.duration}}\ttraceID = {{.traceID}}\t{{ printf \"%-100.100s\" .query }} "
Label formatting is used to sanitize the query while the line format reduce the amount of information and creates a tabular output.
For these given log lines:
level=info ts=2020-10-23T20:32:18.094668233Z caller=metrics.go:81 org_id=29 traceID=1980d41501b57b68 latency=fast query="{cluster=\"ops-tools1\", job=\"loki-ops/query-frontend\"} |= \"query_range\"" query_type=filter range_type=range length=15m0s step=7s duration=650.22401ms status=200 throughput_mb=1.529717 total_bytes_mb=0.994659
level=info ts=2020-10-23T20:32:18.068866235Z caller=metrics.go:81 org_id=29 traceID=1980d41501b57b68 latency=fast query="{cluster=\"ops-tools1\", job=\"loki-ops/query-frontend\"} |= \"query_range\"" query_type=filter range_type=range length=15m0s step=7s duration=624.008132ms status=200 throughput_mb=0.693449 total_bytes_mb=0.432718
The result would be:
2020-10-23T20:32:18.094668233Z 650.22401ms traceID = 1980d41501b57b68 {cluster="ops-tools1", job="loki-ops/query-frontend"} |= "query_range"
2020-10-23T20:32:18.068866235Z 624.008132ms traceID = 1980d41501b57b68 {cluster="ops-tools1", job="loki-ops/query-frontend"} |= "query_range"
Metric queries
Metric queries extend log queries by applying a function to log query results. This powerful feature creates metrics from logs.
Metric queries can be used to calculate the rate of error messages or the top N log sources with the greatest quantity of logs over the last 3 hours.
Combined with parsers, metric queries can also be used to calculate metrics from a sample value within the log line, such as latency or request size. All labels, including extracted ones, will be available for aggregations and generation of new series.
Range Vector aggregation
LogQL shares the range vector concept of Prometheus. In Grafana Loki, the selected range of samples is a range of selected log or label values.
The aggregation is applied over a time duration. Loki defines Time Durations with the same syntax as Prometheus.
Loki supports two types of range vector aggregations: log range aggregations and unwrapped range aggregations.
Log range aggregations
A log range aggregation is a query followed by a duration. A function is applied to aggregate the query over the duration. The duration can be placed after the log stream selector or at end of the log pipeline.
The functions:
rate(log-range): calculates the number of entries per secondcount_over_time(log-range): counts the entries for each log stream within the given range.bytes_rate(log-range): calculates the number of bytes per second for each stream.bytes_over_time(log-range): counts the amount of bytes used by each log stream for a given range.absent_over_time(log-range): returns an empty vector if the range vector passed to it has any elements and a 1-element vector with the value 1 if the range vector passed to it has no elements. (absent_over_timeis useful for alerting on when no time series and logs stream exist for label combination for a certain amount of time.)
Examples:
-
Count all the log lines within the last five minutes for the MySQL job.
count_over_time({job="mysql"}[5m]) -
This aggregation includes filters and parsers. It returns the per-second rate of all non-timeout errors within the last minutes per host for the MySQL job and only includes errors whose duration is above ten seconds.
sum by (host) (rate({job="mysql"} |= "error" != "timeout" | json | duration > 10s [1m]))
Unwrapped range aggregations
Unwrapped ranges uses extracted labels as sample values instead of log lines. However to select which label will be used within the aggregation, the log query must end with an unwrap expression and optionally a label filter expression to discard errors.
The unwrap expression is noted | unwrap label_identifier where the label identifier is the label name to use for extracting sample values.
Since label values are string, by default a conversion into a float (64bits) will be attempted, in case of failure the __error__ label is added to the sample. Optionally the label identifier can be wrapped by a conversion function | unwrap <function>(label_identifier), which will attempt to convert the label value from a specific format.
We currently support the functions:
duration_seconds(label_identifier)(or its short equivalentduration) which will convert the label value in seconds from the go duration format (e.g5m,24s30ms).bytes(label_identifier)which will convert the label value to raw bytes applying the bytes unit (e.g.5 MiB,3k,1G).
Supported function for operating over unwrapped ranges are:
rate(unwrapped-range): calculates per second rate of the sum of all values in the specified interval.rate_counter(unwrapped-range): calculates per second rate of the values in the specified interval and treating them as “counter metric”sum_over_time(unwrapped-range): the sum of all values in the specified interval.avg_over_time(unwrapped-range): the average value of all points in the specified interval.max_over_time(unwrapped-range): the maximum value of all points in the specified interval.min_over_time(unwrapped-range): the minimum value of all points in the specified intervalfirst_over_time(unwrapped-range): the first value of all points in the specified intervallast_over_time(unwrapped-range): the last value of all points in the specified intervalstdvar_over_time(unwrapped-range): the population standard variance of the values in the specified interval.stddev_over_time(unwrapped-range): the population standard deviation of the values in the specified interval.quantile_over_time(scalar,unwrapped-range): the φ-quantile (0 ≤ φ ≤ 1) of the values in the specified interval.absent_over_time(unwrapped-range): returns an empty vector if the range vector passed to it has any elements and a 1-element vector with the value 1 if the range vector passed to it has no elements. (absent_over_timeis useful for alerting on when no time series and logs stream exist for label combination for a certain amount of time.)
Except for sum_over_time,absent_over_time and rate, unwrapped range aggregations support grouping.
<aggr-op>([parameter,] <unwrapped-range>) [without|by (<label list>)]
Which can be used to aggregate over distinct labels dimensions by including a without or by clause.
without removes the listed labels from the result vector, while all other labels are preserved the output. by does the opposite and drops labels that are not listed in the by clause, even if their label values are identical between all elements of the vector.
Unwrapped examples
quantile_over_time(0.99,
{cluster="ops-tools1",container="ingress-nginx"}
| json
| __error__ = ""
| unwrap request_time [1m]) by (path)
This example calculates the p99 of the nginx-ingress latency by path.
sum by (org_id) (
sum_over_time(
{cluster="ops-tools1",container="loki-dev"}
|= "metrics.go"
| logfmt
| unwrap bytes_processed [1m])
)
This calculates the amount of bytes processed per organization ID.
Built-in aggregation operators
Like PromQL, LogQL supports a subset of built-in aggregation operators that can be used to aggregate the element of a single vector, resulting in a new vector of fewer elements but with aggregated values:
sum: Calculate sum over labelsavg: Calculate the average over labelsmin: Select minimum over labelsmax: Select maximum over labelsstddev: Calculate the population standard deviation over labelsstdvar: Calculate the population standard variance over labelscount: Count number of elements in the vectortopk: Select largest k elements by sample valuebottomk: Select smallest k elements by sample value
The aggregation operators can either be used to aggregate over all label values or a set of distinct label values by including a without or a by clause:
<aggr-op>([parameter,] <vector expression>) [without|by (<label list>)]
parameter is required when using topk and bottomk. topk and bottomk are different from other aggregators in that a subset of the input samples, including the original labels, are returned in the result vector.
by and without are only used to group the input vector. The without clause removes the listed labels from the resulting vector, keeping all others. The by clause does the opposite, dropping labels that are not listed in the clause, even if their label values are identical between all elements of the vector.
Vector aggregation examples
Get the top 10 applications by the highest log throughput:
topk(10,sum(rate({region="us-east1"}[5m])) by (name))
Get the count of log lines for the last five minutes for a specified job, grouping by level:
sum(count_over_time({job="mysql"}[5m])) by (level)
Get the rate of HTTP GET requests to the /home endpoint for NGINX logs by region:
avg(rate(({job="nginx"} |= "GET" | json | path="/home")[10s])) by (region)
-
Log queries
- Log stream selector
-
Log pipeline
- Line filter expression
- Label filter expression
- Parser expression
- JSON
- logfmt
- Pattern
- Regular expression
- unpack
- Line format expression
- Labels format expression
-
Log queries examples
- Multiple filtering
- Multiple parsers
- Formatting
-
Metric queries
-
Range Vector aggregation
- Log range aggregations
- Unwrapped range aggregations
- Unwrapped examples
-
Built-in aggregation operators
- Vector aggregation examples
-