"strict mode" ansible

Weird queries

weighted cost

      - record: diskuse:node_filesystem_avail_bytes:predict10m_1h
        expr: predict_linear(node_filesystem_avail_bytes[10m], 1 * 3600)
      - record: diskuse:node_filesystem_avail_bytes:predict30m_1h
        expr: predict_linear(node_filesystem_avail_bytes[30m], 1 * 3600)
      - record: diskuse:node_filesystem_avail_bytes:predict60m_1h
        expr: predict_linear(node_filesystem_avail_bytes[60m], 1 * 3600)
      - record: diskuse:node_filesystem_avail_bytes:weighted_1h
        # Each of the predictions is weighted differently; it is important that the sum of the fractions
        # exactly equals one.
        expr: >-
          diskuse:node_filesystem_avail_bytes:predict60m_1h / 6
          + diskuse:node_filesystem_avail_bytes:predict30m_1h / 3
          + diskuse:node_filesystem_avail_bytes:predict10m_1h / 2

usually we have a usage graph like this one ~~~~~~~~~~~~ 60m predict, too slow

or like this \ ----

10m predict, too aggressive

Graph

symmetric difference

# We use `up` as a proxy for inventory_status="InService", as `prodconfig_host_isolate_cores`
# does not carry the inventory_status label.
# We only have data on each exporter when there's an isolated core, so we need
# to do a symmetric difference on the data sets:
# - From the list of isolated cores in prodconfig, remove the list of isolated cores in node_exporter
# - From the list of isolated cores in node_exporter, remove the list of isolated cores in prodconfig
# If there's any output, there's a mismatch between intent and truth
# Then we `+ on() group_left()` to re-add the labels
expr: >
  (
    (
      group by(inventory_name) (
        up and on(inventory_name)
          (
           (prodconfig_host_isolate_cores unless on(inventory_name) node_cpu_isolated)
           or
           (node_cpu_isolated unless on(inventory_name) prodconfig_host_isolate_cores)
          )
      )
    )
    + on(inventory_name) group_left(inventory_pool, job, inventory_infra_environment) up{job="node"}
  )