rbed

IT operations

Every minute, a web service reports its error count and its CPU load. A further Channel reports deployments; it is a Pulse Channel, so its value exists only in the Sample that carries it, here at minute 10. An error spike soon after a deployment is probably caused by it.

The live chart and its controls need JavaScript. The tables show the Events at the default parameters.

The site cannot run the default parameters: (JavaScript) TypeError: Cannot read properties of undefined (reading 'length')

import static io.github.romanhaendler.rbed.Conditions.*;

import io.github.romanhaendler.rbed.*;
import java.time.Duration;
import java.util.HashMap;
import java.util.Map;

Channel cpu = Channel.named("cpu");
Channel deploy = Channel.pulse("deploy");
Channel errors = Channel.named("errors");

Rule deployment = Rule.named("Deployment").when(deploy.above(0.5));
Rule errorSpike = Rule.named("ErrorSpike")
    .when(errors.above(5)).forDuration(Duration.ofMinutes(2));
Rule regression = Rule.named("Regression")
    .when(after(deployment).within(Duration.ofMinutes(30)).and(errors.above(5)))
    .forDuration(Duration.ofMinutes(2));
Rule highCpu = Rule.named("HighCpu")
    .when(cpu.above(90)).forDuration(Duration.ofMinutes(5));
Detector detector = Detector.builder()
    .rules(deployment, errorSpike, regression, highCpu)
    .build();

for (int m = 0; m < 120; m++) {   // every minute
  Map<Channel, Double> values = new HashMap<>();
  values.put(errors, m >= 25 && m < 35 ? 12.0 : m >= 80 && m < 85 ? 9.0 : 1.0);
  values.put(cpu, m >= 90 && m < 100 ? 95.0 : 50.0);
  if (m == 10) values.put(deploy, 1.0);   // a deployment, in this Sample only
  detector.push(m * 60_000L, values);
}
detector.close();

The Rules

  • Deployment: a deployment is reported. It Ends with the next Sample, which carries no deployment.
  • ErrorSpike: more than 5 errors per minute for 2 minutes.
  • Regression: more than 5 errors per minute for 2 minutes, all within 30 minutes after a Deployment Ended.
  • HighCpu: the CPU above 90 % for 5 minutes.

At the defaults, the deployment at minute 10 is a Deployment from minute 10 to 11. The errors are above 5 per minute from minute 25 to 35 and from 80 to 85, an ErrorSpike each, Confirmed after 2 minutes. The spike at minute 25 starts 14 minutes after the Deployment Ended, so it is a Regression as well; the one at minute 80 is too long after it. The CPU is above 90 % from minute 90 to 100, and HighCpu is Confirmed at minute 95.

Things to try

  • Narrow Regression within to 15 minutes: the spike at minute 25 starts inside the window, but its 2 minutes are not reached before the window closes at minute 26, so the Pending Regression is Discarded.

Reference

The example is DeploymentRegressionExample, a test with its expected Events, listed in the README's Examples.

Methods used: Channel.named, Channel.pulse, Rule.named, Rule.when, Expression.above, Rule.forDuration, Conditions.after, After.within, Condition.and, Detector.builder, Detector.Builder.rules, Detector.Builder.build, Detector.push, Detector.close.

The concepts at work, each on a page of its own: Pulse Channels, After another Event.