Day 77 : Grafana Alerting

Whether you're new to Grafana Alerting or have some experience, discover the basics and features that can assist you in making, handling, and reacting to alerts. This will enhance your team's capability to quickly address and solve problems.

Grafana Alerting allows you to alert on your metrics and logs, no matter where they are stored. Create, manage, and take action on your alerts in a single, consolidated view, and improve your team’s ability to identify and resolve issues quickly.

Easily manage your alerts: No need to look at multiple pages for managing alerts in Grafana. A single Grafana Alerting page consolidates both Grafana-managed alerts and alerts that reside in your Prometheus-compatible data source in one single place.
One alert rule, multiple alert instances: Grafana Alerting can give you system-wide visibility with a single multi-dimensional alert. One alert rule can alert on many items at once, creating one alert instance for each entity that needs your attention. You can also group alert instances based on labels to not receive too many notifications!

Alerting rules are defined in Grafana and are evaluated in the Grafana backend.
Labels are extracted from the metric data and added to the alert instance.Also match alert instances to notification policies and silences and can be used to group your alerts by severity.
Severity=warning is added to the alert instance.
Notification Policy is evaluated and the alert instance is sent to the Slack channel or Email.It is the set of rules for where, when, and how the alerts get routed. Notification policies have a tree structure, where each policy can also match specific alert labels.
Contact Points define how your contacts are notified when an alert fires. We support a multitude of ChatOps tools to ensure the alerts come to your team not the other way around.

Alerts are sent to the alert receiver where they are routed, grouped, inhibited, silenced and notified. In Grafana Alerting, the default alert receiver is the Alertmanager embedded inside Grafana, and is referred to as the Grafana Alertmanager.

Before building the Grafana Alerting, we need to install the Grafana in our system or on cloud.
For detailed installation steps, please visit my Day 73 Blog for complete steps on how to Install Grafana on AWS EC2.
Blog Link : Grafana Installation on AWS EC2 Instance
After installing Grafana, we need to configure Grafana Loki and Promtail to collect the logs from the system which I have already explained in my Day 75 Blog.
Blog Link : Grafana Loki and Promtail Setup

Task-01

Step-02: So in Alerting section, we have Alert Rules and add the data source which we have already configured in our previous blog Loki-1. So we will be using the same data source.

Step-03: Now we will create a new alert rule for Error Alerting. So click on New Alert Rule.We can also add other Alert Rules like Warning and Critical.

Step-04: Now we will define the Query for the Error Alerting. So we will use the same query which we have used in our previous blog Loki-1.

Step-05: Here we will Set the Evaluation Behavior in which we will store the Rule State and we will also set the Evaluation Delay to 5 minute.

Step-06: We will keep the Annotations and Notification as default and then click on Save.

Thus we have successfully created the Error Alerting in Grafana which will alert us if there is any error in the system.

So we will be getting the Error Alerting report in every 5 minutes as we have set the Evaluation Delay to 5 minutes.