Day 75: Grafana Loki Hands-on Tutorial

📊 What is Grafana, and why is it used for monitoring?

• Grafana is an 📈 open-source analytics and monitoring platform that allows users to visualize, query, and analyze data from various data sources. It's commonly used for monitoring because it provides a user-friendly interface for creating interactive dashboards and alerts, making it easier to track the performance of systems and applications.

Why Grafana?

• Grafana is the top choice for monitoring and data visualization, and here's why: it's open-source and packed with powerful features.

• It seamlessly connects to different data sources and provides user-friendly dashboards for making data easy to understand.

• Grafana is particularly great with data that changes over time, making it ideal for tracking evolving metrics.

• It also has strong alerting capabilities to catch issues early and a robust plugin system that lets you customize and integrate with various tools.

• With Grafana, exploring historical data is a breeze, allowing for deep analysis. It's a versatile tool that encourages teamwork and can be tailored to your specific needs, making it perfect for teams and organizations looking for effective monitoring and observability solutions.

Grafana Installation

• For detailed installation steps, please visit my Day 73 Blog for complete steps on how to install Grafana on AWS EC2.

• Blog Link: Grafana Installation on AWS EC2 Instance

• As we have already installed Grafana on AWS EC2, we will now move on to the next step.

What is Grafana Loki?

• Grafana Loki is a horizontally -scalable,highly-available, multi-tenant log aggregation system inspired by Prometheus. It is designed to be very cost-effective and easy to operate.

• Loki is built around the idea of only indexing labels for logs and leaving the original logs messages unindexed. This means that Loki is cheaper to operate and can be order of magnitude more efficient.

• The Loki project was started at Grafana Labs in 2018. Loki is released under the AGPLv3 license.

How does Grafana Loki work?

• Pull in any logs with Promtail: Promtail is a logs collector built specifically for Loki. It uses the same service discovery as Prometheus and includes analogous features for labeling, transforming, and filtering logs before ingestion into Loki.

• Store the logs in Loki: Loki does not index the text of logs. Instead, entries are grouped into streams and indexed with labels.Not only does this reduce costs, it also means log lines are available to query within milliseconds of being received by Loki.

• Use LogQL to explore: Use Loki’s powerful query language, LogQL, to explore your logs. Run LogQL queries directly within Grafana to visualize your logs alongside other data sources, or with LogCLI, for those who prefer a command-line experience.

• Alert on your logs: Set up alerting rules for Loki to evaluate your incoming log data. Configure Loki to send the resulting alerts to a Prometheus Alert-manager so they can then get routed to the right team.

What is Promtail?

• Promtail is an agent that ships the contents of local logs to a private Grafana Loki instance or Grafana Cloud. It is usually deployed to every machine that runs applications that need to be monitored.

1. It primarily:

• Discovers targets

• Attaches labels to log streams

• Pushes them to the Loki instance.

Now let's install Grafana Loki on the AWS EC2 Instance.

• But before that, we need to install Docker on our EC2 Instance because Grafana Loki is a Dockerized Application.

Installing Docker on AWS EC2 Instance.

• Step-01: Update the instance.

sudo apt update
sudo apt-get install docker.io

• Step-02: Start the Docker Service.

sudo systemctl start docker
sudo systemctl enable docker
sudo systemctl status docker

• Step-03: Giving Docker Privileges to the User.

sudo usermod -aG docker $USER

• Step-04: Now we will install Grafana Loki on our EC2 Instance in a folder names grafana-configs

mkdir grafana-configs
cd grafana-configs

• Step-05: Now we will Download the Grafana Loki Configuration File inside the grafana-configs folder.

wget https://raw.githubusercontent.com/grafana/loki/v2.8.0/cmd/loki/loki-local-config.yaml -O loki-config.yaml

• Step-06: Now we will run the Grafana Loki Docker Container which will run on port 3100 inside the grafana-configs folder.

docker run -d --name loki -v $(pwd):/mnt/config -p 3100:3100 grafana/loki:2.8.0 --config.file=/mnt/config/loki-config.yaml

• Step-07: Now we will check the status of the Docker Container.

docker ps

• Step-08: Now we will check the logs of the Docker Container. So go to EC2 instance and in Security Group, add a new rule for port 3100 Selecting ipv4 or My IP and then open the port in the browser. You will see the logs of the Docker Container.

• Step-09: Open the browser and enter the public IP of the instance with the port number 3100 and then click on the enter.

<Public-ip-address>:3100/ready

• Step-10: To see the metrics of the Grafana Loki, enter the below command.

<Public-ip-address>:3100/metrics

Now we will install Promtail on our EC2 Instance.

• Step-01: Download the Promtail Configuration File inside the grafana-configs folder.

wget https://raw.githubusercontent.com/grafana/loki/v2.8.0/clients/cmd/promtail/promtail-docker-config.yaml -O promtail-config.yaml

• Step-02: Now we will run the Promtail Docker Container inside the grafana-configs folder.

docker run -d --name promtail -v $(pwd):/mnt/config -v /var/log:/var/log --link loki grafana/promtail:2.8.0 --config.file=/mnt/config/promtail-config.yaml

• Step-03: Now we will check the status of the Docker Container.

docker ps

Now we will add the Grafana Loki Data Source to Grafana.

• Step-01: Open the Grafana Dashboard and then click on Configuration and then click on the Data Sources.

• Step-02: Now click on the Add Data Source and then select Loki and do the connection settings of URLs as Localhost (Because we yaml file has IP-address as localhost) and then click on Save & Test.

• Rest all the settings will be the default.

• Step-03: Now we will explore data in Grafana Loki which is at the top right corner of the Grafana Dashboard.

• Step-04: Now in the query section, we will enter the below query.

• For Example here I have job = varlogs and then click on the Run Query for the last hour.

Let's see how can we create a new log stream in Grafana Loki.

• Step-01: For example, we want logs from "grafana.logs"

• Step-02: So copy the path of grafana.log.

pwd

• Step-03: Now we will add the path in the promtail-config.yaml file.

sudo vim promtail-config.yaml

• Step-04: Now we will restart the Promtail Docker Container.

docker restart <promtail-container-id>

• Step-05: Now in the query section, we will enter the below query.

• For Example, here I have job = grafanalogs and then click on the Run Query for the last hour.

• Step-06: Now Add this Query to the Dashboard.

• Step-07: You can add more Visualization to the Dashboard.

Now Let's install Nginx on our EC2 Instance and add this logs to the Grafana Loki Dashboard.

• Step-01: Install Nginx on EC2 Instance.

sudo apt-get install nginx -y

• Step-02: Click on Add Visualizations and then select the varlogs and then click on the Operation and the select Rate Count and then click on the Apply.

• Thus it shows how many times Nginx is called in the last hour.

• Step-03: Now we will add these Nginx logs to Visualization.

• Step-04: Now if we want logs from Docker. So do the similar steps as we did for Nginx. Thus we have Docker logs for Visualization.

• Step-05: See the Dashboard of Grafana Loki which have varlogs, Nginx and Docker logs.

Thus we have successfully installed Grafana Loki on AWS EC2 Instance and added Nginx and Docker logs to Grafana Loki Dashboard.