Setting up Monitoring
|
What can we help you find? |
Random | ||||||||||||||||
| Home | Testing | Automation | AI | IoT | OS | RF | Data | Development | References | Tools | Wisdom | Inspiration | Investing | History | Fun | POC | Help |
The following steps were used to set up a monitoring solution.
Prepare a base VM
Let's begin by making a new VM based on the VM_Deb12_Master_v02
Install Prometheus
Ensure apt is up to date
sudo apt update
Install Prometheus
sudo apt install prometheus
Verify Prometheus is installed
systemctl status prometheus
Configuring Prometheus
- By default, Prometheus uses a basic configuration.
- You can customize the configuration by editing the Prometheus configuration file located at /etc/prometheus/prometheus.yml.
- Add your targets (endpoints to monitor) and other settings as needed.
cd /etc/etc/prometheus
Remember to run nano with sudo so you can write this file.
sudo nano prometheus.yml
Start Prometheus
sudo systemctl start prometheus
Stop Prometheus
sudo systemctl stop prometheus
Restarting Prometheus
sudo systemctl restart prometheus
Accessing Prometheus
- Prometheus runs as a service and listens on port 9090 by default.
- Open a web browser and navigate to http://192.168.2.243:9090
Getting Started with Prometheus
https://prometheus.io/docs/prometheus/latest/getting_started/
Install Prometheus - Blackbox Exporter
Ensure apt is up to date
sudo apt update
Install Prometheus - Blackbox Exporter
sudo apt install prometheus-blackbox-exporter
Configure Blackbox Exporter with permissions to perform a ping
sudo setcap cap_net_raw+ep /usr/bin/prometheus-blackbox-exporter
Configure Blackbox Exporter
cd /etc/prometheus
sudo nano blackbox.yml
(At the moment I don't think I need to do anything in here)
Launch Blackbox Exporter
sudo systemctl start prometheus-blackbox-exporter
Enable Blackbox Exporter to start on boot
sudo systemctl enable prometheus-blackbox-exporter
Install Prometheus Pushgateway
Create a directory for prometheus pushgateway
sudo mkdir /var/lib/prometheus-pushgateway
Create the Pushgateway config directories
for i in rules rules.d files_sd; do sudo mkdir -p /etc/prometheus-pushgateway/${i} done
Create a temporary directory for Pushgateway download
sudo mkdir -p /tmp/prometheus-pushgateway
cd /tmp/prometheus-pushgateway
Download the latest Prometheus Pushgateway binary package:
Extract the downloaded file
sudo tar xvf pushgateway*.tar.gz
Relocate the binary files extracted
cd pushgateway*/
sudo mv pushgateway /usr/local/bin/
Create a service unit file for Prometheus Pushgateway
sudo nano /etc/systemd/system/prometheus-pushgateway.service
Add the following code (using Nano)
[Unit]
Description=Prometheus Pushgateway
After=network.target
[Service]
User=prometheus
Group=prometheus
ExecStart=/usr/local/bin/pushgateway --web.listen-address=:9091
Restart=always
[Install]
WantedBy=multi-user.target
Reload systemd to pick up the new service unit
sudo systemctl daemon-reload
Enable to start on boot
sudo systemctl enable prometheus-pushgateway
Start the service
sudo systemctl start prometheus-pushgateway
Verify the service is running
sudo systemctl status prometheus-pushgateway
Access Prometheus Pushgateway Dashboard via Web Interface
Install Go
sudo apt update
sudo apt upgrade -y
sudo apt install golang-go -y
Verify Go is installed
go version
Add Go to the path
TBD
Install NodeJS
sudo apt update
sudo apt upgrade -y
sudo apt install nodejs npm -y
Check NodeJS installed
nodejs --version
Check that Node Package Manager is Installed
npm --version
Install PromLens
Clone Repo
Change directory to Downloads folder
cd Downloads
wget https://github.com/prometheus/promlens/releases/download/v0.3.0/promlens-0.3.0.linux-amd64.tar.gz
tar -xvzf promlens-0.3.0.linux-amd64.tar.gz
chmod +x promlens-0.3.0.linux-amd64
mv promlens-0.3.0.linux-amd64 promlens-linux-adm64
sudo mv promlens-linux-amd64 /usr/local/bin/promlens
Create a service unit file for PromLens
sudo nano /etc/systemd/system/promlens.service
Add the following code to this file
[Unit]
Description=PromLens Service
After=network.target
[Service]
ExecStart=/usr/local/bin/promlens/promlens
WorkingDirectory=/usr/local/bin/promlens
User=gregpaskal
Restart=always
[Install]
WantedBy=multi-user.target
Reload systemd to pick up the new service
sudo systemctl enable promlens.service
Enable the service to start on boot
sudo systemctl enable promlens.service
Start the service
sudo systemctl start promlens.service
verify the service is running
sudo systemctl status promlens.service
Install Grafana
Ensure apt is up to date
sudo apt update
Add Grafana APT Repository
Import the GPG key to allow installation of signed Grafana packages:
sudo mkdir -p /etc/apt/keyrings/
wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null
Choose the stable repository
echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
Update the package list
sudo apt-get update
Install Grafana using apt
sudo apt-get install grafana
Start Grafana
sudo systemctl start grafana-server
Configure Grafana to start on boot
sudo systemctl enable grafana-server.service
Check if Grafana is enabled
sudo systemctl status grafana-server
Access Grafana
Setup a username and password for Grafana
When you install Grafana, it comes with a default administrator account. To set up a new username and password, follow these steps:
Enter the default username: admin and password: admin
Enter a new password: (Whatever you choose)
Create a new user
Install Apache 2
Check if Apache is already installed
apachectl -v
Update the package list
sudo apt update
sudo apt upgrade
Install Apache2
sudo apt install apache2
sudo systemctl status apache2
apachectl -v
sudo systemctl is-enabled apache2
Install Dokuwiki
Download Docuwiki
cd /var/www/html
sudo wget https://download.dokuwiki.org/src/dokuwiki/dokuwiki-stable.tgz
Extract
sudo tar xzvf dokuwiki-stable.tgz -C /var/www/html/
Rename folder
sudo mv dokuwiki-*/ dokuwiki/
Set proper permissions
sudo chown -R www-data:www-data /var/www/html/dokuwiki/
Setup Docuwiki
Install Cockpit
Cockpit monitors local system resources.
Update the package list
sudo apt-get update
Install Cockpit
sudo apt install cockpit
Setup Cockpit to use a specific port
sudo mkdir -p /etc/systemd/system/cockpit.socket.d/
sudo nano /etc/systemd/system/cockpit.socket.d/listen.conf
Enter the following in nano
[Socket]
ListenStream=
ListenStream=1111
Save and exit Nano
Reload the system daemon
sudo systemctl daemon-reload
Start Cockpit
sudo systemctl start cockpit.socket
Verify Cockpit is installed correctly
sudo systemctl status cockpit
Go to Cockpit
Setup a ping to monitor using Prometheus and Grafana
Configure Prometheus for ping
cd /etc/prometheus/
sudo nano prometheus.yml
Add the following code as a scrape_config
- job_name: 'website-ping-gregpaskal.com'
metrics_path: /probe
params:
module: [icmp]
static_configs:
- targets:
- gregpaskal.com # Replace with the actual website URL
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: localhost:9115 # Blackbox exporter address
Client Dashboards
Grafana is an excellent solution for creating separate dashboards for different clients while managing user access and permissions.
Here’s how Grafana can address your requirements:
Multiple Organizations:
- Grafana allows you to create multiple organizations, each acting as a separate workspace.
- Each organization can have its own set of users, dashboards, and data sources.
- This separation ensures that clients’ data and configurations remain isolated.
User Management:
- Grafana provides robust user management features:
- User Accounts: You can create individual user accounts for each client.
- Roles and Permissions: Assign roles (e.g., Viewer, Editor, Admin) to control what each user can access.
- LDAP/AD Integration: Grafana can integrate with LDAP or Active Directory for centralized user management.
- SAML Authentication: Use SAML for single sign-on (SSO) and secure authentication.
Dashboard Access Control:
- Grafana allows you to set permissions at the dashboard level:
- Folder Permissions: Control access to specific folders containing dashboards.
- Team Permissions: Assign teams to dashboards and manage access for those teams.
- Data Source Permissions: Restrict access to specific data sources per dashboard.
Customization:
- Grafana lets you customize each client’s dashboards:
- Variables: Create client-specific variables (e.g., for filtering data by client ID).
- Annotations: Add annotations to highlight events or specific data points relevant to each client.
White-Labeling:
- Grafana Enterprise offers white-labeling options:
- Customize the Grafana interface with your clients’ branding (logo, colors, etc.).
In summary, Grafana’s multi-organization support, user management, and fine-grained access control make it an ideal choice for creating separate dashboards tailored to different clients’ needs. 🌐📊🔒