This post will walk through the basisc of getting Prometheus installed and configured on an Azure Virtual Machine (VM).
Create a new VM
Add a custom domain
Connect to VM and install Prometheus
Login to your Azure Portal
Go to portal.azure.com and click the Virtual Machines option in the left “Favorites” navigation bar.
Then click the “Create virtual machine” button.
Note: If you already have a Virtual Machine (VM) running, press the (+) to add a new one.
Create a virtual machine
From the next screen you’ll see a row of configuration options to choose from (Basics, Disks, Networking, etc).
Let’s go through each of those now.
There are a number of fields we will need to enter information in to on this screen.
Choose the subscription you’d like your VM to associated with, as well as the related Resource Group.
The name of the virtual machine can be anything you’d like. I’m naming mine Prometheus in this example.
I’m going to use the latest supported version of Ubuntu for my Operating System (image).
I chose the EAST US region. Not all regions offer the same size instances so choose the one that makes the most sense for you.
The administrator account I’m creating for this VM is jhand (you can use whatever makes sense for you here).
I’ll be using an SSH public/private key combo to remotely access the VM from my local laptop. If you aren’t familiar with the process of generating your own SSH keys for authentication in to a remote server, check this post out and then return here to continue.
The last configuration option (Inbound Port Rules) is where allow traffic to our virtual machine on an inbound port.
Prometheus uses port 9090 to serve it’s dashboard up in a browser (as opposed to the standard port 80).
Thus we need to allow traffic on that port in order to view Prometheus from a browser on our machine (or any machine for that matter).
However, from the dashboard we are only given the option to open a few standard ports (80, 443,22, and 3389).
So, we’ll skip this step for now and I’ll show you how to open port 9090 once the virtual machine isup and running.
For now, leave this option as “None”, and continue with the configuration by clicking the button near the bottom that says “Next:Disks >”.
In this part of the configuration, I’m going to stick with the defaults. You have the option of choosing several different OS disk types.
If you aren’t sure which disk option to choose from, check out this document to understand the differences of each type.
I recommend leaving the “Use unmanaged disks” option as the default of No.
There is no need for additional data disks at this time, so we can leave the last option alone.
Press the “Next:Networking >” button to proceed.
You can choose an existing Virtual network if you’ve already set a previous one up.
You can leave the default (new suggested network name) if you’d like.
However, I’m going to create a new one and name it Prometheus-vnet.
These are the only changes I’m making on this screen. Everything else is left as the default.
Press the “Ok” button and then proceed to the next configuration option by pressing the “Next:Management >” button.
From the management configuration screen, the only thing I’m going to do is create a new “Diagnostic storage account”. I could accept the default but I want to create one with a name I can associate to Prometheus a little easier.
By clicking the “Create New” I was able to name the storage account prometheusvnetstorage. Pro tip: This field only accepts numbers and letters. I originally had hyphens in the name but that isn’t allowed.
Once you’ve given your storage account a name, press the Ok button followed by the button that says “Next:Guest config >”
There are no changes to be made on this screen so let’s move on to the next option by clicking the button marked “Next:Tags >”.
The same for the tags configuration. We don’t need to make any changes here. Tags can be added later.
Let’s move on to the last step by pressing the button marked “Next:Review + create”.
After a few moments, you’ll see a summary of the configuration you agreed to including an estimated cost of the virtual machine. Give it a quick review and then press the “Create” button at the bottom.
Now, your virtual machine will begin building and configuring itself.
Once the deployment is complete you’ll see a button marked “Go to resource”. Clicking this will take us straight to more information and configuration options for our virtual machine.
From here we’ll make a few more adjustments including opening up port 9090 that we need in order to access Prometheus from a web browser.
Our virtual machine was assigned a public IP address. However, I’m going to be shutting down my instance fairly regularly at which the VM will be assigned a new public IP.
Each time I access my VM from an SSH session I will need to know the address of the instance. Since the IP address will often change, I’m going to assign the VM a “DNS Name Label”. That way I can use something easier to read and remember and no matter what IP address my instance has the DNS name will always resolve to it.
By clicking on the IP address, you’ll then have an option to change a few things with regards to the IP address.
Because we are adding a DNS entry to resolve dynamically changing IP addresses, I’m only going to make a change to the “DNS name label” field.
Here I will give the instance the domain name of prometheus-dev
Once you have made the changes, press the “Save” button near the top of the configuration screen.
This change allows me to access the instance using the domain prometheus-dev.eastus.cloudapp.azure.com
This will be useful for both accessing the instance from an SSH connection as well as through the web browser on port 9090 regardless of the public IP address of my virtual machine.
We can confirm resolution of the IP address by doing a simple ping from a terminal session on our local machine.
From iTerm I’ll ping the full domain name
We can see that the name resolves to the IP address displayed above (18.104.22.168).
We can also see that our virtual machine isn’t responding to the ping request. Don’t worry. That’s a good thing. If we wanted our instance to respond to ping requests, we will have to manually change the configuration to allow it, but we have no use for it.
Create Virtual Network
In order to remotely access our instance we need to allow inbound traffic on a few specific ports. In order to configure this, first let’s return to the “Prometheus” Virtual Machine’s configuration screen.
I typically return by clicking the “All Resources” link on the left navigation and then clicking on the Virtual Machine with the name Prometheus to return to this screen. Additionally, you can pin this screen to your dashboard by clicking the pin icon in the upper right hand corner of the screen.
To begin opening a port, click on the “Networking” link in your VM’s navigation bar from the “Settings” category.
From the network interface configuration screen, we only want to create a few additional inbound port rules so click the button marked “Add inbound port rule”.
The only fields to change on the new security rule screen is the “Destination port ranges” and the “Name”.
We are opening up port 22 and giving it a name in our configuration of Port_22.
NOTE: We typically do NOT want to allow port 22 to remain open on our server to avoid any security breaches. You would like want to allow only specific IP adddresses or domains to be able to access your instances on that port. A VPN in to the VM would be even better. For the purposes of this demonstration I’m going to keep it simple knowing that I will be shutting this instance down later today.
Continue by pressing the “Add” button at the bottom.
You’ll be returned to the previous screen where we can see our newest inbound rule.
Now, repeat the process and add port 9090.
The next thing we will do is check to see if we can access our server through a SSH connection on port 22.
Remote Connection to Virtual Machine
One of the very early configurations we made for our vm was to create an admin account. I named mine jhand.
This username, along with the custom DNS that I created will be used to remotely connect to the server using the following (from my iTerm app):
ssh [email protected]
You will be prompted to confirm the connection. You can type ‘yes’ or just press enter. Your remote virtual machine will then compare the private key information we generated early (stored on your machine) to the information saved on the server during our intial configuration steps.
As long as those keys match, the connection will establish itself and you’ll be taken to the command line of your remote Azure virtual machine.
That’s it. Now you can access your virtual machine quickly and safely without the need of a password simply by running the ssh command.
Now we are ready to actually download and install Prometheus to our instance.
You’ll want to use the most recent version of Prometheus available. AT the time of this writing it is version 2.4.2. The link below reflects that version, but you should use the most recent version available to you at the time you read this.
Bein by connecting to your remote virtual machine using the ssh command previously mentioned.
Once you have logged in and are at the prompt, run the following:
Now that we have downloaded the most recent version of Prometheus, we will need to unpack the contents with the following command:
tar xvfz prometheus-*.tar.gz
Next, let’s change directories (cd) to the new one that contains our files.
If we take a look at the contents of this directory, we can see two executables (prometheus and promtool) along with a YML file (promethes.yml) used to configure Prometheus, two additional directories (console_libraries and consoles) as well as LICENSE and NOTICE text files.
It is the executable named prometheus that we need to invoke to start our service. To do so, we need to pass a few flags along with the execution command. The following will start Prometheus using the prometheus.yml as it’s configuration file as well as enable a flag that allows us to restart Prometheus will a simple CURL command rather than a hard kill of the process using
./prometheus --config.file=prometheus.yml --web.enable-lifecycle
Give it a shot.
You should see the following if Prometheus is running on your virtual machine.
Last, we can confirm that Prometheus is running by accessing it in a web browser using the custom DNS we created along with specifying the port (9090).
prometheus-dev.eastus.cloudapp.azure.com is the DNS for my instance so if I browse to: http://prometheus-dev.eastus.cloudapp.azure.com:9090 I should see the following.
Next we will work on getting data in to Prometheus using the configuration file (prometheus.yml) as well as a few other methods to collect data. We will also go over reloading the configuration file after a change has been made to it without shutting the Prometheus service all the way down.
If you have any questions about any of this reach out to me on Twitter.