I wrote an article back in April titled SFS DevOps 2: Kubernetes, Docker, Selenium, Jenkins, Python, and Prometheus. This is a long article with a basic project setup. Although I definitely recommend combing through it to help your understanding and portfolio, I decided to write a smaller spin off tutorial to serve as an intro to monitoring with Prometheus.If you haven't heard it by now, monitoring is critical for any software development position, especially DevOps. Employers LOVE someone that can catch issues before their clients.I hope this intro serves as a brief and helpful walk through to get Prometheus up and running on your local machine.
What do we need?
An ExporterWe are going to need an application that has an exporter. An exporter is what we tie into our application to present application metrics so that Prometheus can scoop them up and interpret them. In our case, we will have a standard falcon http app with a falcon exporter library included.
A Kube ClusterSince we are practicing and really don't need anything special, we are going to use minikube on our local machine. We will be applying permissions yaml files for cluster roles, cluster role bindings, and a service account. Don't worry if that sounds confusing...this is going to be a copy/paste job almost directly from kubernetes docs.
ConfigMaps with Prometheus/Alertmanager ConfigsWe will take the Prometheus configs and tie them to configmaps in our cluster. Although you don't see why now, this will be beneficial for rapid deployments when we don't want to rebuild an image every time.AND THAT'S IT.Now, before we begin here is what my working directory looks like at the end. If you'd like to show a repo off to employers, try this setup (folders in bold):├── configs│ ├── alertmanager.yml│ ├── prometheus.yml│ └── rules.yml├── deployment.yml└── permissions└── permissions.yml
The Falcon App with Exporter:There are really two ways for you to go about doing this section.A.) The easy way. You can pull scottyfullstack/hello from my docker. Its a falcon app that when hit the route /metrics will show the response from the falcon exporter.B.) Make your own app below and create your own docker image.
- First you will need to make sure you pip install (for docker put these in a requirements.txt file:
- Create hello.pyAnd thats all there is to the falcon app. Notice the Prometheus Middleware used with the /metrics route.
The Minikube Cluster & Permissions:If you are new to kubernetes and have not downloaded minikube or kubectl yet, navigate here and follow the quick installation instructions.Fire up minikube withThis will take a few minutes. Once your cluster starts, check it out withCool, your kube cluster is up and running on your laptop. Let's now deploy the falcon app there and expose it (then use minikube to open it in browser), as well as deploying our permissions.check the status of the pod with:once its up run the following to open it in your browser:Then I will add my permissions file that includes three yamls in one (clusterRole, clusterRoleBinding, serviceAccount). For more information, check out the official kubernetes docs.Lastly for this section, let's deploy our basic Prometheus deployment with their official image.Awesome, you should be able to see the basic Prometheus Dashboard now.In the menu, navigate to Status > Targets to see your Prometheus instance monitoring itself.Let's wrap this up and start reporting.
Prometheus/Alertmanager Configs and Enhancing the Deployment YamlNow that we have Prometheus standing we need to create a configmap for Prometheus that our deployment will pick up each time. If you haven't seen the prometheus docs, check them out. There is a prometheus.yml file that comes standard that tells prometheus what to monitor, how to alert, and at what interval. The basic template can be found here.This is what our prometheus.yml (inside my configs folder) will look like:Note: In the 'hello' targets section at the bottom, you will need to include the output of (WITHOUT http://):!!! IMPORTANT !!! Make sure you keep the yaml tabbing/spacing...Otherwise your deployment will fail (VSCode gave me issues with this, if all else fails use vim).Next lets create an alertmanager config in our configs folder. I've configured my configuration for my slack in the #alerts channel. To set this up check out this link and add Incoming Webhooks to slack. Once you have your webhook url add it to slack_api_url below :Finally, create the rules.yml file for our rules volume:Now create the configmaps from those files:While we are at it, lets create the rules config too (found in the prom docs)Now, lets grab the prometheus deployment yaml and add to it. This is a great way to SHOW what you know, by the way.Compare this to your file and add what's needed.There is a lot going on here, so note the Volumes, the new alertmanager image, and the Service account added.When done run:You should now have all of your services, pods, and configs in your minikube cluster. Let's try it out by bringing down our hello app.after the service is down for 1 minute, you should receive a critical alert for hello deployment in your slack channel. Also, navigate toand check the targets dropdown item. You should see the failure in your prometheus admin panel.And that's how to get started with Prometheus. As you can imagine, there is so much more to do from here! Check back for more!See you next time.
Disclosure: scottyfullstack.com is a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to amazon.com.