As I observe the current job market (yes even with COVID-19 lurking in my area), I have noticed that several tools keep showing up in the needs of hiring companies. The job post below from Glassdoor is a fantastic example of the most critically needed skills today.• DevOps Development Frameworks: Experience working with multiple development frameworks and methodologies with the ability to identify opportunities for innovation in development teams and address identified problems. Experience working in SCRUM and AGILE environments highly desired. Experience leading development teams highly desired.Note: Check boxes show what we cover in this tutorial.DevOps Engineering Technical Skills:
- Source Code Version Control (e.g. Bitbucket, Git, Github, Subversion)✔
- Source Code Review/Quality (e.g. Coverity, Fortify, Crucible, Fisheye)✖
- Software Build Tools (e.g. Gradle, Apache Ant, Maven)✔
- Automation server (e.g. Bamboo, Jenkins)✔
- Automation testing (e.g. Selenium, Unit)✔
- Software Containerization (e.g. Docker)✔
- Container Orchestration (e.g. Kubernetes)✔
- Configuration Management and Deployment (e.g. Ansible, Puppet, Chef, SaltStack)✖
- Binary Artifact Repository (e.g. Artifactory, Nexus)✖
- Issue and Project Tracking (e.g. Jira, Bugzilla)✔
- System Monitoring (e.g. Splunk, Raygun, Nagios, ELK, Zabbix)✔
We are going to cover as many of these as we can, while keeping this free tier.
Let's make a plan!
We will be briefly planning out our project with JIRA and keep track of the tasks. Then we are going to create a local kubernetes cluster with Minikube that is going to have a web and a prometheus (with alert manager) deployment for monitoring. We will also set up Jenkins locally (not in kube for reasons that I will explain) and run selenium tests from there.
- A Hypervisor (i.e. VirtualBox)
- Minikube & Kubectl
- An Atlassian JIRA Account
- A github account
- At least 8gb of RAM (unless you want to move very slowly)
- Slack (optional if you want to see your alerts in action)
I decided to make a Kops tutorial separate in the blogs, so check that out if you want to move away from the local cluster!
Now for the Juice.
Part One: JIRA Project Tracking
This section should be quick, because the main goal here is exposure.
Once you are all set with your Atlassian account, start a new project in your JIRA Software and call it whatever you'd like. Drill down into that project and you will see a tab called "Kanban". This is the board that many agile companies use to track the components of a project like bugs, features, and forward thinking plans.
Let's create a few tasks for our project by clicking the "Create" button.
We will need to track our minikube cluster, web app, monitoring setup with prometheus, and our jenkins (albeit local).
see example below:
I am not going to go through each ticket, as time is valuable. But by looking at your board, you can see the flow.
backlog > selected for development > in progress > done
This is a common process for modern software development companies.
Part 2: Minikube Cluster
Let's start our minikube cluster on our local. Open a terminal and fire away:
once it fires up, lets check the node with kubectl
You should also now have a hidden folder available in your home dir called .kube. This will hold your kube config and auth information to your cluster.
Awesome, our cluster is ready. Let's get to work.
Note: I personally am organizing my repo in this way:
We will be adding files to those directories throughout the project.
Part 3: Hello-App (Prepped for Monitoring)
Navigate to devops_02/apps/
Ok, so the first thing we want to stand up is our Hello-App. If you missed "Devops_01", we made a falcon app. We are going to reuse that app, but make an important change. We need to add a python library for prometheus exports and then add that middleware, as well as create a route for it.
Now save that file and open a dockerfile and requirements.txt file.
Our requirements file needs the following:
Our Dockerfile should look like this:
Now if you want you can build this image and push it to your repo or feel free to pull and use the image scottyfullstack/hello:latest.
That's all we need for the app, so lets get Kube prepped for each deployment.
Part 4: Kubernetes
If you haven't already create and cd into a kubernetes folder in your working directory
Note: We are going to create a lot of these deployments and services using the kubernetes cli. However, we will store those yaml files in our project to show employers that we understand the purpose of each.
Start out creating a deployment for the hello app that we made in the last section, as well as exposing it as a service. The last command below gets the pods of the deployment.
If you want to view this in your browser, open it with
Just to make sure we have our exporter working for our hello app, navigate to the route /metrics and make sure the prometheus exporter is working. The html should return similar to below:
If you want to store this deployment yaml, perform the below steps within the kubernetes directory.
That wraps up the hello deployment and now we can move on to Prometheus for monitoring our hello application.
Part 5: Monitoring with Prometheus
This deployment will have several extra steps, as we need to give prometheus permission to monitor other pods in our cluster.
Checking out the Prometheus documentation, we know we will need a config file that prometheus will reference to know what to monitor. We will use a kubernetes configmap resource for our deployment to use.
Inside ~/devops_02/kubernetes/prometheus create a file called prometheus.yml and place the following inside. On the last line replace the text with the output of (without 'https://' ):
Create and check the configmap with:
We will be adding alertmanager to this in the next section.
Now create the deployment, expose it, and then pull the yaml into a file called deployment.yml:
We will be updating this deployment file, but we need a few things first. We must create the following to ensure that prometheus can communicate with the entire cluster.
- service account
- cluster role
- cluster role binding
I am not going to explain these, but check out the docs for more. Just copy the code below and add it to a new file called permissions.yml
Those are essentially three separate yaml configs in one file. So lets apply them:
Now open up your deployment.yml file and add the service account information and volume for our configmap. Mine looks like the following (notice the second spec container for prometheus. There is now a service account name right aboce "container" and a volume added and referenced by the container):
At this point, you may get errors if you try to apply -f, so I am going to use the --force flag. This is not recommended in production. Always use caution when forcing. This is a learning demo and we will want to override this rather than delete and recreate the deployment.
Now open prometheus in the browser and check the status of Hello
Navigate to "Targets" and check to see if your service is being monitored and says UP
to test, jump over to your console and run
The status should now say DOWN
then you can scale it back up
Part 6: AlertManager
What good would just monitoring alone be, if the application does not alert us when a failure happens? That's where alertmanager comes in. In our case, we want alert manager to alert us when our hello application goes down.
Looking at the docs, we need rules set up for prometheus, as well as the alertmanager image running in tandem with our prometheus image (sharing the deployment).
lets add the rules to our configmap:
open configmap.yml and add the rules.yml | section within the data segment (see below).
Then delete the old configmap and apply it
Finally, let's create the alertmanager configmap and add the alertmanager image (and volumes) to our deployment.yml:
open a new file called alertmanager.yml from within our alert manager directory
For this step you will need to have a slack account with admin access. Install the 'Incoming Webhook' app and then generate a webhook url in the slack portal. Replace 'Webhook URL' with your slack webhook url. Also, I have created a channel called #alerts for AlertManager to send messages to.
Save it and create the configmap
Open ~/devops_02/kubernetes/prometheus/deployment.yml and make sure your spec looks like this (notice the alertmanager container and volume mounts)
Save it and apply it
If done correctly, your prometheus deployment should be standing with both containers in the pod.
Go ahead and test your alerts by scaling the hello deployment down
And we are done with the monitoring portion! Be sure to bring it back up for the next session
Part 7: Jenkins and Selenium Testing
For this part I installed Jenkins to my local machine and will be using python selenium to run my test cases with chromedriver.
Go ahead and set up your jenkins and log in (checkout devops_01 for initial setup information if needed)
Now lets jump back into our directory and if you haven't already mkdir testing
Open a new file called hello.py
Replace the driver URL with your URL from
We also need a requirements.txt file that includes
Then download chromedriver and place it in testing
Add and commit your repo and then push it. Then head over to your jenkins and create a new freestyle project.
For source code use 'GIT' and add your git creds if you haven't already.
Then under 'Build' select 'Execute Shell' and paste the following
Save that and run it! You should see it as a success. If you brought down the hello deployment, it would fail.
I know this was a long post, but I think it will be valuable to you going forward in your learning process. If you want some quick maven experience, check out Maven in 5 Minutes.
Thanks, and good luck!