Run Jenkins in k8s
So yeah, why not use Jenkins in a k8s? Lets get one up an running.
First we need to create an image with some plugins we want pre installed. Normally if we were to install to a VM with internet connection, they could be installed via the Jenkins UI.
Our Dockerfile is like follows. I went a bit overboard with the plugins but I do want to try them all out. Plugins are here
Create a file called Dockerfile with the below content. We will be using the default jenkins image as a base.
Build our image and stuff it into our Harbor image repo.
docker build -t 192.168.56.109:443/core-sys/jenkins-mod:v1 .So the image is gigantic, but we will look into that later :)
Lets get Jenkins running with our new image. Below is the yaml we will use to deploy Jenkins. I’ll explain each yml seperated by — -
- We first create a cluster role binding for the default service account with cluster-admin permission for jenkins to use. Sounds a bit over-kill, yes, but we’ll fix that later.
- Then a persistent vol in master node within the host path to pass as vol to Jenkins
- Then the PVC to claim above vol
- A nodeport service that allows us to use port 30007 to point to Jenkins 8080 port.
- The Jenkins Deploy yaml itself, describing the port we will use, Java env options, our modified image
After running above with kubectl apply -f . running below should get you something like the bottom image.
kubectl get pod,svc,pvc,pv,sa,clusterrolebinding
So we can open our jenkins now from our master ip: http://192.168.56.105:30007/configureSecurity/
The first thing would be to change the security to user based like below.
Once thats out of the way. Lets set the workers to … work.
# K8s url
kubectl config view --minify | grep server | cut -f 2- -d ":" | tr -d " "# Server certificate key
kubectl get secret $(kubectl get sa default -o jsonpath={.secrets[0].name}) -o jsonpath={.data.'ca\.crt'} | base64 --decode# jenkins url is the servicename with the cluster ip port we used.
Credential provider part should be selected as k8s SA and global. We then click add.
We then have to add pod templates for jenkins slaves.
So lets give it a test with a new job: http://192.168.56.105:30007/newJob
Added just a single sleep command and saved.
Before testing, lets tick a box in the build config to be able to run the same unchanged build multiple times. Normally Jenkins doesn’t allow such.
Started a few builds together from below
Thats about it. Thanks for reading.
if you want you can modify the slave images too.
use in configurations like: 192.168.56.109:443/core-sys/jenkins/jnlp-slavemod:v1.0Dockerfile:
FROM jenkinsci/jnlp-slave
USER root
RUN apt-get update \
&& apt-get install -y maven \
&& apt-get install -y gradle
