Jenkins Pipeline Project for DevOps Engineers - Part 1

Jenkins Pipeline Project for DevOps Engineers - Part 1

#90DaysofDevOps Challenge - Days 23/24

Francisco Souza's photo
Francisco Souza
·

13 min read

Table of contents

▶What is CI/CD?

A CI/CD pipeline is the full set of processes that run when you trigger work on your projects. Pipelines contain your workflows, which coordinate your jobs, and this is all defined in your project configuration YAML file.

Continuous Integration(CI) - The automated building and testing of your application on every new commit.

Continuous Deployment(CD) - The automation of building, testing, and deploying. If all tests pass, every new commit will push new code through the entire development pipeline to production with no manual intervention.

Continuous Delivery(CD) - A state where your application is always ready to be deployed. A manual step is required to actually deploy the application.

▶What is a Build Job?

A build job in Jenkins contains the configuration for automating a specific task or step in the application-building process. These tasks include gathering dependencies, compiling, archiving, or transforming code, and testing and deploying code in different environments.

Many different types of build jobs can be created in Jenkins, including:

  • Freestyle projects: These are the most basic type of build job, and they allow you to define the steps that are executed in the build process.

  • Pipelines: Pipelines are a more advanced type of build job that allows you to define the steps in the build process using a declarative syntax.

  • Multi-configuration projects: These allow you to create multiple builds of the same project, each with different configurations.

  • Folders: Folders can be used to organize your build jobs.

  • Multibranch pipelines: These allow you to create pipelines that are based on branches in a Git repository.

  • Organization folders: These allow you to organize your build jobs by organization.

In this blog, I am going to do a Pipeline Project.

▶What are Jenkins Pipeline Projects?

A Pipeline Job is that the Pipeline Scripted job runs on the Jenkins master. This uses a lightweight executor which uses only some resources to translate in the master to atomic commands that execute or send to the agents.

Jenkins Pipeline is a feature of the Jenkins build server, deployed as a plugin, that lets you implement continuous delivery (CD) pipelines on the Jenkins automation server.

Continuous delivery pipelines are automated sequences of processes to deliver software from version control to customers and end-users. Each software changes a developer commits in source control passes through a set of automated processes before being released to production. The pipeline involves building software using repeatable, reliable steps and pushing the build through various testing and deployment stages.

Jenkins Pipelines offers an extensible toolset to model delivery pipelines of varying complexity.

Each Jenkins pipeline has a definition written in a text-based Jenkinsfile, which development teams can commit to their source control repository.

Creating a Jenkinsfile and committing it to source control provides several immediate benefits:

  • Automatically creates a Pipeline build process for all branches and pull requests.

  • Code review/iteration on the Pipeline (along with the remaining source code).

  • Audit trail for the Pipeline.

  • Single source of the stage for the Pipeline, which can be viewed and edited by multiple members of the project.

While the syntax for defining a Pipeline, either in the web UI or with a Jenkinsfile is the same, it is generally considered best practice to define the Pipeline in a Jenkinsfile and check that into source control.

Let's complete today's tasks, which include deploying a web app.

I tried a different project than the one proposed on Day 23 by Shubham Londhe on his #90daysofdevops challenge, trying to push me to improve my knowledge.

▶ Tasks

Step 1 - Run Jenkins in Docker

First, open up a terminal window.

Create a bridge network in Docker using the following docker network create command:

$ docker network create jenkins

To execute Docker commands inside Jenkins nodes, download and run the docker:dind Docker image using the following docker run command:

$ docker run \
  --name jenkins-docker \
  --rm \
  --detach \
  --privileged \
  --network jenkins \
  --network-alias docker \
  --env DOCKER_TLS_CERTDIR=/certs \
  --volume jenkins-docker-certs:/certs/client \
  --volume jenkins-data:/var/jenkins_home \
  --publish 2376:2376 \
  --publish 3000:3000 --publish 5000:5000 \
  docker:dind \
  --storage-driver overlay2

Customize the official Jenkins Docker image, by executing below two steps:

Create Dockerfile with the following content:

FROM jenkins/jenkins:2.401.1
USER root
RUN apt-get update && apt-get install -y lsb-release
RUN curl -fsSLo /usr/share/keyrings/docker-archive-keyring.asc \
  https://download.docker.com/linux/debian/gpg
RUN echo "deb [arch=$(dpkg --print-architecture) \
  signed-by=/usr/share/keyrings/docker-archive-keyring.asc] \
  https://download.docker.com/linux/debian \
  $(lsb_release -cs) stable" > /etc/apt/sources.list.d/docker.list
RUN apt-get update && apt-get install -y docker-ce-cli
USER jenkins
RUN jenkins-plugin-cli --plugins "blueocean docker-workflow"

Build a new docker image from this Dockerfile and assign the image a meaningful name, e.g. "myjenkins-blueocean:2.401.1-1":

$ docker build -t myjenkins-blueocean:2.401.1-1 .

Keep in mind that the process described above will automatically download the official Jenkins Docker image if this hasn’t been done before.

Run your own myjenkins-blueocean:2.401.1-1 image as a container in Docker using the following docker run command:

docker run \
  --name jenkins-blueocean \
  --detach \
  --network jenkins \
  --env DOCKER_HOST=tcp://docker:2376 \
  --env DOCKER_CERT_PATH=/certs/client \
  --env DOCKER_TLS_VERIFY=1 \
  --publish 8080:8080 \
  --publish 50000:50000 \
  --volume jenkins-data:/var/jenkins_home \
  --volume jenkins-docker-certs:/certs/client:ro \
  --volume "$HOME":/home \
  --restart=on-failure \
  --env JAVA_OPTS="-Dhudson.plugins.git.GitSCM.ALLOW_LOCAL_CHECKOUT=true" \
  myjenkins-blueocean:2.401.1-1

Step 2 - Accessing the Docker container

This means you could access your docker container (through a separate terminal/command prompt window) with a docker exec command like:

$ docker exec -it jenkins-blueocean bash

Step 3 - Accessing the Docker logs

There is a possibility you may need to access the Jenkins console log, for instance, when Unlocking Jenkins as part of the Post-installation setup wizard.

$ docker logs <docker-container-name>

Your <docker-container-name> can be obtained using the docker ps command.

Step 4 - Accessing the Jenkins home directory

There is a possibility you may need to access the Jenkins home directory, for instance, to check the details of a Jenkins build in the workspace subdirectory.

If you mapped the Jenkins home directory (/var/jenkins_home) to one on your machine’s local file system (i.e. in the docker run …​ the command above), then you can access the contents of this directory through your machine’s usual terminal/command prompt.

Otherwise, if you specified the --volume jenkins-data:/var/jenkins_home option in the docker run …​ command, you can access the contents of the Jenkins home directory through your container’s terminal/command prompt using the docker container exec command:

$ docker container exec -it <docker-container-name> bash

Step 4 - Jenkins Setup Wizard

Unlocking Jenkins

When you first access a new Jenkins instance, you are asked to unlock it using an automatically-generated password.

After the 2 sets of asterisks appear in the terminal/command prompt window, browse to http://localhost:8080 and wait until the Unlock Jenkins page appears.

Display the Jenkins console log with the command:

$ docker logs jenkins-blueocean

From your terminal/command prompt window again, copy the automatically-generated alphanumeric password (between the 2 sets of asterisks).

On the Unlock Jenkins page, paste this password into the Administrator password field and click Continue.

Step 5 - Customizing Jenkins with plugins

After unlocking Jenkins, the Customize Jenkins page appears.

On this page, click Install suggested plugins.

The setup wizard shows the progression of Jenkins being configured and the suggested plugins being installed. This process may take a few minutes.

Step 6 - Creating the first administrator user

Finally, Jenkins asks you to create your first administrator user.

When the Create First Admin User page appears, specify your details in the respective fields and click Save and Finish.

When the Jenkins is ready page appears, click Start using Jenkins.
Notes:

This page may indicate Jenkins is almost ready! instead and if so, click Restart.

If the page doesn’t automatically refresh after a minute, use your web browser to refresh the page manually.

If required, login to Jenkins with the credentials of the user you just created and you’re ready to start using Jenkins!

Commit this new file. We aren’t pushing any changes yet because we still need to create a Jenkinsfile for the Pipeline job to execute correctly.

Step 7 - Fork and clone the sample repository

Ensure you are signed in to your GitHub account. If you don’t yet have a GitHub account, sign up for a free one on the GitHub website.

Fork the simple-node-js-react-npm-app on GitHub into your local GitHub account.

Clone your forked simple-node-js-react-npm-app repository (on GitHub) locally to your machine.

Run the following command to continue/complete cloning your forked repo:
git clone https://github.com/YOUR-GITHUB-ACCOUNT-NAME/simple-node-js-react-npm-app
where YOUR-GITHUB-ACCOUNT-NAME is the name of your GitHub account.

Step 8 - Create your Pipeline project in Jenkins

  • Go back to Jenkins, log in again if necessary and click create new jobs under Welcome to Jenkins!
    Note: If you don’t see this, click New Item at the top left.

  • In the Enter an item name field, specify the name for your new Pipeline project (e.g. simple-node-js-react-npm-app).

  • Scroll down and click Pipeline, then click OK at the end of the page.

  • ( Optional ) On the next page, specify a brief description for your Pipeline in the Description field (e.g. An entry-level Pipeline demonstrating how to use Jenkins to build a simple Node.js and React application with npm.)

  • Click the Pipeline tab at the top of the page to scroll down to the Pipeline section.

  • From the Definition field, choose the Pipeline script from SCM option. This option instructs Jenkins to obtain your Pipeline from Source Control Management (SCM), which will be your locally cloned Git repository.

  • From the SCM field, choose Git.

  • In the Repository URL field, specify the directory path of your locally cloned repository above, which is from your user account/home directory on your host machine, mapped to the /home directory of the Jenkins container - i.e.

    • For macOS - /home/Documents/GitHub/simple-node-js-react-npm-app

    • For Linux - /home/GitHub/simple-node-js-react-npm-app

    • For Windows - /home/Documents/GitHub/simple-node-js-react-npm-app

  • Click Save to save your new Pipeline project. You’re now ready to begin creating your Jenkinsfile, which you’ll be checking into your locally cloned Git repository.

Step 9 - Create your initial Pipeline as a Jenkinsfile

Your Pipeline will be created as a Jenkinsfile, which will be committed to your locally cloned Git repository (simple-node-js-react-npm-app).

First, create an initial Pipeline to download a Node Docker image and run it as a Docker container (which will build your simple Node.js and React application). Also add a "Build" stage to the Pipeline that begins orchestrating this whole process.

  • Using your favourite text editor or IDE, create and save the new text file with the name Jenkinsfile at the root of your local simple-node-js-react-npm-app Git repository.

  • If you want to install an editor inside the Docker container

      $ docker container exec --user="root" -it jenkins-blueocean bash -c 'apt-get update && apt-get install -y vim'

    Copy the following Declarative Pipeline code and paste it into your empty Jenkinsfile:

pipeline {
    agent {
        docker {
            image 'node:lts-bullseye-slim' 
            args '-p 3000:3000' 
        }
    }
    stages {
        stage('Build') { 
            steps {
                sh 'npm install' 
            }
        }
    }
}

Save your edited Jenkinsfile and commit it to your local simple-node-js-react-npm-app Git repository. E.g. Within the simple-node-js-react-npm-app directory, run the commands:
git add . then git commit -m "Add initial Jenkinsfile"

Go back to Jenkins again, log in again if necessary and click Open Blue Ocean on the left to access Jenkins’s Blue Ocean interface.

In the This job has not been run message box, click Run, then quickly click the OPEN link which appears briefly at the lower-right to see Jenkins building your Pipeline project. If you weren’t able to click the OPEN link, click the row on the main Blue Ocean interface to access this feature.

  • Initially queues the project to be run on the agent.

  • Downloads the Node Docker image and runs it in a container on Docker.

Runs the Build stage (defined in the Jenkinsfile) on the Node container. During this time, npm downloads many dependencies necessary to run your Node.js and React application, which will ultimately be stored in the node_modules workspace directory (within the Jenkins home directory).

The Blue Ocean interface turns green if Jenkins built your Node.js and React application successfully.

Step 10 - Add a test stage to your Pipeline

Go back to your text editor/IDE and ensure yours Jenkinsfile is open.

Copy and paste the following Declarative Pipeline syntax immediately under the Build stage:

stage('Test') {
            steps {
                sh './jenkins/scripts/test.sh'
            }
        }

so that you end up with:

pipeline {
    agent {
        docker {
            image 'node:lts-bullseye-slim'
            args '-p 3000:3000'
        }
    }
    stages {
        stage('Build') {
            steps {
                sh 'npm install'
            }
        }
        stage('Test') { 
            steps {
                sh './jenkins/scripts/test.sh' 
            }
        }
    }

Save your edited Jenkinsfile and commit it to your local simple-node-js-react-npm-app Git repository. e.g. Within the simple-node-js-react-npm-app directory, run the commands:
git stage . then git commit -m "Add 'Test' stage"

Go back to Jenkins again, log in again if necessary and ensure you’ve accessed Jenkins’s Blue Ocean interface.

Click Run at the top left, then quickly click the OPEN link which appears briefly at the lower-right to see Jenkins running your amended Pipeline project. If you weren’t able to click the OPEN link, click the top row on the Blue Ocean interface to access this feature.

Note: You’ll notice from this run that Jenkins no longer needs to download the Node Docker image. Instead, Jenkins only needs to run a new container from the Node image downloaded previously. Also, from now on, no (new) npm dependencies should need to be downloaded during the "Build" stage. Therefore, running your Pipeline this subsequent time should be much faster.
If your amended Pipeline ran successfully, here’s what the Blue Ocean interface should look like. Notice the additional "Test" stage. You can click on the previous "Build" stage circle to access the output from that stage.

Step 11 - Add a final delivery stage to your Pipeline

Go back to your text editor/IDE and ensure your Jenkinsfile is open.

Copy and paste the following Declarative Pipeline syntax immediately under the Test stage of your Jenkinsfile:

 stage('Deliver') {
            steps {
                sh './jenkins/scripts/deliver.sh'
                input message: 'Finished using the web site? (Click "Proceed" to continue)'
                sh './jenkins/scripts/kill.sh'
            }
        }

so that you end up with:

pipeline {
    agent {
        docker {
            image 'node:lts-buster-slim'
            args '-p 3000:3000'
        }
    }
    stages {
        stage('Build') {
            steps {
                sh 'npm install'
            }
        }
        stage('Test') {
            steps {
                sh './jenkins/scripts/test.sh'
            }
        }
        stage('Deliver') { 
            steps {
                sh './jenkins/scripts/deliver.sh' 
                input message: 'Finished using the web site? (Click "Proceed" to continue)' 
                sh './jenkins/scripts/kill.sh' 
            }
        }
    }
}

Save your edited Jenkinsfile and commit it to your local simple-node-js-react-npm-app Git repository. E.g. Within the simple-node-js-react-npm-app directory, run the commands:
git stage . then git commit -m "Add 'Deliver' stage"

Go back to Jenkins again, log in again if necessary and ensure you’ve accessed Jenkins’s Blue Ocean interface.

Click Run at the top left, then quickly click the OPEN link which appears briefly at the lower-right to see Jenkins running your amended Pipeline project. If you weren’t able to click the OPEN link, click the top row on the Blue Ocean interface to access this feature.

If your amended Pipeline ran successfully, here’s what the Blue Ocean interface should look like. Notice the additional "Deliver" stage. Click on the previous "Test" and "Build" stage circles to access the outputs from those stages.

Step 12 - Add a final delivery stage to your Pipeline

Ensure you are viewing the "Deliver" stage (click it if required), then click the green ./jenkins/scripts/deliver.sh step to expand its content and scroll down until you see the http://localhost:3000 link.

Deliver stage output only

Click the http://localhost:3000 link to view your Node.js and React application running (in development mode) in a new web browser tab. You should see a page/site with the title Welcome to React on it.

When you are finished viewing the page/site, click the Proceed button to complete the Pipeline’s execution.

Conclusion

Well done! You’ve just used Jenkins to build a simple Node.js and React application with npm!

The "Build", "Test" and "Deliver" stages of CI/CD that you created above are the basis for building more complex CI/CD workflows in Jenkins.

DevopsCloudCloud Computing