Jenkins

Jenkins pipeline

Jenkins pipeline is probably the most important concept that a newbie needs to learn. Whenever you write a Jenkinsfile, you write a pipeline. When I started to use and learn Jenkins, it is so hard for me to get the pipeline correct because of syntax errors and so many methods/tokens provided by various plugins. OK. So let me document what I have learnt so far.

First, Jenkins pipeline is not a core part of Jenkins. It is provided as a collection of plugins. workflow-aggregator-plugin is the umbrella plugin for Jenkins pipeline. This repo does not have any code. It just declares in the pom file the component repos that make up Jenkins pipeline.

Pipeline syntax

One of the plugins that make up pipeline is pipeline-model-definition. It implements the pipeline parser. Pipeline blocks are parsed as Groovy AST nodes. Therefore, pipeline share the same syntax as Groovy.

Step syntax

The steps clause defines a block of statements. Each statement defines a step. Each step is parsed a method call. I said “method call”, not anything else! In Groovy, a method call can omit the parentheses, so for a shell step, both sh "xxx" and sh("xxx") are correct syntax.

Each step has the format function_name arg1, arg2, ... or function_name, arg1, arg2, ..., argN, closure. See code. Basically, the last element in the statement can be a closure, which defines child steps. Using the second format, we can write slightly more complicated steps like one below

steps {
    timeout(time: 3, unit: 'MINUTES') {
        retry(5) {
            sh './flakey-deploy.sh'
        }
    }
}

A step also can be a script block.

Environment syntax

Environment values can be string constant or Groovy gstring. See code.

Btw, I once tried to update environment variables inside a stage and got blocked for about 5 hours. Basically, environment variables cannot be changed once set inside environment {} block. There is a stack overflow question about it, and this post is the ultimate guide on this topic.

Pipeline execution.

workflow-cps-plugin has a good introduction about the design of execution engine of Jenkins pipeline.

Step exeuction

invokeStep shows the distinction of a step with and without a ending closure.

TODO: read more about jenkins execution engine. Try to understand the design philosophy of a JVM based execution engine. This will be super useful in future if I need to design a DSL in Java. Also, compare it with logstash. Logstash is Ruby based and it can execute arbitrary Ruby code without DSL execution engine.

Plugins

1. jenkins/inbound-agent. Quote from docker hub “This is an image for Jenkins agents using TCP or WebSockets to establish inbound connection to the Jenkins master.” so each jenkins agent pod will have a jnlp container, which runs this image.
2. Kubernetes plugin: allow you to spin up new plugin agent pod to run jobs.
3. Pipeline: Nodes and Processes: shell command step.

API

Remote access

Ex:

export JC='curl --user admin:<TOKEN>'
$JC  'https://jenkins.tryevergreen.com//api/xml' | xq

References:

• https://www.jenkins.io/doc/book/using/remote-access-api/

Gotchas

java.io.NotSerializableException

You may see this error when you use groovy.json.JsonSlurper in your pipeline. This is because Jenkins has a paradigm where all jobs can be interrupted, paused, and resumable through server reboots. To achieve this the pipeline and its data must be fully serializable. See this doc for the background and ways to fix it.

Scripts not permitted to use method …

For example

Scripts not permitted to use method java.time.chrono.ChronoLocalDateTime isBefore java.time.chrono.ChronoLocalDateTime.

Jenkins’ Security Plugin requires whitelisting methods that can be used in Groovy sandbox. The default whitelist is here. Also, you can whitelist new methods: Manage Jenkins > In-process Script Approval.