Job Definitions¶
The job definitions for Jenkins Job Builder are kept in any number of
YAML or JSON files, in whatever way you would like to organize them. When you
invoke jenkins-jobs you may specify either the path of a single
YAML file, or a directory. If you choose a directory, all of
the .yaml/.yml or .json files in that directory will be read, and all the
jobs they define will be created or updated.
Definitions¶
Jenkins Job Builder understands a few basic object types which are described in the next sections.
Job¶
The most straightforward way to create a job is simply to define a Job in YAML. It looks like this:
- job:
name: job-name
That’s not very useful, so you’ll want to add some actions such as Builders, and perhaps Publishers. Those are described later.
These are job parameters that are common to every type of Jenkins job.
Example:
- job:
name: job-name
project-type: freestyle
defaults: global
description: 'Do not edit this job through the web!'
disabled: false
display-name: 'Fancy job name'
concurrent: true
workspace: /srv/build-area/job-name
quiet-period: 5
block-downstream: false
block-upstream: false
retry-count: 3
node: NodeLabel1 || NodeLabel2
logrotate:
daysToKeep: 3
numToKeep: 20
artifactDaysToKeep: -1
artifactNumToKeep: -1
| Job Parameters: |
|
|---|
Job Template¶
If you need several jobs defined that are nearly identical, except perhaps in their names, SCP targets, etc., then you may use a Job Template to specify the particulars of the job, and then use a Project to realize the job with appropriate variable substitution. Any variables not specified at the project level will be inherited from the Defaults.
A Job Template has the same syntax as a Job, but you may add
variables anywhere in the definition. Variables are indicated by
enclosing them in braces, e.g., {name} will substitute the
variable name. When using a variable in a string field, it is good
practice to wrap the entire string in quotes, even if the rules of
YAML syntax don’t require it because the value of the variable may
require quotes after substitution. In the rare situation that you must
encode braces within literals inside a template (for example a shell
function definition in a builder), doubling the braces will prevent
them from being interpreted as a template variable.
You must include a variable in the name field of a Job Template
(otherwise, every instance would have the same name). For example:
- job-template:
name: '{name}-unit-tests'
Will not cause any job to be created in Jenkins, however, it will define a template that you can use to create jobs with a Project definition. It’s name will depend on what is supplied to the Project.
If you use the variable {template-name}, the name of the template
itself (e.g. {name}-unit-tests in the above example) will be
substituted in. This is useful in cases where you need to trace a job
back to its template.
Sometimes it is useful to have the same job name format used even where the template contents may vary. Ids provide a mechanism to support such use cases in addition to simplifying referencing templates when the name contains the more complex substitution with default values.
Default Values for Template Variables¶
To facilitate reuse of templates with many variables that can be substituted, but where in most cases the same or no value is needed, it is possible to specify defaults for the variables within the templates themselves.
This can be used to provide common settings for particular templates. For example:
- project:
name: template_variable_defaults
jobs:
- 'template-variable-defaults-{num}':
num: 1
disabled_var: true
- 'template-variable-defaults-{num}':
test_var: Goodbye World
num: 2
- job-template:
# template specific defaults
# empty value causes disabled_var to be ignored internally
disabled_var:
test_var: Hello World
type: periodic
# template settings
name: 'template-variable-defaults-{num}-{type}'
id: 'template-variable-defaults-{num}'
disabled: '{obj:disabled_var}'
builders:
- shell: |
echo "Job Name: template-variable-defaults-{num}-{type}"
echo "Variable: {test_var}"
To use a default value for a variable used in the name would be uncommon unless it was in addition to another variable. However you can use Ids simplify such use cases.
Project¶
The purpose of a project is to collect related jobs together, and provide values for the variables in a Job Template. It looks like this:
- project:
name: project-name
jobs:
- '{name}-unit-tests'
Any number of arbitrarily named additional fields may be specified,
and they will be available for variable substitution in the job
template. Any job templates listed under jobs: will be realized
with those values. The example above would create the job called
‘project-name-unit-tests’ in Jenkins.
The jobs: list can also allow for specifying job-specific
substitutions as follows:
- project:
name: project-name
jobs:
- '{name}-unit-tests':
mail-to: developer@nowhere.net
- '{name}-perf-tests':
mail-to: projmanager@nowhere.net
If a variable is a list, the job template will be realized with the variable set to each value in the list. Multiple lists will lead to the template being realized with the cartesian product of those values. Example:
- project:
name: project-name
pyver:
- 26
- 27
jobs:
- '{name}-{pyver}'
If there are templates being realized that differ only in the variable used for its name (thus not a use case for job-specific substitutions), additional variables can be specified for project variables. Example:
- job-template:
name: '{name}-{pyver}'
builders:
- shell: 'git co {branch_name}'
- project:
name: project-name
pyver:
- 26:
branch_name: old_branch
- 27:
branch_name: new_branch
jobs:
- '{name}-{pyver}'
You can also specify some variable combinations to exclude from the matrix with
the exclude keyword, to avoid generating jobs for those combinations. You
can specify all the variables of the combination or only a subset, if you
specify a subset, any value of the omited variable will match:
- project:
name: project-name
axe1:
- axe1val1
- axe1val2
axe2:
- axe2val1
- axe2val2
axe3:
- axe3val1
- axe3val2
exclude:
- axe1: axe1val1
axe2: axe2val1
axe3: axe3val2
- axe2: axe2val2
axe3: axe3val1
jobs:
- build-{axe1}-{axe2}-{axe3}
- job-template:
name: build-{axe1}-{axe2}-{axe3}
builders:
- shell: "echo Combination {axe1}:{axe2}:{axe3}"
The above example will omit the jobs:
- build-axe1val1-axe2val1-axe3val2
- build-axe1val1-axe2val2-axe3val1
- build-axe1val2-axe2val2-axe3val1
To achieve the same without the exclude tag one would have to do something
a bit more complicated, that gets more complicated for each dimension in the
combination, for the previous example, the counterpart would be:
- project:
name: project-name_comb1
axe1:
- axe1val1
- axe1val2
axe2: axe2val1
axe3: axe3val1
jobs:
- build-{axe1}-{axe2}-{axe3}
- project:
name: project-name_comb2
axe1:
- axe1val1
- axe1val2
axe2: axe2val2
axe3: axe3val2
jobs:
- build-{axe1}-{axe2}-{axe3}
- project:
name: project-name_comb3
axe1: axe1val2
axe2: axe2val1
axe3: axe3val2
jobs:
- build-{axe1}-{axe2}-{axe3}
- job-template:
name: build-{axe1}-{axe2}-{axe3}
builders:
- shell: "echo Combination {axe1}:{axe2}:{axe3}"
Job Group¶
If you have several Job Templates that should all be realized together, you can define a Job Group to collect them. Simply use the Job Group where you would normally use a Job Template and all of the Job Templates in the Job Group will be realized. For example:
- job-template:
name: '{name}-unit-tests'
builders:
- shell: unittest
publishers:
- email:
recipients: '{mail-to}'
- job-template:
name: '{name}-perf-tests'
builders:
- shell: perftest
publishers:
- email:
recipients: '{mail-to}'
- job-group:
name: '{name}-tests'
jobs:
- '{name}-unit-tests':
mail-to: developer@nowhere.net
- '{name}-perf-tests':
mail-to: projmanager@nowhere.net
- project:
name: project-name
jobs:
- '{name}-tests'
Would cause the jobs project-name-unit-tests and project-name-perf-tests to be created in Jenkins.
Views¶
A view is a particular way of displaying a specific set of jobs. To create a view, you must define a view in a YAML file and have a variable called view-type with a valid value. It looks like this:
- view:
name: view-name
view-type: list
Views are processed differently than Jobs and therefore will not work within a Project or a Job Template.
Macro¶
Many of the actions of a Job, such as builders or publishers, can be defined as a Macro, and then that Macro used in the Job description. Builders are described later, but let’s introduce a simple one now to illustrate the Macro functionality. This snippet will instruct Jenkins to execute “make test” as part of the job:
- job:
name: foo-test
builders:
- shell: 'make test'
If you wanted to define a macro (which won’t save much typing in this case, but could still be useful to centralize the definition of a commonly repeated task), the configuration would look like:
- builder:
name: make-test
builders:
- shell: 'make test'
- job:
name: foo-test
builders:
- make-test
This allows you to create complex actions (and even sequences of actions) in YAML that look like first-class Jenkins Job Builder actions. Not every attribute supports Macros, check the documentation for the action before you try to use a Macro for it.
Macros can take parameters, letting you define a generic macro and more specific ones without having to duplicate code:
# The 'add' macro takes a 'number' parameter and will creates a
# job which prints 'Adding ' followed by the 'number' parameter:
- builder:
name: add
builders:
- shell: "echo Adding {number}"
# A specialized macro 'addtwo' reusing the 'add' macro but with
# a 'number' parameter hardcoded to 'two':
- builder:
name: addtwo
builders:
- add:
number: "two"
# Glue to have Jenkins Job Builder to expand this YAML example:
- job:
name: "testingjob"
builders:
# The specialized macro:
- addtwo
# Generic macro call with a parameter
- add:
number: "ZERO"
# Generic macro called without a parameter. Never do this!
# See below for the resulting wrong output :(
- add
Then <builders /> section of the generated job show up as:
<builders>
<hudson.tasks.Shell>
<command>echo Adding two</command>
</hudson.tasks.Shell>
<hudson.tasks.Shell>
<command>echo Adding ZERO</command>
</hudson.tasks.Shell>
<hudson.tasks.Shell>
<command>echo Adding {number}</command>
</hudson.tasks.Shell>
</builders>
As you can see, the specialized macro addtwo reused the definition from
the generic macro add.
Macro Notes¶
If a macro is not passed any parameters it will not have any expansion
performed on it. Thus if you forget to provide any parameters to a
macro that expects some, the parameter-templates ({foo}) will be
left as is in the resulting output; this is almost certainly not what
you want. Note if you provide an invalid parameter, the expansion
will fail; the expansion will only be skipped if you provide no
parameters at all.
Macros are expanded using Python string substitution rules. This can
especially cause confusion with shell snippets that use { as part
of their syntax. As described, if a macro has no parameters, no
expansion will be performed and thus it is correct to write the script
with no escaping, e.g.:
- builder:
name: a_builder
builders:
- shell: |
VARIABLE=${VARIABLE:-bar}
function foo {
echo "my shell function"
}
However, if the macro has parameters, you must escape the { you
wish to make it through to the output, e.g.:
- builder:
name: a_builder
builders:
- shell: |
PARAMETER={parameter}
VARIABLE=${{VARIABLE:-bar}}
function foo {{
echo "my shell function"
}}
Note that a job-template will have parameters by definition (at
least a name). Thus embedded-shell within a job-template should
always use {{ to achieve a literal {. A generic builder will need
to consider the correct quoting based on its use of parameters.
Item ID’s¶
It’s possible to assign an id to any of the blocks and then use that to reference it instead of the name. This has two primary functions:
- A unique identifier where you wish to use the same naming format for multiple templates. This allows to follow a naming scheme while still using multiple templates to handle subtle variables in job requirements.
- Provides a simpler name for a job-template where you have multiple variables including default values in the name and don’t wish to have to include this information in every use. This also makes changing the template output name without impacting references.
Example:
- project:
name: test_template_id
jobs:
- 'simple-template':
test_var: Hello World
type: periodic
num: 1
- 'not-as-simple-template':
test_var: Goodbye World
type: canary
num: 2
- job-template:
name: 'template-test-ids-{num}-{type}'
id: simple-template
builders:
- shell: |
echo "Template name: {template-name}"
echo "Job name: template-test-ids-{num}-{type}"
echo "{test_var}"
- job-template:
name: 'template-test-ids-{num}-{type}'
id: not-as-simple-template
builders:
- shell: |
echo "Template name: {template-name}"
echo "Job name: template-test-ids-{num}-{type}"
- shell: |
echo "{test_var}"
Raw config¶
It is possible, but not recommended, to use raw within a module to inject raw xml into the job configs.
This is relevant in case there is no appropriate module for a Jenkins plugin or the module does not behave as you expect it to do.
For example:
wrappers:
- raw:
xml: |
<hudson.plugins.xvnc.Xvnc>
<takeScreenshot>true</takeScreenshot>
<useXauthority>false</useXauthority>
</hudson.plugins.xvnc.Xvnc>
Is the raw way of adding support for the xvnc wrapper.
To get the appropriate xml to use you would need to create/edit a job in Jenkins and grab the relevant raw xml segment from the config.xml.
The xml string can refer to variables just like anything else and as such can be parameterized like anything else.
You can use raw in most locations, the following example show them with arbitrary xml-data:
- project:
name: complete002
version:
- 1.2
jobs:
- 'complete001_{version}'
- job-template:
name: 'complete001_{version}'
project-type: maven
scm:
- raw:
xml: |
<!-- <scm> for raw replaces the whole scm section.
where as for others the raw part is added to the existing.
-->
<scm>
<scmraw/>
</scm>
triggers:
- raw:
xml: |
<triggersraw/>
wrappers:
- raw:
xml: |
<wrappersraw/>
builders:
- raw:
xml: |
<buildersraw/>
publishers:
- raw:
xml: |
<publishersraw/>
properties:
- raw:
xml: |
<propertiesraw/>
parameters:
- raw:
xml: |
<parametersraw/>
notifications:
- raw:
xml: |
<metadataraw/>
reporters:
- raw:
xml:
<reportersraw/>
Note: If you have a need to use raw please consider submitting a patch to add or fix the module that will remove your need to use raw.
Defaults¶
Defaults collect job attributes (including actions) and will supply
those values when the job is created, unless superseded by a value in
the ‘Job’_ definition. If a set of Defaults is specified with the
name global, that will be used by all Job (and Job Template)
definitions unless they specify a different Default object with the
defaults attribute. For example:
- defaults:
name: global
description: 'Do not edit this job through the web!'
Will set the job description for every job created.
You can define variables that will be realized in a Job Template.
- defaults:
name: global
arch: 'i386'
- project:
name: project-name
jobs:
- 'build-{arch}'
- 'build-{arch}':
arch: 'amd64'
- job-template:
name: 'build-{arch}'
builders:
- shell: "echo Build arch {arch}."
Would create jobs build-i386 and build-amd64.
Variable References¶
If you want to pass an object (boolean, list or dict) to templates you can
use an {obj:key} variable in the job template. This triggers the use
of code that retains the original object type.
For example:
- project:
name: test_custom_distri
disabled: true
distributions: !!python/tuple [precise, jessie]
architectures: !!python/tuple &architectures
- amd64
- i386
axis_a:
type: user-defined
name: architectures
values: *architectures
jobs:
- '{name}-source'
- job-template:
name: '{name}-source'
project-type: matrix
disabled: '{obj:disabled}'
axes:
- axis:
type: user-defined
name: distribution
values: '{obj:distributions}'
- axis: '{obj:axis_a}'
JJB also supports interpolation of parameters within parameters. This allows a little more flexibility when ordering template jobs as components in different projects and job groups.
For example:
- job-template:
name: '{value-stream}_{project-id}_foo'
display-name: '{value-stream} {project-id} foo'
publishers:
- trigger-parameterized-builds:
- project: '{downstream}'
current-parameters: False
condition: ALWAYS
git-revision: True
- job-template:
name: '{value-stream}_{project-id}_bar'
display-name: '{value-stream} {project-id} bar'
publishers:
- trigger-parameterized-builds:
- project: '{downstream}'
current-parameters: False
condition: ALWAYS
git-revision: True
- job-group:
name: 'pipeline2'
project-id: 'p2'
jobs:
- '{value-stream}_{project-id}_foo':
downstream: '{value-stream}_{project-id}_bar'
- '{value-stream}_{project-id}_bar':
- job-group:
name: 'pipeline1'
project-id: 'p1'
jobs:
- '{value-stream}_{project-id}_bar':
downstream: '{value-stream}_{project-id}_foo'
- '{value-stream}_{project-id}_foo':
- project:
name: derp
jobs:
- 'pipeline1':
value-stream: 'production'
- 'pipeline2':
value-stream: 'experimental'
- defaults:
name: 'global'
downstream: ''
By default JJB will fail if it tries to interpolate a variable that was not defined, but you can change that behavior and allow empty variables with the allow_empty_variables configuration option.
For example, having a configuration file with that option enabled:
[job_builder]
allow_empty_variables = True
Will prevent JJb from failing if there are any non-initialized variables used and replace them with the empty string instead.
Yaml Anchors & Aliases¶
The yaml specification supports anchors and aliases which means that JJB definitions allow references to variables in templates.
For example:
- wrapper_defaults: &wrapper_defaults
name: 'wrapper_defaults'
wrappers:
- timeout:
timeout: 180
fail: true
- timestamps
- job_defaults: &job_defaults
name: 'defaults'
<<: *wrapper_defaults
- job-template:
name: 'myjob'
<<: *job_defaults
- project:
name: myproject
jobs:
- myjob
The anchors and aliases are expanded internally within JJB’s yaml loading calls and are not limited to individual documents. That means you can’t use the same anchor name in included files without collisions.
A simple example can be seen in the specs full length example with the following being more representative of usage within JJB:
- wrapper_defaults: &wrapper_defaults
name: 'wrapper_defaults'
wrappers:
- timeout:
timeout: 180
fail: true
- timestamps
- job_defaults: &job_defaults
name: 'defaults'
<<: *wrapper_defaults
- job-template:
name: 'myjob'
<<: *job_defaults
Which will be expanded to the following yaml before being processed:
- wrapper_defaults:
name: wrapper_defaults
wrappers:
- timeout:
fail: true
timeout: 180
- timestamps
- job_defaults:
name: defaults
wrappers:
- timeout:
fail: true
timeout: 180
- timestamps
- job-template:
name: myjob
wrappers:
- timeout:
fail: true
timeout: 180
- timestamps
Custom Yaml Tags¶
Custom application specific yamls tags are supported to provide enhancements when reading yaml configuration.
These allow inclusion of arbitrary files as a method of having blocks of data managed separately to the yaml job configurations. A specific usage of this is inlining scripts contained in separate files, although such tags may also be used to simplify usage of macros or job templates.
The tag !include: will treat the following string as file which should be
parsed as yaml configuration data.
Example:
- job: name: test-job-1 builders: !include: include001.yaml.inccontents of include001.yaml.inc:
- timeout-wrapper - pre-scm-shell-ant - copy-files
The tag !include-raw: will treat the given string or list of strings as
filenames to be opened as one or more data blob, which should be read into
the calling yaml construct without any further parsing. Any data in a file
included through this tag, will be treated as string data.
Examples:
- job: name: test-job-include-raw-1 builders: - shell: !include-raw: include-raw001-hello-world.sh - shell: !include-raw: include-raw001-vars.shcontents of include-raw001-hello-world.sh:
#!/bin/bash # # Sample script showing how the yaml include-raw tag can be used # to inline scripts that are maintained outside of the jenkins # job yaml configuration. echo "hello world" exit 0contents of include-raw001-vars.sh:
#!/bin/bash # # sample script to check that brackets aren't escaped # when using the include-raw application yaml tag VAR1="hello" VAR2="world" VAR3="${VAR1} ${VAR2}" [[ -n "${VAR3}" ]] && { # this next section is executed as one echo "${VAR3}" exit 0 }using a list of files:
- job: name: test-job-include-raw-1 builders: - shell: !include-raw: - include-raw001-hello-world.sh - include-raw001-vars.sh
The tag !include-raw-escape: treats the given string or list of strings as
filenames to be opened as one or more data blobs, which should be escaped
before being read in as string data. This allows job-templates to use this tag
to include scripts from files without needing to escape braces in the original
file.
Warning
When used as a macro !include-raw-escape: should only be used if
parameters are passed into the escaped file and you would like to escape
those parameters. If the file does not have any jjb parameters passed into
it then !include-raw: should be used instead otherwise you will run
into an interesting issue where include-raw-escape: actually adds
additional curly braces around existing curly braces. For example
${PROJECT} becomes ${{PROJECT}} which may break bash scripts.
Examples:
- job-template: name: test-job-include-raw-{num} builders: - shell: !include-raw-escape: include-raw001-hello-world.sh - shell: !include-raw-escape: include-raw001-vars.sh - project: name: test-job-template-1 num: 1 jobs: - 'test-job-include-raw-{num}'contents of include-raw001-hello-world.sh:
#!/bin/bash # # Sample script showing how the yaml include-raw tag can be used # to inline scripts that are maintained outside of the jenkins # job yaml configuration. echo "hello world" exit 0contents of include-raw001-vars.sh:
#!/bin/bash # # sample script to check that brackets aren't escaped # when using the include-raw application yaml tag VAR1="hello" VAR2="world" VAR3="${VAR1} ${VAR2}" [[ -n "${VAR3}" ]] && { # this next section is executed as one echo "${VAR3}" exit 0 }using a list of files:
- template-job: name: test-job-include-raw-{num} builders: - shell: !include-raw-escape: - include-raw001-hello-world.sh - include-raw001-vars.sh - project: name: test-job-template-1 num: 1 jobs: - 'test-job-include-raw-{num}'
For all the multi file includes, the files are simply appended using a newline character.
To allow for job templates to perform substitution on the path names, when a filename containing a python format placeholder is encountered, lazy loading support is enabled, where instead of returning the contents back during yaml parsing, it is delayed until the variable substitution is performed.
Example:
- wrapper: !include: lazy-load-jobs-timeout.yaml.inc - project: name: test version: - 1.1 jobs: - 'build_myproject_{version}' - job-template: name: 'build_myproject_{version}' wrappers: !include: lazy-load-wrappers-{version}.yaml.inc builders: - shell: !include-raw: echo_vars_{version}.shusing a list of files:
- wrapper: !include: lazy-load-jobs-timeout.yaml.inc - project: name: test num: "002" version: - 1.1 jobs: - 'build_myproject_{version}' - job-template: name: 'build_myproject_{version}' wrappers: !include: lazy-load-wrappers-{version}.yaml.inc builders: - shell: !include-raw: - lazy-load-scripts/echo_vars_{version}.sh - include-raw{num}-cool.sh
Note
Because lazy-loading involves performing the substitution on the file
name, it means that jenkins-job-builder can not call the variable
substitution on the contents of the file. This means that the
!include-raw: tag will behave as though !include-raw-escape: tag
was used instead whenever name substitution on the filename is to be
performed.
Given the behaviour described above, when substitution is to be performed
on any filename passed via !include-raw-escape: the tag will be
automatically converted to !include-raw: and no escaping will be
performed.
Modules¶
The bulk of the job definitions come from the following modules.
Module Execution¶
The jenkins job builder modules are executed in sequence.