OSDC 2017 | Automating Jenkins by Julien Pivotto

of 104

Please download to get full document.

View again

All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.
104 pages
0 downs
Jenkins is a unique piece of software, lots of people and enterprises use it to deploy and build their software and also their infrastructure. It has tons of plugins, and can do virtually anything. It is important for both devs and ops. This talk will be about how you can automate and test your Jenkins instances. In the past, the tooling around it was not so great, but it has changed. Tools like Jenkins Pipeline and Job DSL plugin has entered the game and are here to stay.
  • 1. Jenkins Automation Julien Pivotto (@roidelapluie) Open Source Data Center Conference Berlin
  • 2. whoami Julien "roidelapluie" Pivotto @roidelapluie Sysadmin at inuits Automation, monitoring, HA Jenkins user for 5+ years
  • 3. inuits
  • 4. Jenkins in 2 lines Open Source Pluggable
  • 5. The Jenkins Project Java Hudson (2005) Jenkins (2011)
  • 6. The evolution of Jenkins Jenkins is moving fast (there is competition: travis-ci, gitlab-runners) Far away from just the java world (e.g. mvn specifics are gone in Pipeline world)
  • 7. What can you do with Jenkins? Testing, building, deploying Software Services Infrastructure
  • 8. Mission Critical Public Domain https://commons.wikimedia.org/wiki/File:Roaddamage59quake.JPG
  • 9. Without Jenkins... How do I build this? Where do I deploy this? Who knows where to upload this? When did we deploy this?
  • 10. Automating Jenkins Creative Commons Attribution-ShareAlike 2.0 https://www.flickr.com/photos/machu/3131057286
  • 11. Automating Jenkins Why is it needed? Make build processes transparent Make build processes improvable Make build process reproducible Lower the barrier to update Jenkins security - audit - bugfixes - plugins
  • 12. Automating Jenkins Why is it hard? Building Software is hard For long, Jenkins has been strongly UI-Driven It is "easy" to deploy (.war file) Hey I've deployed Jenkins on my Laptop! It has lots of plugins (and you need them)
  • 13. XML everywhere! Creative Commons Attribution 2.0 https://www.flickr.com/photos/generated/6035308729
  • 14. What can you automate in Jenkins?
  • 15. 1. The Operating System Jenkins is running on The platform/node/host/.. which runs your Jenkins damon
  • 16. 2. The Jenkins Service The start/stop/reload of Jenkins JAVA_OPTIONS
  • 17. 3. Plugins Which plugins to install Which version Which configuration
  • 18. 4. Jenkins Global Configuration SMTP Server Email address Authentication/Authorization
  • 19. 5. Jobs - Views - Folder Views - That really help people Jobs and folders - Creation AND deletion,
  • 20. 6. Jobs definition Jobs content Build steps Deployments configuration
  • 21. 7. Operating system you build your software on Jenkins nodes Where your jobs run (Linux, Android, ...)
  • 22. Automating the OS Creative Commons Attribution-Share Alike 2.0 https://www.flickr.com/photos/timdorr/168925884
  • 23. Jenkins deserves attention Jenkins Server should be automated Receive OS updates Be monitored like any other server
  • 24. Automating the service CC0 Public Domain https://pixabay.com/en/butler-tray-beverages-wine-964007/
  • 25. Installing Jenkins Do NOT use war file (yes I know it's easy) Use Jenkins LTS (3 months lifecycle vs 1 week) Think about: user, directories, backups,... Packages provides: upgrade path, downgrade path, control, checksums, signatures, ...
  • 26. Automating the Jenkins Service Chef: Jenkins in the supermarket Puppet: module rtyler/jenkins Playbooks/etc for other tools as well Docker
  • 27. Jenkins Plugins You NEED them (never seen a Jenkins setup without 20-100 plugins) Fetched from https://updates.jenkins-ci.org Can be installed from the UI :( Can be installed from the CLI
  • 28. Packaging Jenkins Plugins Plugins have dependencies (against plugins & Jenkins core) They have a fixed download path They are listed in updates.jenkins.io/update- center.json https://github.com/roidelapluie/Jenkins- Plugins-RPM
  • 29. Mirroring Jenkins Plugins Mirror http://updates.jenkins-ci.org By default, "latest" will be fetched Don't cache too much github.com/jenkinsci/docker install- plugins.sh
  • 30. Global configuration CC0 Public Domain https://commons.wikimedia.org/wiki/File:Waiting_room_bell.jpg
  • 31. Modifying XML files DON'T
  • 32. system-config-dsl-plugin Like Job DSL, but for the system https://github.com/jenkinsci/system-config- dsl-plugin JENKINS-31094 Downside: it does not exist yet
  • 33. Creative Commons Attribution-Share Alike 3.0 https://commons.wikimedia.org/wiki/File:Groovy-logo.svg
  • 34. Groovy Yet another language to learn? yes. Programming language for the Java platform Scripting language Fully integrated in Jenkins Used by automation tools (chef cookbook, puppet module...)
  • 35. The Jenkins Script Console A groovy console is available at /script http://jenkins.example.com/script also with curl Requires "Overall/Run Scripts" permission
  • 36. Sample Groovy scripts Creative Commons Attribution-Share Alike 2.0 https://www.flickr.com/photos/bjornb/88101376
  • 37. Set number of executors import jenkins.model.Jenkins; Jenkins.instance.setNumExecutors(0)
  • 38. Set HTML formatter import jenkins.model.Jenkins; import hudson.markup.RawHtmlMarkupFormatter; Jenkins.instance.setMarkupFormatter( new RawHtmlMarkupFormatter(false));
  • 39. Set system message import jenkins.model.Jenkins; Jenkins.instance.setSystemMessage(""" Welcome to <em>Jenkins</em>. """);
  • 40. Configure simple-theme import jenkins.model.Jenkins; def simpleThemePlugin =    Jenkins.instance.getDescriptor(   "org.codefirst.SimpleThemeDecorator")    simpleThemePlugin.cssUrl =    "/userContent/example.css" simpleThemePlugin.save() Everything in $JENKINS_HOME/userContent is served under /userContent
  • 41. Email import jenkins.model.Jenkins def desc = Jenkins.instance.getDescriptor(   "hudson.tasks.Mailer") desc.setSmtpHost("") desc.save()
  • 42. Disable usage statistics import hudson.model.UsageStatistics; hudson.model.UsageStatistics.DISABLED = true;
  • 43. But also... Setup Authorization/Authentication Setup prefix url Setup slaves, clouds Setup global libraries Setup credentials Setup first jobs
  • 44. Bonus import org.jenkinsci.plugins.pipeline.utility .steps.shaded.org.yaml.snakeyaml.Yaml Yaml reader = new Yaml() Map config = reader.load(text) Thanks to pipeline utility step (readYaml() step), you can easily read yaml files
  • 45. Scripts sources https://wiki.jenkins-ci.org/display/JENKINS/Jenkins+Script+Console https://github.com/jenkinsci/jenkins-scripts/tree/master/scriptler https://github.com/infOpen/ansible-role-jenkins/tree/master/files/groovy_scripts https://github.com/samrocketman/jenkins-bootstrap-jervis/tree/master/scripts https://github.com/jenkinsci/puppet- jenkins/blob/master/files/puppet_helper.groovy http://pghalliday.com/jenkins/groovy/sonar/chef/configuration/management/2014 /09/21/some-useful-jenkins-groovy-scripts.html
  • 46. Jenkins Javadocs https://jenkins.io/doc/developer/guides/ http://javadoc.jenkins.io/archive/jenkins- 2.32/ http://javadoc.jenkins.io/ http://javadoc.jenkins.io/plugin/gerrit-trigger/
  • 47. Now what? Creative Commons Attribution 2.0 https://www.flickr.com/photos/51029297@N00/5275403364
  • 48. Jenkins init.groovy.d Upon startup, Jenkins will run  $JENKINS_HOME/init.groovy.d/*.groovy  scripts Meanwhile, "Jenkins is getting ready..." message is displayed Drop files, restart Jenkins Allows you to preconfigure everything -- without the GUI
  • 49. init.groovy.d directory behaviour Scripts executed sequentally (alphanum order) Scripts that throws exceptions make startup fail
  • 50. Inside Jenkins Creative Commons Attribution-ShareAlike 4.0 International https://commons.wikimedia.org/wiki/File:Mystery_Cave_passage.jpg
  • 51. The Multiple Approaches GUI .. but this talk is about automation, right? init.groovy.d: to create your seed job Jenkins Job Builder: declarative, python, yaml Jenkins Job DSL: imperative, groovy
  • 52. Jenkins Job Builder An Openstack Project Python (not a Jenkins Plugin!) Support templates Extensible Can do raw xml Limited support for plugins and pipeline Put Jobs config under SCM
  • 53. Jenkins Job DSL A Jenkins Plugin 2012 Groovy DSL to create views & jobs Put Jobs config under SCM
  • 54. init.groovy.d def jobManagement = new JenkinsJobManagement(   System.out, [:], new File('.')) new DslScriptLoader(jobManagement).runScript(""" folder('jenkins') {     displayName('Jenkins') } pipelineJob("jenkins/seed"){   definition {         cpsScm {             scm {                 git {                     remote {                         credentials('jenkins­git­na')                         url('git.example.com/seed.git')                     }                     branches('master')                 }             }             scriptPath('Jenkinsfile')         }     } } """)
  • 55. Creative Commons Attribution-Share Alike 3.0 https://commons.wikimedia.org/wiki/File:Groovy-logo.svg
  • 56. Job DSL support Large community of users Lots of plugins supported
  • 57. Create a job job('test') {  scm {   git('git://example.com/foo.git' 'master')  }  steps {   shell('make test')  } }
  • 58. Create a Pipeline job pipelineJob('test'){  definition {    cps {      script(buildScript)    }  } }
  • 59. Set job properties pipelineJob('test'){  description('Test Build')  parameters {   booleanParam('verbose', false, 'Be Verbose')  }  logRotator {   numToKeep(10)  } }
  • 60. Loops config.each(){ jobConfig ­>   pipelineJob(jobConfig.name) {     triggers {       if (jobConfig.triggers?.scm) {         scm(jobConfig.triggers.scm)       }     }   } }
  • 61. Create a view listView('Project A') {     recurse()     jobs {         regex('myprj/[^/]+')     }     columns {         status()         weather()         name()         lastSuccess()         lastFailure()         lastDuration()         lastBuildConsole()         buildButton()         progressBar()     } }
  • 62. Work with plugins buildMonitorView("OnScreen Status") {  jobs {   name('WatchA')  } } ${JENKINS_URL}/plugin/job-dsl/api-viewer/
  • 63. In Pipeline jobDsl removedJobAction: 'DELETE',   removedViewAction: 'DELETE',   targets: 'seeds/*.groovy'
  • 64. Unleash the power of groovy import jenkins.model.Jenkins; dslCfg = Jenkins.instance.getDescriptor(   "javaposse.jobdsl.plugin.    GlobalJobDslSecurityConfiguration") dslCfg.useScriptSecurity = false == (sec issue with old versions of the plugin) (or power user trick)
  • 65. Load extra libraries jobDsl additionalClasspath: 'src/yaml.jar',   removedJobAction: 'DELETE',   removedViewAction: 'DELETE',   targets: 'seeds/*.groovy'
  • 66. Read yaml import org.yaml.snakeyaml.Yaml Yaml reader = new Yaml() Map config = reader.load(   readFileFromWorkspace('cfg.yaml') Drop the snakeyaml jar file and add it to classpath
  • 67. Jenkins Pipeline Creative Commons Attribution 2.0 https://www.flickr.com/photos/amerune/9294639633
  • 68. Jenkins Pipeline Jenkins Jobs as Code "Jenkins file" Imperative (aka scripted) Pipeline Declarative Pipeline (2017)
  • 69. What is a Jenkinsfile (aka Pipeline) A file that contains the definition of a job No need of Gui Defines Steps, Reports, Environments, Nodes,... Plugins can provide steps Generic "step"
  • 70. How to write Pipelines? Visual Pipeline editor (WIP) "Pipeline Syntax" link in jobs
  • 71. Scriped pipeline node ("Ubuntu && amd64") {   stage('checkout') {     checkout scm   }   stage('build') {     sh 'make'   }   stage('test') {     sh 'make test'   } }
  • 72. Real World Scripted Pipeline node ("Ubuntu && amd64") {   checkout scm   stage('build') {     try {       sh 'make'     } catch(err) {       currentBuild.result = 'UNSTABLE'       publishArtifacts('err.log')       throw err     }   }   stage('test') {     sh 'make test'   }   step([$class: 'JavadocArchiver',   javadocDir: "target/site", keepAll: true]) }
  • 73. Declarative Pipeline (1/2) pipeline {   agent {       label("Ubuntu && amd64")   }   options {       timeout(time: 235, unit: 'MINUTES')       timestamps()       ansiColor('xterm')   }   stages {     stage('build') {       steps {           sh('make')       }    } }
  • 74. Declarative Pipeline (2/2) pipeline {  post {   always {    junit params.jUnitReports   }  } }
  • 75. Declarative vs scripted Declarative checkout code by default Declarative get a workspace "OOTB" Declarative enforces everything in stages Declarative allow Pipeline-wide wrappers (e.g ansiColor, timestamps)
  • 76. Going further with Pipeline Global Libraries Plugins Job DSL Plugin
  • 77. Jenkins Nodes CC0 Public Domain https://www.flickr.com/photos/133259498@N05/27077266322/
  • 78. Docker Docker Docker Run jobs inside containers Clean, short lived containers Easy to update Docker Plugin
  • 79. Docker nodes pattern Build Container Tag it with a tag "candidate" Push it to your registry Run normal testing Run actual builds with that "candidate" If success -> tag with "release" && push
  • 80. In practice Docker Plugin config automated with groovy Candidate and Release tags are setup as slaves They get two labels: "image" "tag" (e.g. "build- centos-7" "candidate") Jobs get a parameter "tag"
  • 81. In the build Jenkinsfile pipeline {  agent {   label("build­centos­7 && ${params.tag}")  } }
  • 82. In the docker-build Jenkinsfile dockerImage = docker.build("build­centos­7",   "­­no­cache") dockerImage.tag('candidate') docker.withRegistry(url, credentials) {   dockerImage.push('candidate') } node("build­centos­7 && candidate") {   sh('test­script') } build job: 'myjob', parameters:   [string(name:'tag', value:'candidate')]    dockerImage.tag('release') docker.withRegistry(url, credentials) {   dockerImage.push('release') }
  • 83. Pros/Cons Updated containers won't block the builds (e.g on packages updates) Containers stay up to date Don't forget that builds release artifacts (sometime you don't want that in nodes tests)
  • 84. Docker Moby Docker
  • 85. Jenkins Master in Docker WHY WOULD YOU DO THAT? Atomic upgrade of all plugins at the same time Easy rollback / update Easy to test Reduce the cost to upgrade
  • 86. Jenkins master in Docker At which price????? Do not run ANYTHING on master; slave everything Keep a Docker Registry with history Assume Docker instability Think docker deployment etc... (A pipeline for your Jenkins master - how cool is this?)
  • 87. Docker and Jenkins Jenkins upstream images: alpine/ubuntu LTS and weekly
  • 88. Dockerfile FROM jenkins:2.46.2­alpine MAINTAINER dev­team@example.com USER root COPY jenkins/userContent/    /var/lib/jenkins/userContent RUN mkdir /var/log/jenkins &&    mkdir /var/lib/jenkins &&    mkdir /var/cache/jenkins   chown ­R jenkins: /var/log/jenkins &&    chown ­R jenkins: /var/lib/jenkins &&    chown ­R jenkins: /var/cache/jenkins
  • 89. Timezone (if needed) yay alpine RUN apk add ­­no­cache tzdata &&    cp /usr/share/zoneinfo/Europe/Brussels      /etc/localtime &&    echo "Europe/Brussels" > /etc/timezone &&    apk del tzdata
  • 90. Install plugins USER jenkins COPY jenkins/plugins.txt /tmp/plugins.txt RUN cat /tmp/plugins.txt | xargs   /usr/local/bin/install­plugins.sh /usr/local/bin/install-plugins.sh is provided in the upstream docker image plugins.txt cloudbees­folder cucumber­reports:4.3.2 docker
  • 91. init.groovy.d COPY jenkins/init.groovy.d/    /usr/share/jenkins/ref/init.groovy.d/ Note: if you keep Jenkins config in a volume, you need to suffix your files .override to update them
  • 92. JAVA_OPTS ENV JAVA_OPTS="­Xmx8192m    ­Djenkins.install.runSetupWizard=false" ENV JENKINS_OPTS="­­handlerCountMax=300    ­­logfile=/var/log/jenkins/jenkins.log    ­­webroot=/var/cache/jenkins/war"
  • 93. Testing the image Jenkins Pipeline init.groovy.d switch: if (System.getenv()['DO_NOT_BUILD']){     Jenkins.instance.doQuietDown() }
  • 94. Block until started That script will fail until Jenkins is started: #!/bin/bash ­xe curl | grep 'All [Jenkins]' ­q || grep "WARNING: Failed to run script file"  /var/log/jenkins/jenkins.log || grep "SEVERE: Failed GroovyInitScript.init"  /var/log/jenkins/jenkins.log retry(30) {   sleep 10   sh("""docker exec ${dockerContainer.id}      bash ­xe /tmp/block­until­started.sh""") }
  • 95. Testing init.groovy.d set ­xe if grep "SEVERE: Failed GroovyInitScript"     /var/log/jenkins/jenkins.log ||   grep "WARNING: Failed to run script file"     /var/log/jenkins/jenkins.log then     sleep 600 # Let some time to investigate     echo "FOUND ERRORS IN THE LOGS"     false fi
  • 96. Seed jobs Somewhere in init.groovy.d cause = new RemoteCause('localhost',  'Build triggered by init.groovy.d') job = Jenkins.instance.getItem("seed") cAction = new CauseAction(cause) build = job.scheduleBuild2(0, cAction) res = build.get().getResult().toString() assert res == 'SUCCESS' Last lines check for completion of seed job
  • 97. Benefits A Pipeline to Deploy Jenkins Automated checks of init.groovy.d etc Check that the seed job works
  • 98. Going further Will the fun ever stop?
  • 99. Trash the Jenkins config Only keep build history Do notput config in persistent volume Let init.groovy.d/job dsl build your config only from code Still, keep job history
  • 100. The glue import jenkins.model.Jenkins; import static groovy.io.FileType.FILES def rawBuildsBaseDir = '/srv/jenkins/state' Jenkins.instance.setRawBuildsDir(   "${rawBuildsBaseDir}/${ITEM_FULL_NAME}") /srv/jenkins/state is a persistent volume it will keep build history
  • 101. Conclusion Public Domain https://commons.wikimedia.org/wiki/File:YellowOnions.jpg
  • 102. Jenkins can be FULLY automated My recommendations: init.groovy.d Jenkins Job DSL Pipeline Try dockerized Jenkins ; might work for you (spoiler: works for us)
  • 103. Julien Pivotto roidelapluie roidelapluie@inuits.eu Inuits https://inuits.eu info@inuits.eu Contact
  • Related Search
    We Need Your Support
    Thank you for visiting our website and your interest in our free products and services. We are nonprofit website to share and download documents. To the running of this website, we need your help to support us.

    Thanks to everyone for your continued support.

    No, Thanks

    We need your sign to support Project to invent "SMART AND CONTROLLABLE REFLECTIVE BALLOONS" to cover the Sun and Save Our Earth.

    More details...

    Sign Now!

    We are very appreciated for your Prompt Action!