Pentaho

 View Only

How to use GitHub and Jenkins to create a CI/CD environment for Pentaho using its REST APIs

This thread has been viewed 12 times
  • 1.  How to use GitHub and Jenkins to create a CI/CD environment for Pentaho using its REST APIs

    Posted 11-18-2022 13:17

    - Introduction

     The idea of this document is to present a step-by-step procedure on how to use GitHub and Jenkins to create a CI/CD environment in a Pentaho context without having to install the PDI Client into Jenkins server, i.e., using only RESTful API calls.

     The picture below represents a typical customer environment.

     The Pentaho Transformations and Jobs are developed using PDI Client (Spoon) and commited/pushed to GitHub using GitHub Desktop or CLI.

     At Jenkins, a build is manually or automatically executed, which will temporarily download such Transformations and Jobs to its workspace and

    subsequently inject them into the Pentaho Server Enterprise Repository, which can be a QA or Production system.

     Repeating this process, as the development evolves, will create multiple versions in GitHub and in the Pentaho Server Enterprise Repository.   

     

     

    - Procedure

     Note: all the names, directories, IP addresses and ports are for exemplification only.

     

    1- At the PDI Client (Spoon), create a File Repository in the local desktop.

    C:\MyWorkspace\Pentaho\Repositories\FileRepository1

     

    2- At the PDI Client (Spoon), in Tools  ->  Repository  ->  Explore...  create folders to organize code

    /demos/ContinousIntegration

     

    3- Create a new repository at GitHub

    https://github.com/rrizzio/pentaho-test

     

    4- At the GitHub Desktop, select  File  ->  Clone repository

    5- Select the URL tab, fill in the fields and click Clone

    URL:  rrizzio/pentaho-test

    Local path: C:\MyWorkspace\Pentaho\Repositories\FileRepository1\demos\ContinousIntegration



    6- At the PDI Client (Spoon), save the Jobs and Transformations into /demos/ContinousIntegration 

    Looking at the Windows Desktop Explorer, you can check the directory content:



    7- At the GitHub Desktop, fill in the Summary field and click Commit to main



    8- To upload to the GitHub, at the GitHub Desktop click on Publish branch



    9- The repository at GitHub has been updated



    10- At host sandbox1 (192.168.120.132), pull a Jenkins Docker image and start a Jenkins container specifying a persistent storage:

    # docker pull jenkins/jenkins

    # docker run -d -v /fs2/jenkins/data:/var/jenkins_home -p 8084:8080 -p 50000:50000 --name jenkins jenkins/jenkins

     

    11- Browse to http://192.168.120.132:8084

    user / password

     

    Follow the instructions to unlock Jenkins using the password at  /var/jenkins_home/secrets/initialAdminPassword

    Customize Jenkins installing the recommended plugins.

     

    12- In Jenkins, verify if the Git plugin is already installed. If you have already installed the Git plugin in response to the prompt during the Jenkins' installation setup, you don´t need to install the GitHub plugin.

     

    At the Dashboard, go to the Manage Jenkins



    Click on Manage Plugins:



    In the Plugins Page, select the Installed tab and verify if the Git plugins are installed.

    If not, select the Available tab and install them.



    13- At the Jenkins Dashboard, create a new job



    Insert an item name and create a Freestyle project



    At the project page, under Source Code Management, there will be an option for Git, if the Git plugin has been installed correctly.

    If such option does not appear, re-install the Git plugins and restart Jenkins.

     

    Select the Git option and enter the repository name used for your Pentaho project



    After specifying the Repository URL, if you get the error "Failed to connect to repository...",  that is because the git command is not present in the host in which Jenkins is installed.

    If the repository is not public, it is a private one, Jenkins wil validate the Github credentials when pulling the source code.

     

    Click the Add drop-down and select Jenkins to use  "Jenkins Credential Provider", this will display the following screen



    Expecify the Git username and password and save.


    Back to the project screeen, at Credentials, select the username specified.

     



    At the Branches to build, the field Branch Specifier, leave it blank.



    At the Build Triggers, you can specify how to automate the build. In this document, the build will be executed manually.

     

    At the Build Environment, you can specify build options. In this document, none option is used.

    At the drop-down menu Add build step, select Execute shell.



    Insert the following commands:

     

    curl -H "Content-Type: application/xml" -d @Job1.kjb -X PUT -u admin:password -i http://192.168.120.105:8080/pentaho/api/repo/files/:home:demos:CI:Job1.kjb

     

    curl -H "Content-Type: application/xml" -d @Transformation1.ktr -X PUT -u admin:password -i http://192.168.120.105:8080/pentaho/api/repo/files/:home:demos:CI:Transformation1.ktr

     

    curl -H 'Authorization:Basic YWRtaW46cGFzc3dvcmQ=' -H "Content-Type: application/xml" -d @Transformation2.ktr -X PUT -i http://192.168.120.105:8080/pentaho/api/repo/files/:home:demos:CI:Transformation2.ktr



    Upon build, the Pentaho Jobs and Transformations from GitHub will be downloaded into the Jenkins workspace.

     

    Then, these commands will inject the Pentaho Jobs and Transformations from the Jenkins workspace into the Pentaho Enterprise Repository in a Pentaho Server at host sandbox5 (192.168.120.105). 

     The directory in the Pentaho Enterprise Repository will be /home/demos/CI

     The third command is using base64 to encode the credentials, as an example on how to hide username and password.

     Click Save

    14- At the Dashboard, execute Build Now and verify in the Build History the status of the job



    Click on the build number and then Click on console output to see the status of the build



    15- Verifying that the Jobs and Transformations have been injected into the target Pentaho Enterprise Repository



    16 –As the CI/CD process evolves, multiple versions are tracked by the Pentaho repository, that can be restored at will.






    ------------------------------
    Rogerio de Rizzio
    Master Technical Expert
    Hitachi Vantara
    ------------------------------