The idea of this document is to present a step-by-step procedure on how to use GitHub and Jenkins to create a CI/CD environment in a Pentaho context without having to install the PDI Client into Jenkins server, i.e., using only RESTful API calls.
The picture below represents a typical customer environment.
The Pentaho Transformations and Jobs are developed using PDI Client (Spoon) and commited/pushed to GitHub using GitHub Desktop or CLI.
At Jenkins, a build is manually or automatically executed, which will temporarily download such Transformations and Jobs to its workspace and
subsequently inject them into the Pentaho Server Enterprise Repository, which can be a QA or Production system.
Repeating this process, as the development evolves, will create multiple versions in GitHub and in the Pentaho Server Enterprise Repository.
Note: all the names, directories, IP addresses and ports are for exemplification only.
1- At the PDI Client (Spoon), create a File Repository in the local desktop.
2- At the PDI Client (Spoon), in Tools -> Repository -> Explore... create folders to organize code
3- Create a new repository at GitHub
4- At the GitHub Desktop, select File -> Clone repository
5- Select the URL tab, fill in the fields and click Clone
Local path: C:\MyWorkspace\Pentaho\Repositories\FileRepository1\demos\ContinousIntegration
6- At the PDI Client (Spoon), save the Jobs and Transformations into /demos/ContinousIntegration
Looking at the Windows Desktop Explorer, you can check the directory content:
7- At the GitHub Desktop, fill in the Summary field and click Commit to main
8- To upload to the GitHub, at the GitHub Desktop click on Publish branch
9- The repository at GitHub has been updated
10- At host sandbox1 (192.168.120.132), pull a Jenkins Docker image and start a Jenkins container specifying a persistent storage:
# docker pull jenkins/jenkins
# docker run -d -v /fs2/jenkins/data:/var/jenkins_home -p 8084:8080 -p 50000:50000 --name jenkins jenkins/jenkins
11- Browse to http://192.168.120.132:8084
user / password
Follow the instructions to unlock Jenkins using the password at /var/jenkins_home/secrets/initialAdminPassword
Customize Jenkins installing the recommended plugins.
12- In Jenkins, verify if the Git plugin is already installed. If you have already installed the Git plugin in response to the prompt during the Jenkins' installation setup, you don´t need to install the GitHub plugin.
At the Dashboard, go to the Manage Jenkins
Click on Manage Plugins:
In the Plugins Page, select the Installed tab and verify if the Git plugins are installed.
If not, select the Available tab and install them.
13- At the Jenkins Dashboard, create a new job
Insert an item name and create a Freestyle project
At the project page, under Source Code Management, there will be an option for Git, if the Git plugin has been installed correctly.
If such option does not appear, re-install the Git plugins and restart Jenkins.
Select the Git option and enter the repository name used for your Pentaho project
After specifying the Repository URL, if you get the error "Failed to connect to repository...", that is because the git command is not present in the host in which Jenkins is installed.
If the repository is not public, it is a private one, Jenkins wil validate the Github credentials when pulling the source code.
Click the Add drop-down and select Jenkins to use "Jenkins Credential Provider", this will display the following screen
Expecify the Git username and password and save.
Back to the project screeen, at Credentials, select the username specified.
At the Branches to build, the field Branch Specifier, leave it blank.
At the Build Triggers, you can specify how to automate the build. In this document, the build will be executed manually.
At the Build Environment, you can specify build options. In this document, none option is used.
At the drop-down menu Add build step, select Execute shell.
Insert the following commands:
curl -H "Content-Type: application/xml" -d @Job1.kjb -X PUT -u admin:password -i http://192.168.120.105:8080/pentaho/api/repo/files/:home:demos:CI:Job1.kjb
curl -H "Content-Type: application/xml" -d @Transformation1.ktr -X PUT -u admin:password -i http://192.168.120.105:8080/pentaho/api/repo/files/:home:demos:CI:Transformation1.ktr
curl -H 'Authorization:Basic YWRtaW46cGFzc3dvcmQ=' -H "Content-Type: application/xml" -d @Transformation2.ktr -X PUT -i http://192.168.120.105:8080/pentaho/api/repo/files/:home:demos:CI:Transformation2.ktr
Upon build, the Pentaho Jobs and Transformations from GitHub will be downloaded into the Jenkins workspace.
Then, these commands will inject the Pentaho Jobs and Transformations from the Jenkins workspace into the Pentaho Enterprise Repository in a Pentaho Server at host sandbox5 (192.168.120.105).
The directory in the Pentaho Enterprise Repository will be /home/demos/CI
The third command is using base64 to encode the credentials, as an example on how to hide username and password.
14- At the Dashboard, execute Build Now and verify in the Build History the status of the job
Click on the build number and then Click on console output to see the status of the build
15- Verifying that the Jobs and Transformations have been injected into the target Pentaho Enterprise Repository
16 –As the CI/CD process evolves, multiple versions are tracked by the Pentaho repository, that can be restored at will.
A proud part of Hitachi Vantara
© Hitachi Vantara Corporation. All Rights Reserved.