SFTP File Transfer Testing Via Jenkins In Docker
Overview
This blog will shows you one of practical testing scenarios which clearly demonstrates how to test SFTP file transfer.
ref: What is SFTP Server?
The test is really simple but to achieve that goal, we need to have some steps to setup a SFTP server which is accessible via Jenkins in docker. It means that Jenkins is able to upload/download file from the server in docker environment.
Additionally, the test is using Python language so some steps to install Python on Jenkins environment are necessary. You can use Dockerfile
to make it done.
ref: install Python3 using Dockerfile
The blog post does not explain docker in depth but you can visit some reference links for details. Instead, it focuses on configuring Jenkins job and SFTP file transfer testing using paramiko
library.
And it is designed to work on Windows 10 Pro only, not on a macOS or Linux machine.
Before continuing, ensure you have installed the following tools
- Docker
You can install the latest version of Docker Desktop which contains both Docker and Docker-Compose tools
ref: download link - GitLab
During the steps, you will be required to download the source code to GitLab. Ensure you have installed and configured Git and authentication to your GitLab
Jenkins & SFTP Servers Startup
Follow the procedure below to set up a SFTP server to be accessible via Jenkins.
If you are currently working from home, VPN connection may prevent installing Jenkins plugins. Please consider disconnecting VPN temporarily before continuing the next steps.
-
Run Docker Desktop
-
Start Jenkins & SFTP server(the ssh server) containers using docker-compose
- Checkout the source code from GitLab(Link is TBD)
- Run the command
ssh-keygen -t rsa -m PEM -f ssh-key
inside the foldercentos
to generate public keyssh-key.pub
(skip passphrase by ENTER key)
- Change directory to project root folder and then run 2 following commands sequentially
docker-compose build
&docker-compose up -d
- Wait until both containers are successfully up and running
- Confirm by accessing Jenkins container
docker exec -it jenkins-server bash
- In the bash terminal of the container, do SFTP connection
sftp ftpuser@sftp-server
with passwordftpuser
- Check the connection established from Jenkins container to the SFTP server container
ref: SSH connection between 2 containers - You can connect to the server using IP instead of hostname
-
Access localhost:8080 to perform initial configuration on Jenkins
- Wait until the Unlock Jenkins page appears
- Get password by accessing Jenkins container
docker logs jenkins-server
- Paste the password into the Administrator password field and click Continue
- Install suggested plugins
- When doing Create the first administrator user then select Skip and continue as admin as screenshot
- And in the remaining screens select option as default (blue highlight)
- Then welcome screen is displayed
- Change password using Configure button
- Log in again with a new password
- Then Welcome page is displayed again
ref: Setup Wizard
- Wait until the Unlock Jenkins page appears
-
Install
Pyenv Pipeline
plugin from Jenkins Dashboard localhost:8080 >> Manage Jenkins >> Manage Plugins (Restart Jenkins after installed it)
Ref: Pyenv Pipeline plugin
Create & Configure Pipeline Job
Follow the procedure below to create and configure pipeline job to do the following
- Download the latest source code from GitLab
- Run test with current source code inside virtual environment
In the upcoming configuration, we will use an existing GitLab repository(Link is TBD) which contains 3 simple Python tests.
The test focuses on
- uploading a file to SFTP server using
paramiko
library - downloading a file from SFTP server using
paramiko
library - checking specific content of the text file using regular expression and
lxml
library to parse/get content file
Now please follow the next step-by-step
- Create a new item for pipeline job
- Copy the script below to pipeline definition and save it
node {
stage('Clone code from Git') {
// The below will clone your repo and will be checked out to master branch by default.
git credentialsId: 'auto-credentials', url: 'https://gitlab.shiftasia.com/thuanpham/jenkins-sftp-sample.git', branch: 'master'
}
stage('Run test') {
// Run test inside virtual environment
withPythonEnv('/usr/bin/python3') {
sh 'pip install -r requirements.txt'
sh 'pytest'
}
}
}
The path /usr/bin/python3
has been utilized for Python virtual environment.
For pipeline syntax, click the link for details
You can generate the following command by using it
git credentialsId: 'auto-credentials', url: 'https://gitlab.shiftasia.com/thuanpham/jenkins-docker-sample.git'
- Run the job
If you are currently working from home, VPN connection is required to run the job. The job will download source code from GitLab.
- Click Console Output to view the log
- Stop Jenkins server and SFTP server by the command
docker-compose down
- Restart 2 servers by the command
docker-compose up -d
That's all for the example, thanks for reading.
Enjoy!