SFTP File Transfer Testing Via Jenkins In Docker

Overview

This blog will shows you one of practical testing scenarios which clearly demonstrates how to test SFTP file transfer.

ref: What is SFTP Server?

The test is really simple but to achieve that goal, we need to have some steps to setup a SFTP server which is accessible via Jenkins in docker. It means that Jenkins is able to upload/download file from the server in docker environment.

Additionally, the test is using Python language so some steps to install Python on Jenkins environment are necessary. You can use Dockerfile to make it done.

ref: install Python3 using Dockerfile

The blog post does not explain docker in depth but you can visit some reference links for details. Instead, it focuses on configuring Jenkins job and SFTP file transfer testing using paramiko library.

And it is designed to work on Windows 10 Pro only, not on a macOS or Linux machine.

Before continuing, ensure you have installed the following tools

1. Docker
   You can install the latest version of Docker Desktop which contains both Docker and Docker-Compose tools
   ref: download link
2. GitLab
   During the steps, you will be required to download the source code to GitLab. Ensure you have installed and configured Git and authentication to your GitLab

Jenkins & SFTP Servers Startup

Follow the procedure below to set up a SFTP server to be accessible via Jenkins.

If you are currently working from home, VPN connection may prevent installing Jenkins plugins. Please consider disconnecting VPN temporarily before continuing the next steps.

1. Run Docker Desktop
   

2. Start Jenkins & SFTP server(the ssh server) containers using docker-compose

   • Checkout the source code from GitLab(Link is TBD)
   • Run the command ssh-keygen -t rsa -m PEM -f ssh-key inside the folder centos to generate public key ssh-key.pub (skip passphrase by ENTER key)
     
   • Change directory to project root folder and then run 2 following commands sequentially docker-compose build & docker-compose up -d
     
   • Wait until both containers are successfully up and running
   • Confirm by accessing Jenkins container docker exec -it jenkins-server bash
   • In the bash terminal of the container, do SFTP connection sftp ftpuser@sftp-server with password ftpuser
   • Check the connection established from Jenkins container to the SFTP server container
     
     ref: SSH connection between 2 containers
   • You can connect to the server using IP instead of hostname
     

3. Access localhost:8080 to perform initial configuration on Jenkins

   • Wait until the Unlock Jenkins page appears
     
   • Get password by accessing Jenkins container docker logs jenkins-server
     
   • Paste the password into the Administrator password field and click Continue
   • Install suggested plugins
   • When doing Create the first administrator user then select Skip and continue as admin as screenshot
     
   • And in the remaining screens select option as default (blue highlight)
   • Then welcome screen is displayed
   • Change password using Configure button
     
   • Log in again with a new password
   • Then Welcome page is displayed again
     
     ref: Setup Wizard

4. Install Pyenv Pipeline plugin from Jenkins Dashboard localhost:8080 >> Manage Jenkins >> Manage Plugins (Restart Jenkins after installed it)
   
   Ref: Pyenv Pipeline plugin

Create & Configure Pipeline Job

Follow the procedure below to create and configure pipeline job to do the following

• Download the latest source code from GitLab
• Run test with current source code inside virtual environment

In the upcoming configuration, we will use an existing GitLab repository(Link is TBD) which contains 3 simple Python tests.

The test focuses on

• uploading a file to SFTP server using paramiko library
• downloading a file from SFTP server using paramiko library
• checking specific content of the text file using regular expression and lxml library to parse/get content file

Now please follow the next step-by-step

1. Create a new item for pipeline job
   
2. Copy the script below to pipeline definition and save it

node {
    stage('Clone code from Git') {
        // The below will clone your repo and will be checked out to master branch by default.
        git credentialsId: 'auto-credentials', url: 'https://gitlab.shiftasia.com/thuanpham/jenkins-sftp-sample.git', branch: 'master'
    }

    stage('Run test') {
        // Run test inside virtual environment
        withPythonEnv('/usr/bin/python3') {
            sh 'pip install -r requirements.txt'
            sh 'pytest'
        }
    }

}

The path /usr/bin/python3 has been utilized for Python virtual environment.
For pipeline syntax, click the link for details

You can generate the following command by using it

git credentialsId: 'auto-credentials', url: 'https://gitlab.shiftasia.com/thuanpham/jenkins-docker-sample.git'

3. Run the job
   If you are currently working from home, VPN connection is required to run the job. The job will download source code from GitLab.
   
4. Click Console Output to view the log
   
   
   
   
5. Stop Jenkins server and SFTP server by the command
   docker-compose down
6. Restart 2 servers by the command docker-compose up -d

That's all for the example, thanks for reading.

Enjoy!