Shallow Git Clone On Jenkins Pipeline

Overview

On a beautiful day, your Jenkins can't clone from the Git repository like every day before.

The error may appear GitException with status code 128

Due to the error, you cannot clone the repository, run the test, commit the result by Jenkins.

How to deal with it?

You should first determine what the cause of the error is. In my case, the current working repository contains a lot of commit history and there will be more in the future.

There is a simple solution called "shallow clone", it is a feature of git to reduce server load. These clone truncate the commit history to reduce the clone size and copy only recent revisions. It’s fast clone, save time and disk space.

ref: What is Shallow Clone?

You just use the –depth option. For example

git clone --depth [depth] --branch [branch] [remote-url] [directory]

Apply shallow git clone to Jenkins Pipelines

Assuming you have a Jenkins Pipeline to clone the complete repository as follows, the pipeline works fine before the above error appears

node {
    stage('Download source code from Git') {
        // the below will clone your repo completely and will be checked out to master branch by default
        git credentialsId: 'auto-credentials', url: 'https://gitlab.shiftasia.com/thuanpham/jenkins-docker-sample.git', branch: 'master'
    }
	
    stage('Run Test') {
        // download docker image with specific version
        docker.image('python:3.6.8-alpine').inside {
            // check Python version
            sh 'python --version'
            sh 'pip list'
            withEnv(["HOME=${env.WORKSPACE}"]) {
                // install libraries required for testing
                sh 'pip install -r requirements.txt --user'
                // check libraries installed
                sh 'pip list'
                // execute test and update test result to input.csv
                sh 'python -m pytest -n 2 --dist=each'
            }
        }
    }
	
    stage('Upload test result to Git') {
	// save local changes to stash
	sh 'git stash -u'
	// pull the latest commit if any
	withCredentials([usernamePassword(credentialsId: 'auto-credentials', passwordVariable: 'PASSWORD', usernameVariable: 'USERNAME')]) {
            sh "git pull https://'$USERNAME:$PASSWORD'@gitlab.shiftasia.com/thuanpham/jenkins-docker-sample.git master"
        }
	// get local changes from stash
	sh 'git stash apply'
    
        sh 'git add input.csv'
        sh '''git commit -m "Commit from Jenkins $(date +"%d-%m-%Y")" '''
        withCredentials([usernamePassword(credentialsId: 'auto-credentials', passwordVariable: 'PASSWORD', usernameVariable: 'USERNAME')]) {
            sh "git push https://'$USERNAME:$PASSWORD'@gitlab.shiftasia.com/thuanpham/jenkins-docker-sample.git master"
        }
    }
}

In the first stage, you need to update as follows to change complete clone to shallow clone
And the remaining stages, please keep them as they are.

stage('Download source code from Git') {
	// clean workspace if the current workspace is not empty 
	//cleanWs()

	// check txt file in current directory existing or not
	def exitCode = sh script: 'find . -name "*.txt" | egrep .'
	boolean exists = exitCode == 0
	
	// if the file is existing
	if (existing) {
		// pull the latest commits if the current repo is shallow clone
		withCredentials([usernamePassword(credentialsId: 'auto-credentials', passwordVariable: 'PASSWORD', usernameVariable: 'USERNAME')]) {
			sh "git fetch --depth 1"
			sh "git reset --hard origin/master"
			sh "git fetch -dfx"
		}
	} else {
		// shallow clone the repo without project folder to the workspace
		// the command works only when the current workspace is empty. Otherwise, try to run cleanWs()
		withCredentials([usernamePassword(credentialsId: 'auto-credentials', passwordVariable: 'PASSWORD', usernameVariable: 'USERNAME')]) {
			sh "git clone --depth 1 --branch master https://'$USERNAME:$PASSWORD'@gitlab.shiftasia.com/thuanpham/jenkins-docker-sample.git ."
		}
	}
}

ref: how-to-update-a-git-shallow-clone

Not only the error has been resolved, but it has also significantly improved the time to clone the repository on Jenkins. The solution works fine in my case.

Just note that the current repo is shallow clone. If you want to convert it to complete/full clone, please refer to the link how-to-convert-a-git-shallow-clone-to-a-full-clone

That's all for the example, thanks for reading.

Enjoy it!