[Infrastructure as Code (IaC)] Troubleshooting Common Issues with GitLab Runner Shell Executor on CentOS

Posted on Edited on In , , Views: Word count in article: 3.7k Reading time ≈ 3 mins.

Troubleshooting Common Issues with GitLab Runner Shell Executor on CentOS

As a senior software engineer, I often encounter various challenges while working with GitLab Runner on CentOS. This blog post aims to address some common issues and provide solutions that can help streamline your CI/CD pipeline. Whether you’re upgrading Git, managing permissions for Docker, or dealing with runtime errors, this guide will offer practical steps to resolve these issues efficiently.

1. Fatal Error: git fetch-pack: expected shallow list

Issue:

When running your GitLab CI/CD pipeline, you might encounter the following error:

1
fatal: git fetch-pack: expected shallow list

Solution:

This issue often arises due to an outdated version of Git. To resolve it, upgrade Git to the latest version.

1
2
3
yum install http://opensource.wandisco.com/centos/7/git/x86_64/wandisco-git-release-7-2.noarch.rpm

yum upgrade git -y

2. Permission Denied: Docker Daemon Socket

Issue:

You may see an error stating:

1
ERROR: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock

Solution:

This error indicates that the gitlab-runner user does not have the necessary permissions to access the Docker daemon. You can either add the gitlab-runner user to the Docker group or run gitlab-runner as the root user.

Option 1: Add gitlab-runner to Docker group

1
2
3
4
5
# Create the Docker Group (if it doesn't already exist):
sudo groupadd docker

# Add gitlab-runner to the Docker group
sudo usermod -aG docker gitlab-runner

Option 2: Run gitlab-runner as root

Modify the GitLab Runner service file:

1
2
3
# /etc/systemd/system/gitlab-runner.service

ExecStart=/usr/bin/gitlab-runner "run" "--working-directory" "/home/gitlab-runner" "--config" "/etc/gitlab-runner/config.toml" "--service" "gitlab-runner" "--user" "root"

Then, check if the GitLab Runner is running with the correct user:

1
ps aux | grep gitlab-runner

3. Dubious Ownership in Repository

Issue:

Another common error you might face is:

1
fatal: detected dubious ownership in repository at

Solution:

To resolve this, add the repository to the list of safe directories.

1
git config --global --add safe.directory "*"

4. OCI Runtime Create Failed

Issue:

While starting a container, you might encounter the following error:

1
OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:319: getting the final child's pid from pipe caused "EOF": unknown

Solutions:

This issue can be caused by several factors. Here are three potential solutions:

Way 1: Increase kernel.pid_max

1
2
3
$ sysctl -n kernel.pid_max
32768
$ sysctl -w kernel.pid_max=100000

Way 2: Increase user.max_user_namespaces

1
2
3
4
$ sysctl -n user.max_user_namespaces
0  
# if zero try this
$ sysctl -w user.max_user_namespaces=15000

Way 3: Resolve Page Allocation Failure

Check for page allocation failures:

1
2
3
4
5
$ grep -w 'runc:\[1:CHILD\]: page allocation failure' /var/log/messages | tail -n 4
Nov 20 16:13:54 ETL010080 kernel: runc:[1:CHILD]: page allocation failure: order:4, mode:0x10c0d0
Nov 20 16:15:46 ETL010080 kernel: runc:[1:CHILD]: page allocation failure: order:4, mode:0x10c0d0
Nov 20 16:16:28 ETL010080 kernel: runc:[1:CHILD]: page allocation failure: order:4, mode:0x10c0d0
Nov 20 16:16:41 ETL010080 kernel: runc:[1:CHILD]: page allocation failure: order:4, mode:0x10c0d0

Solution 1: Drop Caches

1
echo 3 > /proc/sys/vm/drop_caches

Solution 2: Compact Memory

1
2
3
echo 1 > /proc/sys/vm/compact_memory
# or
sysctl -w vm.compact_memory=1

After applying these solutions, remember to restart the GitLab Runner for changes to take effect:

1
systemctl restart gitlab-runner

Conclusion

Working with GitLab Runner on CentOS can sometimes present challenges, but with the right approach and solutions, you can overcome these hurdles efficiently. By keeping your tools updated, managing permissions correctly, and tweaking system parameters, you can ensure a smooth CI/CD pipeline. If you encounter any other issues, feel free to explore the GitLab and Docker documentation or reach out to the community for further assistance.

Happy coding!