Gitlab-CI include: from private repository

Published: 2018-10-24 Tags: gitlab, CI Category: Automation

Note: The native support for including files from other non-public repositories was added in Gitlab 11.7, so this workaround is no longer needed.

Gitlab added new include statement for the CE edition of their product with the 11.4.0 release, which is great news. However there's this one minor thing mentioned in the documentation:

Note: The remote file must be publicly accessible through a simple GET request, as we don't support authentication schemas in the remote URL.

This makes it a little bit tricky to use the functionality if you want to fetch the file from private repository. Here's how I got it working. (TL/DR at the end)

Including the file🔗

Let's pretend we have repository called templates/ci. In this repository there's to_include.yml file that we would like to include.

In completely different repository, we have .gitlab-ci.yml file. From this file we'd like to include the to_include.yml. The naive approach is to open the remote file in gitlab web interface and click the Open raw button. We'll get an URL that's gonna look something like:

https://example.com/templates/ci/raw/master/to_include.yml

So let's try that:

# cat .gitlab-ci.yml
include: "https://example.com/templates/ci/raw/master/to_include.yml"

But this won't work as our file is in private repository and gitlab will throw us 302 instead trying to redirect us to login page. So that's a fail. We need to authenticate somehow.

Preparing the access account🔗

So the idea is, that we will use personal token of a user with read-only access to the repository that's going to serve us the file we want to include. If you're Gitlab admin, you can create such user from the Admin area.

In the same area you can create impersonation token for this user. (alternatively if you just want to use your personal user account directly, you can just create personal access token, that will work as well. The token will look something like: eYjPgAuz4Avr2LtZacmA. (I've made this one up, yours will be different) Make sure the api scope is checked.

Next step is to add that user to the repository we want to fetch the file from. Make sure the user is at least Reporter to allow fetching files. (Guest won't work)

Adding token🔗

Now that we have the token, let's use it for the request. We are limited to GET parameters, because include: can only do GET requests, but fortunately Gitlab supports that as well. Here's our CI configuration with token:

# cat .gitlab-ci.yml
include: "https://example.com/templates/ci/raw/master/to_include.yml?private_token=eYjPgAuz4Avr2LtZacmA"

Now that should work? Right? Well wrong:

Error: Remote file 'https://example.com/templates/ci/raw/master/to_include.yml?private_token=eYjPgAuz4Avr2LtZacmA' is not valid. 

Uhh. So this is a bit cryptic. What actually went wrong is, that include: is really meant to be used for simple GET calls and it expects to GET a static yml file. It tries to read our URL and sees that this is some weird file called:

to_include.yml?private_token=eYjPgAuz4Avr2LtZacmA

It's not even yml bro, sorry. Unless..

Adding .yml "extension"🔗

Fortunately Gitlab will happily ignore any parameters it doesn't know. So we can add one like this:

# cat .gitlab-ci.yml
include: "https://example.com/templates/ci/raw/master/to_include.yml?private_token=eYjPgAuz4Avr2LtZacmA&.yml"

And that finally works.

TL/DR:🔗

include: "https://example.com/some/repo/raw/branch/file.yml?private_token=<token>&.yml"

Hope that helps.