Pipeline notifications from Gitlab to Matrix via Webhook Proxy

Published: 2019-03-11 Tags: gitlab, CI, matrix Category: Automation

Currently there's no official out-of-the-box integration available in Gitlab that would allow you to post messages via Matrix protocol. There are some 3rd party options available, like Gitlab plugin for Maubot, or the Slack-compatible Webhooks Matrix appservice, but it might not be ideal solution in some cases for one reason or another.

For me the reasons to look further can be mostly boiled to three things:

1. Some tool I'd like to use doesn't even support Slack
2. I don't want to have full featured bot running in the room, I just need to get notifications.
3. Lack of flexibility - mostly in terms of how the messages look like, but also in ability to pick and choose only specific messages.

Fortunately there is a tool that isn't really specific to Gitlab, nor is it aimed at usage with Matrix, but it is flexible enough to glue the two (and other tools) together. Let me introduce you to Webhook Proxy

Webhook Proxy🔗

The whole thing is pretty simple Python application, that is quite readable and reasonably well documented, which is already a good start. Go ahead and read the Readme, but to boil it down a bit, you essentially just configure the service using a yaml configuration file, where you define endpoints and then you define actions that should happen when the endpoint is reached.

To give you some quick introduction, start with this simple configuration:

server:
host: '127.0.0.1'
port: '5000'

endpoints:
- /endpoint/path:
    method: 'POST'

    headers:
        X-Sender: 'itsme .+'

    body:
        message: '.+'

    actions:
        - log:
            message: 'Posted message: {{ request.json.message }}'

The above configuration will start on port 5000 and if you POST json to /endpoint/path, it will do the following:

1. Check whether the headers contain X-Sender with value matching regular expression. In our case itsme buddy would work, itsnotme bro would not match.
2. Check if the client POST-ed valid Json and if it contains non-empty message string.

If above is true, the actions will happen. In our case we just log the received message to standard output. Not too useful, but it's a start.

It shows us four powerful features of Webhook Proxy:

1. You can define arbitrary endpoints that can receive various input.
2. You can easily verify that the submitted data are in proper format and contain expected values.
3. You can handle such requests with actions.
4. Actions can be templated using Jinja syntax, so they can react to received data.

Actions and templating🔗

You can probably already see where this is going. So let's zoom in on the actions and templating in scope of Gitlab-Matrix integration. We want to receive data via Gitlab webhook, then post it to Matrix.

The act of posting the message needs to be done in two steps:

Log in to obtain the token🔗

# ...
actions:
    - http:
        target: "https://server.example.com/_matrix/client/r0/login"
        json: true
        method: POST
        body:
          type: "m.login.password"
          identifier:
            type: "m.id.user"
            user: "username"
          password: "password"
          device_id: "DEVICEID"
          initial_device_display_name: "Gitlab"
        output: >
          Got token
          {% set _ = context.set('token', response.json().get('access_token')) %}
# ...

We're using the http action here. As you can see, we're doing POST request to /_matrix/client/r0/login endpoint and in the body formatted as json we send the credentials.

The tricky bit is that we're (ab)using the output template to actually save the received token to the context for the following step to use. This way, at the end of this action, we should see "Got token" in the logs and we should have token available in following templates as {{ context.token }}.

For the sake of simplicity we're putting the login values straight into the configuration, bu you could also just use some request GET parameter via {{ request.args.username }} for username for example. (Or POST it in the json if the tool supports that, in that case you would have it available as {{ request.json.username }}.)

Send a message to a room🔗

# ...
actions:
    # ... getting the token omitted here ...
    - http:
        target: "https://server.example.com/_matrix/client/r0/rooms/!nEXWXEcguUyLEXWXE:example.com/send/m.room.message/1"
        json: true
        method: PUT
        headers:
          Authorization: "Bearer {{ context.token }}"
        body:
          msgtype: m.text
          body: "Webhook from Gitlab received"
          format: "org.matrix.custom.html"
          formatted_body: "<h5>Webhook from Gitlab received</h5>"

Pretty simple eh? Note that we're using the context.token that we set in previous step to provide the Authorization header.

Again, there is room id hard coded for simplicity, but that can be provided as POST or GET data.

The sent message is quite simple "Webhook from Gitlab receives" in plaintext and formatted form. This is not all that much useful. We can make the message way more helpful, but first let's look at posted data.

Sending a webhook from Gitlab🔗

The webhook setup is quite well documented here. The webhook needs to point to our endpoint, that we defined above. In the introductory example, you'd point webhook URL to something like http://webhook.example.com/endpoint/path. Obviously it makes more sense to configure the endpoints in a way that makes sense to you - something like /ci/pipeline/updates might be more self explanatory

If you scroll further down the webhook documentation, you'll see sample data sent from Gitlab for different kind of events. You can use posted event data in your webhook configuration to make it more useful.

Using the POST data from Gitlab🔗

Let's try to work with the Push Event. You'll normally use this data in two places.

Validating the request🔗

First you probably want to make sure that the received callback is actually hitting the correct endpoint and that the data you expect to be present is actually there.

endpoints:
  - /gitlab/pipeline:
    headers:
      X-Gitlab-Event: Push Hook
    body:
      object_kind: push
      user_name: ".+"
      project:
        name: ".+"

In the above code, we tell Webhook Proxy to validate the request before proceeding with actions. It will make sure the proper X-Gitlab-Event header is sent. So for example pointing pipeline event webhook to the same endpoint by mistake will be caught early. We also check whether object_kind is set to push and that there's user and project name information present in POST data. This check is by no means extensive, but it's gonna be enough for our simple use case.

Using the posted data🔗

I'm going to use the posted data to show appropriate message in Matrix room:

# ...
actions:
    # ... getting the token omitted here ...
    - http:
        target: "https://server.example.com/_matrix/client/r0/rooms/!nEXWXEcguUyLEXWXE:example.com/send/m.room.message/1"
        json: true
        method: PUT
        headers:
          Authorization: "Bearer {{ context.token }}"
        body:
          msgtype: m.text
          body: "{{ request.json.user_name}} just pushed to {{ request.json.project.name }}"
          format: "org.matrix.custom.html"
          formatted_body: "<b>{{ request.json.user_name}}</b> just pushed to <i>{{ request.json.project.name }}</i>"

Notice how we use the posted data to send informative message to Matrix room. Now that's definitely more useful message to see. Applying the same principles, you can now announce pipeline status, reported issue, merge request and so on. Using Jinja template tests, conditions and other features can make the notification handling extremely flexible.