Continuous integration at home with Drone

Published: 2016-09-16 Tags: automation, NAS, drone, CI, docker Category: Linux

Has science gone too far? It might look so, because:

1. It feels like a huge over-engineering to implement CI at home.
2. What would you integrate anyway?
3. Just why?

Well let me explain my reasons.

My home setup provides a great opportunity to try out new stuff, because:

• There's no approval process, I decide what gets done.
• There is some space for experimenting, trying things just because they are interesting, not just because business needs them.
• Failures are generally of low impact.

However there are a lot of similarities with the professional environment:

• There are some quite strict constraints. The time and budget available is quite possibly much more limited, than what we work with in our day-to-day work life.
• The impact might be huge if stuff doesn't work - loosing photos from last holiday due to a mess up or explaining that my family can't watch movies, because I broke Plex again is sometimes harder, than doing a really painful post-mortem investigation at work.

With that in mind, let's see what kind of CI we can try to implement in short amount of time and with very limited resources.

Choosing the CI tools🔗

I want something self-hosted as SAAS solutions that I've found would be either too pricy, or very problematic to integrate - most of my home deployment isn't meant to be externally accessible. Also privacy is a factor here.

I'm using Docker heavily in my home NAS setup, so I'd like to have a solution that is very docker-native.

I need something simple - I have very limited HW resources and not much time to get it working. Also I like simple tools.

And last, but not least, it should be very easy to maintain - again, time is a precious resource here. Less time maintaining means more time doing other interesting stuff.

Meet Drone🔗

Finding Drone was a bit of a revelation. It is very docker-native, in fact it's using containers for almost everything - even for its own installation. It's dead simple to install and by default, all its configuration fits into a single sqlite and individual .drone.yml files (not unlike Travis's .travis.yml files) in individual repositories configuring the pipeline.

Also, there are no plugins to manage (in my opinion, the biggest blessing and curse of Jenkins) as these are just docker images, that are pulled as needed during a build. This also makes sure, that your local self-hosted deployment won't become that special snowflake server over time. Rebuilding it from scratch would probably be a matter of minutes. This all makes it very easy to maintain. Because plugins and build steps are just Docker images, you can extend your CI in any language you want. As long as you can read json on stdin and output json on stdout, you're sorted. All the plugin dependencies are in the docker image and the plugins don't really interact with each other, so there's no dependency hell to go through and plugins work equally well on your laptop as they do on your CI server.

Speaking about laptop, Drone comes with commandline utility, that you can install. (or just copy to be exact) In true Go fashion it comes with all dependencies in single binary, so you just need to provide Docker. With that you can start your builds locally and the build will fetch all the same images and do exact same steps as the server would do. That way you can test as much as you like and once you're done, you can just push changes to origin and you know it's just going to work exactly the same. You basically have your entire CI stack at your laptop. If you ask me, that's the coolest thing I've seen in a while.

Drone inherits the user management from your git server of choice, so you don't have new set of credentials to handle here. Again, one less thing to maintain.

If you ever worked with Travis, it will feel somewhat familiar with the configuration. If you worked with Concourse, it will feel like Déjà vu - Concourse is also very docker-native citizen, but I've found it a bit too limiting for various (mostly well intended) reasons.

Let's do some CI🔗

Enough words, let's do some continuous integration and deployment. By the way, I'm not going to describe the actual Drone installation and configuration - that is simple enough and well documented in the official documentation. Just follow it.

As I said, I'm using docker heavily. My NAS doesn't have samba installed directly, it runs a custom build samba container that fits my needs. My reasons are good flexibility, isolation between services and fast and easy way to rebuild the whole system if the need arises.

One friction point is keeping the container images up-to-date. So let's do something about it.

First, let's add a .drone.yml to our repository with samba image Dockerfile:

publish:
  docker:
    repo: samba
    tag: latest
    storage_driver: vfs
    insecure: true
    registry: registry.local:5000

deploy:
  ssh:
    host: nas.local
    port: 22
    user: drone
    commands:
      - sudo docker pull registry.local:5000/samba
      - sudo systemctl restart smb.service

What just happened? Here's what Drone does for us once we push the changes:

1. Checkout the repository code with the commit we just pushed. Note, that we didn't configure this anywhere. While you could define it explicitly, (for example to alter some default behavior) it does the right thing by default.
2. Run a container that will build image from the Dockerfile and if successful, will push it to the local registry. This is the publish part.
3. Connect to the NAS, pull the freshly built image and restart the systemd service to run the new image.

Not bad for 15 lines of code, right?

In somewhat unusual way, we're skipping the build phase as the publish docker plugin does the build for us. Again, Drone does what we need by default without explicitly configuring it.

Let's automate some more🔗

Let's create a new repository and drop two files there.

.drone.yml:

build:
  image: registry.local:5000/samba
  pull: true
  commands:
  - bash drone_check_updates.sh

notify:
  downstream:
    repositories:
      - docker/samba
    token: $$TOKEN
    server: http://drone.local:8000
    when:
      success: true

drone_check_updates.sh:

#!/usr/bin/env bash

# Check apt for updates
apt-get update -y && \
apt-get upgrade -sy && \
if apt-get upgrade -sy | fgrep -q '0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded' ;
then
    echo '== NO UPDATES FOR APT =='
    exit 1
else
    echo '== APT UPDATES AVAILABLE =='
    exit 0
fi

Commit, push and Drone will:

1. Pull the latest samba image we have
2. Run the drone_check_updates.sh script in it. This is a simple one, that just checks if there are any updates available, but you can do more here if you want.
3. If the scripts exits with zero (which means there are some updates) it will trigger a build on the docker/samba repository, which triggers rebuild and deployment we did above.

In other words, we now have a completely automated process of checking the update, rebuilding and redeploying the image if needed. All we need to do is to fire the build of the update repository via cron or something similar. Once you're familiar with Drone, it's probably about 10 minute job, possibly less.

Also notice how easy is to read the yaml files. Just a short glance and you have pretty good idea what will Drone do with these. If you ever had to go through numerous pages and hundreds of clicks just to understand someone's else CI configuration, because he wasn't around and you had to do a little change, this might be a dream come true right there.

So should we migrate from Jenkins or what?🔗

Well, it depends. In a way Drone is very simple tool, almost just a skeleton or a CI framework rather than a stack. It doesn't hide much from you like - say - Jenkins does. You can really see the clockwork behind the scenes. For some people that might be a scary view. They might prefer the cushion of Jenkin's web interface. However, the more you're adopting the DevOps methodology, the closer you get to the bones of the system anyways. Just be prepared, that the web interface really doesn't do much - it's merely a tool to watch the build status. You can restart a build, do some basic configuration and that's about it. Everything else is configured in the .drone.yaml - and I would consider that feature, but some people might think otherwise.

Also Drone will only do the CI for one git server. It's just the way it works. Obviously you can spin up more of them, so you're not really limited here, it's just something to keep in mind. For small to middle-sized team, this probably won't be a problem anyway. If you have entire company on GitHub, Bitbucket or similar, this won't be a problem at all. It should be able to use a shared DB and have a pool of workers, (those are just hosts with installed Docker) but I haven't tried that myself.

I found that sometimes new repositories don't get detected in Drone, I had to restart it to have them available in the interface. Although that might be due to the very early implementation of integration with Gogs.

I'm missing some built-in way to trigger builds periodically. (nightly builds for example) Sure you can do it from outside via API, but that's just one extra thing to take care of. I see how it doesn't really fit into the concept of yaml files scattered acros repositories - the whole tool is in a way quite decentralized. A single scheduler might be out of the scope, but I still miss it a bit. (But hey, there are many tools out there that will do just that..)

And last, as everything Docker based, it won't run on Windows. You could maybe try to bend Boot2Docker to do what you need, but just don't. If you ask me, this is feature, not a bug. You can still develop on Windows, obviously, you can push your code, it will get built as usual, so no harm there, but if any step of your build process requires windows executable you might be out of luck.

All these are just minor issues as far as I'm concerned. Overall I like it a lot. Should you migrate? Maybe. I'd certainly recommend to at least give it a try, it's very easy to do a trial run. Chances are you'll get hooked just like I did.

By the way, the webpage you're reading was generated by Nikola and then uploaded to S3 - all with a single git push. Whole process was implemented in about 10 minutes.

Not bad huh?