Nameless

GitOps for the Free Cloud: Bridging Kubernetes and OpenStack with ORC

2026-06-18T00:00:00+00:00

Following up on the ORC intro from last year’s OpenInfra Summit, I presented at DevConf.cz how ORC brings true GitOps to OpenStack infrastructure management. Since ORC exposes OpenStack resources as Kubernetes objects, standard GitOps tools like Argo CD or Flux can manage them like any other.

Also check out the recorded demo and its companion repository.

Manage your OpenStack resources from Kubernetes with ORC

2025-10-18T00:00:00+00:00

Stephen and I introduced the OpenStack Resource Controller (ORC), a set of kubernetes CRDs and controller to manage OpenStack resources, at the 2025 OpenInfra Summit.

Hello Hubtty 👋

2021-10-10T00:00:00+00:00

Part of my responsibilities at my day job involves reviewing other people’s code. There can be a fairly large number of Pull Requests in many different repositories that I need to monitor. In the past, while I was working on OpenStack, I was quite fond of Gertty to give me a quick glance of the new PRs and help me spend my time better. I missed this tool after I changed teams and most of the code reviews happened on Github rather than Gerrit.

About a year ago, I decided to scratch my own itch and looked at how hard it would be to port Gertty to the Github API and workflow. It turned out to not be so difficult: Gertty is well designed and it’s a joy to work with its source code. Hubtty was born.

Meet Hubtty

How Hubtty came to be

OK, so where do I start? How do I port Gertty from Gerrit to Github? A quick look at the Github REST API tells me it should be possible to port most of the Gertty’s features relatively easily. I considered for a minute using a python library to talk to the Github API but rapidly put this idea aside as it would mean rewriting a lot of the sync module: Gertty uses the requests library to talk to the Gerrit REST API directly and I figured I could simply reuse the same call, but with the Gitub API instead as the two APIs are more or less compatible.

After experimenting a bit, I was quickly able to retrieve the list of PRs for each repositories the user had subscribed to. Excited with this early success I set myself a goal to send the first review from Hubtty before the end of the year (it was November) which I barely did.

At this time, Hubtty was still relying on an existing authorization token that you had to create manually from the Github UI or “borrow” from another application. Next step was to implement the device flow so that Hubtty could create Authorization tokens on the user’s behalf. This part was straightforward but I realized it now forces the organizations to explicitely approve Hubtty when they have enabled third-party application restrictions otherwise it wouldn’t be able to submit reviews. This is still an issue to this day.

After I started subscribing to more repositories I ran into issues with the Github API rate limiting. The issue was rather obvious and quickly fixed. A less obvious issue gave me a hard time and only manifested when I got back to using Hubtty after some time – think PTO – where it would only sync part of the PRs. It was an problem with how I handled the pagination for some queries, leading to this nasty synchronization issue.

Another big part was to port the search syntax to the Github search. As of today, I’ve implemented the most important search operators and left out things we do not yet synchronize with Hubtty.

Finally, I’ve also made Hubtty consistently use the Github terminology. For example, it now uses Repository over Project (which means something entirely different for Github), and Pull Request over Change to list the most obvious. This was quite painful.

I used Hubtty as my inbox for PRs for the major part of this year and it was getting more stable, to the point I’m no longer ashamed of recommending it to others.

What it looks like now

On Friday I released the first version of Hubtty, v0.1. It has reached a state where it has feature parity with Gertty, or close enough, and is also stable enough to be used by other developers.

It’s available through the cheese shop, install it with:

pip install hubtty

Take a look:

List of subscribed repositories

List of PRs in a subscribed repository

Opening a PR

Viewing code changes

Plans ahead

In the coming versions, I’d like to have it integrate better with the Github features and improve UX. There is a lot to do, among other things:

Markdown rendering
Highlight PR with merge conflicts
Highlight draft PRs
Sync draft reviews
Autocompletion for usernames
Cherry-pick PRs
Reactions
…

I would also like to have hubtty packaged for the major distros. This should be relatively straightforward considering gertty has already done the work for us.

I hope it’s going to be as useful to you as it is to me.

Test it, give feedback, and report issues and feature requests on the Github project page.

Happy hacking!

Flashing Qualcomm-powered Android device from Linux

2020-09-27T00:00:00+00:00

Today I got fed up with the stock Android ROM – and the lack of security updates – that came with my tablet and decided to switch to a better alternative.

The instruction for my device said to use QFIL to flash it. However, this is a Windows program and none of my computers run Windows. The instruction had a few steps with the word important written in red bold UPPERCASE, probably as a mean to discourage you experimenting and risk bricking your device. The forum thread wasn’t also very helpful to provide instruction for Linux.

A quick Internet search later, I was on the qdl page where the tool description says:

As open source tool (for Linux) that implements the Qualcomm Sahara and Firehose protocols has been developed by Linaro, and can be used for program (or unbrick) MSM based devices, such as Dragonboard 410c or Dragonboard 820c.

This seems like it! However, no mention of QPST or QFIL, the two tools I read about earlier. I decided to trust the nice folks at Linaro and indeed, this was exactly the tool that I needed.

As of today, there are no distribution packages available so we’ll have to build from source:

$ git clone https://git.linaro.org/landing-teams/working/qualcomm/qdl.git
$ cd qdl
$ sudo dnf install libxml2-devel libudev-devel
$ make

When in EDL mode, the device identifies itself as a Qualcomm modem and we need to turn off ModemManager to prevent it from interfering:

sudo systemctl stop ModemManager

From there, I would change to directory where I unzipped the TWRP archive and run qdl:

$ sudo ./qdl/qdl --debug prog_emmc_firehose_8953_ddr.mbn rawprogram0.xml patch0.xml
Waiting for EDL device

After getting my device into EDL 9008 mode, in my case connecting it to my computer while being powered off and holding the volume up button, qdl noticed the device and started flashing it:

HELLO version: 0x2 compatible: 0x1 max_len: 1024 mode: 0
READ image: 13 offset: 0x0 length: 0x34
READ image: 13 offset: 0x34 length: 0x120
...
READ image: 13 offset: 0x53504 length: 0x720
END OF IMAGE image: 13 status: 0
DONE status: 0
qdl: failed to read: Connection timed out
FIREHOSE WRITE: <?xml version="1.0"?>
...
[PROGRAM] flashed "recovery" successfully at 15323kB/s
...

I could then follow the rest of the instruction and finish rooting my device.

After installing the custom ROM from TWRP and fixing a bootloop because I stupidly forgot to install the gapps, I got everything working and am ready to enjoy using my tabled again.

Bringing Kano to non-English speakers

2019-05-05T00:00:00+00:00

What is Kano?

Kano is a small educational Linux distribution to teach computers to kids. Kano is great: if you have kids, definitely check it out.

It runs on a cheap Raspberry Pi, the system is free. It’s perfect. Or almost… It’s only available in English or Spanish and the company behind Kano, despite a few promises in that sense, hasn’t yet made it available to other languages.

However because Kano is Open Source, meaning you can modify it to your liking, we do not rely on them to provide us with localized builds to enjoy the distro in another language. With a few tweaks, you can do it yourself. Here is how.

Make the system available in your language

Turn on your Kano computer and make sure it has network connectivity. Go to Advanced and enable SSH. Then you can connect to it via SSH and change the language of the system:

sudo dpkg-reconfigure locales

Select which locales you want to enable. For instance for French, I’m enabling fr_FR.UTF-8. It’s usually a good idea to always build en_US.UTF-8. On the next screen you’ll be able to select which locale will be the default one.

To make your change take effect, log out and log in again. If you don’t know what that means, simply restart your Kano computer.

Now that you’re back in Kano you’ll notice that the interface is… still in English. What happened? Didn’t we just changed the system locales? Yes we did. In fact if you open an application such as the file browser or the calculator, you’ll see it applied your language preference. These applications are originated from the wider Open Source community and include translation files, contrary to the Kano developed applications.

We need to create those files. This is where you can help.

Translating Kano applications

Translating software can be fun and doesn’t require any particular technical knowledge. All the sentences or words from the original software are extracted into a file that serves as a template for new translation files.

There are several services that allow translating files online. Transifex, Pootle, and Zanata are among the most well known.

I took the liberty to upload the Kano template files to Zanata because it is free and offers a public instance. So far, I’ve created translation projects for:

Make Snake - the snake game
Terminal Quest - teach the use of the terminal
Kano Overworld - the application behind the Story Mode

For these projects I uploaded the existing translation files I could find on the Internet and started contributing French translations.

Create yourself an account, join the projects, and start translating.

Apply the changes

This is where it gets more complex. In an ideal world, the translation files are included with the software and shipped with the next release. In this case however, we’ll have to upload the files to the Kano computer. This location is different for each soft.

For Make Snake

Clone the make-snake repository:

git clone git@github.com:KanoComputing/make-snake.git

Download the .po file, for example for French, and copy it to the po directory as fr.po. The file name has to match the language code. Finally run make from the po directory:

cd po
make

Copy the resulting locale/fr/LC_MESSAGES/make-snake.mo file to /usr/share/locale/fr/LC_MESSAGES/make-snake.mo on the Kano computer.

For Terminal Quest

For Terminal Quest, you’ll follow the same process:

git clone git@github.com:KanoComputing/terminal-quest.git
# Add the translated .po file to the po directory
cd po
make

Then copy the resulting terminal-quest.mo to /usr/share/locale/YOUR_LOCAL/LC_MESSAGES/ on the Kano computer.

For Story Mode

Kano-overworld is a different beast. It’s a LÖVE application built using the lua programming language. The translation template comes from the kano-overworld-i18n-orig package, but I have yet to figure a way to turn the translated .po files back into a lua dict that the application expects. So what I did instead was to copy the original language resource and edit it in place.

The “Story Mode” files live in /usr/share/kano-overworld/build/kanoOverworld.love. It’s simply a zip archive.

sudo apt-get install zip
mkdir kano-overworld
cd kano-overworld
unzip /usr/share/kano-overworld/build/kanoOverworld.love
cp -r res/locales/en_US/ res/locales/fr_FR/
vi res/locales/fr_FR/lang.lua

Then you can update the resources with:

zip -9 -r /usr/share/kano-overworld/build/kanoOverworld.love res/

Here is how it looks in French:

And now what?

This is only the beginning. I’m hoping that people will jump on board and help translate the Kano applications in their languages.

I’ll certainly push scripts to make this whole experience less hacky and make is easier to update the distribution with the latest translations.

It’s a bit unfortunate the Kano developers don’t take i18n too seriously. I understand they have business imperative and may not want to deal with support for non-English users but that shouldn’t prevent them from sharing the tools they use internally and make it easier for the community to provide translation files.

Perhaps one day Kano – the company – will integrate these language files in their projects and finally open Kano – the distribution – to more users?

Toggle Debug for containerized OpenStack services

2018-03-28T00:00:00+00:00

The general case with kolla based containers

The configuration file is generally bind-mounted to a staging directory in the container then copied via the set_configs.py script to the place as specified in /var/lib/kolla/config_files/config.json file when the container starts. This means that changes to the config files on the host will not affect the running containers, ensuring better isolation. You would have to take a conscious decision and restart the container for the config change to take effect. This is for example how Kolla-ansible and TripleO proceed.

Say we want to bump the verbosity for the nova_scheduler service. In a TripleO deployment, we would do:

$ sudo crudini --set /var/lib/config-data/puppet-generated/nova/etc/nova/nova.conf DEFAULT debug true 
$ sudo docker restart nova_scheduler

Now we’re getting the DEBUG messages:

$ sudo grep DEBUG /var/log/containers/nova/nova-scheduler.log | wc -l
325

To revert back, we need to do the inverse change:

$ sudo crudini --set /var/lib/config-data/puppet-generated/nova/etc/nova/nova.conf DEFAULT debug false 
$ sudo docker restart nova_scheduler

Reload config at runtime

Another – and better – option would be to take advantage of the oslo.config mutable config mechanism that allows changing service settings at runtime. This has the big advantage that we’re only changing the configuration in the container and we can easily revert back by restarting the container.

Toggling the debug option at runtime was accepted as a community goal for Rocky meaning that all services should support it in the near future. This is already the case for nova to let’s see how it works in practice to turn debug on with our previous nova_scheduler example. This time, we’re editing the config file directly in the container and send it a signal telling it to reload the configuration:

$ sudo docker exec -u root nova_scheduler sed -ie 's/^[# ]*debug=false/debug=true/I' /etc/nova/nova.conf
$ sudo docker kill -s SIGHUP nova_scheduler

After we’re done debugging, turning the log severity back to normal is just a matter of restarting the container:

$ sudo docker restart nova_scheduler

Essential tools of an OpenStack contributor

2017-12-01T00:00:00+00:00

This year again, I had the chance to attend the OpenStack Day France one day event in Paris and present about the tools we use in the OpenStack community to contribute to the project.

My talk covered the different kinds of contribution and highlighted some of the lesser known – but essential – tools available, giving tips and tricks learned from years of practice. I hope both the casual contributor and the more experienced one can find useful nuggets of information in this presentation.

This time, all the sessions were recorded and the videos should be available shortly. Overall, I found this event was very well organized with a nice crowd of around 400 attendants. Kudos to the OpenStack-fr team for making this second edition a success. See you there next year!

Cleanup local git branches - the Gerrit case

2017-10-22T00:00:00+00:00

If you’ve been working with git for long enough you’ve probably noticed the local branches tend to accumulate. The git branch command provides a handy option to filter branches where the tip commits also belongs to the current branch:

git branch --merged

This is great and works fine in the majority of case and is often found in oneliners to cleanup local branches but this is not enough when working with workflows where the commits are often rewritten a few times before they eventually make it in tree, for example the Gerrit workflow we use in OpenStack. Gerrit uniquely identifies the commit with the Change-Id, a small label it inserts in the commit message and we can make advantage of this to check if a commit has been merged or not and delete the local branch.

First we need to get the Change-Id:

# Get Change-Id of $commit
change_id=$(git log -n1 --pretty=format:%b $commit | awk '/Change-Id:/ {print $0}')

With this Change-Id we can now verify if it was merged or not in the current branch:

# Check that commit was merged into $current_branch
merged_commit=$(git log --pretty=format:%h --grep "$change_id" ${current_branch})
if [ -z "$merged_commit" ]; then
    # This change is missing from $current_branch
fi

Great, so now we can find commits that have not yet been merged in the current branch. But what about the ones that were merged? It is entirely possible someone pushed a new version of the patch in Gerrit, or maybe I made local changes to a patch I haven’t yet submitted for review and the change was merged in the meantime. So if I blindly delete the branch I may lose important local changes. How can I check the two versions of the patch are the same? The interdiff tool compares diff files and we can assume they’re the same when the output is empty:

if [[ $(interdiff <(git show $commit) <(git show $merged_commit) 2>&1) ]]; then
    # The patch that was merged differs from what I have in local branch
fi

Putting it all together:

#!/bin/bash

function prompt_for_missing_commit {
    commit=$1
    branch=$2
    current_branch=$3
    git log --oneline -n1 $commit
    read -p "Commit $commit in $branch is missing from $current_branch. Inspect? [Yn] " answer
    if ! [[ "${answer,,}" =~ ^(n|no)$ ]]; then
        git show $commit
    fi
}

function prompt_for_commit_diff {
    local_commit=$1
    merged_commit=$2
    local_branch=$3
    current_branch=$4
    git log --oneline -n1 $commit
    read -p "Commit $local_commit in $local_branch and $merged_commit in $current_branch differ. Inspect? [Yn] " answer
    if ! [[ "${answer,,}" =~ ^(n|no)$ ]]; then
        interdiff <(git show $local_commit) <(git show $merged_commit) | colordiff
    fi
}

current_branch=$(git symbolic-ref --short HEAD)

for branch in $(git for-each-ref --format='%(refname:short)' refs/heads/); do
    if [ "$branch" == "$current_branch" ]; then
        continue
    fi
    echo
    echo "Checking branch $branch"
    branch_differs=0
    for commit in $(git log --no-merges --pretty=format:"%h" ${current_branch}..${branch}); do
        change_id=$(git log -n1 --pretty=format:%b $commit | awk '/Change-Id:/ {print $0}')
        if [ -z "$change_id" ]; then
            branch_differs=1
            prompt_for_missing_commit $commit $branch $current_branch
            continue
        fi
        merged_commit=$(git log --pretty=format:%h --grep "$change_id" ${current_branch})
        if [ -z "$merged_commit" ]; then
            branch_differs=1
            prompt_for_missing_commit $commit $branch $current_branch
            continue
        else
            # Check that the merged patch is similar to what is in local branch
            # NOTE needs interdiff from patchutils and colordiff
            if [[ $(interdiff <(git show $commit) <(git show $merged_commit) 2>&1) ]]; then
                branch_differs=1
                prompt_for_commit_diff $commit $merged_commit $branch $current_branch
            fi
        fi
    done
    if [ $branch_differs -eq 0 ]; then
        read -p "$branch fully merged. Delete? [yN] " answer
        if [[ "${answer,,}" =~ ^(y|yes)$ ]]; then
            git branch -D $branch
        fi
    else
        read -p "$branch differs from $current_branch. Delete anyway? [yN] " answer
        if [[ "${answer,,}" =~ ^(y|yes)$ ]]; then
            git branch -D $branch
        fi
    fi
done

Challenges in containerizing OpenStack

2016-12-12T00:00:00+00:00

A presentation I made for the first edition of OpenStack Day France in Paris end of November that covers some of the challenges faced by the Kolla project in its effort to containerize OpenStack:

Setting up Gertty for effective code review

2016-09-20T00:00:00+00:00

Gertty is a console-based client for the Gerrit code review tool. Once you start doing a lot of code reviews as part of your daily work, gertty proves to be an excellent alternative to the Gerrit web interface. For one, it can work offline so it means it’s possible to review code while on the move. It also means the interface is more reactive since everything is cached locally. Plus the fact that it’s a console tool is really appealing to programmers who spend their life in the terminal.

However, like all great tools, it needs a bit of configuration it in order to deliver its full potential. I’m going to document here a few of the settings I’m using in the context of OpenStack.

Vim key bindings

If like me you like using the vim key bindings whenever possible, then you’re in luck. Gertty ships with a vi keymap and allows you to remap most of the controls. To enable the vi bindings simply set keymap: vi in your .gertty.yaml file. However, don’t stop here or you’ll be heading for serious disappointment. The default vi keymap is rudimentary with only the four direction keys hjkl and you’ll need to redefine most of the combinations to feel at home.

Gertty supports key sequences in the form of [key1, key2] so you can easily emulate a leader key or a command like :q.

Here is the configuration I am using for reference:

keymaps:
  - name: vi
    cursor-down: ['j', 'down']
  - name: vi
    cursor-up: ['k', 'up']
  - name: vi
    cursor-left: ['h', 'left']
  - name: vi
    cursor-right: ['l', 'right']
  - name: vi
    cursor-page-up: ['ctrl u', 'page up']
  - name: vi
    cursor-page-down: ['ctrl d', 'page down']
  - name: vi
    cursor-max-left: ['^', 'home', 'ctrl a']
  - name: vi
    cursor-max-right: ['$', 'end', 'ctrl e']
  - name: vi
    quit: [[':', 'q'], 'ctrl q']
  - name: vi
    interactive-search: ['/']
  - name: vi
    toggle-hidden: [['t', 'h']]
  - name: vi
    toggle-star: [['t', '*']]
  - name: vi
    toggle-reviewed: [['t', 'r']]
  - name: vi
    toggle-list-reviewed: [['t', 'R']]
  - name: vi
    toggle-subscribed: [['t', 's']]
  - name: vi
    toggle-list-subscribed: [['t', 'S']]

Right now, the only thing I’m missing are the G and gg movements to go to the bottom or top of the panel. They are apparently missing from the underlying urwid library.

Dashboards

In the spirit of the web interface, gertty also allows you to define your own dashboards. The syntax is a bit different though, and as a consequence the official OpenStack dashboards can’t be imported directly and need to be adapted slightly. Unfortunately, I couldn’t find documentation for gertty’s query interface, so until someone finds the courage and time to write it down, the source code remains your best resource.

Here is for example what I’m using for the Kolla project:

dashboards:
  - name: "My changes"
    query: "owner:self status:open"
    key: "f2"
  - name: "Incoming reviews"
    query: "is:open is:reviewer"
    key: "f3"
  - name: "Kolla: Needs feedback"
    query: "project:^openstack/kolla.* status:open NOT owner:self label:Workflow>=0 NOT (label:Code-Review<=-1 or label:Code-Review>=1) age:2d"
    key: "f4"
  - name: "Kolla: Needs Approval"
    query: "project:^openstack/kolla.* status:open NOT owner:self label:Workflow>=0 (label:Verified>=1,jenkins or label:Verified>=1,zuul) label:Code-Review=2 NOT label:Code-Review<=-1"
    key: "f5"
  - name: "Kolla: No negative feedback"
    query: "project:^openstack/kolla.* status:open NOT owner:self (label:Verified>=1,jenkins or label:Verified>=1,zuul) NOT (label:Code-Review<=-1 or label:Code-Review>=1)"
    key: "f6"
  - name: "Kolla: Backports"
    query: "project:^openstack/kolla.* status:open NOT owner:self branch:stable/mitaka OR branch:stable/liberty"
    key: "f7"
  - name: "Kolla: Disagreement"
    query: "project:^openstack/kolla.* status:open (label:Verified>=1,jenkins or label:Verified>=1,zuul) label:Code-Review<=-1 label:Code-Review>=1"
    key: "f8"

Keyword replacement

The OpenStack Gerrit instance conveniently replaces the git commit message tags it understands with links to Launchpad issues or blueprints. Gertty offers that feature through keyword replacements.

Here is for instance how to mimic that behavior:

commentlinks:
  # Match external references to bugs on Launchpad
  - match: "(?P<bug_str>(?:[Cc]loses|[Pp]artial|[Rr]elated)-[Bb]ug *: *#?(?P<bug_id>\\d+))"
    replacements:
      - link:
          text: "{bug_str}"
          url: "https://launchpad.net/bugs/{bug_id}"
  # Match external references to blueprints on Launchpad
  - match: "(?P<bp_str>(?:[Bb]lueprint|bp) +(?P<blueprint>[\\w\\-.]+))"
    replacements:
      - link:
          text: "{bp_str}"
          url: "https://blueprints.launchpad.net/openstack/?searchtext={blueprint}"

Filter job results by pipelines

That last one hasn’t yet made it to gertty. This is somewhat specific to OpenStack gerrit instance and how it works with Zuul. The above commit lets you filter the job results by pipeline and displays useful information like the date or the number of rechecks.

See what gertty looks like with the patch applied: