Nameless There are only two hard things in Computer Science

Toggle Debug for containerized OpenStack services

The general case with kolla based containers

The configuration file is generally bind-mounted to a staging directory in the container then copied via the set_configs.py script to the place as specified in /var/lib/kolla/config_files/config.json file when the container starts. This means that changes to the config files on the host will not affect the running containers, ensuring better isolation. You would have to take a conscious decision and restart the container for the config change to take effect. This is for example how Kolla-ansible and TripleO proceed.

Say we want to bump the verbosity for the nova_scheduler service. In a TripleO deployment, we would do:

$ sudo crudini --set /var/lib/config-data/puppet-generated/nova/etc/nova/nova.conf DEFAULT debug true 
$ sudo docker restart nova_scheduler

Now we’re getting the DEBUG messages:

$ sudo grep DEBUG /var/log/containers/nova/nova-scheduler.log | wc -l
325

To revert back, we need to do the inverse change:

$ sudo crudini --set /var/lib/config-data/puppet-generated/nova/etc/nova/nova.conf DEFAULT debug false 
$ sudo docker restart nova_scheduler

Reload config at runtime

Another – and better – option would be to take advantage of the oslo.config mutable config mechanism that allows changing service settings at runtime. This has the big advantage that we’re only changing the configuration in the container and we can easily revert back by restarting the container.

Toggling the debug option at runtime was accepted as a community goal for Rocky meaning that all services should support it in the near future. This is already the case for nova to let’s see how it works in practice to turn debug on with our previous nova_scheduler example. This time, we’re editing the config file directly in the container and send it a signal telling it to reload the configuration:

$ sudo docker exec -u root nova_scheduler sed -ie 's/^[# ]*debug=false/debug=true/I' /etc/nova/nova.conf
$ sudo docker kill -s SIGHUP nova_scheduler

After we’re done debugging, turning the log severity back to normal is just a matter of restarting the container:

$ sudo docker restart nova_scheduler

Essential tools of an OpenStack contributor

This year again, I had the chance to attend the OpenStack Day France one day event in Paris and present about the tools we use in the OpenStack community to contribute to the project.

My talk covered the different kinds of contribution and highlighted some of the lesser known – but essential – tools available, giving tips and tricks learned from years of practice. I hope both the casual contributor and the more experienced one can find useful nuggets of information in this presentation.

This time, all the sessions were recorded and the videos should be available shortly. Overall, I found this event was very well organized with a nice crowd of around 400 attendants. Kudos to the OpenStack-fr team for making this second edition a success. See you there next year!

Cleanup local git branches - the Gerrit case

If you’ve been working with git for long enough you’ve probably noticed the local branches tend to accumulate. The git branch command provides a handy option to filter branches where the tip commits also belongs to the current branch:

git branch --merged

This is great and works fine in the majority of case and is often found in oneliners to cleanup local branches but this is not enough when working with workflows where the commits are often rewritten a few times before they eventually make it in tree, for example the Gerrit workflow we use in OpenStack. Gerrit uniquely identifies the commit with the Change-Id, a small label it inserts in the commit message and we can make advantage of this to check if a commit has been merged or not and delete the local branch.

First we need to get the Change-Id:

# Get Change-Id of $commit
change_id=$(git log -n1 --pretty=format:%b $commit | awk '/Change-Id:/ {print $0}')

With this Change-Id we can now verify if it was merged or not in the current branch:

# Check that commit was merged into $current_branch
merged_commit=$(git log --pretty=format:%h --grep "$change_id" ${current_branch})
if [ -z "$merged_commit" ]; then
    # This change is missing from $current_branch
fi

Great, so now we can find commits that have not yet been merged in the current branch. But what about the ones that were merged? It is entirely possible someone pushed a new version of the patch in Gerrit, or maybe I made local changes to a patch I haven’t yet submitted for review and the change was merged in the meantime. So if I blindly delete the branch I may lose important local changes. How can I check the two versions of the patch are the same? The interdiff tool compares diff files and we can assume they’re the same when the output is empty:

if [[ $(interdiff <(git show $commit) <(git show $merged_commit) 2>&1) ]]; then
    # The patch that was merged differs from what I have in local branch
fi

Putting it all together:

#!/bin/bash

function prompt_for_missing_commit {
    commit=$1
    branch=$2
    current_branch=$3
    git log --oneline -n1 $commit
    read -p "Commit $commit in $branch is missing from $current_branch. Inspect? [Yn] " answer
    if ! [[ "${answer,,}" =~ ^(n|no)$ ]]; then
        git show $commit
    fi
}

function prompt_for_commit_diff {
    local_commit=$1
    merged_commit=$2
    local_branch=$3
    current_branch=$4
    git log --oneline -n1 $commit
    read -p "Commit $local_commit in $local_branch and $merged_commit in $current_branch differ. Inspect? [Yn] " answer
    if ! [[ "${answer,,}" =~ ^(n|no)$ ]]; then
        interdiff <(git show $local_commit) <(git show $merged_commit) | colordiff
    fi
}

current_branch=$(git symbolic-ref --short HEAD)

for branch in $(git for-each-ref --format='%(refname:short)' refs/heads/); do
    if [ "$branch" == "$current_branch" ]; then
        continue
    fi
    echo
    echo "Checking branch $branch"
    branch_differs=0
    for commit in $(git log --no-merges --pretty=format:"%h" ${current_branch}..${branch}); do
        change_id=$(git log -n1 --pretty=format:%b $commit | awk '/Change-Id:/ {print $0}')
        if [ -z "$change_id" ]; then
            branch_differs=1
            prompt_for_missing_commit $commit $branch $current_branch
            continue
        fi
        merged_commit=$(git log --pretty=format:%h --grep "$change_id" ${current_branch})
        if [ -z "$merged_commit" ]; then
            branch_differs=1
            prompt_for_missing_commit $commit $branch $current_branch
            continue
        else
            # Check that the merged patch is similar to what is in local branch
            # NOTE needs interdiff from patchutils and colordiff
            if [[ $(interdiff <(git show $commit) <(git show $merged_commit) 2>&1) ]]; then
                branch_differs=1
                prompt_for_commit_diff $commit $merged_commit $branch $current_branch
            fi
        fi
    done
    if [ $branch_differs -eq 0 ]; then
        read -p "$branch fully merged. Delete? [yN] " answer
        if [[ "${answer,,}" =~ ^(y|yes)$ ]]; then
            git branch -D $branch
        fi
    else
        read -p "$branch differs from $current_branch. Delete anyway? [yN] " answer
        if [[ "${answer,,}" =~ ^(y|yes)$ ]]; then
            git branch -D $branch
        fi
    fi
done

Challenges in containerizing OpenStack

A presentation I made for the first edition of OpenStack Day France in Paris end of November that covers some of the challenges faced by the Kolla project in its effort to containerize OpenStack:

Setting up Gertty for effective code review

Gertty is a console-based client for the Gerrit code review tool. Once you start doing a lot of code reviews as part of your daily work, gertty proves to be an excellent alternative to the Gerrit web interface. For one, it can work offline so it means it’s possible to review code while on the move. It also means the interface is more reactive since everything is cached locally. Plus the fact that it’s a console tool is really appealing to programmers who spend their life in the terminal.

However, like all great tools, it needs a bit of configuration it in order to deliver its full potential. I’m going to document here a few of the settings I’m using in the context of OpenStack.

Vim key bindings

If like me you like using the vim key bindings whenever possible, then you’re in luck. Gertty ships with a vi keymap and allows you to remap most of the controls. To enable the vi bindings simply set keymap: vi in your .gertty.yaml file. However, don’t stop here or you’ll be heading for serious disappointment. The default vi keymap is rudimentary with only the four direction keys hjkl and you’ll need to redefine most of the combinations to feel at home.

Gertty supports key sequences in the form of [key1, key2] so you can easily emulate a leader key or a command like :q.

Here is the configuration I am using for reference:

keymaps:
  - name: vi
    cursor-down: ['j', 'down']
  - name: vi
    cursor-up: ['k', 'up']
  - name: vi
    cursor-left: ['h', 'left']
  - name: vi
    cursor-right: ['l', 'right']
  - name: vi
    cursor-page-up: ['ctrl u', 'page up']
  - name: vi
    cursor-page-down: ['ctrl d', 'page down']
  - name: vi
    cursor-max-left: ['^', 'home', 'ctrl a']
  - name: vi
    cursor-max-right: ['$', 'end', 'ctrl e']
  - name: vi
    quit: [[':', 'q'], 'ctrl q']
  - name: vi
    interactive-search: ['/']
  - name: vi
    toggle-hidden: [['t', 'h']]
  - name: vi
    toggle-star: [['t', '*']]
  - name: vi
    toggle-reviewed: [['t', 'r']]
  - name: vi
    toggle-list-reviewed: [['t', 'R']]
  - name: vi
    toggle-subscribed: [['t', 's']]
  - name: vi
    toggle-list-subscribed: [['t', 'S']]

Right now, the only thing I’m missing are the G and gg movements to go to the bottom or top of the panel. They are apparently missing from the underlying urwid library.

Dashboards

In the spirit of the web interface, gertty also allows you to define your own dashboards. The syntax is a bit different though, and as a consequence the official OpenStack dashboards can’t be imported directly and need to be adapted slightly. Unfortunately, I couldn’t find documentation for gertty’s query interface, so until someone finds the courage and time to write it down, the source code remains your best resource.

Here is for example what I’m using for the Kolla project:

dashboards:
  - name: "My changes"
    query: "owner:self status:open"
    key: "f2"
  - name: "Incoming reviews"
    query: "is:open is:reviewer"
    key: "f3"
  - name: "Kolla: Needs feedback"
    query: "project:^openstack/kolla.* status:open NOT owner:self label:Workflow>=0 NOT (label:Code-Review<=-1 or label:Code-Review>=1) age:2d"
    key: "f4"
  - name: "Kolla: Needs Approval"
    query: "project:^openstack/kolla.* status:open NOT owner:self label:Workflow>=0 (label:Verified>=1,jenkins or label:Verified>=1,zuul) label:Code-Review=2 NOT label:Code-Review<=-1"
    key: "f5"
  - name: "Kolla: No negative feedback"
    query: "project:^openstack/kolla.* status:open NOT owner:self (label:Verified>=1,jenkins or label:Verified>=1,zuul) NOT (label:Code-Review<=-1 or label:Code-Review>=1)"
    key: "f6"
  - name: "Kolla: Backports"
    query: "project:^openstack/kolla.* status:open NOT owner:self branch:stable/mitaka OR branch:stable/liberty"
    key: "f7"
  - name: "Kolla: Disagreement"
    query: "project:^openstack/kolla.* status:open (label:Verified>=1,jenkins or label:Verified>=1,zuul) label:Code-Review<=-1 label:Code-Review>=1"
    key: "f8"

Keyword replacement

The OpenStack Gerrit instance conveniently replaces the git commit message tags it understands with links to Launchpad issues or blueprints. Gertty offers that feature through keyword replacements.

Here is for instance how to mimic that behavior:

commentlinks:
  # Match external references to bugs on Launchpad
  - match: "(?P<bug_str>(?:[Cc]loses|[Pp]artial|[Rr]elated)-[Bb]ug *: *#?(?P<bug_id>\\d+))"
    replacements:
      - link:
          text: "{bug_str}"
          url: "https://launchpad.net/bugs/{bug_id}"
  # Match external references to blueprints on Launchpad
  - match: "(?P<bp_str>(?:[Bb]lueprint|bp) +(?P<blueprint>[\\w\\-.]+))"
    replacements:
      - link:
          text: "{bp_str}"
          url: "https://blueprints.launchpad.net/openstack/?searchtext={blueprint}"

Filter job results by pipelines

That last one hasn’t yet made it to gertty. This is somewhat specific to OpenStack gerrit instance and how it works with Zuul. The above commit lets you filter the job results by pipeline and displays useful information like the date or the number of rechecks.

See what gertty looks like with the patch applied:

Gertty in action