Categories on dd if=/dev/brain of=/var/log/site

Custom Inventories with OpenStack-Ansible

Fri, 31 Aug 2018 00:00:00 +0000

Over the years, many users of OpenStack-Ansible came to me with simple requests: How can I add an ansible group into OpenStack-Ansible? How can I add hosts to this group?

Please allow me to answer this in more detail here, with an explanation of how OpenStack-Ansible inventories work.

The `openstack-ansible` CLI’s inventory

One of the early steps of an OpenStack-Ansible deployment is ensuring Ansible is installed, with its right version and environment. This is done by our bootstrap-ansible.sh script.

During this process, we ensure ansible and openstack-ansible commands are in the user’s PATH¹. Both commands are, in fact, a wrapper around Ansible’s binaries with a little twist: We load environment variables from an openstack-ansible.rc².

This openstack-ansible.rc contains environment variables altering the runtime behavior of Ansible. This is done thanks to Ansible itself reading some predefined environment variables³. One of these environment variables is ANSIBLE_INVENTORY.

For your information, ansible<=2.3 only accepts one inventory at a time. When the OpenStack-Ansible project was created, all the flexibility had to be built into a single inventory.

That’s how OpenStack-Ansible dynamic inventory appeared, with its env.d appeared. The env.d is a way to structure the inventory seen by Ansible. It is a very powerful but unfriendly mechanism. Its usage has been described in OpenStack-Ansible’s documentation for many branches and releases now⁴.

However, because of the complexity behind it, many deployers struggle adapting the configuration of the inventory.

This is one of the reasons I very early brought the idea of layering inventories. My first attempt at it was a dynamic inventory (again!) loading the usual OpenStack-Ansible dynamic inventory, and merging it with a user provided static inventory.

This sadly increased complexity of the inventory code, but decreased the complexity for the users to manage their inventory.

It was very easy to override default OpenStack-Ansible inventory, by doing export ANSIBLE_INVENTORY=/opt/my_own_inventory.py.

Hopefully, this proof of concept never made it into OpenStack-Ansible tree, as we would have increased the code complexity and made the system harder to maintain.

After discussing with many people involved in Ansible (and particularily interested in Ansible’s inventory), I realised what we needed in terms of inventories was matching other requests in the community.

Ansible 2.4 made possible to load and merge multiple inventories in one command.

So when OpenStack-Ansible was updated to Ansible 2.4, I added the fact to load, by default, a static inventory.ini next to the dynamic inventory⁵.

How does this static inventory work in Queens?

In Queens, you could add your own inventory.ini in /etc/openstack_deploy and it would be taken automatically using the openstack-ansible CLI. That would allow you to extend your inventory with new hosts, and new host_vars and group_vars using static files, which is easier to extend than env.d.

In fact, if you don’t want to extend your inventory but add extra host and group vars, the openstack-ansible wrapper automatically creates the inventory.ini for you⁶.

However, there is no default structure present in OpenStack-Ansible Queens, which forced users to a write a complete inventory for OpenStack-Ansible in their /etc/openstack_deploy/inventory.ini.

So, what has improved in Rocky?

Before Rocky, users had to still rely on the OSA dynamic inventory to get a basic deployment of Openstack-Ansible, and were only using this static inventory to extend the dynamic inventory.

In Rocky, I ensured consistency and simplification of the playbooks’ targets.

I also introduced a default inventory file structure⁷, which allow users to populate those groups to get a working deployment with only a simple, static, inventory in userspace.

Here is an example for a bare-metal deployment.

aio1 ansible_connection=local ansible_host=127.0.0.1
# is_metal=True
## Setup hosts
[physical_hosts]
aio1
## Setup infra
[etcd_all]
[galera_all]
aio1
[haproxy]
[memcached]
aio1
[rabbitmq_all]
aio1
[repo_all]
aio1
[rsyslog]
[unbound]
[utility_all]
aio1
## Setup openstack
[keystone_all]
aio1
# Glance
[glance_api]
aio1
[glance_registry]
aio1
# Neutron
[neutron_agent]
aio1
[neutron_dhcp_agent]
aio1
[neutron_l3_agent]
aio1
[neutron_linuxbridge_agent]
aio1
[neutron_metadata_agent]
aio1
[neutron_server]
aio1
# Cinder
[cinder_api]
aio1
[cinder_backup]
aio1
[cinder_scheduler]
aio1
[cinder_volume]
aio1
# Nova
[nova_conductor]
aio1
[nova_api_placement]
aio1
[nova_console]
aio1
[nova_scheduler]
aio1
[nova_api_os_compute]
aio1
[nova_consoleauth]
aio1
[nova_api_metadata]
aio1
[nova_compute]
aio1
# Heat
[heat_api]
aio1
[heat_api_cfn]
aio1
[heat_api_cloudwatch]
aio1
[heat_engine]
aio1

The openstack_user_config has extra variables that would then be missing. You can add them as group vars or extra variables (depending on your needs). The minimum required variables to set up an openstack-ansible deployment could look like the following:

---
internal_lb_vip_address: 172.29.236.100
# The external IP is quoted simply to ensure that the .aio file can be used as input
# dynamic inventory testing.
external_lb_vip_address: "10.0.2.15"
rabbitmq_hosts_entries: []
neutron_provider_networks:
network_types: "vxlan,flat"
network_mappings: "flat:eth12"
network_vxlan_ranges: "1:1000"
neutron_local_ip: "{{ ansible_host }}"

Conclusion

With this article, I hope that you have enough information to extend the inventories the way you want for your OpenStack-Ansible version.

See also: https://github.com/openstack/openstack-ansible/blob/7189ca7f1ed950944911c3418bf4afee47699315/scripts/bootstrap-ansible.sh#L157-L171
^[return]
See also: https://github.com/openstack/openstack-ansible/blob/master/scripts/openstack-ansible.rc
^[return]
The predefined environment variables have changed over time. Check for example their definition as of today here: https://raw.githubusercontent.com/ansible/ansible/653d9c0f87f681ac386864bad4cb48f0c5e2ddfe/lib/ansible/config/base.yml
^[return]
See also: https://docs.openstack.org/openstack-ansible/rocky/reference/inventory/understanding-inventory.html
^[return]
See also: https://github.com/openstack/openstack-ansible/commit/ba6a3ed899de5f0b98386c20e736f61e58807c9b
^[return]
https://github.com/openstack/openstack-ansible/blob/39b718a5c12779bc15d8efc432cbadbe69745323/scripts/bootstrap-ansible.sh#L245-L249 ^[return]
Extra structure for inventory.ini: https://review.openstack.org/#/c/580368/5
^[return]

A review of OpenStack's LOCI and its extensibility

Tue, 28 Aug 2018 00:00:00 +0000

LOCI is a very simple openstack project focusing on building OCI images for OpenStack Services using a Dockerfile and a series of shell scripts.

Currently, 9 projects (Cinder, Glance, Heat, Horizon, Ironic, Keystone, Neutron, Nova, and Octavia) can be built with it.

Flexibility and extensibility

LOCI is relying on passing arguments to the Docker image building for its flexibility. It requires you to pass at least the PROJECT as CLI argument, to tell which OpenStack project you want to build.

The other parameters are optional. You can pass a base image as starting point (FROM) in argument if you do not want to build from the ubuntu base image. Other variables can be provided, like your personal pip web mirror, or proxy settings.

The image building’s shell scripts receives the appropriate arguments as environment variables. Many shell scripts do not need any updating to leverage this flexibility. If your use case is not met, I doubt it would be a problem to change the scripts to adapt your use case.

On top of that, you can of course build a child image to your needs.

The build process

The build process currently behaves like this:

python and virtualenv are installed on the base image, using distribution packages.
A system user is created for the project.
virtualenv is upgraded, unconstrained, in a virtualenv. I suppose if you want to have fully reproducible builds with no surprise on the virtualenv version, you could implement a bypass using a docker build argument.
pip get installed in previous virtualenv, unconstrained again. Previous comment applies here.
The bindep pip package and extra pip packages based on the project + profile (coming from a pydep.txt file at the root of the LOCI repo) are installed.

If a wheel built image was created, its upper constraints file is used for the installation of those pydeps + bindep pip packages. These packages are installed in a binary form (only), from your pip mirror of choice.

If no pip mirror can be trusted, it would be technically possible to implement a bypass by having an argument that flips the --only-binary parameter to --no-binary, and copying your own constraints file.
The destination project is cloned with the user provided reference (default is master branch’s HEAD).
The destination project, and any extra PIP_PACKAGES (should PIP_PACKAGES be not empty), are installed based on projects’ setup() content.
Distribution packages are installed at this point using bindep.txt file.
Cleanup of installed distro packages (and other extra artifacts) happen, and allows the system to use global packages.

Security and reproducibility

While I did not have the chance to do a thorough security review, here is my opinion: The “requirements” building, while optional, is in my opinion necessary for security.

LOCI can build wheels for all the openstack requirements’ upper-constraints in your environment, which can then be published in a secure location for your next project building. It means you would rely on OpenStack security team’s work, which would eventually update the requirements/constraints of an upstream project by blacklisting an insecure python package. If you do not build the wheels with the LOCI requirements project, there is no reason to think you will pass a constraints file matching your OpenStack projects versions. Therefore, you are probably using anything from PyPI matching each project’s requirements, which can be very variable.

Build speed

The image building itself is quite fast, due to the simplicity of the process. If you have a CI pipeline, you can first build your ‘requirements’ image, then use its artifact for all the other projects building in parallel, which should take around a total of 10 to 20 minutes maximum.

Gating expectations

It is expected in the current gating process to expose the wheels built in the ‘requirements’ image on a web server. You cannot use locally built files, unless you carry your own patch changing the base Dockerfile.

Learning pace

Because the project itself is very simple, it is easy to track what is happening there. It is easy to contribute, and a few minutes/hours should be enough to get a grip of the whole codebase.

Conclusions

LOCI is a very simple and efficient way to build OpenStack images. It relies on Dockerfiles, simple shell scripts, and the power of bindep to install OpenStack software from sources.

If you are looking for a way to build images based on distribution packages, there is little to no point of using LOCI, as you could as well build your image with a simple Dockerfile and a command like zypper install ${PACKAGE_LIST}. But that would also mean you will have to package your software and its dependencies (maybe for two different versions of Python), handle package conflicts, and other packaging work.

The packaging can be completely avoided if you build from source relying on the python ecosystem, OpenStack creation of upper constraints, and bindep management by upstream team(s).

Switching to the new gandi mail service

Sun, 22 Apr 2018 00:00:00 +0000

If you are a Gandi customer, you probably already know they offer an email service. What you might have missed, is Gandi’s services refactor to be more modern. Emails are following the same treatment, starting with the replacement of Roundcube.

The new gandi email service is based on SOGo, and was introduced in January last year in beta.

First, the impact of the change (not mentioning mailpack changes):

New domains can only have the new service, and only 2 mailboxes are provided by default.
Existing domains can keep their 5 mailboxes.
The new service now has 3GB storage per mailbox, instead of the 1GB storage total.
Switching to the new system prevents you from using the legacy platform and api, whether for renewing your domain or for managing your emails.
The new webmail has a new address.
SOGo brings new features, namely CalDAV and CardDAV support.

Switching your domain to the new service

Make sure you have a gandi v5 account on https://id.gandi.net
Go to domains > (your domain name) > Email. There is a switch button on the page to use the new service.

If you DNS Servers are outside gandi, ensure you have the following records:

@ 10800 IN MX 10 spool.mail.gandi.net.
@ 10800 IN MX 50 fb.mail.gandi.net.
@ 10800 IN TXT "v=spf1 include:_mailcust.gandi.net ?all"

Next to that, configure CalDAV/CardDAV records for the DAV clients:

imap IN CNAME access.mail.gandi.net.
pop IN CNAME access.mail.gandi.net.
smtp IN CNAME relay.mail.gandi.net.
_caldavs._tcp IN SRV 10 1 443 sogo3.gandi.net.
_carddavs._tcp IN SRV 10 1 443 sogo3.gandi.net.
_imaps._tcp IN SRV 10 1 993 mail.gandi.net.
_submission._tcp IN SRV 10 1 587 mail.gandi.net.
_caldavs._tcp IN TXT "path=/SOGo/dav/"
_carddavs._tcp IN TXT "path=/SOGo/dav/"

With all those records set, it should be easy for SOGo clients to work with your domain, if they respect RFCs like the RFC6764.

For Android, I strongly suggest you to use f-droid, and DAVDroid. With the above records set, you can just add your email, and everything will be auto-discovered.

Feedback

The service’s web interface is slow, terribly slow. It prevents a daily usage as a gmail/google calendar replacement. If you are only using another interface, like your phone, it is easier to switch to this new service, as you will not miss this new web interface.

The web interface option for WebCAL files seems broken. The import of the openstack combined releases calendar comes up just empty, while working perfectly fine in google calendar.

Thanks Gandi for this welcomed change!

ansible-inventory tips

Mon, 09 Apr 2018 00:00:00 +0000

The ansible-inventory purpose is to show Ansible inventory information, as stated by its --help message.

However, this quite youg CLI has an interesting trick up its sleeve.

Quick reminder, Ansible allows by default 4 kinds of inventories:

ini
script
host_list
yaml

Interesting fact number 1: All the inventory types above can be given as input of the ansible-inventory CLI.

Interesting fact number 2: The output of the ansible-inventory CLI is a valid json inventory.

Interesting fact number 3: ansible-inventory can output inventories into the yaml format.

You can therefore use ansible-inventory to chain, merge, or convert ansible inventories!

Here is an example of how it can be used:

The first type, ini inventories, is the what you probably have seen in your first usage of ansible, and may look like this:

[glance_api]
localhost ansible_connection=local
[glance_registry]
localhost2 ansble_connection=local
[glance_all:children]
glance_api
glance_registry

This ini file can therefore be converted into a yaml file:

$ ansible-inventory -i inventory.ini -y --list > inventory.yaml

The resulting yaml file would be:

all:
children:
glance_all:
children:
glance_api:
hosts:
localhost:
ansible_connection: local
glance_registry:
hosts:
localhost2:
ansble_connection: local
ungrouped: {}

If the -y was omitted, the output would be a json inventory:

$ ansible-inventory -i inventory.ini --list > inventory.json

Looking like:

{
"_meta": {
"hostvars": {
"localhost": {
"ansible_connection": "local"
},
"localhost2": {
"ansble_connection": "local"
}
}
},
"all": {
"children": [
"glance_all",
"ungrouped"
]
},
"glance_all": {
"children": [
"glance_api",
"glance_registry"
]
},
"glance_api": {
"hosts": [
"localhost"
]
},
"glance_registry": {
"hosts": [
"localhost2"
]
},
"ungrouped": {}
}

As you might have noticed, there is no json inventory type. The json is the standard interface used for the \“script\” inventory type.

A valid inventory \“script\” (also named dynamic inventory), is an executable which, when called with --list, outputs a json whose content is following ansible conventions, like the json above.

As such, the json above can be converted into a dynamic inventory script, by making it executable and showing its content.

Create a file yourscript.sh with the following content:

#!/bin/bash
cat <<EOANSIBLESCRIPT
<the content of the yaml above>
EOANSIBLESCRIPT

and then chmod +x yourscript.sh.

That file can now be converted into yaml with:

$ ansible-inventory -i ./yourscript.sh -y --list > inventory.yaml

Hope it helps you understand what can be done with the ansible-inventory CLI!

My view on Ansible 2.5

Sun, 08 Apr 2018 00:00:00 +0000

Ansible 2.5 was released last month.

If you didn’t read/hear about it, you should first read the ansible 2.5 release notes, ansible 2.5 changelog, and the ansible 2.5 porting guide.

Here is what I like and don’t like about this release:

Each module documentation is far more readable. Look at the stat module in 2.5 vs its documentation in 2.4 .
The module index documentation is far harder to automatically search. That’s a big nuisance if you know your modules and want to quickly search for modules parameters for example.
While this isn’t shown up in the release notes, we finally have an etcd module in ansible, for etcd3. The etcd2 module is still in the works, so is its lookup.
I do not have a strong opinion about using loop: vs with_. I just do not like the fact I will have to do a grep and replace for no foreseeable reason. It\’s replacing one evil with another. Same goes for replacing the pipe (|) with [is]{.title-ref} for the conditions (Although I agree that the latter is more readable).
I like the namespace ansible_facts. In fact (pun intended), I prefer using namespacing in many places. I even suggested using this mechanism in openstack-ansible for automatic variable loading in test environment in a test or required namespace.
The ansible config lookup might be interesting in the future.
We now have new modules for InfluxDB, and grafana.
A netns module was introduced, to have basic functionality of creation/deletion of namespaces.
An interesting package_facts module has been introduced. It automatically registers the list of packages installed on the host in the host vars, under the ansible_facts namespace. Similarily, you can use the service_facts module to know the current state of a service.
A vdo module was added. If you want to know more about it, check out this page for VDO install, and for checking the VDO benefits.

I’d like to add a big thanks to ricardocarillocruz, gundalow, and bcoca for their help on irc!

OpenStack Summit Sydney Recap

Sun, 19 Nov 2017 00:00:00 +0000

The venue

Due to its remote location, the OpenStack Summit in Sydney was smaller than other summits I attended. The venue was excellent, it wasn’t hard to find the rooms, and everything was provided to be able to work constantly (energy, wifi, food, and the usual beverages!). Sydney itself was a great location. The only downside was that nightlife in Sydney: it almost becomes a dead city after midnight!

The sessions

OpenStack summits never lack interesting sessions. You have to balance what you want to attend and be a good time juggler, to capitalize every minute. For this recap, I will separate my session overview in three tracks: The recorded sessions, the etherpad based conversations, and the hallway track.

The recorded sessions

I usually skip this track because:

Sessions are recorded, so I can see them later
You can generally have 1-to-1 with presenters and attendees in the hallway track.

This summit was no different.

Kevin Carter, Amy Marrich, and I had our own recorded session for the OpenStack-Ansible project, named unsurprisingly “OpenStack-Ansible Project Update”. Sadly, due to technical issues, our session didn’t run for its full 20 minutes length, and the presentation didn’t meet my excellence criteria. Next time, I’ll prepare for the worst! If you want to see the extend of the new features we want to introduce during the Queens cycle, have a look at this instead.

I also attended the session “Practical k8s on OpenStack-Ansible”. I am glad it had a proper attendance, but sad the session wasn’t recorded. Well done Florian, I received good feedback from attendees!

The etherpad based conversations

Those conversations are structured discussions, where the moderator lets the attendance discuss how to improve things on a specific topic, in non-interrupt based conversations. I generally spend the majority of my time in those sessions.

The first session I attended was the “OpenStack Health” session, talking about its relevance. I learned many things about the future of Zuul v3 dashboard there too, and how things will be improved in the future. I’ll stay in touch with the team to learn more about these dashboards.

Our OpenStack-Ansible feedback session had a very good attendance. It had new, as well as existing faces, which is encouraging. We mainly discussed Ansible’s OpenStack modules, and offline installs. Here is what I take from the conversation:

We need to improve our documentation for offline installs, and make it simpler by removing all external dependencies outside the apt/pip/git parts (LXC download template files, extra packages outside the repositories, cirros images…). Having an apt mirror, an PyPI server, and git repos should be enough to deploy OpenStack from source in an offline fashion. All of this was already planned for this cycle, so we are in sync with our community.
We need to extend our test coverage for roles by testing idempotency of Ansible’s OpenStack modules. Example: Designate role test should test the os_zone and os_recordset modules. On top of that, we could extend Ansible’s test coverage by automatically testing changes in their OpenStack modules by using our role tests with the help of Zuul v3.

The “Installation Guide” changes session was interesting, talking about its relevance nowadays, and how to improve it. Because there is no radical change on the installation guide, OpenStack-Ansible won’t be impacted: the installation of OpenStack using OpenStack-Ansible is described in the “deploy guide”.

The “skip level upgrade” session ended up by saying “We need a proper cross-project documentation that all projects can apply”. Someone volunteered for the documentation, and I suggested to compare their work with our “Leapfrog” work, available in our openstack-ansible-ops repository.

The Privsep session explained its impact on deployers in the future. Here is a short summary:

Documentation will be written on how to use privsep (sudo rootwrap that contains privsep or sudo privsep directly). We will need to adapt our roles based on it.
I conveyed the need to implement this in a standard way accross all the projects, to simplify deployer’s life.

The “stable policy” conversation agreed that deployment tools have different needs than the other projects. We decided to make sure NO deployment project should have this tag, and each deployment project can pick and choose what’s best for its customers, because they are already opiniated.

A Self-Healing SIG was created, with as first action points creating mailing lists and IRC communication channels. There was a divergence of use cases already: Telcos wanted the self-healing to go as far as inside the openstack hosted applications. The group will first focus on the openstack applications themselves. I reminded the group we should not be re-writing a monitoring solution, but instead creating a common base and/or modifying projects to fullfill our self-healing needs. The first item would be to implement a proper health checking system, usable by monitoring solutions and inside the self-healing framework.

The “Ansible Onboarding” session was a full house. The audience was split between Ansible beginners, OpenStack-Ansible beginners, and OpenStack-Ansible experienced users. We covered the topic by browsing the docs, telling how it installs Ansible, and how everything works together, mentionning all the pain points people usually see. I hope this will help those new contributors!

The “Ansible State of the Union” session was explaining where Ansible was, and is heading towards. I currently don’t see anything conflicting with our strategy. We can improve Ansible’s modules by bringing real use cases, as mentionned above.

The “Neutron Pain Points” session was unsuccessful to me on one topic: Failure to agree there was a pain point for people who want to use OpenStack with Neutron like they would use VMware. The rest of the session required some in-depth Neutron knowledge I didn’t have.

Last, but not least, the (in my opinion, controversial) “Upstream LTS” session. I suggest you to visit the etherpad of the conversation, and the etherpad of the “after conversation” to get a solid understanding of the issues. I had plenty of hallway conversations about these topics, sharing my concerns on how things can go wrong, and how hard it will be to do it right. There are other ways to solve the initial problem that spawned the need of an “LTS” for our users, but that’s probably worth another blog post, and another series of discussions.

The “hallway track”

The “hallway track” is my favorite part. Every OpenStack gathering (summit/PTG/mid-cycles) allows you to meet more and more people, whether they are long time contributors or new community members. This summit was no exception. I got the chance to meet with people from my own country, from the other side of the world, and this time also with people from down under which don’t usually get the chance to travel to Europe or America. It was very nice to meet all of you!

Here are a few highlights of some “hallway track” conversations:

Explained how to increase test coverage of the telemetry stack, with the hope of seeing the test matrix increased in roles and our integrated gates.
Discussed with Miguel Lavalle (Neutron PTL) about the inclusion of OpenStack-Ansible neutron testing as non-voting jobs, like we do for Keystone, nova-lxd, and Dragonflow already. There was no opposition to this idea. With the addition of Glance and Neutron, only Cinder would be missing to have a compute starter kit fully tested with our deployment tooling before it even get tested in our integration repository.
Talked with Jay Bryant (Cinder PTL) about how we can help him having a “standalone cinder” test in our cinder role.
Launched the idea of a collaboration channel with the Ansible’s team with the help of Ricardo Carrillo Cruz. This has to be followed up after the summit.
Drafted “battle plans” on how to improve Dragonflow in Pike for Ubuntu. (Includes an etcd3 conversation).
Prepared the work to spin off “config_template” outside of our plugins repo, to be able to use it as a meta dependency for any role, not only the OpenStack-Ansible ones (for example Sebastien Han’s ceph-ansible, or Flavio Percoco’s projects).

Conclusion

I am glad people are still interested in Ansible, and using it with OpenStack. OpenStack-Ansible has a sweet spot for deployers that don’t want to go for the full k8s experience to deploy OpenStack. We’ve proven once more our flexibility, by showing we are in line with our deployer needs, and demonstrating kubernetes can still be used (on top of your OpenStack-Ansible deployed cloud!).

Group and host variables overriding in OpenStack-Ansible

Tue, 15 Aug 2017 00:00:00 +0000

You probably know it already, Ansible has different means of overriding a variable, and also different levels of precedence. OpenStack-Ansible is no different, and ships with different levels itself. I’ll talk about the group and host variables here.

If you manage your own Ansible configurations, you probably have roles (inside their defaults and vars folders) variables, playbook variables, and you probably tie all of these up with group and host vars. You probably have noticed that these variables are often modified, and are the core of your Ansible installation.

Editing those is something highly inconvenient for an Ansible based project that ships its own group/host variables: it means your users have to edit the code of your project (for example, OpenStack-Ansible), to match their needs.

We still have ways to override group and host vars in OpenStack-Ansible, but it depends on what you need, and which branch you are on. Let’s talk about that.

For Pike

During Pike cycle, we moved the group variables into a different folder. This allowed us to use our variables plugin, loading group and host variables from different folders, with the precedence completely up to the deployer (wiring can be seen here, if you want to replicate this in your own environment)

This series of commits allows you to override any group or host variable, surgically. See also our upstream documentation.

As usual, you could still use OpenStack-Ansible user variables, if you want to set a variable for all your nodes. This has the highest level of precedence, because it passes the user variable file in the CLI, like you would if you were using ansible-playbook <...> -e @filename.yml.

For Newton, and Ocata

There was no consensus (yet) on whether or not OpenStack-Ansible community should allow this backport of the variables into the different folder. Newton and Ocata being stable branches, it’s unlikely that you’ll be able to override OpenStack-Ansible variables surgically, like you would in Pike.

The variable plugin is still loaded, which means you can still have extra group variables and extra host variables, if need be. It’s a convenient way to add new variables, properly scoped, but that won’t help you if you want to override, for example lxc_container_config_list, for one or a group of hosts.

Let’s see what is possible.

The lxc_container_config_list example

For this example, we want to override lxc_container_config_list for cinder_api nodes.

The task creating the container and using this list uses two variables you can technically override: lxc_container_default_config_list and lxc_container_config_list.

The simplest case

If you want to keep the lxc_container_config_list (the existing group variable override), and add something on top, you could simply redefine your lxc_container_default_config_list into your /etc/openstack_deploy/group_vars/cinder_api.yml (please double check that the vars plugin is loaded for you, by having a look at your in your /usr/local/bin/openstack-ansible.rc).

Overriding the group variable, elegantly.

Overriding the lxc_container_config_list would be different, because the precedence would force you to use our user variables.

You could add into your own user variable file (let’s say /etc/openstack_deploy/user_cinder.yml):

lxc_container_config_list: "{{ properties.lxc_container_config | default(['lxc.aa_profile=lxc-openstack']) }}"

So, you will be updating the lxc_config_list for all containers to the lxc-openstack aa profile, but at the same time, you allow a container property to override it!

You can then create your /etc/openstack_deploy/env.d/cinder.yml with the following content:

container_skel:
cinder_api_container:
properties:
lxc_container_config: [ "lxc.aa_profile=unconfined" ]

However, you still need to ship env.d content for each of the groups requiring a different value than lxc-openstack. In other words, in this case, you need to override the env.d content for the cinder_volume and neutron_agent groups.

Overriding the group variable, centrally.

There is an alternative:

You could directly use group membership inside your user variables!

Example:

cinder_lxc_container_config_list: [ "lxc.aa_profile=override" ]
others_lxc_container_config_list: [ "lxc.aa_profile=lxc-openstack" ]
lxc_container_config_list: "{% if 'cinder_api' in group_names %}{{ cinder_lxc_container_config_list }}{% else %}{{ others_lxc_container_config_list }}{% endif %}"

Or even better, if you want to add the cinder config list on top of the existing ones:

lxc_container_config_list: |-
{% set config_list = [] %}
{% set _ = config_list.extend(others_lxc_container_config_list) %}
{% if 'cinder_api' in group_names %}
{% set _ = config_list.extend(cinder_lxc_container_config_list) %}
{% endif %}
{{ config_list }}

Overriding a host variable.

What would you do if you need to override the cinder backends per host on the storage hosts, and define their limit_container_types amongst other data?

For that you can configure your conf.d/cinder.yml:

storage_hosts:
aio1:
ip: 172.29.236.100
container_vars:
cinder_backends: "{{ cinder_backends_per_host.get(inventory_hostname) }}"

After that, you have to configure a user variable, where cinder_backends_per_host is a dictionary, and each host of your inventory would be a key:

cinder_backends_per_host:
aio1_aodh_container-c065ea73:
limit_container_types: cinder_volume
lvm:
volume_group: cinder-volumes
volume_driver: cinder.volume.drivers.lvm.LVMVolumeDriver3
volume_backend_name: LVM_iSCSI
iscsi_ip_address: "172.29.236.100"

Conclusion

It is possible to override your group variables and host variables in every version of OpenStack-Ansible. The more recent your version is, the more streamlined and user-friendly your experience will be. Stay up to date, because Ansible 2.4 will probably bring changes improving things even further…

Usage of ansible local facts

Sat, 05 Aug 2017 00:00:00 +0000

If you haven’t looked at ansible ‘local’ facts yet, maybe this article will encourage you to do so. Ansible ‘local’ facts is a very powerful tool to get information, statically or dynamically, on your hosts WHEN the hosts are the authoritative source of truth.

The example in the documentation is so simple it could be overlooked. You should use it if you don’t already use facter or ohai.

First usage

Let’s start with the example of the documentation:

If you write a file named something.fact into /etc/ansible/facts.d/, you may use its content as local fact by using:

{{ ansible_local.something }}

after running the setup module.

So, if you have a file, named a.fact, containing:

[b]
c=True

You could use {{ ansible_local.a.b.c }} for anything. Tasks, conditionals… (Remember you could use set_fact for caching in your local play!)

As long as this file is readable under a json or a ini form, ansible will take it.

So, when is it useful, and what could I use it for?

If you have to configure a software depending on its state, and that software doesn’t report to a centralized state management system, that’s where your local fact will shine.

Example, you want to ensure your Galera cluster is in a correct state, and erase the state of an incorrect node. For that, you may need to compare what each node “sees”, and therefore a centralized view can’t help you.

You can write a script (let’s call it galera.fact) with the simple following content¹:

#!/bin/bash
echo '[cluster_status']
mysql mysql -e "SHOW GLOBAL STATUS LIKE 'wsrep_cluster_status'\G" | awk '/Value/{print "status="$2}'

Don’t forget to mark it as executable.

You can now see the status of your node(s) by looking at {{ ansible_local.galera.cluster_status.status }}".

Very useful.

What’s the catch?

Slowness

The local facts will slow up your playbooks!

Calling the setup module (which happens by default on a playbook unless gather_facts is set to no) will trigger the “local” data generation. If your script is slow to execute, you will have to face the slowdown.

On top of that, you’ll carry extra variables in your vars/hostvars, which burdens ansible on very large environments.

The alternative approach would be that your script pushes to a central local location, like a redis k/v store, or etcd.

Unfriendliness

To do an action when your galera cluster status is not “Primary”, you’d have to ensure the full fact chain is set. If you are not sure about whether your fact is defined or not, you could end up writing a waterfall of checks just to be able to read {{ ansible_local.galera.cluster_status.status }}.

Example:

 when:
- ansible_local is defined
- ansible_local.galera is defined
- ansible_local.galera.cluster_status is defined
- ansible_local.galera.cluster_status.status is defined
- ansible_local.galera.cluster_status.status != "Primary"

Hopefully here, if you can install jmespath3, ansible got you covered with the json_query filter.

You can run in this case:

 when: "{{ ansible_local | json_query('galera.cluster_status.status') | default('Error',True) != 'Primary' }}"

For your information, json_query will return an empty string, not an error, if the element is not found. That’s why I am using default('something',True).

Recap

Ansible local facts is a double-edged sword. It’s very powerful to get a complete a view of your environment if you don’t already use another method. However, it comes with slowness.

I would still recommend use local facts as much as possible if you can’t use a central approach, they are more flexible and faster to write than writing an inventory, a var plugin, or an inventory plugin!

I know this example isn’t the best, because you could use galera notification scripts better, but that’s just an example. ^[return]

My view on Ansible 2.3 release notes

Fri, 21 Apr 2017 00:00:00 +0000

Ansible 2.3 was released. I’m pretty sure you’ve seen things online about it, and probably read the changelog.

Networking modules were a big thing this release.

So big, it’s easy to overlook the rest. Here are, in my opinion, many other things worth glancing over for 2.3 (discovered from commits, or from the changelog):

ansible_user doesn’t work anymore if you don’t have a variable explicitly set (depends on your connection plugin). If you have facts gathered, you may want to use ansible_user_id. This bug has already been fixed in the 2.3 branch and will probably be shipped in version 2.3.0.1
Vars precedence and inheritence have changed. Please have a look at the docs for variable precedence. I have to test if it broke my vars_plugins.
module_utils for custom modules can be shipped with roles. \o/
a passwordstore lookup was introduced (see also passwordstore.org). Maybe it was there before, but I can’t remember it, and I love it.
a keyring lookup was introduced.
You can now have a more terse run output with the new dense callback plugin.
YAML and pickle cache support! Interesting, I think I will prefer YAML compared to all of the existing cache plugins.
There are new combinations and permutations filters.
We now finally have archive, long after unarchive.
New/Updated toys (modules) are available for ansible stats, wait for connection, nginx, openssl, jenkins groovy scripts, parted, tempfile, zfs, Clear Linux, HP ILO, FreeIPA, iso extraction, ldap, logstash, netapp, omapi, ovirt, serverless, openbsd, let’s encrypt, the usual clouds, …
Time to flatten your package lists yourself again! removed ‘package’ from default squash actions as not all package managers support it and it creates errors when using loops, any user can add back via config options if they don’t use those package managers or otherwise avoid the errors. Great. Moar slowness.
Blocks can now have a name field. I like this improved readability.
SO MUCH WOW with strategies: Default strategy is now configurable via ansible.cfg or environment variable. That’s a change I love, mainly because I now have a different way to use a strategy I wrote (with the help of Kevin Carter, see in openstack-ansible-plugins) without changing my playbook.
There are changes in the way multiple --tags given in the CLI behave. On this note, if you’re tired of all of these changes, you can use my/our tag filtering strategy ;)
restructured how async works to allow it to apply to action plugins that choose to support it. Great, we’ll have to adapt config-template action plugin, if need be.
I am now forced to have a look at our custom connection plugin because of the multiple connection/retry behavior changes. For example:
- On platforms that support it, use more modern system polling API instead of select in the ssh connection plugin. This removes one limitation on how many parallel forks are feasible on these systems
- added optional ‘piped’ transfer method to ssh plugin for when scp and sftp are missing, ssh plugin is also now ‘smarter’ when using these options
- On top of that, there were other changes linked to connections/retries.
I. FEEL. THE. DANGER. : Fixed issues with inventory formats not handling ‘all’ and ‘ungrouped’ in an uniform way.
Yum state: list now works with disable repo and enable repo.
Split on newlines when searching become prompt (5).
LXC module doesn’t have a problem with HOSTNAME in bashrc anymore.
The ansible-galaxy init can now take a skeleton argument, and/or read env vars for skeleton role location.
The message “skipping: no hosts matched” has returned!
The apt_repository module got a bug fixed.

Installing an OpenStack-Ansible multi-node cloud in the cloud

Thu, 29 Dec 2016 00:00:00 +0000

Introduction

Installing openstack is not that hard, but it could be time consuming. When running your first installs, you probably want to prototype on instances instead of hardware, to shave time. It is basically one of the “cloud” principles: Focus on your goal (a business man would have said “value”), and rely on your providers whenever possible.

So, for me, it makes 100% sense to deploy OpenStack-Ansible in the cloud. So what are the steps?

Note: This post’s relevance can change overtime, as OpenStack and OpenStack-Ansible evolves.

General overview

If you want to deploy your OpenStack-Ansible cloud in the cloud, here are the top level steps.

You want to have, like you would in hardware, an “OpenStack-Ansible”-ready operating system.
Because currently OpenStack-Ansible requires¹ your nodes to be on the same management network, you probably want them connected in some way.
You then want to follow the usual steps for deploying ansible.

These steps are similar to steps needed to run OpenStack-Ansible nodes in your datacenter. The “cloud” nature brings more flexibility and constraints at the same time, so let’s go through these a little further

In the future I’m presuming you have a host/node that can:

do git operations (clone)
run python and ansible
connect to your cloud provider(s) to create your instances.

This node is NOT the deploy node (but it could), it’s simply your laptop or whatever.

Step 1: Prepare the operating system for OpenStack-Ansible

OpenStack-Ansible requires an Ubuntu 14.04 (up to Newton) or 16.04 (from Newton onwards) with specific packages. CentOS 7 is on the works, but let’s focus on Ubuntu 16.04 for this exercice. Using a cloud provider already makes sense: Ubuntu being popular, you probably don’t have to roll your own image, everything is probably already ready to work. Give your ssh keypair, select ubuntu 16.04 and DONE!

Let’s say for the sake of the exercice you’ve created the following instances in your cloud provider:

deploy
haproxy01
haproxy02
infra01
infra02
infra03
compute01

You probably want to install:

git, python, and some other ansible dependencies on the deploy node
python2.7 on haproxy nodes
a recent kernel, python2.7, enough RAM/disk on the infra nodes.

Please check the current requirements for your version in the OpenStack-Ansible installation/deployment guide.

You could do install these packages manually by running apt-get install x on each of those nodes.

If you’re like me, you probably want to automate it. Enters the preliminary step: Using ansible to prepare your operating systems.

Warning: Using Ansible to bootstrap your hosts in order to deploy OpenStack-Ansible on your instances (one of these will be bootstraping Ansible for OpenStack-Ansible) can cause dizziness. But don’t worry, we are not looping.

Step 1.0 Preparing ansible inventory

To use Ansible, you better have an inventory. You cannot use the OpenStack-Ansible inventory at this point of the lab yet, so you need to write your own.

Ansible did most of the job for you by providing dynamic inventories scripts for AWS or OpenStack clouds. Put these scripts under your inventory/ and chmod +x them (²). I am assuming you’re creating your own repo or folder for this.

The Ansible OpenStack dynamic inventory script relies on shade and your clouds.yaml (for example ~/.config/openstack/clouds.yaml), so make sure they are ready.

For user-friendliness, you may want to add in your clouds.yaml:

ansible:
use_hostnames: True
expand_hostvars: False
fail_on_errors: True

Check your inventory by issuing python <your inventory script> --list.

Step 1.1 Targetting the proper nodes

Now that your Ansible inventory is ready, you can see its layout is flat, except if you have given metadata to your instances. This lack of groups could be problematic.

Two ways to fix this:

Make sure what you run is targetting the proper nodes by using an in-memory inventory features
Use the ansible inventory merge behavior and complete the inventory with a static inventory file to make your life easier.

For the in-memory inventory, you need tasks holding the group/host names. If these tasks are not using variables, you will prevent the reuse of the plays. If using variables, their location would be, in my opinion, far less obvious than the inventory (in play, in a variable file, in group/host vars, …).

In this case, I think the solution two seems better. The node targetting should be done with an ansible inventory file (like an INI file) and benefit from ansible merge behavior. So here is an extract of what you’d need in the INI file for this step:

[pre_osa]
infra01
infra02
infra03
deploy
haproxy01
haproxy02
compute01

Not that hard, right?

If you skipped the step 1.0, your only choice would be to write an INI file with the following content:

[pre_osa]
infra01 ansible_host=xx.xx.xx.xx
infra02 ansible_host=xx.xx.xx.xx
infra03 ansible_host=xx.xx.xx.xx
deploy ansible_host=xx.xx.xx.xx
haproxy01 ansible_host=xx.xx.xx.xx
haproxy02 ansible_host=xx.xx.xx.xx
compute01 ansible_host=xx.xx.xx.xx

Everytime you’d change your cloud node, you’d have to rewrite your ansible_host value. Tedious but ok for small tests too. Your pick.

Step 1.2 Installing the OpenStack-Ansible requirements

Now, you need to install the basic requirements for Ansible. This is technically required on the deploy node, but some other nodes may require them too. For safety, install the packages everywhere!

So, for Ubuntu you need the following:

ansible pre_osa -m raw -a "test -e /usr/bin/python || (apt-get -y update && apt-get install -y python-minimal)"
ansible pre_osa -m raw -a "apt-get update -y && apt-get install -y git build-essential python-simplejson libssl-dev libffi-dev bridge-utils"

Note: The above commands don’t specify a user or elevation method, nor the path to the inventory you just created. Adapt it to your needs.

Note: The above commands only list the packages needed at the time of the writing of this article. Please check what is required in OpenStack-Ansible by checking the bindep.txt file in the root of your openstack-ansible repo. Or just install bindep and use it.

Step 1.3 Test your ansible installation

You can now run:

ansible pre_osa -m ping

And move on the real stuff.

Step 2.0 Configure networking

Cloud multi-node problem statement

The odds are really high that your cloud provider will prevent you from having network traffic coming to/from multiple ethernet addresses for one network. In general, you have one tuple (IP, hardware address) per network configured at your provider. Because OpenStack-Ansible is relying on bridging to put containers together on the same networks, only the host traffic will be able to go outside. It’s perfectly fine from an all-in-one (AIO) standpoint, because in this case the external world will only speak to the load balancer on the host, and the rest of the OpenStack traffic will stay inside the hosts and don’t need external world forwarding. This won’t do in our multi-node scenario.

The solutions

You can do NAT, but I will not recommend it. This is a radical change from OpenStack-Ansible standard networking. You’ll be on your own, probably will have issues with some software, and your future hardware deploy of OpenStack-Ansible will probably not be deployed with NAT for scalability reasons (therefore the reproducibility of this exercice is quite limited). On top of that NAT isn’t really a good idea if you plan to have an OpenStack cloud from this century, i.e. with IPv6.

You could deploy all your OpenStack-Ansible components on metal instead of in containers. You’d then multiply the amount of cloud instances you need, and complexity. Also I’d rather have the same configuration for OpenStack-Ansible in my cloud deploy as I would have in my hardware deploy.

The other solution would be to encapsulate your Layer 2 traffic between your cloud instances: Your configuration won’t change for OpenStack-Ansible, you’ll only have different networking (and a smaller MTU in the case of cloud instances vs the hardware installation). Keep in mind that there are many roles in the ansible galaxy to configure your networking for your needs.

Encapsulation and tunneling using VXLAN

Please note that the OpenStack-Ansible repo hides a hidden gem here, in the bootstrap-host role. OpenStack-Ansible uses this role (located in the tests folder) in its CI testing for configuring many things required to deploy/configure OpenStack.

The bootstrap-host role has the ability to configure basic networking, but also VXLAN encapsulation. If you want to use this role (³), you probably want to select only its networking bits. Create your own playbook, target your cloud nodes, include the bootstrap-host role, and run the playbook with the proper tags. It should look like this:

ansible-playbook osa-cloud-multinode-networking.yml --tags=prepare-networking

Note: the bootstrap-host role need the excellent config_template plugin to work. Make sure you have the plugin in your Ansible plugins path. You can override the plugins folder location by adapting your ansible.cfg file, or using environment variables. Alternatively, you may want to consider that the bootstrap-host role as a meta dependency to the openstack-ansible-plugins role, which makes possible to use the config_template easily. That’s what I usually prefer.

Adapt variables to trick the bootstrap-host role into doing your bidding.

For example, my osa-cloud-multinode-networking.yml looks like this:

---
- hosts: pre_osa
vars:
bootstrap_host_aio_config: no
bootstrap_host_encapsulation_interface: "{{ ansible_default_ipv4.interface }}" #Assuming you only have one interface, a public one, you want to assign this as exit interface and encapsulation carrying interface
bootstrap_host_public_interface: "{{ ansible_default_ipv4.interface }}"
bootstrap_host_encapsulation_enabled: True #not really necessary to define it, but hell it doesn't hurt to be explicit.
roles:
- bootstrap-host

You probably want to give each of your nodes a node_id. You can either assign the node_id in the inventory (node per node or in a group var), or defining it in the playbook (set_facts/vars):

node_id: "{{ groups.pre_osa.index(inventory_hostname) }}"

The trick here is that the bootstrap-host can configure a VXLAN overlay between all your nodes. This will be used to encapsulate traffic for br-mgmt, br-storage, br-vxlan, and br-vlan!

Step 2.1 Verify network configuration

That’s the simplest of the steps:

 ansible pre_osa -m shell -a "cat /etc/network/interfaces.d/*"
ansible pre_osa -m setup

Alternatively, you can check the connectivity on the br-mgmt of the nodes by pinging the nodes manually.

Step 3.0 Deploy and use!

From now on, the nodes will be “like if they were on the same layer 3”, which should be enough for your OpenStack-Ansible deployment.

Use your br-mgmt IPs when configuring your conf.d/openstack_user_config and everything should be fine!

Conclusion

Building overlays accross multiple datacenters/clouds is quite easy, and work by default in OpenStack-Ansible. Make use of it!

We don’t really require all the nodes to be in the same management network, but it makes your life quite simpler. This is the path chosen for this article. ^[return]
Please remind me to write a tool for that. ^[return]
To use the bootstrap-host role, you have to clone the role in the appropriate folder! ^[return]

ansible-galaxy, my journey to install golang

Sun, 02 Oct 2016 00:00:00 +0000

Introduction

I wanted to self-host transfer.sh, and didn’t want to start using docker or new tech.

TL:DR; As usual, I turned to ansible, and (as usual) I was disappointed by the quality of the roles on the ansible-galaxy website. I wanted to submit PR to the best role, but at the end I’d have to modify it.

So, what is really the point of galaxy.ansible.com? What could we do to make it better?

The journey

Transfer.sh is a simple tool to upload your files from the command line. It’s server side is developped with go. To install this in a repeatable fashion, I wanted to use the available roles in ansible-galaxy.

First step: Search a role on the galaxy website.

To find the best role, I trust the other users. I check the download count. I had 2 targets: one role with 192 downloads and one with 164. I then asked myself: how is it possible that a popular language like golang doesn’t have more download counts?

Obviously there was an issue.

Second step: Verify if galaxy website isn’t borked again.

galaxy.ansible.com has caused me a lot of pain in the past. The most memorable for me was this one time where I could not import the latest version of my role, and hoped disabling/enabling it could help the situation. Obviously(?) the role disable worked, but the enable failed, and all shell scripts relying on the presence of this repo started to fail. When this kind of things happen at 6PM, you know you are gonna stay up late to introduce fixes everywhere. All the trust you have for this website is now gone.

Let’s get back to the matter here. A simple refresh (or “back”) on the page, shows me different results! Interesting. I wonder how much people tried a terrible role because the best role was not shown in the results.

Let’s see what we have now:

Oh, that’s better, 6k downloads! This is where my focus will be.

But before that, should I try once again? Do I really have only 10 records? I suppose I have at least 11 of them, since a new one showed up. Let’s try that search again.

Finally, a proper list! So, I can now (after multiple tentatives) confirm there are 24 roles listed in the galaxy for golang.

On another note, did you notice that the role with the most downloads did not appear in the first PAGE of results when sorted by relevance?

Third step: Elect the best role

Obviously galaxy website relevance sort was not reliable, so I tried to sort with downloads. There were 4 of them with more than 100 downloads.

The first one is supporting my distribution and installs the latest release of golang! Sadly, this could stop anytime, because the variables used for the version have to be manually set in default/main.yml . This seems brittle, but it will be fine if the maintainer of the role is answering to the community.

Let’s therefore check the PRs!

2 open PR. Once from December last year, one in march this year. We are in september. That’s a long period for triaging/fixing for a role that small.
Now the closed PRs. There are many of them, some of them closed recently. A good sign. Many of them are basically jumping the version to a more recent one. It makes sense, that’s what I’ve seen!

At that point, here are my options:

PR to adapt the role to make it resilient to version changes
Check the other roles

I decided to check the other roles, maybe the grass is greener on the other side of the fence.

The second in terms of downloads forced me to edit my ansible.cfg to change the hash_merge behavior. Terrible. It didn’t support latest golang version either.

The third is installing latest version, but has a dependant role for something I can’t explain yet. It also forces me to become: and become_user: root, on top of having terrible task readability.

The last one was the debops one. debops has lots of roles, and probably fit use cases which isn’t mine: I don’t want to have this complex series of roles and dependencies, or being forced to use a playbook. Ansible should be simple.

I give the golang example here, but trust me, it’s the same story for every single role I’m trying to install.

Fourth step: Analyse how long would it take me to adapt the best role vs recreating one

This role has 6 tasks. I can keep 3 of them, and modify the rest.

I’ve decided to adapt the role and submit a PR. My own golang role is still published on the galaxy, until the PR are merged. It seems a good intermediary.

Conclusion

Even basic roles that should be properly done are done in a terrible way. IMO, RedHat should now invest in:

a proper galaxy website
implementing a way to cleanup old role based on community feedback on the galaxy website
ensuring roles on the galaxy website are self-contained and concise. Reduce the relevance of roles based on the amount of dependencies, and at the same time, increase the relevance of roles with a minimum set of tasks.
ensuring the community understand and applies the best practices.

Orchestrate a service restart during a maintenance window with ansible

Sat, 18 Jun 2016 00:00:00 +0000

Introduction

Someone recently asked me:

“How do I run all the playbooks as usual, but make sure that ansible DOESN’T restart my service, and restart my service later?”

I didn’t get the purpose of this question. I suspected it might have to do with how people write their playbooks, and how possibly long playbooks must run to fit in tight maintenance windows.

The goal here was indeed to reduce the length of the maintenance windows by having ansible applying all the changes (like generating a daemon configuration), but handling the notifications for restarts at a later time. Of course, on the future run, most of the tasks should be entirely skipped and only the handler (the service restart) should run.

What you’ll learn in this article is how to modify a role to get this behavior, not if it’s a good idea or not. (Hint: It’s not. Terrible idea. You should properly build your application/architecture to respond to signals, being HA-able and avoid these kind of tricks to always be in a consistent state.)

I can haz code?

You can find all the examples/methods for this exercice in my PoC repository, under the conditionalhandler role.

For the sake of the simplicity, the tasks are tagged with the name of each method they belong to, and have additional tags like service_config (for the tasks simulating a service configuration generation) or service_restart (for the tasks simulating a service restart).

With this simple example, a standard task and a handler are enough to represent a real life example. The method here will explain what’s needed on top of these two steps to bring this “timely” restart feature.

If you’re testing these, please add -e "changed=True" if you want to simulate a service configuration change.

How would you do this?

Method 1: Use an intermediary handler

My initial thought was to trick ansible’s handlers/notification systems.

Remember this: if you have a generate service configuration task notifying an intermediate handler task named handling service restart instead of directly notifying the service restart handler task, the only way to have a conditional behavior on the service restart task is using a variable (i.e. using a when: clause). Using the tags system won’t work for this case.

In other words, this:

ansible-playbook conditional-handler-playbook.yml -t method1 --skip-tags=service_restart -e "changed=True"

will always run the service_restart tagged task. So this is not the intended behavior.

Method 2: Introduce two variables, a task and make the tasks run conditionally on these variables

You can define in your role’s default variables the following:

# Set this to False to deny service restarts after a configuration change
conditionalhandler_allow_restart: True
# Set this to True to always restart the service.
conditionalhandler_force_restart: False

A new task would be required to handle the force restart, while a condition should be added to the service restart handler.

For me, this is the least elegant way of doing it. This is really verbose, and requires variables that could be avoided. Readability is possible thanks to proper variable names. Replace the variable names by a and b, and read the code again. It will be far more difficult to read.

Generating the configuration without restart is done by executing:

ansible-playbook conditional-handler-playbook.yml -t method2 -e "conditionalhandler_allow_restart=False"

Forcing the service restart:

ansible-playbook conditional-handler-playbook.yml -t method2 -e "conditionalhandler_force_restart=True"

Method 3: Remove the handler, use register and add one variable

In this method, the service restart isn’t run as a handler, but as a standard task. This service restart task can therefore run traditionally (when generate service configuration get its status as “changed”) OR run when the new variable conditionalhandler_force_restart is set to True .

Avoiding/forcing a service restart becomes respectively:

ansible-playbook conditional-handler-playbook.yml -t method3 --skip-tags=service_restart
ansible-playbook conditional-handler-playbook.yml -t method3 -e "conditionalhandler_force_restart=True"

Method 4: Using a marking file and flushing handlers (or a post_task)

In this method, we’ll have one more task, one more handler. We don’t use ansible register: directive and we don’t add additional variables. We just make sure everything runs in two steps, by having either a post_task or a flush_handlers meta. The flush_handlers makes possible to execute the handlers, and then continue with the tasks. Here it’s what we need to have a temporary file created and/or deleted (depending on the case).

Avoiding/forcing service restart is respectively:

ansible-playbook conditional-handler-playbook.yml -t method4 --skip-tags=service_restart
ansible-playbook conditional-handler-playbook.yml -t method4 --skip-tags=service_config

Final Words - What is the most elegant way to do it?

My preferred method is the method 4, because it’s far more readable, it avoids variable crawl, the tags are consistent, and it’s understandable from a “non ansible” point of view. As I said before, the most elegant method would be to NOT do this at all, and make sure the system is always consistent after a run. Don’t forget that some services don’t require a restart to change their live status. Another signal could be used to take the change into consideration. This signal should be then used in the ansible handler.

Photo credit: Maker-9070

My YubiKey Quickstart guide

Mon, 13 Jun 2016 00:00:00 +0000

It’s been a while I’m thinking to step in the smart card world. It’s been years I’ve got a smart card on me all the time. I only use it once a year, for paying my taxes… Everytime I use it, I’m complaining. A lot.

Here are a few reasons for it:

Everything linked to its usage as a smart card is poorly documented. I really hope it’s not by design.
It requires specific hardware: a special kind of smart card reader (so not every smart card reader works!)
It requires specific software: a specific version of java (use java64 for an unreliable/unpleasant experience!), and sometimes even your version of java32 doesn’t work with it
It doesn’t work on all the browsers! You need to have a specific plugin. Of course, if you insert your smart card AFTER launching Firefox, you’re good for its restart.
You can’t do whatever you want with it (can’t set your own keys …)

Taxes feeling + grumpy {soft,hard}ware makes me unhappy. It’s unlikely I’ll use this smartcard in other circumstances (anyway, it’s maybe better -security wise- to avoid mixing the same device for different security purposes).

So, I decided to give YubiKeys a go.

What are YubiKeys, and why would I want one?

I recently found the YubiKeys more interesting. The YubiKey 4 packs a big list of features, for a really low cost. Moreover, there is currently a 20% discount for github users on yubico website (“temporary offer”).

In terms of (hardware) requirements, you only need a good ol’ USB port to work. The YubiKey Neo adds NFC to the mix for your mobile (but the Neo have drawbacks, check the cypto specs here: https://www.yubico.com/products/yubikey-hardware/). For your information, the Neo is currently cheaper on amazon than it is on yubico website.

In terms of features, the YubiKey works as HID for 2-Factor Authentication mechanisms (fido u2fa, yubico OTP, …) and it also works as CCID (smartcards! you can use it for signing, encryption, authentication with applications like pgp, openssh, openssl and many others).

Ok I’m in! Where should I start ?

Simple question, long answer. Don’t start with the “documentation” of the dev.yubico website! There is so much documentation on their website, you’ll be lost in all the links.

Here is what I’d do:

Simply start on your device page (yubikey 4/yubikey neo) and absorb what’s there.
You should then find the getting started page interesting.
You should then go ahead with the personalisation tools page.

You have plenty links on these three pages to follow what’s really needed to your use case. The last page of this list have obviously a link for the yubico personalisation tool download. This tool makes possible in-depth changes of your YubiKey.

Any other links?

For github/google, simply head to your account security settings. U2FA is very simple and doesn’t really need much explanation. You don’t need the personalisation tool to make it work. For non-U2FA, that’s where the fun begins…

Here are a few interesting explanations I found by browsing the interwebz/yubico website:

Configuring Yubikey as generic OATH HOTP token generator. I read somewhere was possible to enable this kind of 2FA on Ubuntu One website, but I didn’t find the option there yet.
Configuring YubiKey as a smartcard for ssh authentication, CA or code signing. On this page, you should probably have a look at the SSH with PIV and PKCS#11 section. It’s easy to use a certificate stored on the yubikey for SSH authentication: you just have to pass the library interface (opensc) to the PKCS#11 as identity in your openssh command line. This is supported with a portable openssh version > 5.4p1. (It’s also easy to export your public key to set it in the authorized keys for example). Check also here for an explanation of your Yubikey’s PIV slots.
Configuring PAM and SSH server for a 2FA with Yubico OTP (1 2 3 4 ). Take these with a pinch of salt (Some pages claim to improve security by using their method, where it clearly doesn’t. More on that in a later article).
Configuring PGP with YubiKeys. This is where you should head to for file encryption, mail encryption/signing, ssh authentication…

Last words

I think the YubiKey is a simple yet efficient device. It protects your accounts in a simple manner, and everyone interested in security should at least consider it (or one of its alternatives).

Photo credit: Thomas Flenstad

Auto deployment of Pelican webpages to github user pages

Sun, 05 Jun 2016 00:00:00 +0000

Introduction

As you may already have seen, this website is static html. I want to share how I do it. My goals:

have the technical out of the way when writing content
have an automated way of publish content
be simple to use wherever I am

TL;DR: I have a tox configuration that does the heavy lifting for publishing content written with pelican, check how it’s done on the appropriate github repo

Pelican

Multiple solutions for the automation

So I had multiple choices:

Integrate with a CI/CD like Travis CI, semaphore (my two favorites) or jenkins.
Work in a manual way.
Use git hooks.
Use tox

The first solution is simple to setup and relies on a lot of different parties (“vendors”).

The second seems more tedious on the long run.

The git hooks is more contained (it’s pure git without fuss!) but requires to either have a proper git server that will handle my python builds, or a way to setup the client hooks and/or dependencies on the node that is used for writing the blog.

As I’ll host everything on a public git (github in my case), I decided to go for the last one. The github server hooks are limited to curl things, So I can’t have proper testing there. And because I want an easy clone/work way of things, I can’t use use client side hooks. That’s why I’ll use tox for testing.

Tox to the rescue

If you don’t know tox, I invite you to look at this wonder. In our case here, it’s basically a python’s virtualenv-aware make command. It’s a good python automation & testing tool.

Thanks to tox behavior, I can just focus on content: The tox.ini will hold the information about the commands to run (I’ll definitely forget these, so having one ini file as a reminder is good for me!) and run each test in its own virtualenv, which I find good for avoiding pollution on your “writing node”. The requirements for this “writing node” is simple: python, tox, and virtualenv. If you really want to reduce your host dependencies, you could even run tox in a virtualenv.

So, I have one repo with my environment, holding the pelican and tox configuration files, the markdown-rst-whatever files and other static content. When I want to work, I clone this repository and use my text editor to write in content/. Simple, right?

Let’s have a look at my tox.ini configuration file:

sdist

One of the first steps invoked when running tox is python setup.py sdist, that should generate your package.zip.

In our case, we don’t have a setup.py or setup.cfg holding our code. We’re barely using python stuff. So, we can skip this step by setting:

skipsdist = true

the virtual envs creation

Tox then creates the virtualenvs. There are lots of ways to affect this, I invite you to check at the tox documentation. Tox, by default, creates its virtualenvs in .tox/ folder. Every tox command will create its own virtualenv inside this folder.

I defined two different virtual environments, one for the linting (facultative), and one for pelican jobs. I want to keep them separate because they can have different dependencies.

Inside the virtualenvs

So, the next steps for tox is to calculate/install the dependencies.

tox reads the deps section of each environment configuration (if any). This overrides the default testenv deps configuration.

tox then installs the result of the python setup.py sdist (for example, a package.zip). Here this step is skipped.

tox reads the command section relative to the virtualenv to run the job.

The linters case

I have a “linters” test command (tox -e linters), that is used for … linting the restructured text files, obviously.

The linter tool is not being nice with the others, and like I said before, I’ve isolated the linter tool from the others. It has therefore its own configuration section, to setup a different list of dependencies. For the linters, I just needed restructuredtext_lint as dependency.

For the command part, I wrote a small python script that dumps (in a friendly manner) my next mistake.

the pelican virtualenv

My main testenv section is for my pelican work. This section has multiple commands: one that can generates the content with the testing profile (build), watch for saves changes (watch), serve the web content using twisted (serve) and finally deploy to production (publish).

To avoid having multiple virtualenvs (one per command), I make sure that all the commands have the same environment by setting envdir in testenv:

envdir = {toxinidir}/.tox/functional

Because testenv is the parent and testenv:linters inherit from it, the previous change made linters’ virtualenv the same as the others. Thefore, I need to override the linters’s envdir configuration:

envdir = {toxinidir}/.tox/linters

It means that the linters virtualenv is now unlinked (again) from the “functional” virtualenv.

Last words

With the fact that python can easily be found on most OSes, I think this workflow is quite simple to follow.

Photo credit: MDreibelbis

Categories on dd if=/dev/brain of=/var/log/site

Custom Inventories with OpenStack-Ansible

The openstack-ansible CLI’s inventory

How does this static inventory work in Queens?

So, what has improved in Rocky?

Conclusion

A review of OpenStack's LOCI and its extensibility

Flexibility and extensibility

The build process

Security and reproducibility

Build speed

Gating expectations

Learning pace

Conclusions

Switching to the new gandi mail service

Switching your domain to the new service

Feedback

ansible-inventory tips

My view on Ansible 2.5

OpenStack Summit Sydney Recap

The venue

The sessions

The recorded sessions

The etherpad based conversations

The “hallway track”

Conclusion

Group and host variables overriding in OpenStack-Ansible

For Pike

For Newton, and Ocata

The lxc_container_config_list example

The simplest case

Overriding the group variable, elegantly.

Overriding the group variable, centrally.

Overriding a host variable.

Conclusion

Usage of ansible local facts

First usage

So, when is it useful, and what could I use it for?

What’s the catch?

Slowness

Unfriendliness

Recap

My view on Ansible 2.3 release notes

Installing an OpenStack-Ansible multi-node cloud in the cloud

Introduction

General overview

Step 1: Prepare the operating system for OpenStack-Ansible

Step 1.0 Preparing ansible inventory

Step 1.1 Targetting the proper nodes

Step 1.2 Installing the OpenStack-Ansible requirements

Step 1.3 Test your ansible installation

Step 2.0 Configure networking

Cloud multi-node problem statement

The solutions

Encapsulation and tunneling using VXLAN

Other solutions

Step 2.1 Verify network configuration

Step 3.0 Deploy and use!

Conclusion

ansible-galaxy, my journey to install golang

Introduction

The journey

First step: Search a role on the galaxy website.

Second step: Verify if galaxy website isn’t borked again.

Third step: Elect the best role

Fourth step: Analyse how long would it take me to adapt the best role vs recreating one

Conclusion

Orchestrate a service restart during a maintenance window with ansible

Introduction

I can haz code?

How would you do this?

Method 1: Use an intermediary handler

Method 2: Introduce two variables, a task and make the tasks run conditionally on these variables

Method 3: Remove the handler, use register and add one variable

Method 4: Using a marking file and flushing handlers (or a post_task)

Final Words - What is the most elegant way to do it?

My YubiKey Quickstart guide

What are YubiKeys, and why would I want one?

Ok I’m in! Where should I start ?

Any other links?

The `openstack-ansible` CLI’s inventory