Monthly Archives: May 2015

Fernet Tokens for Fun & Profit

I’ve been digging into Fernet tokens this past week and getting ready to switch us over to using them. This is the first in a series of blog posts I plan on writing about them. This one will mainly be background on why we’re switching and what we hope to gain. The next post will cover rolling them out which will probably be in a few weeks. For now we’re running these keys in our dev environments for more testing while we focus resources on Kilo upgrades.

What are Fernet Tokens?
How do you explain Fernet tokens? Rather than some lengthy treatise mathematical and identity management theory, just know this: Fernet tokens use shared private keys to avoid having to store or replicate tokens in your database. This makes them super fast, reduces load on your database, and solves replication lag between data centers and nodes within a data center. If your manager asks you, “they’re faster, small, and reduce load on the db”, that should suffice. Dolph Mathews has a good write-up on how much faster they are here. You can also dive into the different token formats for comparison on another of his posts, here.

What Issues will this solve for us?

Now about the DB replication issues… I cannot tell you how much stuff we had to do to deal with database and replication issues with UUID tokens, here’s a few samples:

  • custom cron job to reap expired tokens
  • force db transactions to a master, despite us being active/active so tokens would be there when asked for
  • hacks to our cross-region icinga checks to allow the tokens to replicate, literally sleep(3)

We’ve even had a service accidentally DOS us by requesting so many tokens the DB couldn’t keep up and keystone ran out of DB threads. Hopefully all this is solved by Fernet tokens.

Will this cause an outage to switch to?

Switching token providers will cause an outage. All the old tokens you’ve issued are now 100% useless. So prep accordingly. I will give some updates to the next blog post on how long this was and what issues we saw when we did it.

Do I need to be only Kilo/Liberty?

  • Horizon – you need a newish copy of django_openstack_auth which I think is in Liberty
  • Keystone – you need to be on Kilo
  • python-keystonemiddleware it’s best to have at least 1.1.0. If you have 1.0, you MUST restart all OpenStack services are switching tokens
  • Everything Else – Shouldn’t matter!

A note on python-keystonemiddleware. In 1.0.0 if a service (say Nova) can’t use it’s token for some reason, it won’t try to get a new one until the old one expires. So if you switch to Fernet’s you have to restart all OpenStack services that talk to Keystone or they will not work. We already have some ansible to do this mainly in response to RabbitMQ issues but it works here too.

How do I get Keys onto the boxes?

All keystone nodes in your cluster need to have the same keys. Fortunately there is the concept of rotation so there’s no outage when switching keys, there’s always a key thats “up next” or “on-deck” so that when you’re rotating you switch to a key that’s already on every box. Now as for getting the keys there. I’m going to use puppet to deploy keys that I store in hiera and rotate with a jenkins job, but there are other ways like a shared FS or rsync. More details on my method once I know it works in a later blog post!

How does key rotation work?

What Fernet Rotation Looks Like

What Fernet Rotation Looks Like

If you read through the information on Fernet tokens, key rotation is by far the most confusing. I’ve sat down with pen and paper and now think I get it, so allow me to explain. I’m going to use a 3 key example here, they keys are named with numbers. I highly encourage you to setup a throwaway Keystone box and use keystone-manage fernet_rotate if you don’t follow this.

You need to know 4 rules about how these keys work first:

  1. The highest numbered key is the current signing key.
  2. The 0 key is the key that will become the next signing key.
  3. All other keys are old keys, they’ve been used in the past and there might be old tokens out there still signing with them depending on your expiration schedule
  4. New keys are always created as key 0.

Starting position, per the rules above.

  • 0 – this is the on-deck key, after the next rotation, it’s primary.
  • 1 – this is the old key, it used to be primary, and its still here in case any old tokens are still signed with it. Next rotation it gets deleted.
  • 2 – this is the current primary key thats used for signing.

Now we do a Rotation…

  • 0 becomes 3
  • 1 gets deleted
  • 2 stays 2
  • a new key becomes 0
A Fernet Rotation in Action

A Fernet Rotation in Action

So How does this work?

Let’s pretend we have a few tokens since this is a running OpenStack cluster. All tokens before the rotation above are signed with 2. We do the rotation, now new tokens are signed with 3. When a token comes in, Keystone tries both 3 and 2 to decode the token, and either should work. At this point we CANNOT rotate again until no more active keys are signed with 2, because 2 is going to be deleted! This means you need to have more tokens if you plan on rotating more frequently or have a long token expiration time. We’re going to rotate roughly weekly, and we have a 2 hour token timeout, so 3 is plenty.

Homework

If you think you get this, try this a homework problem. Assume that you have max_active_keys set to 5 and that you have 5 keys: 0, 4, 5, 6, 7.

  • Which is the current signing key?
  • Which is on-deck or the next key to be used? What will it’s number be after the rotation?
  • Which key will be deleted on next rotation?
  • What happens if a token comes in signed with key 5?
  • What happens if a token comes in signed with key 3?

Other Sources

I gathered a lot of this info from trying stuff but also a lot from blog posts. I’ve referenced two above, but I also want to recommend Lance Bragstad’s blog. Note, Lance’s blog is the only blog in the world where you can read about quinoa recipes and shotgun shot patterns.

Tagged , ,

What I Hope to Get From the OpenStack Vancouver Summit

reproduced from content I wrote for SuperUser.

Matt Fischer, principal engineer at Time Warner Cable, shares survival tips for the Summit: backup plans, a beer list and a talk that he promises will be better than “Cats.”

May 11, 2015

The Vancouver summit is about a week away and so it’s time to start my prep work for the summit. This first means making a list of talks I want to go to. It also means making a list of any people I want to meet in person to talk to or people I owe a beer (or three) to. Finally, it means a list of things I want to accomplish at design sessions. If you’ve never been to a Summit before, what you get out of it really depends on how well you dig into the schedule and do some advanced planning.

So with that in mind, here’s my thinking.

Planning for Talks

I have some main focus areas for talks, things I want to come away from the summit knowing more about. These include things for which I’m the “owner” of in the Time Warner Cable OpenStack cloud, but also things I’m just curious about, so in no particular order:

Operations I always want to know how to do things differently or better, so you’ll see me around these rooms a lot. I’m specifically interested in Upgrades, CI/CD, and integrating new features into our cloud.

Heat I’m specifically interested in application catalog capabilities. Designate – I’ve worked some on this in the past few months. Neutron – I always want and need to know more about Neutron, even though it’s not my focus area.

Now with talks you can simply find the ones you like, and if you create an account and sign-in, you can add them to your personalized schedule. This is what I’d recommend to help plan your day. But, there’s a few more tricks that I’d recommend you use.

Review your schedule every morning. If you’re in Vancouver with a team you can divide up talks if you have time conflicts. You may also find that you’ve changed your mind or the schedule has changed, hence the morning review.

Have a backup talk. Sometimes talks are full or maybe you go and it’s not for you. You always need a backup talk.

Make a list of talks you want to watch later. All the talks end up on the OpenStack Foundation’s Youtube channel. Make a list of talks that you didn’t or won’t get to and watch them later. This does not work well for hands-on sessions, so I always opt to go to them.

If you can’t pick a talk by subject, pick by speaker. The speaker makes the talk sometimes more than the subject.

Don’t be afraid to have free time. The summit can be grueling. Leave a space or two in your schedule and visit the vendors or go take a nap.

Planning for People

OpenStack is community driven and the community is made of people. Take time to say “hi” to the people you’ve talked to on IRC or mailing lists.

Take the time to thank someone who fixed a bug for you or better yet buy them a beer. You cannot underestimate the value of having a beer with someone you’ve only previously met online. I cannot emphasize this enough.

Who are the lucky ones on my beer list this year? You’ll have to wait and see, hopefully I’m on yours!

Planning for Design Sessions

For the first time, OpenStack Puppet will be a real OpenStack project and so while previously we’ve had an hour to discuss stuff, this time we will have a full day for design work. Through lots of work over the past year, I’ve become core in OpenStack Puppet, and I hope to spend a good part of my day Tuesday participating in live discussions and work sessions. We have lots of stuff to discuss, the largest item which is dear to me, is when the master branch can drop support for the old stable release. If you’ve been active in other projects, you may have similar issues like this that need closure. These are usually easier to figure out in a room rather than in an IRC meeting. However, many of the design sessions are already planned so it may be too late to get something on the agenda, but it’s not too late to attend and participate. I’d recommend making a list of the things you want to cover and seeing how that lines up with the design session schedule. Please note that the Design Summit and the OpenStack Summit use different schedules hosted on different pages.

Planning for Parties

The OpenStack Summits have a large after-hours social aspect. These are valuable to attend just for camaraderie — you don’t have to drink beer to go and have fun. I generally go to as many of these as possible, they are usually pretty great.

You’re Invited!

I hope to see everyone in Vancouver, and would like to especially invite you to see some of my talks. The descriptions are pretty good, but I thought I’d say a few things about what you could expect to get out of each of them, here they are in chronological order with links so you can add them to your schedule.

Building Clouds with OpenStack Puppet Modules If you’re curious about how companies use Puppet to deploy OpenStack or about how our community works, you should attend this talk. You will get a couple different views and ideas on using Puppet from Mike Dorman and I, and hear about the community from Emilien Macchi. Monday, May 18 • 4:40 p.m. – 5:20 p.m. http://sched.co/38gd

A CI/CD Alternative to Push and Pray for OpenStack There are lots of CI talks this year, but I promise you will learn something new at this one. Clayton O’Neill and I will cover lots of topics and tools and you will see how we use these tools to get code and config from concept to production. Tuesday, May 19 • 12:05 p.m. – 12:45 p.m. http://sched.co/37lb

Getting DNSaaS to Production with Designate Have your customers been asking for DNSaaS (DNS as-a-Service)? Do you plan on having several people working on it full time as core Designate developers or would you rather just get it deployed with the minimum of pain? If the latter, then this is the talk for you. Clayton and I will cover what work is required, what work we did, and what to watch out for. One special thing that we will cover is how to write your own (or use our) Designate Sink which lets you automatically create records every time a new floating IP is assigned. Wednesday, May 20 • 9:50 a.m. – 10:30 a.m. http://sched.co/380J

Real World Experiences with Upgrading OpenStack at Time Warner Cable There are also lots of Upgrade talks at this summit. In this one, Clayton and I will be telling a story of what happened when we upgraded to Juno. Even though I wasn’t smart enough to put “beer” in the title of my upgrade talk, maybe you can learn some lessons or get some ideas from us. You’ll laugh, you’ll cry, it will be better than “Cats.” Thursday, May 21 • 2:20 p.m. – 3:00 p.m. http://sched.co/37wu

Tagged ,