Monthly Archives: July 2015

Keystone Token Revocations Cripple Validation Performance

Having keystone token revocation events cripples token validation performance. If you’ve been following any of the mailing lists posts on this topic, then you already know this since it’s been discussed (here) and (here). In this post I explore the actual impact and discuss what you can do about it.

What Are Revocations?

A token is revoked for any number of reasons, but basically when it’s revoked, it’s invalid. Here are some of the reasons that revocation events will be generated:

  • The token is intentionally invalidated via the API
  • A user is deleted
  • A user has a role removed
  • A user is removed from a project
  • A user logs out of Horizon
  • A user switches projects in Horizon

Of these events the last two are by far the most common reasons that revocation events are being generated in your cloud.

How Are Revocation Events Used?

How this works varies some based on the token type, but lets assume that a token comes in that is non-expired. We know that either from decrypting it (Fernet) or from looking it up in the DB (UUID). But before Keystone can bless the token, it needs to check the revocation table to ensure that the token is still valid. So it loads the table called revocation_event and takes a peek. Also when it does this load, Keystone does a little house-keeping and removes and revocation events that are on tokens that are already expired. The time a revocation event lives is the same as the token. It does not make sense to have a 3 hour old revocation event when the longest token can live is 1 hour. The unfortunate thing with this algorithm is that it locks the table, slowing down other revocations even more and if it takes too long, leads to deadlocks and failed API calls.

Why Should You Care About Token Validation?

Keystone token validation underlies every single API call that OpenStack makes. If keystone token validation is slow, everything is slow. Validation takes place for example when you make a nova call, nova has to be sure that the token is okay first before performing the action.

Results

If you want to see the experimental setup, skip below, but most of you will want the numbers first!

The chart below shows two runs of the benchmark which checks concurrent token validations. You will see that as soon as you have revocation events, performance falls significantly. There are two lines on the chart. The first line, blue, is our current packaged version of Keystone which is Kilo++/Liberty. The second line in red, shows the performance of a version of Liberty from July 17 with this patch applied. The hope with the patched code is that smarter use of deletes would improve performance, it does not in a measurable way. It may however reduce deadlocks, but I am unable to validate that since my environment is not under any real load.

Benchmark Results

Note: Do not put too much stock into the fact that the red line starts slower than the blue, instead focus on the shape of the curve. There’s too many possible variables in my testing (like what my hypervisor is doing and all the other changes between versions) to compare them apples to apples.

Experimental Setup

For the experimental setup all the systems are guests running in our production cloud built using vagrant-openstack and our standard puppet automation code. The nodes are as follows:

  • 3 keystone nodes
  • 1 haproxy load balancer
  • a puppet master, which also runs the benchmarks

The nodes are running Ubuntu and a version of Keystone from master from May 2015. They are using Fernet tokens that expire after two hours. mysql is setup as a 3 node Galera cluster that preferentially uses one node. The systems were not otherwise busy or doing much else.

The test itself tries to do 20 validations at once up to 4000 of them. It talks to the load balancer which is setup to do round-robin connections.

Given all the variables here, I don’t expect you to replicate these numbers, but rather viewed relative to each other.

Running the Benchmark

For the benchmark code, I used a modified version of Dolph’s benchmark experiment. The modified code is here (note that the detection of whether ab is installed is broken feel free to send me a fix).

Usage:

./benchmark.sh [Keystone Node or LB] [admin_password]

Generating Revoked Tokens

Here’s my kinda hacky script to generate and revoke tokens, it could be better if it just used curls for both. Usage is to pass in a number of tokens to create and then revoke as arg1 and then a valid token as arg2 that you’ve previously generated.

echo "getting & revoking $1 tokens"
for i in $(eval echo "{1..$1}")
do
TOKEN=`keystone token-get | grep id | grep -v tenant_id | grep -v user_id | awk '{ print $4 }'`
curl -X DELETE -i -H "X-Auth-Token: $2" "${OS_AUTH_URL}/tokens/${TOKEN}"
done

Solutions?

Here are a few ideas I’d recommend. First get a baseline of how many revocations you have on a regular basis, this should mainly be from people signing out of Horizon or switching projects in Horizon. For us it’s about 20-30. This is how you check.

mysql -u root keystone -e "select count(id) from revocation_event;"

Once you get a normal number, I’d recommend putting a threshold check into Icinga.

Watch your testing too, we have some regression tests that create users, roles, etc and generates about 500 revocation events.

If you have a spike of events, and you’re not worried about rogue users, you can simply truncate the table.

mysql -u root keystone -e "truncate table revocation_event;"

This has security implications so make sure you know what you are doing.

Another idea is writing a no-op driver for revocations, this essentially disables the feature and again has security implications.

Finally, I’d recommend enabling caching for revocation events you still get the same curve, but you’ll start out at a higher performance value.

Tagged , ,