Keystone: User Enabled Emulation Can Lead to Bad Performance

An update on my previous post about User Enabled Emulation, tl;dr, don’t use it. It’s slow. Here’s what I found:

I just spent parts of today debugging why keystone user-list was so slow. It was taking between 8 and 10 seconds to list 20 users. I spent a few hours tweaking the caching settings, but the database was so small that the cache never filled up, and so I realized that this was not the main issue. A colleague asked me if basic ldapsearch was slow, and no it was fine. Then I dug into what Keystone is doing with the enabled emulation code. Unlike a user_filter, Keystone appears to be querying the user list and then re-querying LDAP for each user to check if they’re in the enabled_emulation group. This leads to a lot of extra queries which slows things down. When I disabled this setting, the query performance improved dramatically to between 2-2.5 seconds, about a 4x speed-up. If you are in a real environment with more than 20 users, the gain will be pretty good in terms of real seconds.

Disabling the enabled_emulation leaves us with no user enabled information. In order to get the Enabled field back, I’m going to add a field to the schema to emulate what AD does with the enabled users. Since this portion of Keystone was designed for AD, this blog post may help clear up what exactly it expects here from an AD point of view. Read that page to the end and you get a special treat, how to use Logical OR in LDAP, see if it makes less sense than the bf language does.

Also to reduce my user count, I did enable a user_filter, which since it’s just part of the initial query does NOT appear to slow things down. You could skip the new field and just use the filter if you want, however it’s not clear what impact a “blank” for Enabled has, other than perhaps some confusion. If it has a real impact, PLEASE comment here!

Tagged , , ,

3 thoughts on “Keystone: User Enabled Emulation Can Lead to Bad Performance

  1. Yuriy Taraday says:

    That’s unfortunate. I didn’t expect performance degradation to be that big.
    I think, there’s a way to fix this but I don’t have free cycles to take on this task. Can you please file a bug on this topic?

  2. Nathan Kinder says:

    The emulated enabled functionality adds a whole bunch of extra LDAP operations (complete with establishing a connection and tearing it down for every operation). I’ve looked at the LDAP operations that are performed by Keystone in a packet trace, and there were a lot of extra operations that really don’t need to be performed. For example, a simple ‘keystone user-show’ performed 4 LDAP operations when I looked earlier this year:

    1) search by id
    2) search by name
    3) search enabled
    4) search enabled

    That’s 4 connections being established (with SSL handshakes hopefully), 4 bind operations, 4 search operations, and 4 disconnects just to show a user. A ‘user show’ was also one of the better cases. You don’t want to know what a user update or checking group membership looks like…

    One thing that would be helpful is to change the way that group membership is checked. If an LDAP server uses the ‘memberOf’ attribute, the list of groups that a user belongs to is right there in the user entry. This would allow a single search operation to retrieve a user, then we could just use the memberOf attributes without searching for the groups. There are some wrinkles to work out, such as the problem of not having all of the mapped group attributes in the group DN. Still, if the group’s RDN is using the attribute that is mapped to the group ID in Keystone, that is likely all we really need for a membership check. It would be a bit of surgery in the Keystone LDAP code, but it would pay off well in terms of performance.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>