Tag Archives: LDAP

Using Keystone’s LDAP Connection Pools to Speed Up OpenStack

If you use LDAP with Keystone in Juno you can give your implementation a turbo-boost by using LDAP connection pools. Connection pooling is a simple idea. Instead of bringing up and tearing down a connection every time you talk to LDAP, you just reuse an existing one. This feature is widely used in OpenStack when talking to mysql and adding it here really makes sense.

After enabling this feature, using the default settings, I got a 3x-5x speed-up when getting tokens as a LDAP authenticated user.

Using the LDAP Connection Pools

One of the good things about this feature is that it’s well documented (here). Setting this up is easy. The tl;dr is that you can just enable two fields and then use the defaults and they seem to work pretty well.

First turn the feature on, nothing else works without this master switch

# Enable LDAP connection pooling. (boolean value)
use_pool=true

Then if you want to use pools for user authentication, add this one:

# Enable LDAP connection pooling for end user authentication. If use_pool
# is disabled, then this setting is meaningless and is not used at all.
# (boolean value)
use_auth_pool=true

Experimental Setup

For my experiment I used a virtual keystone node that we run on top of our cloud, pointing at a corporate AD box using ldaps. Using an LDAP user, I requested 500 UUID tokens in a row. We use a special hybrid driver that uses the user creds to bind against ldap and ensure that the user/pass combo is valid. I also changed my OS_AUTH_URI to point directly at localhost to avoid hitting the load balancer. Finally I’m using the eventlet (keystone-all) vs apache2 to run Keystone. According to the Keystone PTL, Morgan Fainberg, “under apache I’d expect less benefit” If you’re not using the eventlet, ldaps, or my hybrid driver you might get different results, but I’d still expect it to be faster.

Here’s my basic test script:

export OS_TENANT_NAME=admin
export OS_USERNAME=LDAPUSER
export OS_PASSWORD=password
export OS_TENANT_NAME=admin
export OS_REGION_NAME='dev02'
export OS_AUTH_STRATEGY=keystone
export OS_AUTH_URL=http://localhost:5000/v2.0/
echo "getting $1 tokens"
for i in $(eval echo "{1..$1}")
do
curl -s -X POST http://localhost:5000/v2.0/tokens \
-H "Content-Type: application/json" \
-d '{"auth": {"tenantName": "'"$OS_TENANT_NAME"'", "passwordCredentials": {"username": "'"$OS_USERNAME"'", "password": "'"$OS_PASSWORD"'"}}}' > /dev/null
done

Results

Using the default config, It took 7 mins, 25s to get the tokens.

getting 500 tokens
real 7m25.527s
user 0m2.312s
sys 0m1.557s

I then enabled use_pool and use_auth_pool and restarted keystone, the results were quite a bit faster, a 5x speed-up. Wow.

getting 500 tokens
real 1m25.774s
user 0m2.302s
sys 0m1.539s

I ran this several times and the results were all within a few seconds of each other.

I also tried this test using the keystone CLI and the results were closer to 3.5x faster, still a respectable number.

Watching the Connections

I have a simple command so I can see how many connections are being used:

watch -n1 "netstat -an p tcp | grep :3269"

Using this simple code I can see it bounce between 0 and 1 without connection pools.

Using the defaults but with connection pools enabled, the number of connections was a solid 4. Several minutes after the test ran they died after a bit and went to 0.

I’m not sure why I didn’t get more than 4, but raising the pool counts did not change this value. Any ideas on this are welcome. This is because I have 4 workers on this node.

Tokens Are Fundamental

The coolest part of this is that this change speeds everything up. Since you need a token to do anything, I re-ran the test but just had it run nova list, cinder list, and glance image-list 50 times using the clients. Without the pooling, it took 316 seconds but with the pooling it took 231 seconds.

Plans

There are lots of ways to improve the performance of OpenStack, but this one is simple and easy to setup. The puppet code to configure this is in progress now. Once it lands, I plan to move this to dev and then to staging and prod in our environments. If I learn any other interesting things there, I’ll update the post.

Tagged , , ,

Quick Update on Stacked Auth

A quick update on Stacked/Hybrid auth for Keystone. I spent some time this week updating the driver for Icehouse. That branch is available here and includes packaging info. I’ve been too lazy to make a non-packaging copy of the branch but it should be easy to do. During my debugging of the new driver, I realized one thing, even though this driver does a bind using the credentials of the user you are trying to authenticate, you still need a service account to bind with if you are using subtree search. The reason is that the LDAP code does an initial bind to map your user id to a dn so that then it can do the “real” bind. If you see any issues with this code please let me know!

Tagged , ,

Keystone: Using Stacked Auth (SQL & LDAP)

Most large companies have an Active Directory or LDAP system to manage identity and integrating this with OpenStack makes sense for some obvious reasons, like not having multiple passwords and automatically de-authenticating users who leave the company. However doing this in practice can be a bit tricky. OpenStack needs some basic service accounts. Then your users probably want their own service accounts, since they don’t want to stick their AD password in a Jenkins config. So multiple the number of teams that may use your OpenStack by 4 or 5 and you may have a ballpark estimate on the number of service accounts you need. Then take that number to your AD team and be sure to run it by your security guys too. This is the response you might get.

Note: All security guys wear cloaks.

The way you can tell how a conversation with a security guy is going is to count the number of facepalms they do.

In this setup you probably also do not have write access to your LDAP box. This means that you need a way to track role assignment outside of LDAP, using the SQL back-end (a subject I’ve previous covered)

So this leaves us with an interesting problem, fortunately one with a few solutions. The one we chose was to use a “stacked” or “hybrid” auth in Keystone. One where service accounts live in Keystone’s SQL backend and if users fail to authenticate there they fallback to LDAP. For role Assignment we will store everything in Keystone’s SQL back-end. The good news is that Ionuț Arțăriși from SUSE has already written a driver that does exactly what you need. His driver for Identity and Assignment is available here on github and has a nice README to get you started. I did have a few issues getting it to work, which I will cover below which hopefully saves you some time in setting this up.

The way Ionut’s driver works is pretty simple, try the username and password first locally against SQL. If that fails, try to use the combo to bind to LDAP. This saves the time of having to bind with a service account and search, so it’s simpler and quicker. You will need to be able to at least bind RO with your standard AD creds for this to work though.

From this point forward, you probably will only have the issues I have if you have an AD/LDAP server with a large number of users. If you’re in a lab environment, just following the README in Ionut’s driver is all you need. For everyone else, read on…

Successes and Problems

After configuring the LDAP driver, I switched and enabled hybrid identity and restarted keystone. The authentication module seemed to be working great. I did my first test by doing some basic auth and then running a user-list. The auth worked, although my user lacked any roles that allowed him to do a user list. When I set the SERVICE_TOKEN/ENDPOINT and re-ran the user list, well that was interesting. I noticed that since I had enabled the secret LDAP debug option (hopefully soon to be non-secret) that it was doing A TON of queries and I eventually hit the server max before setting page_size=1000. Below is an animation that shows what it was like running keystone user-list:

This isn't really animated, but is still an accurate representation of what happened.

This isn’t really animated, but is still an accurate representation of how slow it was

It did eventually finish, but it took 5 minutes and 25 seconds, and ain’t nobody got time for that. One simple and obvious fix for this is to just modify the driver to not pull LDAP users in the list. There are too many to manage anyway, so I made the change and now it only showed local accounts. This was a simple change to the bottom of the driver, to just return SQL users and not query LDAP.

root@mfisch:~# keystone user-list
+----------------------------------+------------+---------+--------------------+
| id | name | enabled | email |
+----------------------------------+------------+---------+--------------------+
| e0ede62ebcb64e489a41f4a9b18cd63c | admin | True | root@localhost |
| f9d76feb7081463f973eeda82c918547 | ceilometer | True | root@localhost |
| 176e232176c74164a42e2f773695671b | cinder | True | cinder@localhost |
| fc8007f82b7542e88ca28fab5eda1b5c | glance | True | glance@localhost |
| 635b4022aafc4f5488c048109c7b1665 | heat | True | heat@localhost |
| b3175fc7b62748679b4c958fe89fbdf0 | heat-cfn | True | heat-cfn@localhost |
| 4873cd322d0d4582908002b619e3940a | neutron | True | neutron@localhost |
| 37ef623c60474bf5a384a2047719acf4 | nova | True | nova@localhost |
| ff07bd356b8d4959be4326a43c297db0 | swift | True | swift@localhost |
+----------------------------------+------------+---------+--------------------+

After making this change, I went to add the admin role to my AD user and found I had a second problem. It seemed to be relying on my user being in the user list before it would assign them a role.

root@mfisch:~# keystone user-role-add --user-id MyADUser --role-id admin --tenant-id 6031675b7618458b8bb85b92857532b06No user with a name or ID of 'MyADUser' exists.

I didn’t really want to wait five minutes to assign or list a role, so this needed fixing.

Solutions

It was late on Friday afternoon when I started toying with some solutions for the role issue. The first solution is to add OpenStack users into a special ou so that you can limit the scope of the search. This is done via the user_filter setting in keystone.conf. If this is easy for your organization, it’s a simple fix. It also allows tools like Horizon to show all your users. If you do this you can obviously undo the change to not return LDAP users.

The second idea that came to mind was that every time an AD user authenticated, I could just make a local copy of that user, without a password and setup in a way that they’d be unable to authenticate locally. I did manage to get this code functional, but it has a few issues. The most obvious one is that when users leave the company there’s no corresponding clean-up of their records. The second one is that it doesn’t allow for any changes to things like names or emails once they’ve authenticated. In general it’s bad to have two copies of something like this info anyway. I did however manage to get this code working and it’s available on github if you ever have a use for it. It is not well tested and it’s not very clean in the current state. With this change, I had this output (truncated):

root@mfisch:~# keystone user-list
+----------------------------------+------------+---------+--------------------+
| id | name | enabled | email |
+----------------------------------+------------+---------+--------------------+
| MyADUser | MyADUser | True | my corporate email |
| e0ede62ebcb64e489a41f4a9b18cd63c | admin | True | root@localhost |
| f9d76feb7081463f973eeda82c918547 | ceilometer | True | root@localhost |
...
+----------------------------------+------------+---------+--------------------+

Due to the concerns raised above, we decided to drop this idea.

My feelings on this solution...

My feelings on this solution…

The final idea which came after some discussions was to figure out why the heck a user list was needed to add a role. I dug into the client code, hacked a fix, and then asked on IRC. It turns out I wasn’t the first person to notice it. The bug (#1189933) is that when the user ID is not a UUID, it thinks you must be passing in a name and does a search. The fix for this is to pull a copy of python-keystoneclient thats 0.4 or newer. We had 0.3.2 in our repo. If you upgrade this package to the latest (0.7x) you will need a copy of python-six that’s 1.4.0 or newer. This should all be cake if you’re using the Ubuntu Cloud Archive.

Final State

In the final state of things, I have taken the code from Ionut and:

  1. disabled listing of LDAP users in the driver
  2. upgraded python-six and python-keystoneclient
  3. packaged and installed Ionut’s drivers (as soon as I figure out the license for them, I’ll publish the packaging branch)
  4. reconfigured keystone via puppet

So far it’s all working great with just one caveat so far. If users leave your organization they will still have roles and tenants defined in Keystone’s SQL. You will probably need to define a process for cleaning those up occasionally. As long as the users are deactivated in AD/LDAP though the roles/tenant mappings for them should be relatively harmless cruft.

Tagged , , ,

Keystone: User Enabled Emulation Can Lead to Bad Performance

An update on my previous post about User Enabled Emulation, tl;dr, don’t use it. It’s slow. Here’s what I found:

I just spent parts of today debugging why keystone user-list was so slow. It was taking between 8 and 10 seconds to list 20 users. I spent a few hours tweaking the caching settings, but the database was so small that the cache never filled up, and so I realized that this was not the main issue. A colleague asked me if basic ldapsearch was slow, and no it was fine. Then I dug into what Keystone is doing with the enabled emulation code. Unlike a user_filter, Keystone appears to be querying the user list and then re-querying LDAP for each user to check if they’re in the enabled_emulation group. This leads to a lot of extra queries which slows things down. When I disabled this setting, the query performance improved dramatically to between 2-2.5 seconds, about a 4x speed-up. If you are in a real environment with more than 20 users, the gain will be pretty good in terms of real seconds.

Disabling the enabled_emulation leaves us with no user enabled information. In order to get the Enabled field back, I’m going to add a field to the schema to emulate what AD does with the enabled users. Since this portion of Keystone was designed for AD, this blog post may help clear up what exactly it expects here from an AD point of view. Read that page to the end and you get a special treat, how to use Logical OR in LDAP, see if it makes less sense than the bf language does.

Also to reduce my user count, I did enable a user_filter, which since it’s just part of the initial query does NOT appear to slow things down. You could skip the new field and just use the filter if you want, however it’s not clear what impact a “blank” for Enabled has, other than perhaps some confusion. If it has a real impact, PLEASE comment here!

Tagged , , ,

Keystone: LDAP for Identity, SQL for Assignment

This week I volunteered to work on playing around with integrating keystone with LDAP (via a FreeIPA box). Since I basically knew nothing about LDAP nor keystone before starting, I learned a few lessons along the way. And here they are:

The Setup

I started with a FreeIPA server that our team uses, I have full admin rights to it. I was also running an “All In One” openstack Havana instance on a VM that I setup using puppet_openstack_builder.

The Goal

My basic goal here was to learn more about both LDAP and keystone and in the process get the AIO instance to authenticate against LDAP as a preliminary test to authenticating against AD. In our environment, I wanted LDAP to manage Idenity (users, groups, and group memberships) and Keystone’s SQL backend to manage Assignment (roles, tenants, domains). This will work well when we integrate with AD since we cannot just create accounts on the corporate AD server willy-nilly. This is called Read Only LDAP and is covered in more detail here.

Diving In

There are several awesome blog entries about using FreeIPA and Keystone from Adam Young and these helped me get started. I’d configure keystone, restart it, then tail the logs while running keystone user-list.

Configuration

The very basic config is done like this:

First, enable the LDAP identity driver:
[identity]
driver = keystone.identity.backends.ldap.Identity
#driver = keystone.identity.backends.sql.Identity

Then you need to tell Keystone to use SQL for assignment:
[assignment]
driver = keystone.assignment.backends.sql.Assignment

Next we setup the user to which AD will authenticate since we’re using ldaps:

[ldap]
url = ldaps://example.com:636
user = uid=service_acct,cn=users,cn=accounts,dc=example,dc=com
password = UbuntuRulez

Then we tell it about the user and group schema:

user_tree_dn = cn=users,cn=accounts,dc=example,dc=com
user_filter = (memberOf=cn=openstack,cn=groups,cn=accounts,dc=example,dc=com)
user_objectclass = inetUser
user_id_attribute = uid
# this is what is searched on
user_name_attribute = uid
user_mail_attribute = mail
user_pass_attribute =
# XXX FIXME -mfisch: wont work on freeIPA like this
#user_enabled_attribute = (nsAccountLock=False)
user_allow_create = False
user_allow_update = False
user_allow_delete = False
...
group_tree_dn = cn=groups,cn=accounts,dc=example,dc=com
group_filter =
group_objectclass = groupOfNames
group_id_attribute = cn
group_name_attribute = cn
group_member_attribute = member
group_desc_attribute = description
# group_attribute_ignore =
group_allow_create = False
group_allow_update = False
group_allow_delete = False

Lastly we point at the cert info:
use_tls = False
tls_cacertfile = /etc/ssl/certs/ca-certificates.crt

The First Problem

The first stumbling block I hit is that I needed to use ldaps. This is a pretty basic fix, just grab the cert and put it on your box where keystone is running. Unfortunately I found out that unless I had the cert path also defined in /etc/ldap/ldap.conf, the query didn’t work. The fix is to make a pretty much empty /etc/ldap/ldap.conf and add this one line that points to the cert you pulled down:

TLS_CACERT /etc/ssl/certs/ca-certificates.crt

user_name_attribute

My next issue is that the default user_name_attribute is not right for FreeIPA. I checked this using Apache Directory Studio which made it easy for me to browse the tree and test queries. I needed to use uid and not cn. The issue here is that if you have this wrong, then the initial authentication of your LDAP user for the ldaps query fails and the resulting output provides no clue as to the issue, even with debug enabled. Once I solved this, I had a permission issue.

Bootstrapping

The problem once you switch the authentication mechanism to LDAP, you’ve “lost” all the old roles and users that puppet setup, like the admin user who actually has permissions to do stuff like list users. The fix as any keystone expert will know is to bypass the main API and use the service token directly. In the keystone.conf a service token is defined as admin_token. To use it, unset everything like OS_USERNAME and set:

export SERVICE_TOKEN=
export SERVICE_ENDPOINT=http://localhost:35357/v2.0

Then you need to give your user, in my case, “id=mfischer” permission to do stuff. I ended up giving mfischer the admin and _member_ role in both of my tenants since he/me is the new admin. Finally, I switched my settings back to what’s in my rc file, mfischer as the user, my LDAP password, and the normal keystone endpoint, and… finally my user-list query worked and I got results from LDAP.

root@landscape-03:/var/log/keystone# keystone user-list
+------------+------------+---------+--------------------------+
| id | name | enabled | email |
+------------+------------+---------+--------------------------+
| mfischer | mfischer | | matt.fischer@example.com |
....

User Enabled?

keystone wants to know if a user is enabled and shows this as a column in the output of user-list. All my results were blank, and it’s not clear that FreeIPA has a straightforward “enabled” column in our setup. At a glance this stuff seemed optimized for AD’s user-enabled mask stuff. (If you know a fix, let me know please).

Trying Other Services

I decided to try nova list next and guess what, it failed. It’s obvious to me now, but the puppet AIO has users created for almost all the services, like nova, cinder, glance, etc. I needed users for these. This is easy to do in FreeIPA, so I made a batch of users and restarted the node. I bet you can guess, but stuff still failed because these users didn’t have the roles that they needed. At this point, I switched the identity backend back to SQL, restarted keystone and made a map of roles to users and tenants. In my case it was pretty basic, everything needed admin in the services tenant and some needed more, here’s how to do it simply:

for I in 'glance' 'nova' 'cinder' 'neutron' 'heat' 'heat-cfn' 'swift';
do; keystone user-role-add --user-id=$I --tenant-id=de4442c6e54a43459eaab97e26dc21f8 --role id=a5a8ea228b1942e28289ba63fba9b3c0; done

I did this and then bounced the node again (which is frankly simpler than restarting everything but unlikely to work in the real world). And again nothing worked! I’d forgotten one more step, each service has a password defined in it’s config file as “admin_password”. I signed back into FreeIPA and set the passwords and then bounced the node a final time.

This time, I could sign into Horizon and everything worked great! Finally, I was using LDAP only for identity and didn’t need to switch back.

Obviously this is much better to be done before you start your puppet install but in my case it was a great learning experience about roles/tenants/users and the tools that keystone provides.

Debugging Tools/Hints

Here are some more debugging tools and hints that I came across and maybe will help you if you’re dealing with LDAP and keystone:

  1. Enable debug and verbose in keystone, of course.
  2. I’ve mentioned Apache Directory Studio so you can test LDAP queries. This command also helps show what fields are available and is simpler than using ADS: “ipa user-show –all –raw”.
  3. The LDAP code in keystone has double secret logging which you can enable in /usr/lib/python2.7/dist-packages/keystone/common/ldap/core.py. You can look for “ldap.set_option(ldap.OPT_DEBUG_LEVEL, 4095)” and uncomment it. These logs only show on stdout, so you’ll need to stop the service and run it by hand to see this output.
  4. I also traced some code in pdb, in addition to the file listed above you should also look at /usr/lib/python2.7/dist-packages/keystone/identity/backends/ldap.py

Good luck everyone!

Tagged , ,