Monthly Archives: April 2014

Keystone: Using Stacked Auth (SQL & LDAP)

Most large companies have an Active Directory or LDAP system to manage identity and integrating this with OpenStack makes sense for some obvious reasons, like not having multiple passwords and automatically de-authenticating users who leave the company. However doing this in practice can be a bit tricky. OpenStack needs some basic service accounts. Then your users probably want their own service accounts, since they don’t want to stick their AD password in a Jenkins config. So multiple the number of teams that may use your OpenStack by 4 or 5 and you may have a ballpark estimate on the number of service accounts you need. Then take that number to your AD team and be sure to run it by your security guys too. This is the response you might get.

Note: All security guys wear cloaks.

The way you can tell how a conversation with a security guy is going is to count the number of facepalms they do.

In this setup you probably also do not have write access to your LDAP box. This means that you need a way to track role assignment outside of LDAP, using the SQL back-end (a subject I’ve previous covered)

So this leaves us with an interesting problem, fortunately one with a few solutions. The one we chose was to use a “stacked” or “hybrid” auth in Keystone. One where service accounts live in Keystone’s SQL backend and if users fail to authenticate there they fallback to LDAP. For role Assignment we will store everything in Keystone’s SQL back-end. The good news is that Ionuț Arțăriși from SUSE has already written a driver that does exactly what you need. His driver for Identity and Assignment is available here on github and has a nice README to get you started. I did have a few issues getting it to work, which I will cover below which hopefully saves you some time in setting this up.

The way Ionut’s driver works is pretty simple, try the username and password first locally against SQL. If that fails, try to use the combo to bind to LDAP. This saves the time of having to bind with a service account and search, so it’s simpler and quicker. You will need to be able to at least bind RO with your standard AD creds for this to work though.

From this point forward, you probably will only have the issues I have if you have an AD/LDAP server with a large number of users. If you’re in a lab environment, just following the README in Ionut’s driver is all you need. For everyone else, read on…

Successes and Problems

After configuring the LDAP driver, I switched and enabled hybrid identity and restarted keystone. The authentication module seemed to be working great. I did my first test by doing some basic auth and then running a user-list. The auth worked, although my user lacked any roles that allowed him to do a user list. When I set the SERVICE_TOKEN/ENDPOINT and re-ran the user list, well that was interesting. I noticed that since I had enabled the secret LDAP debug option (hopefully soon to be non-secret) that it was doing A TON of queries and I eventually hit the server max before setting page_size=1000. Below is an animation that shows what it was like running keystone user-list:

This isn't really animated, but is still an accurate representation of what happened.

This isn’t really animated, but is still an accurate representation of how slow it was

It did eventually finish, but it took 5 minutes and 25 seconds, and ain’t nobody got time for that. One simple and obvious fix for this is to just modify the driver to not pull LDAP users in the list. There are too many to manage anyway, so I made the change and now it only showed local accounts. This was a simple change to the bottom of the driver, to just return SQL users and not query LDAP.

root@mfisch:~# keystone user-list
| id | name | enabled | email |
| e0ede62ebcb64e489a41f4a9b18cd63c | admin | True | root@localhost |
| f9d76feb7081463f973eeda82c918547 | ceilometer | True | root@localhost |
| 176e232176c74164a42e2f773695671b | cinder | True | cinder@localhost |
| fc8007f82b7542e88ca28fab5eda1b5c | glance | True | glance@localhost |
| 635b4022aafc4f5488c048109c7b1665 | heat | True | heat@localhost |
| b3175fc7b62748679b4c958fe89fbdf0 | heat-cfn | True | heat-cfn@localhost |
| 4873cd322d0d4582908002b619e3940a | neutron | True | neutron@localhost |
| 37ef623c60474bf5a384a2047719acf4 | nova | True | nova@localhost |
| ff07bd356b8d4959be4326a43c297db0 | swift | True | swift@localhost |

After making this change, I went to add the admin role to my AD user and found I had a second problem. It seemed to be relying on my user being in the user list before it would assign them a role.

root@mfisch:~# keystone user-role-add --user-id MyADUser --role-id admin --tenant-id 6031675b7618458b8bb85b92857532b06No user with a name or ID of 'MyADUser' exists.

I didn’t really want to wait five minutes to assign or list a role, so this needed fixing.


It was late on Friday afternoon when I started toying with some solutions for the role issue. The first solution is to add OpenStack users into a special ou so that you can limit the scope of the search. This is done via the user_filter setting in keystone.conf. If this is easy for your organization, it’s a simple fix. It also allows tools like Horizon to show all your users. If you do this you can obviously undo the change to not return LDAP users.

The second idea that came to mind was that every time an AD user authenticated, I could just make a local copy of that user, without a password and setup in a way that they’d be unable to authenticate locally. I did manage to get this code functional, but it has a few issues. The most obvious one is that when users leave the company there’s no corresponding clean-up of their records. The second one is that it doesn’t allow for any changes to things like names or emails once they’ve authenticated. In general it’s bad to have two copies of something like this info anyway. I did however manage to get this code working and it’s available on github if you ever have a use for it. It is not well tested and it’s not very clean in the current state. With this change, I had this output (truncated):

root@mfisch:~# keystone user-list
| id | name | enabled | email |
| MyADUser | MyADUser | True | my corporate email |
| e0ede62ebcb64e489a41f4a9b18cd63c | admin | True | root@localhost |
| f9d76feb7081463f973eeda82c918547 | ceilometer | True | root@localhost |

Due to the concerns raised above, we decided to drop this idea.

My feelings on this solution...

My feelings on this solution…

The final idea which came after some discussions was to figure out why the heck a user list was needed to add a role. I dug into the client code, hacked a fix, and then asked on IRC. It turns out I wasn’t the first person to notice it. The bug (#1189933) is that when the user ID is not a UUID, it thinks you must be passing in a name and does a search. The fix for this is to pull a copy of python-keystoneclient thats 0.4 or newer. We had 0.3.2 in our repo. If you upgrade this package to the latest (0.7x) you will need a copy of python-six that’s 1.4.0 or newer. This should all be cake if you’re using the Ubuntu Cloud Archive.

Final State

In the final state of things, I have taken the code from Ionut and:

  1. disabled listing of LDAP users in the driver
  2. upgraded python-six and python-keystoneclient
  3. packaged and installed Ionut’s drivers (as soon as I figure out the license for them, I’ll publish the packaging branch)
  4. reconfigured keystone via puppet

So far it’s all working great with just one caveat so far. If users leave your organization they will still have roles and tenants defined in Keystone’s SQL. You will probably need to define a process for cleaning those up occasionally. As long as the users are deactivated in AD/LDAP though the roles/tenant mappings for them should be relatively harmless cruft.

Tagged , , ,

Announcing forth-keystoneclient

Over the past few weeks I’ve been working on a new client implementation for keystone. Python is a bit bloated for nimble CLIs like keystone, and so I went back to a language I used at my first job back in 1999, Forth. Forth is interpreted like python but it’s also very simple and memory efficient. And unlike python, forth can fit on a floppy disk. This allows this client to be installed onto satellites and other memory constrained environments. For embedded systems, forth also makes a great alternative to C. Now, onto the client.

Mastering Forth

Mastering Forth

As you probably know, forth is stack-based, and no real programmer likes someone else managing the stack for them. I’ve taken this paradigm into the client design. So let’s dive in. Below is an example usage so you can see how simple it is:

First we remember the definition of what user-role-list does. Python offers the “help” and “dir” for introspection, forth has ‘ and DUMP which work just as well. First let’s make sure user-role-list is defined:

' user-role-list  ok

Then let’s see what it’s doing:

' user-role-list 64 cells dump
7F0A31F7F300: D5 44 40 00  00 00 00 00 - 00 00 00 00  00 00 00 00  .D@.............
7F0A31F7F310: 25 46 40 00  00 00 00 00 - 20 DE F2 31  0A 7F 00 00  %F@..... ..1....
7F0A31F7F320: 88 46 40 00  00 00 00 00 - 99 99 99 99  99 99 99 99  .F@.............

As you can simply tell from the above, we need to pass in the tenant-id and user-id, so let’s make the call.

S" de4442c6e54a43459eaab97e26dc21bc2"  ok
S" 2ebadab9148d421287eee6d264f29736d"  ok
user-role-list 33  
| id | name | user_id | tenant_id |
| 006eaf0730e44756bc679038477d3bbd | Member | 2ebadab9148d421287eee6d264f29736d | de4442c6e54a43459eaab97e26dc21bc2 |
| 03e83b65036a4e0cbd7cff5bff858c76 | admin | 2ebadab9148d421287eee6d264f29736d | de4442c6e54a43459eaab97e26dc21bc2 |

I’ll be handing this out on floppy disks at the OpenStack conference in Atlanta if you’re interested, come find me and I’ll make you a copy.

Prepping for distribution at the conference

Prepping for distribution at the conference

A note on debugging, if you plan on installing this on satellites remember how the speed of light can interact with your token expiration timeout.

Tagged ,