Monthly Archives: January 2014

Keystone: LDAP for Identity, SQL for Assignment

This week I volunteered to work on playing around with integrating keystone with LDAP (via a FreeIPA box). Since I basically knew nothing about LDAP nor keystone before starting, I learned a few lessons along the way. And here they are:

The Setup

I started with a FreeIPA server that our team uses, I have full admin rights to it. I was also running an “All In One” openstack Havana instance on a VM that I setup using puppet_openstack_builder.

The Goal

My basic goal here was to learn more about both LDAP and keystone and in the process get the AIO instance to authenticate against LDAP as a preliminary test to authenticating against AD. In our environment, I wanted LDAP to manage Idenity (users, groups, and group memberships) and Keystone’s SQL backend to manage Assignment (roles, tenants, domains). This will work well when we integrate with AD since we cannot just create accounts on the corporate AD server willy-nilly. This is called Read Only LDAP and is covered in more detail here.

Diving In

There are several awesome blog entries about using FreeIPA and Keystone from Adam Young and these helped me get started. I’d configure keystone, restart it, then tail the logs while running keystone user-list.


The very basic config is done like this:

First, enable the LDAP identity driver:
driver = keystone.identity.backends.ldap.Identity
#driver = keystone.identity.backends.sql.Identity

Then you need to tell Keystone to use SQL for assignment:
driver = keystone.assignment.backends.sql.Assignment

Next we setup the user to which AD will authenticate since we’re using ldaps:

url = ldaps://
user = uid=service_acct,cn=users,cn=accounts,dc=example,dc=com
password = UbuntuRulez

Then we tell it about the user and group schema:

user_tree_dn = cn=users,cn=accounts,dc=example,dc=com
user_filter = (memberOf=cn=openstack,cn=groups,cn=accounts,dc=example,dc=com)
user_objectclass = inetUser
user_id_attribute = uid
# this is what is searched on
user_name_attribute = uid
user_mail_attribute = mail
user_pass_attribute =
# XXX FIXME -mfisch: wont work on freeIPA like this
#user_enabled_attribute = (nsAccountLock=False)
user_allow_create = False
user_allow_update = False
user_allow_delete = False
group_tree_dn = cn=groups,cn=accounts,dc=example,dc=com
group_filter =
group_objectclass = groupOfNames
group_id_attribute = cn
group_name_attribute = cn
group_member_attribute = member
group_desc_attribute = description
# group_attribute_ignore =
group_allow_create = False
group_allow_update = False
group_allow_delete = False

Lastly we point at the cert info:
use_tls = False
tls_cacertfile = /etc/ssl/certs/ca-certificates.crt

The First Problem

The first stumbling block I hit is that I needed to use ldaps. This is a pretty basic fix, just grab the cert and put it on your box where keystone is running. Unfortunately I found out that unless I had the cert path also defined in /etc/ldap/ldap.conf, the query didn’t work. The fix is to make a pretty much empty /etc/ldap/ldap.conf and add this one line that points to the cert you pulled down:

TLS_CACERT /etc/ssl/certs/ca-certificates.crt


My next issue is that the default user_name_attribute is not right for FreeIPA. I checked this using Apache Directory Studio which made it easy for me to browse the tree and test queries. I needed to use uid and not cn. The issue here is that if you have this wrong, then the initial authentication of your LDAP user for the ldaps query fails and the resulting output provides no clue as to the issue, even with debug enabled. Once I solved this, I had a permission issue.


The problem once you switch the authentication mechanism to LDAP, you’ve “lost” all the old roles and users that puppet setup, like the admin user who actually has permissions to do stuff like list users. The fix as any keystone expert will know is to bypass the main API and use the service token directly. In the keystone.conf a service token is defined as admin_token. To use it, unset everything like OS_USERNAME and set:

export SERVICE_ENDPOINT=http://localhost:35357/v2.0

Then you need to give your user, in my case, “id=mfischer” permission to do stuff. I ended up giving mfischer the admin and _member_ role in both of my tenants since he/me is the new admin. Finally, I switched my settings back to what’s in my rc file, mfischer as the user, my LDAP password, and the normal keystone endpoint, and… finally my user-list query worked and I got results from LDAP.

root@landscape-03:/var/log/keystone# keystone user-list
| id | name | enabled | email |
| mfischer | mfischer | | |

User Enabled?

keystone wants to know if a user is enabled and shows this as a column in the output of user-list. All my results were blank, and it’s not clear that FreeIPA has a straightforward “enabled” column in our setup. At a glance this stuff seemed optimized for AD’s user-enabled mask stuff. (If you know a fix, let me know please).

Trying Other Services

I decided to try nova list next and guess what, it failed. It’s obvious to me now, but the puppet AIO has users created for almost all the services, like nova, cinder, glance, etc. I needed users for these. This is easy to do in FreeIPA, so I made a batch of users and restarted the node. I bet you can guess, but stuff still failed because these users didn’t have the roles that they needed. At this point, I switched the identity backend back to SQL, restarted keystone and made a map of roles to users and tenants. In my case it was pretty basic, everything needed admin in the services tenant and some needed more, here’s how to do it simply:

for I in 'glance' 'nova' 'cinder' 'neutron' 'heat' 'heat-cfn' 'swift';
do; keystone user-role-add --user-id=$I --tenant-id=de4442c6e54a43459eaab97e26dc21f8 --role id=a5a8ea228b1942e28289ba63fba9b3c0; done

I did this and then bounced the node again (which is frankly simpler than restarting everything but unlikely to work in the real world). And again nothing worked! I’d forgotten one more step, each service has a password defined in it’s config file as “admin_password”. I signed back into FreeIPA and set the passwords and then bounced the node a final time.

This time, I could sign into Horizon and everything worked great! Finally, I was using LDAP only for identity and didn’t need to switch back.

Obviously this is much better to be done before you start your puppet install but in my case it was a great learning experience about roles/tenants/users and the tools that keystone provides.

Debugging Tools/Hints

Here are some more debugging tools and hints that I came across and maybe will help you if you’re dealing with LDAP and keystone:

  1. Enable debug and verbose in keystone, of course.
  2. I’ve mentioned Apache Directory Studio so you can test LDAP queries. This command also helps show what fields are available and is simpler than using ADS: “ipa user-show –all –raw”.
  3. The LDAP code in keystone has double secret logging which you can enable in /usr/lib/python2.7/dist-packages/keystone/common/ldap/ You can look for “ldap.set_option(ldap.OPT_DEBUG_LEVEL, 4095)” and uncomment it. These logs only show on stdout, so you’ll need to stop the service and run it by hand to see this output.
  4. I also traced some code in pdb, in addition to the file listed above you should also look at /usr/lib/python2.7/dist-packages/keystone/identity/backends/

Good luck everyone!

Tagged , ,

Updating Keystone Endpoints

This morning we ran into an issue when trying to get our jenkins server to talk to an Openstack cloud we have setup. The issue is that the systems are in different data centers with different firewalls and other assorted networking challenges. The network guys solved this for my by giving the control node in the openstack cloud a NAT address that I could ping. However, after doing all this, I still had issues with keystone and assorted commands (like nova) hanging.

The first step to debugging this was to figure out why it was hanging. To solve this, I used the debug option on nova. (Technically to be honest, the first thing I did was check to see if iptables was blocking ports I needed, you may also need to do this) However, with iptables looking good, I used –debug in keystone endpoint-list.

The first thing you can see in the output is that its successfully retrieving the endpoint list from keystone. The second thing, which was causing the hang, is that it was trying to ping an internal IP and not the natted one. Here’s some of that output with a lot of cruft removed and fake IPs:

[mfischer@puppet-ci-01 ~]$ keystone --debug --os-auth-url --os-username mfischer --os-password ubuntu4life --os-tenant-name test endpoint-list
REQ: curl -i -X POST -H "Content-Type: application/json" -H "User-Agent: python-keystoneclient"

REQ BODY: {"auth": {"tenantName": "test", "passwordCredentials": {"username": "mfischer", "password": "ubuntu4life"}}}

connect: (, 5000) ************

bunch of cruft here that basically means it connected

connect: (, 35357) ************

hang here

The last line was troubling because that’s an internal IP that I cannot ping from this box. The info on that line comes from keystone, in the endpoint list. This is what the list of endpoints looks like, truncated and formatted for a normal screen.

| id | region | publicurl | internalurl | adminurl |service_id|
| 28c...| RegionOne ||| | 9f13... |

So the fix here is that I needed to change these internal URLs to a more friendly and portable hostname. This is doable in two ways that I know of:

      Delete and re-create the endpoints
      Hack the mysql db

Since Option 2 sounded more exciting, I got to work. A great overview on how to do just that is here. After reading this, I realized that I’d have to hand-update every line for the public and admin URLs. The issue I have with this process if that changing all the URLs is error prone and tedious, so instead I wrote a tool.

The tool is called update-endpoints, and it’s hosted here. If you try it, please be careful, back up or dump your DB. I’ve only done limited testing on it and it could break your system. The basic usage is that it connects to your DB and updates the hostname/IP portion of the URL for a class of endpoints (admin, public, or internal). For example, to point all public endpoints to,

./ --username root --password debb4rpm --host localhost --endpoint --type public

This change should not change the port or other parts of the URL, so if you want to change those this tool won’t work for you. It seems by default on my installs that mysql can only be talked to from localhost, so I’ve been running this on my control nodes.

After I ran the tool against the public and admin URLs, I rechecked my endpoints, and now they pointed to, which was conveniently NATd for me.

| id | region | publicurl | internalurl | adminurl |service_id|
| 28c...| RegionOne ||| | 9f13... |

Better yet, all the nova and keystone commands I wanted to run worked and Jenkins was able to talk to the controller!

If you’d like to improve the tool, please do a pull request.