This week I volunteered to work on playing around with integrating keystone with LDAP (via a FreeIPA box). Since I basically knew nothing about LDAP nor keystone before starting, I learned a few lessons along the way. And here they are:
I started with a FreeIPA server that our team uses, I have full admin rights to it. I was also running an “All In One” openstack Havana instance on a VM that I setup using puppet_openstack_builder.
My basic goal here was to learn more about both LDAP and keystone and in the process get the AIO instance to authenticate against LDAP as a preliminary test to authenticating against AD. In our environment, I wanted LDAP to manage Idenity (users, groups, and group memberships) and Keystone’s SQL backend to manage Assignment (roles, tenants, domains). This will work well when we integrate with AD since we cannot just create accounts on the corporate AD server willy-nilly. This is called Read Only LDAP and is covered in more detail here.
There are several awesome blog entries about using FreeIPA and Keystone from Adam Young and these helped me get started. I’d configure keystone, restart it, then tail the logs while running keystone user-list.
The very basic config is done like this:
First, enable the LDAP identity driver:
driver = keystone.identity.backends.ldap.Identity
#driver = keystone.identity.backends.sql.Identity
Then you need to tell Keystone to use SQL for assignment:
driver = keystone.assignment.backends.sql.Assignment
Next we setup the user to which AD will authenticate since we’re using ldaps:
url = ldaps://example.com:636
user = uid=service_acct,cn=users,cn=accounts,dc=example,dc=com
password = UbuntuRulez
Then we tell it about the user and group schema:
user_tree_dn = cn=users,cn=accounts,dc=example,dc=com
user_filter = (memberOf=cn=openstack,cn=groups,cn=accounts,dc=example,dc=com)
user_objectclass = inetUser
user_id_attribute = uid
# this is what is searched on
user_name_attribute = uid
user_mail_attribute = mail
# XXX FIXME -mfisch: wont work on freeIPA like this
#user_enabled_attribute = (nsAccountLock=False)
user_allow_create = False
user_allow_update = False
user_allow_delete = False
group_tree_dn = cn=groups,cn=accounts,dc=example,dc=com
group_objectclass = groupOfNames
group_id_attribute = cn
group_name_attribute = cn
group_member_attribute = member
group_desc_attribute = description
# group_attribute_ignore =
group_allow_create = False
group_allow_update = False
group_allow_delete = False
Lastly we point at the cert info:
use_tls = False
tls_cacertfile = /etc/ssl/certs/ca-certificates.crt
The First Problem
The first stumbling block I hit is that I needed to use ldaps. This is a pretty basic fix, just grab the cert and put it on your box where keystone is running. Unfortunately I found out that unless I had the cert path also defined in /etc/ldap/ldap.conf, the query didn’t work. The fix is to make a pretty much empty /etc/ldap/ldap.conf and add this one line that points to the cert you pulled down:
My next issue is that the default user_name_attribute is not right for FreeIPA. I checked this using Apache Directory Studio which made it easy for me to browse the tree and test queries. I needed to use uid and not cn. The issue here is that if you have this wrong, then the initial authentication of your LDAP user for the ldaps query fails and the resulting output provides no clue as to the issue, even with debug enabled. Once I solved this, I had a permission issue.
The problem once you switch the authentication mechanism to LDAP, you’ve “lost” all the old roles and users that puppet setup, like the admin user who actually has permissions to do stuff like list users. The fix as any keystone expert will know is to bypass the main API and use the service token directly. In the keystone.conf a service token is defined as admin_token. To use it, unset everything like OS_USERNAME and set:
Then you need to give your user, in my case, “id=mfischer” permission to do stuff. I ended up giving mfischer the admin and _member_ role in both of my tenants since he/me is the new admin. Finally, I switched my settings back to what’s in my rc file, mfischer as the user, my LDAP password, and the normal keystone endpoint, and… finally my user-list query worked and I got results from LDAP.
root@landscape-03:/var/log/keystone# keystone user-list
| id | name | enabled | email |
| mfischer | mfischer | | email@example.com |
keystone wants to know if a user is enabled and shows this as a column in the output of user-list. All my results were blank, and it’s not clear that FreeIPA has a straightforward “enabled” column in our setup. At a glance this stuff seemed optimized for AD’s user-enabled mask stuff. (If you know a fix, let me know please).
Trying Other Services
I decided to try nova list next and guess what, it failed. It’s obvious to me now, but the puppet AIO has users created for almost all the services, like nova, cinder, glance, etc. I needed users for these. This is easy to do in FreeIPA, so I made a batch of users and restarted the node. I bet you can guess, but stuff still failed because these users didn’t have the roles that they needed. At this point, I switched the identity backend back to SQL, restarted keystone and made a map of roles to users and tenants. In my case it was pretty basic, everything needed admin in the services tenant and some needed more, here’s how to do it simply:
for I in 'glance' 'nova' 'cinder' 'neutron' 'heat' 'heat-cfn' 'swift';
do; keystone user-role-add --user-id=$I --tenant-id=de4442c6e54a43459eaab97e26dc21f8 --role id=a5a8ea228b1942e28289ba63fba9b3c0; done
I did this and then bounced the node again (which is frankly simpler than restarting everything but unlikely to work in the real world). And again nothing worked! I’d forgotten one more step, each service has a password defined in it’s config file as “admin_password”. I signed back into FreeIPA and set the passwords and then bounced the node a final time.
This time, I could sign into Horizon and everything worked great! Finally, I was using LDAP only for identity and didn’t need to switch back.
Obviously this is much better to be done before you start your puppet install but in my case it was a great learning experience about roles/tenants/users and the tools that keystone provides.
Here are some more debugging tools and hints that I came across and maybe will help you if you’re dealing with LDAP and keystone:
- Enable debug and verbose in keystone, of course.
- I’ve mentioned Apache Directory Studio so you can test LDAP queries. This command also helps show what fields are available and is simpler than using ADS: “ipa user-show –all –raw”.
- The LDAP code in keystone has double secret logging which you can enable in /usr/lib/python2.7/dist-packages/keystone/common/ldap/core.py. You can look for “ldap.set_option(ldap.OPT_DEBUG_LEVEL, 4095)” and uncomment it. These logs only show on stdout, so you’ll need to stop the service and run it by hand to see this output.
- I also traced some code in pdb, in addition to the file listed above you should also look at /usr/lib/python2.7/dist-packages/keystone/identity/backends/ldap.py
Good luck everyone!