Tag Archives: keystone

Announcing forth-keystoneclient

Over the past few weeks I’ve been working on a new client implementation for keystone. Python is a bit bloated for nimble CLIs like keystone, and so I went back to a language I used at my first job back in 1999, Forth. Forth is interpreted like python but it’s also very simple and memory efficient. And unlike python, forth can fit on a floppy disk. This allows this client to be installed onto satellites and other memory constrained environments. For embedded systems, forth also makes a great alternative to C. Now, onto the client.

Mastering Forth

Mastering Forth

As you probably know, forth is stack-based, and no real programmer likes someone else managing the stack for them. I’ve taken this paradigm into the client design. So let’s dive in. Below is an example usage so you can see how simple it is:

First we remember the definition of what user-role-list does. Python offers the “help” and “dir” for introspection, forth has ‘ and DUMP which work just as well. First let’s make sure user-role-list is defined:

' user-role-list  ok

Then let’s see what it’s doing:

' user-role-list 64 cells dump
7F0A31F7F300: D5 44 40 00  00 00 00 00 - 00 00 00 00  00 00 00 00  .D@.............
7F0A31F7F310: 25 46 40 00  00 00 00 00 - 20 DE F2 31  0A 7F 00 00  %F@..... ..1....
7F0A31F7F320: 88 46 40 00  00 00 00 00 - 99 99 99 99  99 99 99 99  .F@.............
...
ok

As you can simply tell from the above, we need to pass in the tenant-id and user-id, so let’s make the call.

S" de4442c6e54a43459eaab97e26dc21bc2"  ok
S" 2ebadab9148d421287eee6d264f29736d"  ok
user-role-list 33  
+----------------------------------+--------------------+----------------------------------+----------------------------------+
| id | name | user_id | tenant_id |
+----------------------------------+--------------------+----------------------------------+----------------------------------+
| 006eaf0730e44756bc679038477d3bbd | Member | 2ebadab9148d421287eee6d264f29736d | de4442c6e54a43459eaab97e26dc21bc2 |
| 03e83b65036a4e0cbd7cff5bff858c76 | admin | 2ebadab9148d421287eee6d264f29736d | de4442c6e54a43459eaab97e26dc21bc2 |
+----------------------------------+--------------------+----------------------------------+----------------------------------+
ok

I’ll be handing this out on floppy disks at the OpenStack conference in Atlanta if you’re interested, come find me and I’ll make you a copy.

Prepping for distribution at the conference

Prepping for distribution at the conference

A note on debugging, if you plan on installing this on satellites remember how the speed of light can interact with your token expiration timeout.

Tagged ,

Keystone: User Enabled Emulation Can Lead to Bad Performance

An update on my previous post about User Enabled Emulation, tl;dr, don’t use it. It’s slow. Here’s what I found:

I just spent parts of today debugging why keystone user-list was so slow. It was taking between 8 and 10 seconds to list 20 users. I spent a few hours tweaking the caching settings, but the database was so small that the cache never filled up, and so I realized that this was not the main issue. A colleague asked me if basic ldapsearch was slow, and no it was fine. Then I dug into what Keystone is doing with the enabled emulation code. Unlike a user_filter, Keystone appears to be querying the user list and then re-querying LDAP for each user to check if they’re in the enabled_emulation group. This leads to a lot of extra queries which slows things down. When I disabled this setting, the query performance improved dramatically to between 2-2.5 seconds, about a 4x speed-up. If you are in a real environment with more than 20 users, the gain will be pretty good in terms of real seconds.

Disabling the enabled_emulation leaves us with no user enabled information. In order to get the Enabled field back, I’m going to add a field to the schema to emulate what AD does with the enabled users. Since this portion of Keystone was designed for AD, this blog post may help clear up what exactly it expects here from an AD point of view. Read that page to the end and you get a special treat, how to use Logical OR in LDAP, see if it makes less sense than the bf language does.

Also to reduce my user count, I did enable a user_filter, which since it’s just part of the initial query does NOT appear to slow things down. You could skip the new field and just use the filter if you want, however it’s not clear what impact a “blank” for Enabled has, other than perhaps some confusion. If it has a real impact, PLEASE comment here!

Tagged , , ,

Keystone: User Enabled Emulation (follow-up)

Last week I wrote about Keystone using LDAP for Identity. Thanks to a helpful comment from Yuriy Taraday and a quick email exchange I have solved the issue of the empty “Enabled” column. Here’s how it works.

Emulated user enable checking is useful if your LDAP system doesn’t have a simple “enabled” attribute. It works by using a separate group of users or tenants that you must be a member of in order to be enabled. Yuriy has a simple example for this which shows how we can mark user2 as enabled and user1 as disabled.

*ou=Users
+-*cn=enabled_users
-member=cn=user2,ou=Users
+-*cn=user1
+-*cn=user2

To make this setup work, add these to your keystone.conf file:

user_enabled_emulation = True
user_enabled_emulation_dn = cn=enabled_users,cn=groups,cn=accounts,dc=example,dc=com

The default value for foo_enabled_emulation_dn is cn=enabled_foos,$tree_dn, in other words, user_enabled_emulation_dn has a default of enabled_users (note the s).

Keep in mind that even when you remove a user from the enabled_users group they are still a valid user to LDAP/AD. This means that your service account, which you’re using for ldaps authentication, does not need to be a member of this group.

The user_enabled_emulation_* fields appear to be undocumented to me, so I’ll work on that this week so that the official docs are more helpful.

Tagged ,

Keystone: LDAP for Identity, SQL for Assignment

This week I volunteered to work on playing around with integrating keystone with LDAP (via a FreeIPA box). Since I basically knew nothing about LDAP nor keystone before starting, I learned a few lessons along the way. And here they are:

The Setup

I started with a FreeIPA server that our team uses, I have full admin rights to it. I was also running an “All In One” openstack Havana instance on a VM that I setup using puppet_openstack_builder.

The Goal

My basic goal here was to learn more about both LDAP and keystone and in the process get the AIO instance to authenticate against LDAP as a preliminary test to authenticating against AD. In our environment, I wanted LDAP to manage Idenity (users, groups, and group memberships) and Keystone’s SQL backend to manage Assignment (roles, tenants, domains). This will work well when we integrate with AD since we cannot just create accounts on the corporate AD server willy-nilly. This is called Read Only LDAP and is covered in more detail here.

Diving In

There are several awesome blog entries about using FreeIPA and Keystone from Adam Young and these helped me get started. I’d configure keystone, restart it, then tail the logs while running keystone user-list.

Configuration

The very basic config is done like this:

First, enable the LDAP identity driver:
[identity]
driver = keystone.identity.backends.ldap.Identity
#driver = keystone.identity.backends.sql.Identity

Then you need to tell Keystone to use SQL for assignment:
[assignment]
driver = keystone.assignment.backends.sql.Assignment

Next we setup the user to which AD will authenticate since we’re using ldaps:

[ldap]
url = ldaps://example.com:636
user = uid=service_acct,cn=users,cn=accounts,dc=example,dc=com
password = UbuntuRulez

Then we tell it about the user and group schema:

user_tree_dn = cn=users,cn=accounts,dc=example,dc=com
user_filter = (memberOf=cn=openstack,cn=groups,cn=accounts,dc=example,dc=com)
user_objectclass = inetUser
user_id_attribute = uid
# this is what is searched on
user_name_attribute = uid
user_mail_attribute = mail
user_pass_attribute =
# XXX FIXME -mfisch: wont work on freeIPA like this
#user_enabled_attribute = (nsAccountLock=False)
user_allow_create = False
user_allow_update = False
user_allow_delete = False
...
group_tree_dn = cn=groups,cn=accounts,dc=example,dc=com
group_filter =
group_objectclass = groupOfNames
group_id_attribute = cn
group_name_attribute = cn
group_member_attribute = member
group_desc_attribute = description
# group_attribute_ignore =
group_allow_create = False
group_allow_update = False
group_allow_delete = False

Lastly we point at the cert info:
use_tls = False
tls_cacertfile = /etc/ssl/certs/ca-certificates.crt

The First Problem

The first stumbling block I hit is that I needed to use ldaps. This is a pretty basic fix, just grab the cert and put it on your box where keystone is running. Unfortunately I found out that unless I had the cert path also defined in /etc/ldap/ldap.conf, the query didn’t work. The fix is to make a pretty much empty /etc/ldap/ldap.conf and add this one line that points to the cert you pulled down:

TLS_CACERT /etc/ssl/certs/ca-certificates.crt

user_name_attribute

My next issue is that the default user_name_attribute is not right for FreeIPA. I checked this using Apache Directory Studio which made it easy for me to browse the tree and test queries. I needed to use uid and not cn. The issue here is that if you have this wrong, then the initial authentication of your LDAP user for the ldaps query fails and the resulting output provides no clue as to the issue, even with debug enabled. Once I solved this, I had a permission issue.

Bootstrapping

The problem once you switch the authentication mechanism to LDAP, you’ve “lost” all the old roles and users that puppet setup, like the admin user who actually has permissions to do stuff like list users. The fix as any keystone expert will know is to bypass the main API and use the service token directly. In the keystone.conf a service token is defined as admin_token. To use it, unset everything like OS_USERNAME and set:

export SERVICE_TOKEN=
export SERVICE_ENDPOINT=http://localhost:35357/v2.0

Then you need to give your user, in my case, “id=mfischer” permission to do stuff. I ended up giving mfischer the admin and _member_ role in both of my tenants since he/me is the new admin. Finally, I switched my settings back to what’s in my rc file, mfischer as the user, my LDAP password, and the normal keystone endpoint, and… finally my user-list query worked and I got results from LDAP.

root@landscape-03:/var/log/keystone# keystone user-list
+------------+------------+---------+--------------------------+
| id | name | enabled | email |
+------------+------------+---------+--------------------------+
| mfischer | mfischer | | matt.fischer@example.com |
....

User Enabled?

keystone wants to know if a user is enabled and shows this as a column in the output of user-list. All my results were blank, and it’s not clear that FreeIPA has a straightforward “enabled” column in our setup. At a glance this stuff seemed optimized for AD’s user-enabled mask stuff. (If you know a fix, let me know please).

Trying Other Services

I decided to try nova list next and guess what, it failed. It’s obvious to me now, but the puppet AIO has users created for almost all the services, like nova, cinder, glance, etc. I needed users for these. This is easy to do in FreeIPA, so I made a batch of users and restarted the node. I bet you can guess, but stuff still failed because these users didn’t have the roles that they needed. At this point, I switched the identity backend back to SQL, restarted keystone and made a map of roles to users and tenants. In my case it was pretty basic, everything needed admin in the services tenant and some needed more, here’s how to do it simply:

for I in 'glance' 'nova' 'cinder' 'neutron' 'heat' 'heat-cfn' 'swift';
do; keystone user-role-add --user-id=$I --tenant-id=de4442c6e54a43459eaab97e26dc21f8 --role id=a5a8ea228b1942e28289ba63fba9b3c0; done

I did this and then bounced the node again (which is frankly simpler than restarting everything but unlikely to work in the real world). And again nothing worked! I’d forgotten one more step, each service has a password defined in it’s config file as “admin_password”. I signed back into FreeIPA and set the passwords and then bounced the node a final time.

This time, I could sign into Horizon and everything worked great! Finally, I was using LDAP only for identity and didn’t need to switch back.

Obviously this is much better to be done before you start your puppet install but in my case it was a great learning experience about roles/tenants/users and the tools that keystone provides.

Debugging Tools/Hints

Here are some more debugging tools and hints that I came across and maybe will help you if you’re dealing with LDAP and keystone:

  1. Enable debug and verbose in keystone, of course.
  2. I’ve mentioned Apache Directory Studio so you can test LDAP queries. This command also helps show what fields are available and is simpler than using ADS: “ipa user-show –all –raw”.
  3. The LDAP code in keystone has double secret logging which you can enable in /usr/lib/python2.7/dist-packages/keystone/common/ldap/core.py. You can look for “ldap.set_option(ldap.OPT_DEBUG_LEVEL, 4095)” and uncomment it. These logs only show on stdout, so you’ll need to stop the service and run it by hand to see this output.
  4. I also traced some code in pdb, in addition to the file listed above you should also look at /usr/lib/python2.7/dist-packages/keystone/identity/backends/ldap.py

Good luck everyone!

Tagged , ,