Tag Archives: openstack

Updating Keystone Endpoints

This morning we ran into an issue when trying to get our jenkins server to talk to an Openstack cloud we have setup. The issue is that the systems are in different data centers with different firewalls and other assorted networking challenges. The network guys solved this for my by giving the control node in the openstack cloud a NAT address that I could ping. However, after doing all this, I still had issues with keystone and assorted commands (like nova) hanging.

The first step to debugging this was to figure out why it was hanging. To solve this, I used the debug option on nova. (Technically to be honest, the first thing I did was check to see if iptables was blocking ports I needed, you may also need to do this) However, with iptables looking good, I used –debug in keystone endpoint-list.

The first thing you can see in the output is that its successfully retrieving the endpoint list from keystone. The second thing, which was causing the hang, is that it was trying to ping an internal IP and not the natted one. Here’s some of that output with a lot of cruft removed and fake IPs:

[mfischer@puppet-ci-01 ~]$ keystone --debug --os-auth-url --os-username mfischer --os-password ubuntu4life --os-tenant-name test endpoint-list
REQ: curl -i -X POST -H "Content-Type: application/json" -H "User-Agent: python-keystoneclient"

REQ BODY: {"auth": {"tenantName": "test", "passwordCredentials": {"username": "mfischer", "password": "ubuntu4life"}}}

connect: (, 5000) ************

bunch of cruft here that basically means it connected

connect: (, 35357) ************

hang here

The last line was troubling because that’s an internal IP that I cannot ping from this box. The info on that line comes from keystone, in the endpoint list. This is what the list of endpoints looks like, truncated and formatted for a normal screen.

| id | region | publicurl | internalurl | adminurl |service_id|
| 28c...| RegionOne ||| | 9f13... |

So the fix here is that I needed to change these internal URLs to a more friendly and portable hostname. This is doable in two ways that I know of:

      Delete and re-create the endpoints
      Hack the mysql db

Since Option 2 sounded more exciting, I got to work. A great overview on how to do just that is here. After reading this, I realized that I’d have to hand-update every line for the public and admin URLs. The issue I have with this process if that changing all the URLs is error prone and tedious, so instead I wrote a tool.

The tool is called update-endpoints, and it’s hosted here. If you try it, please be careful, back up or dump your DB. I’ve only done limited testing on it and it could break your system. The basic usage is that it connects to your DB and updates the hostname/IP portion of the URL for a class of endpoints (admin, public, or internal). For example, to point all public endpoints to foo.com,

./update-endpoints.py --username root --password debb4rpm --host localhost --endpoint foo.com --type public

This change should not change the port or other parts of the URL, so if you want to change those this tool won’t work for you. It seems by default on my installs that mysql can only be talked to from localhost, so I’ve been running this on my control nodes.

After I ran the tool against the public and admin URLs, I rechecked my endpoints, and now they pointed to foo.com, which was conveniently NATd for me.

| id | region | publicurl | internalurl | adminurl |service_id|
| 28c...| RegionOne | http://foo.com:8080| http://foo.com:8080| http://foo.com:8080 | 9f13... |

Better yet, all the nova and keystone commands I wanted to run worked and Jenkins was able to talk to the controller!

If you’d like to improve the tool, please do a pull request.


OpenStack: How to Grant or Deny Permissions to Features

I’m now working on OpenStack (still on Ubuntu too in my free time) and part of that switch is lots of learning. Today I tried to answer a basic question, “How do I prevent a user from being able to create a router?” After some mostly fruitless searching and asking I stumbled upon a policy.json file in the neutron config, this looked promising. So from that start to a functional solution, follow along below.

First, Ask the Right Question

As I found out later, the right way to ask this is “how do I deny everyone the right to create a router and allow some people”. This is using the role based security model that OpenStack uses. Previously I’d only used the standard admin and _member_ roles. Now I had a use for a new role.

Create and Assign the Role

Now I needed to create a role. I did this in Horizon, by clicking on Admin->Roles. I called my new role “can_create_router”, which is probably too specific for the real world but works fine here. After creating the role, I needed to grant the role to a user. For my example, I have two users, mfisch, who cannot create routers and router_man, who can. Since I could not find how to grant a role in Horizon (Havana), I used the CLI.

Find tenant ID

[root@co-control-01 ~(keystone_admin)]# keystone tenant-list
| id | name | enabled |
| 0a7685cce7c341fd94a83b5dc5f4b18f | admin | True |
| 0b85137ded2b45418ebfc3278675679e | proj | True |
| 03fae7b406814ea48a4c10255dd855cf | services | True |

Find the user-id for router_man

[root@co-control-01 ~(keystone_admin)]# keystone user-list --tenant-id 0b85137ded2b45418ebfc3278675679e
| id | name | enabled | email |
| 01faa8477702463fa12ea5d6b6950416 | mfisch | True | |
| 0586608123e343f2a0be8237029bdc2d | router_man | True | |

Find the ID for the “can_create_routers” role

[root@co-control-01 ~(keystone_admin)]# keystone role-list
| id | name |
| 0fe2ff9ee4384b1894a90878d3e92bab | _member_ |
| 0c580f80022a4705b49b920772936178 | admin |
| 03e83b65036a4e0cbd7cff5bff858c76 | can_create_routers |

Finally, grant the “can_create_routers” role to router_man

keystone user-role-add --user-id 0586608123e343f2a0be8237029bdc2d --tenant-id 0b85137ded2b45418ebfc3278675679e --role-id 03e83b65036a4e0cbd7cff5bff858c76

And validate the new role

[root@co-control-01 ~(keystone_admin)]# keystone user-role-list --user-id 8586608123e343f2a0be8237029bdc2d --tenant-id 5b85137ded2b45418ebfc3278675679e
| id | name | user_id | tenant_id |
| 006eaf0730e44756bc679038477d3bbd | Member | 0586608123e343f2a0be8237029bdc2d | 0b85137ded2b45418ebfc3278675679e |
| 03e83b65036a4e0cbd7cff5bff858c76 | can_create_routers | 0586608123e343f2a0be8237029bdc2d | 0b85137ded2b45418ebfc3278675679e |

Configure Neutron’s Policy File

Now we need to configure neutron to allow this new role and to block everyone without it. This is not so easy since there’s no CLI yet for this. Like most really cool stuff in OpenStack, it’s probably 6 months away. For now, do it manually.

We start by making a backup of /etc/neutron/policy.json on the control node, because we don’t want to break stuff. After that, open the file and look for the line that has “create_router”: on it. This is the feature we’d like router_man to have. How this file works is explained here in more detail, but what we need to know for now is that we only want admins and anyone with the “can_create_routers” role to be able to do it. I ended up doing it like this:

"create_router": "rule:context_is_admin or role:can_create_routers",

“rule:context_is_admin” basically boils down to “role:admin” so that will also work. Save the file and exit.
Here’s my diff of the file if you’d rather see it that way:
< "create_router": "rule:regular_user", --- > "create_router": "rule:context_is_admin or role:can_create_routers",

Restart neutron

I can never remember all the neutron services, so I usually run service –list-all | grep neutron and restart everything that’s running. This is my set from today:

service neutron-server restart
service neutron-dhcp-agent restart
service neutron-metadata-agent restart
service neutron-openvswitch-agent restart

If you know a way to restart all of neutron/nova/etc with a pseudo-service, please let me know.

Try It!

Log into Horizon as mfisch and then try to create a router.

No router for you!

No router for you!

Now let’s sign in as router_man and see if we have the power.

Great Success

Great Success


I’m just scratching the surface of what you can do with these roles. The policy.json files are full of things that you can allow or deny. In the future, Horizon is supposed to take these into account when laying out the UI, because ideally mfisch in this scenario shouldn’t even see the “Create Router” button. Until then, the error message will suffice. Also thanks to Dave Lyle at HP for pointing me in the right direction this morning when I was fumbling around.