Updating Keystone Endpoints

This morning we ran into an issue when trying to get our jenkins server to talk to an Openstack cloud we have setup. The issue is that the systems are in different data centers with different firewalls and other assorted networking challenges. The network guys solved this for my by giving the control node in the openstack cloud a NAT address that I could ping. However, after doing all this, I still had issues with keystone and assorted commands (like nova) hanging.

The first step to debugging this was to figure out why it was hanging. To solve this, I used the debug option on nova. (Technically to be honest, the first thing I did was check to see if iptables was blocking ports I needed, you may also need to do this) However, with iptables looking good, I used –debug in keystone endpoint-list.

The first thing you can see in the output is that its successfully retrieving the endpoint list from keystone. The second thing, which was causing the hang, is that it was trying to ping an internal IP and not the natted one. Here’s some of that output with a lot of cruft removed and fake IPs:

[mfischer@puppet-ci-01 ~]$ keystone --debug --os-auth-url --os-username mfischer --os-password ubuntu4life --os-tenant-name test endpoint-list
REQ: curl -i -X POST -H "Content-Type: application/json" -H "User-Agent: python-keystoneclient"

REQ BODY: {"auth": {"tenantName": "test", "passwordCredentials": {"username": "mfischer", "password": "ubuntu4life"}}}

connect: (, 5000) ************

bunch of cruft here that basically means it connected

connect: (, 35357) ************

hang here

The last line was troubling because that’s an internal IP that I cannot ping from this box. The info on that line comes from keystone, in the endpoint list. This is what the list of endpoints looks like, truncated and formatted for a normal screen.

| id | region | publicurl | internalurl | adminurl |service_id|
| 28c...| RegionOne ||| | 9f13... |

So the fix here is that I needed to change these internal URLs to a more friendly and portable hostname. This is doable in two ways that I know of:

      Delete and re-create the endpoints
      Hack the mysql db

Since Option 2 sounded more exciting, I got to work. A great overview on how to do just that is here. After reading this, I realized that I’d have to hand-update every line for the public and admin URLs. The issue I have with this process if that changing all the URLs is error prone and tedious, so instead I wrote a tool.

The tool is called update-endpoints, and it’s hosted here. If you try it, please be careful, back up or dump your DB. I’ve only done limited testing on it and it could break your system. The basic usage is that it connects to your DB and updates the hostname/IP portion of the URL for a class of endpoints (admin, public, or internal). For example, to point all public endpoints to foo.com,

./update-endpoints.py --username root --password debb4rpm --host localhost --endpoint foo.com --type public

This change should not change the port or other parts of the URL, so if you want to change those this tool won’t work for you. It seems by default on my installs that mysql can only be talked to from localhost, so I’ve been running this on my control nodes.

After I ran the tool against the public and admin URLs, I rechecked my endpoints, and now they pointed to foo.com, which was conveniently NATd for me.

| id | region | publicurl | internalurl | adminurl |service_id|
| 28c...| RegionOne | http://foo.com:8080| http://foo.com:8080| http://foo.com:8080 | 9f13... |

Better yet, all the nova and keystone commands I wanted to run worked and Jenkins was able to talk to the controller!

If you’d like to improve the tool, please do a pull request.


Leave a Reply

Your email address will not be published. Required fields are marked *