Category Archives: SysAdmin

Landscape Tags to Puppet Facter Facts

I’ve been playing around this week with using Landscape to interact with puppet. I really like Landscape as an administration tool, but I also really like using puppet to manage packages and configuration. So the question is how to get puppet to do something based on Landscape. Landscape tags seemed like an obvious choice for this, tag a system as “compute-node” and it becomes a nova compute node, but how do you make this happen?

Standing on Shoulders

After running through a couple of ideas, I was inspired by a great article from Mike Milner. Mike’s blog post uses Landscape tags to do node classification and I wanted to use tags to set facter facts so I needed a few changes.

Make Some Tags

First I went to Landscape and added some basic tags to my box:

Just a couple tags

Just a couple tags

Get Some Tags

Next, I sat down to write a script that would run on each system and get the tags for the system it was running on. I did this first before I looked into the glue with Facter. This turned out to be pretty since since the Landscape API, is easy to use, well documented, and comes with a CLI implementation that makes testing easy.

ubuntu@mfisch:~$ get_tags.py
nova-compute
vm

Facter Time

Once that worked, it was time to look at Facter. A colleague told me about executable facter facts. The tl;dr for this is, drop a script into /etc/facter/facts.d, make it executable and facter will take the output from it and turn it into facter facts. This was super cool. I had planned on some complex hooks being required, but all I had to do was print the tags to stdout. However, Facter wants a key=value pair and Landscape tags are more like just values, so I decided on a method to generate keys by prepending landcape_tagN, where N is an iterated number. With that change in, I ran facter:

ubuntu@mfisch:~$ facter | grep landscape
landscape_tag0 => nova-compute
landscape_tag1 => vm

Values or Keys?

The puppet master will not know what to do with “landscape_tag0” most likely, so we’ll need a convention to make this more useful. One idea that my colleague had was to actually set a tag with an = sign, like this, “type=compute”. Alas Landscape won’t let us do this, so instead we’ll probably just set a convention that the last _ is the key/value boundary. That would map like this:

  • datacenter_USEast -> datacenter=USEast
  • node_type_compute -> node_type=compute
  • foo_bar -> foo=bar
  • Note the current version of my script that’s in github doesn’t have this convention yet, you’ll probably want to choose your own anyway.

    Issues

    My script is pretty simple, it retrieves a list of computers, filtering on the hostname, as long as it only finds one match, it proceeds. This matching is the weakpoint of the current iteration of this script. It’s possible to have more than one system registered with the same hostname, so I’m thinking of adding a better filter here when I get time. If you’ll note the original script that I based mine on solved this by making you pass in the hostname or id as an argument. My current plan is to look at the system title in /etc/landscape/client.conf and do a 2nd level filter on that, but even that I don’t think will be guaranteed to be unique.

    What is This Good For?

    You probably use an automated tool to install and boot a node and it registers with Landscape (via puppet of course). Now what. We have a box running Ubuntu but not doing much else. Let’s say I want a new compute node in an openstack cluster, so all I’d have to do is tag that box with a pre-determined tag, say “compute-node”, let puppet agent run and wait. The puppet master will see the facter facts that let us know it should be a compute node and act accordingly.

    Code

    Here’s the code and patches always welcome.

    OpenStack: How to Grant or Deny Permissions to Features

    I’m now working on OpenStack (still on Ubuntu too in my free time) and part of that switch is lots of learning. Today I tried to answer a basic question, “How do I prevent a user from being able to create a router?” After some mostly fruitless searching and asking I stumbled upon a policy.json file in the neutron config, this looked promising. So from that start to a functional solution, follow along below.

    First, Ask the Right Question

    As I found out later, the right way to ask this is “how do I deny everyone the right to create a router and allow some people”. This is using the role based security model that OpenStack uses. Previously I’d only used the standard admin and _member_ roles. Now I had a use for a new role.

    Create and Assign the Role

    Now I needed to create a role. I did this in Horizon, by clicking on Admin->Roles. I called my new role “can_create_router”, which is probably too specific for the real world but works fine here. After creating the role, I needed to grant the role to a user. For my example, I have two users, mfisch, who cannot create routers and router_man, who can. Since I could not find how to grant a role in Horizon (Havana), I used the CLI.

    Find tenant ID

    [root@co-control-01 ~(keystone_admin)]# keystone tenant-list
    +----------------------------------+------------+---------+
    | id | name | enabled |
    +----------------------------------+------------+---------+
    | 0a7685cce7c341fd94a83b5dc5f4b18f | admin | True |
    | 0b85137ded2b45418ebfc3278675679e | proj | True |
    | 03fae7b406814ea48a4c10255dd855cf | services | True |
    +----------------------------------+------------+---------+

    Find the user-id for router_man

    [root@co-control-01 ~(keystone_admin)]# keystone user-list --tenant-id 0b85137ded2b45418ebfc3278675679e
    +----------------------------------+------------+---------+----------------------------+
    | id | name | enabled | email |
    +----------------------------------+------------+---------+----------------------------+
    | 01faa8477702463fa12ea5d6b6950416 | mfisch | True | |
    | 0586608123e343f2a0be8237029bdc2d | router_man | True | |
    +----------------------------------+------------+---------+----------------------------+

    Find the ID for the “can_create_routers” role

    [root@co-control-01 ~(keystone_admin)]# keystone role-list
    +----------------------------------+--------------------+
    | id | name |
    +----------------------------------+--------------------+
    | 0fe2ff9ee4384b1894a90878d3e92bab | _member_ |
    | 0c580f80022a4705b49b920772936178 | admin |
    | 03e83b65036a4e0cbd7cff5bff858c76 | can_create_routers |
    +----------------------------------+--------------------+

    Finally, grant the “can_create_routers” role to router_man

    keystone user-role-add --user-id 0586608123e343f2a0be8237029bdc2d --tenant-id 0b85137ded2b45418ebfc3278675679e --role-id 03e83b65036a4e0cbd7cff5bff858c76

    And validate the new role

    [root@co-control-01 ~(keystone_admin)]# keystone user-role-list --user-id 8586608123e343f2a0be8237029bdc2d --tenant-id 5b85137ded2b45418ebfc3278675679e
    +----------------------------------+--------------------+----------------------------------+----------------------------------+
    | id | name | user_id | tenant_id |
    +----------------------------------+--------------------+----------------------------------+----------------------------------+
    | 006eaf0730e44756bc679038477d3bbd | Member | 0586608123e343f2a0be8237029bdc2d | 0b85137ded2b45418ebfc3278675679e |
    | 03e83b65036a4e0cbd7cff5bff858c76 | can_create_routers | 0586608123e343f2a0be8237029bdc2d | 0b85137ded2b45418ebfc3278675679e |
    +----------------------------------+--------------------+----------------------------------+----------------------------------+

    Configure Neutron’s Policy File

    Now we need to configure neutron to allow this new role and to block everyone without it. This is not so easy since there’s no CLI yet for this. Like most really cool stuff in OpenStack, it’s probably 6 months away. For now, do it manually.

    We start by making a backup of /etc/neutron/policy.json on the control node, because we don’t want to break stuff. After that, open the file and look for the line that has “create_router”: on it. This is the feature we’d like router_man to have. How this file works is explained here in more detail, but what we need to know for now is that we only want admins and anyone with the “can_create_routers” role to be able to do it. I ended up doing it like this:

    "create_router": "rule:context_is_admin or role:can_create_routers",

    “rule:context_is_admin” basically boils down to “role:admin” so that will also work. Save the file and exit.
    Here’s my diff of the file if you’d rather see it that way:
    105c105
    < "create_router": "rule:regular_user", --- > "create_router": "rule:context_is_admin or role:can_create_routers",

    Restart neutron

    I can never remember all the neutron services, so I usually run service –list-all | grep neutron and restart everything that’s running. This is my set from today:

    service neutron-server restart
    service neutron-dhcp-agent restart
    service neutron-metadata-agent restart
    service neutron-openvswitch-agent restart

    If you know a way to restart all of neutron/nova/etc with a pseudo-service, please let me know.

    Try It!

    Log into Horizon as mfisch and then try to create a router.

    No router for you!

    No router for you!

    Now let’s sign in as router_man and see if we have the power.

    Great Success

    Great Success

    Conclusion

    I’m just scratching the surface of what you can do with these roles. The policy.json files are full of things that you can allow or deny. In the future, Horizon is supposed to take these into account when laying out the UI, because ideally mfisch in this scenario shouldn’t even see the “Create Router” button. Until then, the error message will suffice. Also thanks to Dave Lyle at HP for pointing me in the right direction this morning when I was fumbling around.

    Tagged

    dconf Settings: defaults and locks

    Last year I worked on a project where I was playing around with system-wide default settings and locks and I thought I’d share a post based on some of my notes. Most all of what I will mention here is covered in depth by the dconf SysAdmin guide, so if you plan on using this, please read that guide as well. UPDATE: Gnome has moved all the dconf stuff into the Gnome SysAdmin guide, it’s a bit more scattered now, but there.

    For most everyone, you have just one dconf database per user. It is a binary blob and it’s stored in ~/.config/dconf/user. Anytime you change a setting, this file gets updated. For system administrators who may want to set a company-wide default value, a new dconf database must be created.

    Create a Profile

    The first step in setting up other databases is to create a dconf profile file. By default you don’t need one since the system uses the default database, user.db, but to setup other databases you will. So create a file called /etc/dconf/profile/user and add the list of databases that you want. Note that this list is a hierarchy and that the user database should always be on top.

    For this example, I will create a company database and a division database. The hierarchy implies that we will have company-wide settings, perhaps a wallpaper, settings on top that are specific to the division, perhaps the IP of a proxy server that’s geographically specific, and each user will have customized settings on top of that.

    To create a profile, we’ll do the following:

    mkdir -p /etc/dconf/profile

    and edit /etc/dconf/profile/user, then add:

    user-db:user
    system-db:division
    system-db:company

    Keyfiles

    (Note: I am doing this on a relatively clean precise install using a user that has not changed their wallpaper setting, that is important later)

    Once you have created the profile hierarchy, you need to create keyfiles that set the values for each database. For this example, we will just set specific wallpaper files for each hierarchy. This is done with key files:

    mkdir -p /etc/dconf/db/division.d/

    and edit /etc/dconf/db/division.d/division.key, add the following:

    [org/gnome/desktop/background]
    picture-uri='file:///usr/share/backgrounds/Flocking_by_noombox.jpg'

    Next we’ll create the company key file:

    sudo mkdir -p /etc/dconf/db/company.d/

    and edit /etc/dconf/db/company.d/company.key, add the following:

    [org/gnome/desktop/background]
    picture-uri='file:///usr/share/backgrounds/Murales_by_Jan_Bencini.jpg'

    Finally, you need to run sudo dconf update so that dconf sees these changes.

    After running dconf update, you will see two changes. The first and most obvious change is that the background is now a bunch of Flocking birds, not the Precise default. The second change is that you will see two new binary dconf database files in /etc/dconf/db, one called company and one called division. If you don’t see these changes then you did something wrong, go back and check the steps.

    flocking

    Since I have no default set the division’s default takes precedence

    The current user and any new users will inherit the Division default wallpaper, Flocking. However, the user still may change the wallpaper to anything they want, and if they change it, that change will be set in the user database, which takes precedence. So this method gives us a soft-default, a default until otherwise modified. If you are trying this test on a user who has already modified the wallpaper, you will notice that it didn’t change due to this precedence.

    If we want to force all users, new and existing, to get a specific wallpaper, we need to use a lock.

    Locks

    For this example, let’s assume that the IS department for our division really really likes the Flocking picture and doesn’t want anyone to change it. In order to force this, we need to set a lock. A lock is simple to make, it just specifies the name of the key that is locked. A locked key takes precedence over all other set keys.

    Before doing this, I will use the wallpaper picker and select a new wallpaper, this will take precedence, until the lock is created. I picked Bloom for my test.

    I like flowers more than birds.

    I like flowers more than birds.

    Now it’s time to make the lock, because the IS department really doesn’t like flowers, so we create the lock as follows.

    sudo mkdir -p /etc/dconf/db/division.d/locks/

    and then edit /etc/dconf/db/division.d/locks/division.lock (note file name doesn’t really matter) and add the following line:

    /org/gnome/desktop/background/picture-uri

    After saving the file, run sudo dconf update. Once doing so, I’m again looking at birds, even though I modified it in my user database to point to Bloom.

    Lock file forces me to use the Division settings

    Lock file forces me to use the Division settings

    One interesting thing to note, any changes the user is making are still being set in their dconf user db, but the lock is overriding what is being seen from outside dconf. So if I change the wallpaper to London Eye in the wallpaper picker and then remove the lock by simply doing sudo rm division.lock && sudo dconf update, I immediately get the London Eye. So it’s important to keep this in mind, the user db is being written into, but the lock is in effect masking the user db value when the setting is read back.

    London Eye wallpaper is shown after I remove the lock

    London Eye wallpaper is shown after I remove the lock

    Lock Hierarchy

    Lock hierarchy is interesting, in that the lowermost lock takes precedence. What this means is that if we lock both the company and division wallpapers, we will see the company one. In the example below I set locks on the wallpaper key for both databases, and I end up seeing Murales, the company default.

    Company setting takes precedence

    Company setting takes precedence with both locked

     

    Locks Without Keys

    It is also possible to set a lock on a hierarchy without a corresponding default key. In this instance the system default is used and the user is unable to change the setting. For this example, I set a company lock but removed the company key. The resulting wallpaper is the system default.

    System default wallpaper for Precise is seen

    System default wallpaper for Precise is seen

    What Value is Seen – A Quiz

    If you’d like to test your knowledge of what key will take precedence when read from dconf, follow the quiz below, answers are at the bottom. For each scenario, see if you can figure out what wallpaper the user will see, assume the same database hierarchy as used in the example.

    1. User Wallpaper: unset, Division Wallpaper: Flock, Company Wallpaper: Murales
    2. User Wallpaper: London Eye, Division Wallpaper: Flock, Company Wallpaper: Murales
    3. User Wallpaper: London Eye, Division Wallpaper: Flock, Company Wallpaper: Murales, Lock file for Company Wallpaper setting
    4. User Wallpaper: London Eye, Division Wallpaper: Flock, Company Wallpaper: Murales, Lock file for Division and Company Wallpaper setting
    5. User Wallpaper: London Eye, Division Wallpaper: Flock, Company Wallpaper: unset, Lock file for Division and Company Wallpaper setting

    Answers: Flock, London Eye, Murales, Murales, Default for Precise

    Testing

    Some notes about testing this if you are trying it:

      • Creating new users and logging in as them is a good way to see what settings are shown, the wallpaper is a great visual test as it’s easy to verify.
      • Do not do this on your development box. I screwed up my settings right before I was going to give a demo. I’d recommend a VM. If you do screw something up, check .xsession-errors, that’s where my problem was apparent.

    Summary

    If you’re a system administrator or you really like pictures of birds, dconf keyfiles and locks are the correct mechanism to make settings that are defaults, soft or hard. Hopefully this has been illustrative on how they work. I’d recommend playing with them in a VM and once you understand the hierarchies and locking, they should be pretty easy to use.

    Tagged , ,

    Sending Custom Data with DHCP Option Tags … and Then Doing Something with the Data

    Over the past week I’ve been playing with DHCP option tags, and using them to send data to DHCP client systems, and then do something with the data. I didn’t know much about DHCP option tags before doing this, I figure DHCP just gave you an IP address and maybe a nameserver. In reality, DHCP can be customized to send a couple hundred different settings, including where the IRC or SMTP servers are, VOIP settings, and lots more. What makes it really powerful is that you can use the unassigned options to create your own settings, and then do actions on the client when they get set. In this simple example, I’m going to send and set a new “message of the day” via dhcp. The message of the day is a (usually) short text message shown when you login via the terminal and is stored in /etc/motd. If you are trying this on a production system, you should back-up your /etc/motd first.

    Server Setup

    On the server, after installing dhcpd, edit /etc/dhcp/dhcpd.conf, create a valid subnet and then in the global space, add your new option. Since options 151-174 are unassigned, I will use option 151. You could also use a ‘site-local’ option between 224-254.

    Add the following outside any subnet sections:
    option motd code 151 = text;
    option motd "Happy Easter from your DHCP server team";

    Client Setup

    On the client, manually run dhclient -d and make sure you’re getting a valid IP from the server. Once you know that is working, edit /etc/dhcp/dhclient.conf and add two parts, first you need to name the new option and second you need to request it:

    Add this to the top, you may see another option code there, add it near that:
    option motd code 151 = text;

    Then look for the big request block, add your new option into that block:
    request subnet-mask, broadcast-address, time-offset, routers,
    domain-name, domain-name-servers, domain-search, host-name,
    netbios-name-servers, netbios-scope, interface-mtu,
    rfc3442-classless-static-routes, ntp-servers,
    dhcp6.domain-search, dhcp6.fqdn,
    dhcp6.name-servers, dhcp6.sntp-servers,
    motd;

    Now, let’s see if we get it. Edit /etc/dhcp/dhclient-exit-hooks.d/debug and switch RUN=”no” to RUN=”yes”. Then edit the large block of variables and add your new option, motd.

    Re-run dhclient -d and then check /tmp/dhclient-script.debug, do you see your motd option? If so, your setup is correct. It should look liks this:

    new_motd='Happy Easter from your DHCP server team'

    Doing Something When You Get an Option

    dhclient defines exit-hooks, one of which is the debug script you edited above. You can create your own shell script in /etc/dhcp/dhclient-exit-hooks.d/ and follow some of the other examples in there. DHCP options come in as variables, so in this case, the script would use ${new_motd}. From what I can tell everything in here has to be a shell script, a python script I tried did not work. Here is my script to set motd:

    if [ -n "${new_motd+x} ]; then
    echo ${new_motd} > /etc/motd
    fi

    Re-run dhclient -d and you should see that the script has run and your message of the day is now a Happy Easter message.

    What About NetworkManager?

    If you are running NetworkManager on your system, and you probably are, then NetworkManager starts dhclient during boot and subverts the running of dhclient’s exit hook scripts. You can use NetworkManager’s own version of “exit hooks” which live in /etc/NetworkManager/dispatcher.d to respond to DHCP events. Scripts in that directory can be written directly in python. The scripts run from NetworkManager’s dispatcher use slightly different environment variables for DHCP options, they’re in all caps and are pre-pended with DHCP4_ (and presumably DHCP6_ if you’re serving IPv6 addresses).

    if [ -n "${DHCP4_MOTD+x} ]; then
    echo ${DHCP4_MOTD} > /etc/motd
    fi

    See the man NetworkManager(8) for more details on actions that your scripts can respond to.

    Options as Expressions

    There are more advanced things you can do. For example, you set a setting on the server based on an expression, for example:

    if option dhcp-user-class = "fortcollins-office" {
    option smtp-server "smtp.ftc-office.net";
    }

    You then configure the client to send up a dhcp-user-class of “fortcollins-office” for all your PCs in Fort Collins.

    Using my example, you could change the holiday in the message based on the date that the lease was given out.

    See man dhcp-eval(5) for more info on expressions.

    Encapsulated Options

    Another thing you can do is vendor encapsulated options. Vendor encapsulated options take a bunch of vendor specific info and encapsulate it into one option. This means that you can pack a bunch of data into one option. I am not sure if all dhcpd servers support this construct however.

    Testing This
    If you want to try DHCP options without messing up your network, the quickest way is to create two virtual machines, a client and a server, and give them each two NICs. One NIC on each machine should be on a private network, and the server should have a static IP on this private network NIC and serve DHCP on that interface. The other non-private NIC is used by you to copy files to and from your virtual machines, but is otherwise not specifically required. I did this setup using Ubuntu 12.04 precise and qemu and it works great.

    Conclusion
    Customized DHCP options let you transmit data to your DHCP clients and then write scripts to act on that data. Although my example is contrived, you could use DHCP to transmit useful system settings and then configure them via script.

    References

    Tagged , , ,