Friday, June 19, 2015

Cobbler login failure for spacewalk proxy

Cobbler login failure for spacewalk proxy


We built out a spacewalk infrasture with one spacewalk_master and several spacewalk_proxy in remote data centers around the world. Once we completed the proxy, spacewalk proxy worked fine but the entire cobbler setup on the proxy did not work.

spacewalk proxy references

These are the document use to build the proxies


Debugging login failure

Once we completed the build out of the proxy, run the comand:

cobbler list

Traceback (most recent call last):
  File "/usr/bin/cobbler", line 35, in <module>
  File "/usr/lib/python2.6/site-packages/cobbler/", line 511, in main
    rc =
  File "/usr/lib/python2.6/site-packages/cobbler/", line 185, in run
    self.token         = self.remote.login("", self.shared_secret)
  File "/usr/lib64/python2.6/", line 1199, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib64/python2.6/", line 1489, in __request
  File "/usr/lib64/python2.6/", line 1253, in request
    return self._parse_response(h.getfile(), sock)
  File "/usr/lib64/python2.6/", line 1392, in _parse_response
    return u.close()
  File "/usr/lib64/python2.6/", line 838, in close
    raise Fault(**self._stack[0])
xmlrpclib.Fault: <Fault 1: "<class 'cobbler.cexceptions.CX'>:'login failed'">
 Debug :
 python -m pdb /usr/bin/cobbler list

> /usr/bin/cobbler(15)<module>()
-> """
(Pdb) n
> /usr/bin/cobbler(17)<module>()
-> import cobbler.cli as app
(Pdb) n
> /usr/bin/cobbler(18)<module>()
-> import sys
(Pdb) n
> /usr/bin/cobbler(20)<module>()
-> PROFILING = False
(Pdb) n
> /usr/bin/cobbler(22)<module>()
(Pdb) n
> /usr/bin/cobbler(35)<module>()
-> sys.exit(app.main())
(Pdb) s
> /usr/lib/python2.6/site-packages/cobbler/
-> def main():
(Pdb) n
> /usr/lib/python2.6/site-packages/cobbler/
-> cli = BootCLI()
(Pdb) n
> /usr/lib/python2.6/site-packages/cobbler/
-> cli.check_setup()
(Pdb) s
> /usr/lib/python2.6/site-packages/cobbler/
-> def check_setup(self):
(Pdb) n
> /usr/lib/python2.6/site-packages/cobbler/
-> s = xmlrpclib.Server(self.url_cobbler_xmlrpc)
(Pdb) n
> /usr/lib/python2.6/site-packages/cobbler/
-> try:
(Pdb) n
> /usr/lib/python2.6/site-packages/cobbler/
(Pdb) n
> /usr/lib/python2.6/site-packages/cobbler/
-> s = xmlrpclib.Server(self.url_cobbler_api)
(Pdb) n
> /usr/lib/python2.6/site-packages/cobbler/
-> try:
(Pdb) n
> /usr/lib/python2.6/site-packages/cobbler/
(Pdb) n
> /usr/lib/python2.6/site-packages/cobbler/
-> if not os.path.exists("/var/lib/cobbler/"):
(Pdb) n
> /usr/lib/python2.6/site-packages/cobbler/
-> if not os.access("/var/lib/cobbler/", os.R_OK):
(Pdb) n
> /usr/lib/python2.6/site-packages/cobbler/>None
-> if not os.access("/var/lib/cobbler/", os.R_OK):
(Pdb) n
> /usr/lib/python2.6/site-packages/cobbler/
-> rc =
(Pdb) s
> /usr/lib/python2.6/site-packages/cobbler/
-> def run(self, args):
(Pdb) n
> /usr/lib/python2.6/site-packages/cobbler/
-> self.token         = self.remote.login("", self.shared_secret)
(Pdb) p self.shared_secret

Note that the shared secret is read from /var/lib/cobbler/ and that this file changes upon cobblerd restart ( service cobblerd restart)

Furthermore :
  module = authn_spacewalk

ProxyPass /cobbler_api https://[spacewalk_master]/download//cobbler_api

What this means is that the proxy is authenicating against the spacewalk_master cobbler.


Each time cobblerd is restarted on the spacewalk_master /var/lib/cobbler/ must be sync'd to the /var/lib/cobbler/ on the spacewalk_proxy.

You can either push from the  spacewalk_master upon restart of cobblerd or pull a cron on the proxy to pull the latest file.

crontab -e
# spacewalk_cobbler_sharedsecret_sync on the proxy
* * * * * /usr/bin/rsync -av [spacewalk_master]:/var/lib/cobbler/ /var/lib/cobbler/

Thursday, January 8, 2015

Building lxml on Centos 6

Building lxml on Centos 6


I seem to find lots of issues with building lxml. I find it annoying to search and not find a good answer. Then there are so many ways to install python modules : tar file/make, easy_install, pip, etc, rpms. I found that my favorite flavor to install for now is pip.


If you performed a previous installation please rm -rf /tmp/pip-build-[username]. This will save you alot of grief.

Install epel repo
rpm -Uhv

Install required rpm

yum install gcc python-pip python-devel libxml2-devel libxslt-devel zlib-devel -y

Run pip

pip install lxml

$ pip list
distribute (0.6.10)
iniparse (0.3.1)
lxml (3.4.1)
pycurl (7.19.0)
pygpgme (0.1)
urlgrabber (3.9.1)
yum-metadata-parser (1.1.2)

$ python
Python 2.6.6 (r266:84292, Jan 22 2014, 09:42:36)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import lxml

Saturday, December 27, 2014

Install Skype for Linux ( Fedora 21 )

Skype install for linux

For the life of me, why microsoft would not port skype to x86_64. Perhaps it is because they are going to kill it for future use, boo bah bah ( in a maniacal laughter ). So we have only one choice, install 32bit version. I'm currently experimenting with fedora 21.


Download skype
In my case, fedora 32 bit :


Install rpms packages and run skype

In a terminal 

yum install glibc.i686 libstdc++.i686 libXv.i686 qtwebkit.i686 libXScrnSaver.i686 alsa-plugins-pulseaudio.i686 -y
rpm -Uhv skype-


  • Download skype rpm
  • rpm -Uhv skype-
    error: Failed dependencies:
    ... is needed by skype- is needed by skype- is needed by skype- is needed by skype- is needed by skype- is needed by skype- is needed by skype-
    qt >= 4.6 is needed by skype-
    qtwebkit is needed by skype-


  • find package the required file comes from:  yum provides */[filename] 
  • yum provides */
      glibc-2.20-5.fc21.i686 : The GNU libc libraries
      Repo        : fedora
      Matched from:
      Filename    : /lib/i686/nosegneg/
      Filename    : /lib/

  • Look for packages such as i686 or noarch
  • yum install glibc.i686 -y
    • specify non version will install the latest same as yum install glibc-2.20-5.fc21.i686 -y
  • Repeat for all the files
  • In the end the 5 rpm packages above requires a total of 147 dependent packages on a clean install of fedora.

Wednesday, July 30, 2014

Gitlab Server Installation on Centos 6.4 x88_64

Gitlab Installation

This blog is to document the steps install Gitlab on a Centos 6.4 x88_64 Server. It will discuss some of the issues we encountered and provide some puppet code to automated the build. We used this wiki as a reference for the installation

Gitlab using several subsystems
  • Postgres database
  • nginx
  • unicorn
  • ruby / rails
  • redis
  • sidekiq
We opted to install postgres 9.3 instead of the default provided in the rpm. Postgres will be running as a local database account gitlab.
Gitlab will run as local user Centos account git.


Install and Configure Postgres

Reference for this postgres installation :

Install postgres yum repo

rpm -Uhv
There was an issue with postgres requiring a new version of openss 10 so we installed a 6.5 package instead. Note this openssl is from updates repo which includes the fix for heartbleed vulnerability.

rpm -Uhv 
yum -y install postgresql93-server-9.3.4-1PGDG.rhel6.x86_64 postgresql93-9.3.4-1PGDG.rhel6.x86_64 postgresql93-libs-9.3.4-1PGDG.rhel6.x86_64
Edit pg_hba.conf
vi /var/lib/pgsql/9.3/data/pg_hba.conf
#local   all        all                                         peer
local   all         postgres                                    trust
local   all         all                                         trust
Initial database and start postgres daemon
service postgresql-9.3 initdb 
service postgresql-9.3 start
For some reason the gitlab configuration script is missing some instructions so we pre-config the db. Note postgres has no password to log in locally.

sudo -u postgres psql
CREATE DATABASE gitlab_production;
\c gitlab_production
CREATE USER gitlab WITH PASSWORD 'my_git_passwd1';
GRANT ALL PRIVILEGES ON DATABASE gitlab_production to gitlab;
Check gitlab database login
psql -U gitlab -W -d gitlab_production 

Install and Configure Gitlab

The download instruction for Centos can be found here :
Select Centos 6.

Download Gitlab
yum install openssh-server
yum install postfix
rpm -i gitlab-7.1.1_omnibus-1.el6.x86_64.rpm
Configure gitlab

vi /etc/gitlab/gitlab.rb 
# Change the external_url to the address your users will type in their browser
git_data_dir "/home/git"

external_url 'http://[your_hostname]'

#custom postgres install
postgresql['enable'] = false

# Fill in the values for database.yml
gitlab_rails['db_adapter'] = 'postgresql'
gitlab_rails['db_encoding'] = 'utf8'
gitlab_rails['db_database'] = 'gitlab_production'
gitlab_rails['db_username'] = 'gitlab'
gitlab_rails['db_password'] = 'dsu-8cq-c5r-poz'
gitlab_rails['db_socket'] = '/tmp/.s.PGSQL.5432'
Since we where using our own postgres install we need to add addition entries by adding
postgresql['enable'] = false

Run gitlab reconfiguration
gitlab-ctl reconfigure 
gitlab-rake gitlab:setup  RAILS_ENV=production 
Login to gitlab
In a browser, type http://[your_hostname]
username :
passwd   : 5iveL!fe 


Couldn't create database for {"adapter"=>"postgresql", "encoding"=>"UTF-8", "database"=>"gitlab_production", "pool"=>10, "username"=>"gitlab", "password"=>"dsu-8cq-c5r-poz", "host"=>"", "port"=>5432, "socket"=>"/tmp/.s.PGSQL.5432"}
-- enable_extension("plpgsql")
rake aborted!
PG::Error: FATAL:  Ident authentication failed for user "gitlab" 

 We had to run numerous time to figure out what was going on.

  • vi /etc/gitlab/gitlab.rb
  • gitlab-ctl reconfigure
  • gitlab-rake gitlab:setup  RAILS_ENV=production

Instead we run the following before editing the final gitlab.rb
  • vi /var/opt/gitlab/gitlab-rails/etc/database.yml
  • gitlab:setup  RAILS_ENV=production

Bonus Puppet code

Git Node definition

node /[your_git_server]/
            version => "9.3",

            version => "7.0.0",
            require => Class["postgres"],

Postgres module

class postgres ( $version = "9.3" )

    case $version {

        "9.3"     : { $package_version = "93" }

        default   : { $package_version = "93" }

        [ "postgresql${package_version}-server", "postgresql${package_version}" , "postgresql${package_version}-libs" ]:
            ensure => latest,
            require => Package["pgdg-redhat93-9.3-1"];
            ensure => latest, 
         #Add this to a local repo : 
         #TODO centos 6.4 requires openssl-1.0.1e-16.el6_5.14.x86_64.rpm for centos 6.5 

             ensure  => "running",
             enable  => "true",
             require => [Package["postgresql${package_version}-server"], Exec["init_postgres"]],


            command => "service postgresql-${version} initdb",
            path    => "/usr/local/bin/:/bin/:/usr/bin/:/usr/sbin/:/sbin",
            onlyif => "ls /var/lib/pgsql/${version}/data",
            require => Package["postgresql${package_version}-server"],



Gitlab Module

class gitlab( $version = '7.0.0' )


            ensure => directory,
            require => Class["gitlab::user"];

            ensure => "present",
            owner  => $::git,
            group  => $::git,
            mode    => 0655,
            notify => Service["gitlab-ctl"],
            content => template("gitlab/gitlab.rb.erb"),
            require => Package["gitlab-${version}_omnibus"];


             ensure => latest,
             require => Class["gitlab::user"];

             ensure => latest,
             require => Package["git"];


            ensure  => "running",
            hasrestart => true,
            restart => "/usr/bin/gitlab-ctl reconfigure && chsh  -s /bin/bash ${::git_user}",
            hasstatus => true,
            status => "/usr/bin/gitlab-ctl status",
            start =>  "/usr/bin/gitlab-ctl start",
            stop =>  "/usr/bin/gitlab-ctl stop",
            require => File["/etc/gitlab/gitlab.rb"],

class gitlab::user
    # base::generic_user is a custom "define" create user. There are many definitions to create users. We made it into a class so we can "require" it
    # git user account information was define on the top level scope e.g. = $::git
            user => "$git_user",
            uid => "${git_user_uid}",
            base_home => "/var/opt",
            home_name => "gitlab",
            password => "${git_user_pass}",
            comment => "${git_user_comment}",
            bash_profile_path => "/opt/gitlab:/opt/gitlab/bin:/opt/gitlab/embedded/bin:/usr/sbin:/sbin:/apps/bin:\$PATH",
            bash_rubylib => "/opt/gitlab/embedded/lib"

# Autconfigured by Puppet. Do not edit

# Change the external_url to the address your users will type in their browser
git_data_dir "/home/git"

external_url 'http://<%=fqdn%>'

#need to install postgres
postgresql['enable'] = false

# Fill in the values for database.yml
gitlab_rails['db_adapter'] = 'postgresql'
gitlab_rails['db_encoding'] = 'utf8'
gitlab_rails['db_database'] = 'gitlab_production'
gitlab_rails['db_username'] = 'gitlab'
gitlab_rails['db_password'] = 'your_git_user_dbpasswd'
gitlab_rails['db_socket'] = '/tmp/.s.PGSQL.5432'

Monday, June 16, 2014

Centos 6.4 GUI installation : Could not allocate requested partitions: not enough space for LVM requests.


I'm building a new vm image of Centos 6.4 using the GUI. Upon creating the lvm disk configuration, I receive the following error message despite having a 5G disk :


Solution:  Use "Create Custom Layout"


Overview of tasks:
  • Create /boot partition
  • Create lvm partition
  • Create / partition using lvm


Detail steps

  • Click back button to return to disk installation menu, select "Create Custom Layout".
  • Click Next" 

  • Click "Create"
  • Select "Standard Partition" and click "Create"
  • In the Mount Point list, Select "/boot"
  • Click "OK"

  •  Click "Create"
  •  Select "LVM Physical Volume", and click "Create"
  • If you want use the rest of the partition for the entire lvm type the "Size" as show below then click OK.

  • Click "Create"
  • Select "LVM Volume Group" and click "Create"
  • If you want to Change the name of the Volume Group Name. We use VolGroup00
    • Note : Volume Group Name maps this part of the filesystem name
      • /dev/mapper/VolGroup00-root
  • Click "Add"
  • In the "Mount Point" list, select "/"
  • If you want to change the "Logical Volume Name". We use root
    • Note : Logical Volume Name maps this part of the filesystem name
      • /dev/mapper/VolGroup00-root
  • Click "OK"
  • Click "OK"
  •  Finally, Click "Next"

  •  Enjoy

Wednesday, May 28, 2014

Opennebula context support for Centos 5


I am a big fan of Opennebula because of some of features such as good support for virtual networking and vmware and documentation. One of the biggest attractions is that is it truly open source. Opennebula has way to pass parameters from the Sunstone interface and pre-defined templates to provision a new virtual machine(vm) via context. We tried to provision some vm's with Centos 6 and it worked great using Opennebula-context rpm package install in the base image. Unfortunately the context package did not work for Centos 5.


We first installed one-context.rpm on a Centos 5 image. Upon provisioning a new guest, We noticed that the context was not being implement. Such as the hostname not being update to what we typed in the Sunstone interface.

We then looked at Opennebula-context in Github. /etc/init.d/vmcontext is the main service which runs all the context.

Upon closer inspection, the cdrom was not being mounted and that /dev/disk/by-label/CONTEXT symlink was missing. /dev/disk/by-label/CONTEXT was symlink to /dev/sr0 on Centos 6 which was named differently on Centos 5.

Centos 6
Centos 5
On the Centos 5 guest vm. We manually symlinked to the cdrom:
cd /dev/disk/by-label
ln -sf ../../hdb CONTEXT
Test vmcontext service:
service vmcontext start

We received : "mount: no such partition found" which was tracked down to this line in /etc/init.d/vmcontext
mount -t iso9660 -L CONTEXT -o ro /mnt
If we run the following first then the line above would work. Weird.
mount /dev/disk/by-label/CONTEXT -o ro /mnt
umount /mnt
mount -t iso9660 -L CONTEXT -o ro /mnt



Perform these steps on the Centos 5 base image before you import into Opennebula
vi /etc/init.d/vmcontext
mount -t iso9660 -L CONTEXT -o ro /mnt
mount /dev/disk/by-label/CONTEXT -o ro /mnt
To make the symlink correct cdrom device we need to update udev

vi /etc/udev/rules.d/50-udev.rules
KERNEL=="hd[a-z]", BUS=="ide", SYSFS{removable}=="1", SYSFS{device/media}=="cdrom", SYMLINK+="cdrom cdrom-%k"
KERNEL=="hd[a-z]", BUS=="ide", SYSFS{removable}=="1", SYSFS{device/media}=="cdrom", SYMLINK+="cdrom cdrom-%k disk/by-label/CONTEXT"
comment out 
BUS=="ide", KERNEL=="hd*[0-9]", SYSFS{../removable}=="1", GOTO="persistent_end"
#BUS=="ide", KERNEL=="hd*[0-9]", SYSFS{../removable}=="1", GOTO="persistent_end"

Friday, May 23, 2014

Restore Classic Firefox toolbars

Again I have to blog about the stupidity of the firefox version 29.0.1 interface designers. I want to refresh the screen but do not want to press F5. Hunting for that tiny refresh icon on the left of the url bar is quite annoying.

 Let's restore the classic theme

  • Restart Firefox
  • Click View > Toolbars > ...Customize
  • Bottom right Click right of Buttons Toggle to Small
  • Drag Refresh Tool button to where you want.
  • Close Customize Tab