Cobbler login failure for spacewalk proxy

Cobbler login failure for spacewalk proxy

Background

We built out a spacewalk infrasture with one spacewalk_master and several spacewalk_proxy in remote data centers around the world. Once we completed the proxy, spacewalk proxy worked fine but the entire cobbler setup on the proxy did not work.

spacewalk proxy references

These are the document use to build the proxies
https://fedorahosted.org/spacewalk/wiki/HowToInstallProxy
https://www.redhat.com/archives/spacewalk-list/2014-May/msg00088.html

 

Debugging login failure

Once we completed the build out of the proxy, run the comand:

cobbler list

Traceback (most recent call last):
  File "/usr/bin/cobbler", line 35, in <module>
    sys.exit(app.main())
  File "/usr/lib/python2.6/site-packages/cobbler/cli.py", line 511, in main
    rc = cli.run(sys.argv)
  File "/usr/lib/python2.6/site-packages/cobbler/cli.py", line 185, in run
    self.token         = self.remote.login("", self.shared_secret)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1489, in __request
    verbose=self.__verbose
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1253, in request
    return self._parse_response(h.getfile(), sock)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1392, in _parse_response
    return u.close()
  File "/usr/lib64/python2.6/xmlrpclib.py", line 838, in close
    raise Fault(**self._stack[0])
xmlrpclib.Fault: <Fault 1: "<class 'cobbler.cexceptions.CX'>:'login failed'">
 
 Debug :
 python -m pdb /usr/bin/cobbler list

> /usr/bin/cobbler(15)<module>()
-> """
(Pdb) n
> /usr/bin/cobbler(17)<module>()
-> import cobbler.cli as app
(Pdb) n
> /usr/bin/cobbler(18)<module>()
-> import sys
(Pdb) n
> /usr/bin/cobbler(20)<module>()
-> PROFILING = False
(Pdb) n
> /usr/bin/cobbler(22)<module>()
-> if PROFILING:
(Pdb) n
> /usr/bin/cobbler(35)<module>()
-> sys.exit(app.main())
(Pdb) s
--Call--
> /usr/lib/python2.6/site-packages/cobbler/cli.py(505)main()
-> def main():
(Pdb) n
> /usr/lib/python2.6/site-packages/cobbler/cli.py(509)main()
-> cli = BootCLI()
(Pdb) n
> /usr/lib/python2.6/site-packages/cobbler/cli.py(510)main()
-> cli.check_setup()
(Pdb) s
--Call--
> /usr/lib/python2.6/site-packages/cobbler/cli.py(153)check_setup()
-> def check_setup(self):
(Pdb) n
> /usr/lib/python2.6/site-packages/cobbler/cli.py(159)check_setup()
-> s = xmlrpclib.Server(self.url_cobbler_xmlrpc)
(Pdb) n
> /usr/lib/python2.6/site-packages/cobbler/cli.py(160)check_setup()
-> try:
(Pdb) n
> /usr/lib/python2.6/site-packages/cobbler/cli.py(161)check_setup()
-> s.ping()
(Pdb) n
> /usr/lib/python2.6/site-packages/cobbler/cli.py(166)check_setup()
-> s = xmlrpclib.Server(self.url_cobbler_api)
(Pdb) n
> /usr/lib/python2.6/site-packages/cobbler/cli.py(167)check_setup()
-> try:
(Pdb) n
> /usr/lib/python2.6/site-packages/cobbler/cli.py(168)check_setup()
-> s.ping()
(Pdb) n
> /usr/lib/python2.6/site-packages/cobbler/cli.py(173)check_setup()
-> if not os.path.exists("/var/lib/cobbler/web.ss"):
(Pdb) n
> /usr/lib/python2.6/site-packages/cobbler/cli.py(177)check_setup()
-> if not os.access("/var/lib/cobbler/web.ss", os.R_OK):
(Pdb) n
--Return--
> /usr/lib/python2.6/site-packages/cobbler/cli.py(177)check_setup()->None
-> if not os.access("/var/lib/cobbler/web.ss", os.R_OK):
(Pdb) n
> /usr/lib/python2.6/site-packages/cobbler/cli.py(511)main()
-> rc = cli.run(sys.argv)
(Pdb) s
--Call--
> /usr/lib/python2.6/site-packages/cobbler/cli.py(181)run()
-> def run(self, args):
(Pdb) n
> /usr/lib/python2.6/site-packages/cobbler/cli.py(185)run()
-> self.token         = self.remote.login("", self.shared_secret)
(Pdb) p self.shared_secret
'91245c18d9cbc4ba2sdfsdf66c4aac4d9d3fd53eb95b91685f3cd24c985a9e2700b53153d1fb8dfc4931e0cd3e5f57951b3217980f1f4fb9b1e2ccb155e5d3882b163f2b2eca52ea4c60c0403ad215b73fc4666d1db6c3c5a9d5ee5f226db6df600ee35895f4be7d503cf0743efe633016e0aeafc84b816181646dec268176a6a447b4a6363c0b04feda03703066055466c3ee53b3026d427b92344da0108b293b3d033a5c159c56ba1b8672d98a9fc683bb4a2c4a913ad8021490df6ffd3e4a90177f6ed91fe1e7903f87e3291ca0fce6ea4c7beb1677d57a3940150b646f84df7e0fa137283a6fd68ba06cf8d5cf38784207a1fdd29eba499772495e71fdb0625ca56d85db74586f2b4866be699113f66ab2e4b835aadad598997819a64f39d4534c8aa6608d6d8dfa2cbad8d86e2ecdc336d4b7b7566dec4a906a60fd2b8299ec0fc8791c576ee23b7f33916ae86e7c8df7f699b767f5d25b12b5ef7f865260b09fb36dcb39c1120946a4364c5082500918a33a93c5d46913215d4df03f6912bfb13dab0fb35996cee9a699eb1095c89be9e1cd0e237aa7a920bdecd0cb1d55d91db9046b28b755843fa41c30ab3caf5d92a2778642f6ba1700431a13fb53212e2ee8c058a079acad7fe0ddfc8d01e931f5af7c3b528d69d06810e88cf752e949f8e2b8c8dd3b57b3734f767ccc16345c859a5700b1754911b9df08829c9'

Note that the shared secret is read from /var/lib/cobbler/web.ss and that this file changes upon cobblerd restart ( service cobblerd restart)

Furthermore :
/etc/cobbler/modules.conf
[authentication]
  module = authn_spacewalk


/etc/httpd/conf.d/cobbler-proxy.conf
ProxyPass /cobbler_api https://[spacewalk_master]/download//cobbler_api


What this means is that the proxy is authenicating against the spacewalk_master cobbler.

Solution

Each time cobblerd is restarted on the spacewalk_master /var/lib/cobbler/web.ss must be sync'd to the /var/lib/cobbler/web.ss on the spacewalk_proxy.

You can either push from the  spacewalk_master upon restart of cobblerd or pull a cron on the proxy to pull the latest file.

e.g.
crontab -e
# spacewalk_cobbler_sharedsecret_sync on the proxy
* * * * * /usr/bin/rsync -av [spacewalk_master]:/var/lib/cobbler/web.ss /var/lib/cobbler/web.ss

Building lxml on Centos 6

Building lxml on Centos 6

Background

I seem to find lots of issues with building lxml. I find it annoying to search and not find a good answer. Then there are so many ways to install python modules : tar file/make, easy_install, pip, etc, rpms. I found that my favorite flavor to install for now is pip.

Installation

If you performed a previous installation please rm -rf /tmp/pip-build-[username]. This will save you alot of grief.

 
Install epel repo
rpm -Uhv http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm

Install required rpm
yum install gcc python-pip python-devel libxml2-devel libxslt-devel zlib-devel -y

Run pip
pip install lxml

Check
$ pip list
distribute (0.6.10)
iniparse (0.3.1)
lxml (3.4.1)
pycurl (7.19.0)
pygpgme (0.1)
urlgrabber (3.9.1)
yum-metadata-parser (1.1.2)

$ python
Python 2.6.6 (r266:84292, Jan 22 2014, 09:42:36)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import lxml
>>>

Install Skype for Linux ( Fedora 21 )

Skype install for linux

For the life of me, why microsoft would not port skype to x86_64. Perhaps it is because they are going to kill it for future use, boo bah bah ( in a maniacal laughter ). So we have only one choice, install 32bit version. I'm currently experimenting with fedora 21.

 

Download skype 

 

http://www.skype.com/en/download-skype/skype-for-computer/
In my case, fedora 32 bit :
http://www.skype.com/go/getskype-linux-beta-fc10

 

Install rpms packages and run skype

In a terminal 

yum install glibc.i686 libstdc++.i686 libXv.i686 qtwebkit.i686 libXScrnSaver.i686 alsa-plugins-pulseaudio.i686 -y
rpm -Uhv skype-4.3.0.37-fedora.i586.rpm
skype

Background

• Download skype rpm
• rpm -Uhv skype-4.3.0.37-fedora.i586.rpm

    error: Failed dependencies:
    ...
    libpthread.so.0(GLIBC_2.3.2) is needed by skype-4.3.0.37-fc16.i586
    librt.so.1 is needed by skype-4.3.0.37-fc16.i586
    librt.so.1(GLIBC_2.2) is needed by skype-4.3.0.37-fc16.i586
    libstdc++.so.6 is needed by skype-4.3.0.37-fc16.i586
    libstdc++.so.6(CXXABI_1.3) is needed by skype-4.3.0.37-fc16.i586
    libstdc++.so.6(GLIBCXX_3.4) is needed by skype-4.3.0.37-fc16.i586
    libstdc++.so.6(GLIBCXX_3.4.9) is needed by skype-4.3.0.37-fc16.i586
    qt >= 4.6 is needed by skype-4.3.0.37-fc16.i586
    qtwebkit is needed by skype-4.3.0.37-fc16.i586
    ...

• find package the required file comes from:  yum provides */[filename] 
• yum provides */libpthread.so.0

      glibc-2.20-5.fc21.i686 : The GNU libc libraries
      Repo        : fedora
      Matched from:
      Filename    : /lib/i686/nosegneg/libpthread.so.0
      Filename    : /lib/libpthread.so.0

• Look for packages such as i686 or noarch
• yum install glibc.i686 -y
• specify non version will install the latest same as yum install glibc-2.20-5.fc21.i686 -y

• Repeat for all the files
• In the end the 5 rpm packages above requires a total of 147 dependent packages on a clean install of fedora.

 

Gitlab Server Installation on Centos 6.4 x88_64

Gitlab Installation

This blog is to document the steps install Gitlab on a Centos 6.4 x88_64 Server. It will discuss some of the issues we encountered and provide some puppet code to automated the build. We used this wiki as a reference for the installation https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/README.md.

Overview
Gitlab using several subsystems

• Postgres database
• nginx
• unicorn
• ruby / rails
• redis
• sidekiq

We opted to install postgres 9.3 instead of the default provided in the rpm. Postgres will be running as a local database account gitlab.
Gitlab will run as local user Centos account git.

Setup

Install and Configure Postgres

Reference for this postgres installation : http://tecadmin.net/install-postgresql-on-centos-rhel-and-fedora/

Install postgres yum repo

rpm -Uhv http://yum.postgresql.org/9.3/redhat/rhel-6-x86_64/pgdg-redhat93-9.3-1.noarch.rpm

There was an issue with postgres requiring a new version of openss 10 so we installed a 6.5 package instead. Note this openssl is from updates repo which includes the fix for heartbleed vulnerability.

rpm -Uhv http://mirror.centos.org/centos/6/updates/x86_64/Packages/openssl-1.0.1e-16.el6_5.14.x86_64.rpm 
yum -y install postgresql93-server-9.3.4-1PGDG.rhel6.x86_64 postgresql93-9.3.4-1PGDG.rhel6.x86_64 postgresql93-libs-9.3.4-1PGDG.rhel6.x86_64

Edit pg_hba.conf

vi /var/lib/pgsql/9.3/data/pg_hba.conf
#local   all        all                                         peer
local   all         postgres                                    trust
local   all         all                                         trust

Initial database and start postgres daemon

service postgresql-9.3 initdb 

service postgresql-9.3 start

For some reason the gitlab configuration script is missing some instructions so we pre-config the db. Note postgres has no password to log in locally.

sudo -u postgres psql

CREATE DATABASE gitlab_production;

\c gitlab_production
CREATE USER gitlab WITH PASSWORD 'my_git_passwd1';
GRANT ALL PRIVILEGES ON DATABASE gitlab_production to gitlab;

\list
\q
 

Check gitlab database login

psql -U gitlab -W -d gitlab_production 

Install and Configure Gitlab

The download instruction for Centos can be found here : https://about.gitlab.com/downloads/
Select Centos 6.

Download Gitlab

wget https://downloads-packages.s3.amazonaws.com/centos-6.5/gitlab-7.1.1_omnibus-1.el6.x86_64.rpm
yum install openssh-server
yum install postfix

rpm -i gitlab-7.1.1_omnibus-1.el6.x86_64.rpm

Configure gitlab

vi /etc/gitlab/gitlab.rb 

# Change the external_url to the address your users will type in their browser
git_data_dir "/home/git"

external_url 'http://[your_hostname]'

#custom postgres install
postgresql['enable'] = false

# Fill in the values for database.yml
gitlab_rails['db_adapter'] = 'postgresql'
gitlab_rails['db_encoding'] = 'utf8'
gitlab_rails['db_database'] = 'gitlab_production'
gitlab_rails['db_username'] = 'gitlab'
gitlab_rails['db_password'] = 'dsu-8cq-c5r-poz'
gitlab_rails['db_socket'] = '/tmp/.s.PGSQL.5432'
 

Since we where using our own postgres install we need to add addition entries by adding

postgresql['enable'] = false

Run gitlab reconfiguration

gitlab-ctl reconfigure 
gitlab-rake gitlab:setup  RAILS_ENV=production 

Login to gitlab
In a browser, type http://[your_hostname]

username : admin@local.host
passwd   : 5iveL!fe 

Debugging

Couldn't create database for {"adapter"=>"postgresql", "encoding"=>"UTF-8", "database"=>"gitlab_production", "pool"=>10, "username"=>"gitlab", "password"=>"dsu-8cq-c5r-poz", "host"=>"127.0.0.1", "port"=>5432, "socket"=>"/tmp/.s.PGSQL.5432"}
-- enable_extension("plpgsql")
rake aborted!
PG::Error: FATAL:  Ident authentication failed for user "gitlab" 

 We had to run numerous time to figure out what was going on.

• vi /etc/gitlab/gitlab.rb

• gitlab-ctl reconfigure

• gitlab-rake gitlab:setup  RAILS_ENV=production

Instead we run the following before editing the final gitlab.rb

• vi /var/opt/gitlab/gitlab-rails/etc/database.yml

• gitlab:setup  RAILS_ENV=production

Bonus Puppet code

Git Node definition

/etc/puppet/manifest/git.pp

node /[your_git_server]/

{

    class
    {
        "postgres":
            version => "9.3",
    }

    class
    {
        "gitlab":
            version => "7.0.0",
            require => Class["postgres"],
    } 

} 

Postgres module

/etc/puppet/modules/postgres/manifests/init.pp

class postgres ( $version = "9.3" )
{

    case $version {

        "9.3"     : { $package_version = "93" }

        default   : { $package_version = "93" }
    }

    package
    {
        [ "postgresql${package_version}-server", "postgresql${package_version}" , "postgresql${package_version}-libs" ]:
            ensure => latest,
            require => Package["pgdg-redhat93-9.3-1"];
         

        "pgdg-redhat93-9.3-1":

            ensure => latest, 

         #Add this to a local repo : http://yum.postgresql.org/9.3/redhat/rhel-6-x86_64/pgdg-redhat93-9.3-1.noarch.rpm 

         #TODO centos 6.4 requires openssl-1.0.1e-16.el6_5.14.x86_64.rpm for centos 6.5 

    }


    service
    {
         "postgresql-${version}":
             ensure  => "running",
             enable  => "true",
             require => [Package["postgresql${package_version}-server"], Exec["init_postgres"]],

    }

    exec
    {
        "init_postgres":
            command => "service postgresql-${version} initdb",
            path    => "/usr/local/bin/:/bin/:/usr/bin/:/usr/sbin/:/sbin",
            onlyif => "ls /var/lib/pgsql/${version}/data",
            require => Package["postgresql${package_version}-server"],

    }

}

Gitlab Module

/etc/puppet/modules/gitlab/manifests/init.pp

class gitlab( $version = '7.0.0' )
{


    class
    {
        "gitlab::user":
    }

    file
    {
        "/home/git":
            ensure => directory,
            require => Class["gitlab::user"];


        "/etc/gitlab/gitlab.rb":
            ensure => "present",
            owner  => $::git,
            group  => $::git,
            mode    => 0655,
            notify => Service["gitlab-ctl"],
            content => template("gitlab/gitlab.rb.erb"),
            require => Package["gitlab-${version}_omnibus"];

    }

    package
    {
         "git":
             ensure => latest,
             require => Class["gitlab::user"];

         "gitlab-7.0.0_omnibus":
             ensure => latest,
             require => Package["git"];

    }

    service
    {
        "gitlab-ctl":
            ensure  => "running",
            hasrestart => true,
            restart => "/usr/bin/gitlab-ctl reconfigure && chsh  -s /bin/bash ${::git_user}",
            hasstatus => true,
            status => "/usr/bin/gitlab-ctl status",
            start =>  "/usr/bin/gitlab-ctl start",
            stop =>  "/usr/bin/gitlab-ctl stop",
            require => File["/etc/gitlab/gitlab.rb"],
    }
}


class gitlab::user
{

    # base::generic_user is a custom "define" create user. There are many definitions to create users. We made it into a class so we can "require" it

    # git user account information was define on the top level scope e.g. = $::git

    base::generic_user 
    {
        "${git_user}":
            user => "$git_user",
            uid => "${git_user_uid}",
            base_home => "/var/opt",
            home_name => "gitlab",
            password => "${git_user_pass}",
            comment => "${git_user_comment}",
            bash_profile_path => "/opt/gitlab:/opt/gitlab/bin:/opt/gitlab/embedded/bin:/usr/sbin:/sbin:/apps/bin:\$PATH",
            bash_rubylib => "/opt/gitlab/embedded/lib"
    }
}

/etc/puppet/modules/gitlab/templates/gitlab.rb.erb

# Autconfigured by Puppet. Do not edit


# Change the external_url to the address your users will type in their browser
git_data_dir "/home/git"

external_url 'http://<%=fqdn%>'

#need to install postgres
postgresql['enable'] = false

# Fill in the values for database.yml
gitlab_rails['db_adapter'] = 'postgresql'
gitlab_rails['db_encoding'] = 'utf8'
gitlab_rails['db_database'] = 'gitlab_production'
gitlab_rails['db_username'] = 'gitlab'
gitlab_rails['db_password'] = 'your_git_user_dbpasswd'
gitlab_rails['db_socket'] = '/tmp/.s.PGSQL.5432'
 

Centos 6.4 GUI installation : Could not allocate requested partitions: not enough space for LVM requests.

Issue:

I'm building a new vm image of Centos 6.4 using the GUI. Upon creating the lvm disk configuration, I receive the following error message despite having a 5G disk :

 

Solution:  Use "Create Custom Layout"

 

Overview of tasks:

• Create /boot partition
• Create lvm partition
• Create / partition using lvm

 

Detail steps

• Click back button to return to disk installation menu, select "Create Custom Layout".
• Click Next" 

• Click "Create"
• Select "Standard Partition" and click "Create"
• In the Mount Point list, Select "/boot"
• Click "OK"

•  Click "Create"
•  Select "LVM Physical Volume", and click "Create"
• If you want use the rest of the partition for the entire lvm type the "Size" as show below then click OK.

• Click "Create"
• Select "LVM Volume Group" and click "Create"
• If you want to Change the name of the Volume Group Name. We use VolGroup00
• Note : Volume Group Name maps this part of the filesystem name
• /dev/mapper/VolGroup00-root

• Click "Add"
• In the "Mount Point" list, select "/"
• If you want to change the "Logical Volume Name". We use root
• Note : Logical Volume Name maps this part of the filesystem name
• /dev/mapper/VolGroup00-root

• Click "OK"
• Click "OK"

•  Finally, Click "Next"

•  Enjoy

Opennebula context support for Centos 5

Issue

I am a big fan of Opennebula because of some of features such as good support for virtual networking and vmware and documentation. One of the biggest attractions is that is it truly open source. Opennebula has way to pass parameters from the Sunstone interface and pre-defined templates to provision a new virtual machine(vm) via context. We tried to provision some vm's with Centos 6 and it worked great using Opennebula-context rpm package install in the base image. Unfortunately the context package did not work for Centos 5.

Diagnosis

We first installed one-context.rpm on a Centos 5 image. Upon provisioning a new guest, We noticed that the context was not being implement. Such as the hostname not being update to what we typed in the Sunstone interface.

We then looked at Opennebula-context in Github. /etc/init.d/vmcontext is the main service which runs all the context.

Upon closer inspection, the cdrom was not being mounted and that /dev/disk/by-label/CONTEXT symlink was missing. /dev/disk/by-label/CONTEXT was symlink to /dev/sr0 on Centos 6 which was named differently on Centos 5.

Centos 6

/dev/sr0

Centos 5

/dev/hdb

On the Centos 5 guest vm. We manually symlinked to the cdrom:

cd /dev/disk/by-label
ln -sf ../../hdb CONTEXT

Test vmcontext service:

service vmcontext start

We received : "mount: no such partition found" which was tracked down to this line in /etc/init.d/vmcontext

mount -t iso9660 -L CONTEXT -o ro /mnt

If we run the following first then the line above would work. Weird.

mount /dev/disk/by-label/CONTEXT -o ro /mnt
umount /mnt
mount -t iso9660 -L CONTEXT -o ro /mnt

 

Solution

Perform these steps on the Centos 5 base image before you import into Opennebula
vi /etc/init.d/vmcontext

change
mount -t iso9660 -L CONTEXT -o ro /mnt
to
mount /dev/disk/by-label/CONTEXT -o ro /mnt

To make the symlink correct cdrom device we need to update udev

vi /etc/udev/rules.d/50-udev.rules

change
KERNEL=="hd[a-z]", BUS=="ide", SYSFS{removable}=="1", SYSFS{device/media}=="cdrom", SYMLINK+="cdrom cdrom-%k"
to 

KERNEL=="hd[a-z]", BUS=="ide", SYSFS{removable}=="1", SYSFS{device/media}=="cdrom", SYMLINK+="cdrom cdrom-%k disk/by-label/CONTEXT"

comment out 
BUS=="ide", KERNEL=="hd*[0-9]", SYSFS{../removable}=="1", GOTO="persistent_end"
to
#BUS=="ide", KERNEL=="hd*[0-9]", SYSFS{../removable}=="1", GOTO="persistent_end"

Restore Classic Firefox toolbars

Again I have to blog about the stupidity of the firefox version 29.0.1 interface designers. I want to refresh the screen but do not want to press F5. Hunting for that tiny refresh icon on the left of the url bar is quite annoying.

 Let's restore the classic theme

• In firefox url bar, go to https://addons.mozilla.org/en-US/firefox/addon/classicthemerestorer.
• Click "Add to Firefox"

• Restart Firefox
• Click View > Toolbars > ...Customize
• Bottom right Click right of Buttons Toggle to Small

• Drag Refresh Tool button to where you want.
• Close Customize Tab