Force Bios Setup from PowerCLI

I was trying to troubleshoot a Linux VM yesterday and the VM Console from vCenter would be too slow after a reset, so that I could not modify the kernel boot parameters, because by the time it displayed anything, there was already a kernel panic message and I had an unresponsive VM.

So I ticked the Force BIOS Setup option, to force the VM to go on to the BIOS screen on booting up. I removed quiet from the kernel entry in GRUB but all I got was a message saying that I should disable SElinux and after I went through the whole thing again and managed to boot up again without making any changes to the kernel entry, I thought I had better look for a PowerCLI solution, so here it is:

1. Fire up VMware vSphere PowerCLi.
2. Connect to vCenter (Connect-VIServer vcenterserver)
3. Import-Module -Name .\VMware.Vim.dll
4. $ida = get-vm -name ida
5. $idaVMBO = New-Object VMware.Vim.VirtualMachineBootOptions
6. $idaVMBO.EnterBIOSSetup=$true
7. $idaVMCS = New-Object VMware.Vim.VirtualMachineConfigSpec
8. $idaVMCS.BootOptions = $idaVMBO
9. $idaView = Get-View -VIObject $ida
10. $idaView.ReconfigVM($idaVMCS)
11. restart-vm $ida

To repeat the process after step 11, just do:

1. $idaView.ReconfigVM($idaVMCS)
2. restart-vm $ida -confirm:$false

I think this is a good candidate for a script.

An experiment

I thought I would try a little experiment today. How easy is it to manipulate Google search results?

Google algorithm(s) is not public, but one of the well known criteria that it uses, or at least used to use, was the number of links to a page, so with that in mind, I have created a few blogs in tumblr, wordpress and posterous and simply provided a link to my post on how to set up the virtual center server appliance to use multiple VLANs.

For the record, all the blogs are called dumbrepeater.<platform> (e.g. http://dumbrepeater.posterous.com/) and the posts have been tagged with vCenter and vCSA.

A search on Google for VCSA VLAN already returns my post first, but I want to see whether I can get my post showing up as the first result when searching for VCSA with this simple trick.

I might add a few more blogs on other platforms to see if I can fool Google.

Setting up vCenter Server Appliance (vCSA) - Using multiple VLANs

A few weeks ago I attended a VMware training course, VMware vSphere: Install, Configure, Manage [V5.0], and one of the things we learned about was, the vCenter Server Appliance, which is a virtual machine that runs vCenter. The main advantage of this pre-configured virtual machine is that it runs Suse Enterprise Linux and thus it does not require you to give your hard earned cash to Microsoft, just VMware.

At any rate, the appliance, which can be download here, can be configured through a web interface, however the web interface is very limiting, almost useless for production use of the vCSA. You can only configure a single IPv4 address, you don't believe me:

There is nothing preventing you from adding an extra IPv4 address to the single NIC the appliance comes with or from adding a second NIC to the appliance and giving it a new IP address, but you will not be able to configure it from the web interface, which is a bit daft.

Our environment networks are segmented into various VLANs, so that all the VMware hosts are on the same VLAN, which is separate from the VM Network, I guess this could be called the production network, which is where all the guests are. This is also where the active directory domain controllers are, as they are virtualized, which presents a little bit of a problem when trying to set up the vCSA to authenticate with AD.

In our setup, we run vMotion traffic on VLAN 12, Management Network on VLAN 11 and VM Network or production on VLAN 10. The thing is you cannot run any virtual machine on the Management Network, because it is of VMKernel type, which means that we have to create a new connection type for the vCSA on VLAN 11.

This is what you need to do to add a new connection type:

1. Select your host
2. Select Configuration tab
3. Select Networking
4. Click Add Networking
5. Accept the default of Virtual Machine and Click Next
6. Choose whether you want to use a new vswitch or not and click next.
7. Give the network a meaning label and select the VLAN tag and click next.
8. Click Finish

In my case, I created a connection type called Virtual Appliance with VLAN 11 so that the vCSA could communicate with the ESXi hosts and modified VM Network (PROD) so that it had a VLAN ID of 10 so that it can communicate with the rest of the guests.

I finally created a VMKernel connection on VLAN 12 to enable vMotion traffic. In order to do this, you can use the above instructions but on step 5, change the connection type to VMKernel.

This is the result:

The vCSA can now be imported from the template downloaded from the VMware site. Please make sure that you also download the disks and place them in the same directory as the template file. In order to import the template, simply click File | Deploy OVF Template and follow the wizard.

When the vCSA finished importing, I made a few changes to it:

1. Reduced RAM to 4 GB. This instance is only managing 2 hosts.
2. Set the network connection for NIC 1 to Virtual Appliance, i.e. VLAN 11.
3. Add a second NIC and set its network connection to VM Network (PROD), i.e. VLAN 10.

The Virtual Appliance NIC can now be configured from the console interface but the VM Network (PROD) will need to be configured directly in Suse. Remember that the password for the root account is vmware.

To configure the Virtual Appliance, from the guest console, simply select Configure Network and follow the wizard. (Note that I took the screenshot after I configured the first NIC). I think you get https://0.0.0.0:5480 before you configure it.

In our case we have a DHCP server on the VM Network (PROD), so configuration for the VM Network (PROD) is simple. Create this file /etc/sysconfig/networking/devices/ifcfg-eth1 and add this:

DEVICE=eth1
BOOTPROTO='dhcp'
STARTMODE='auto'
TYPE=Ethernet
USERCONTROL='no'

However, in order for this to work a symbolic link needs to be created in /etc/sysconfig/network/ and  network service should be restarted (only eth1 needs to be brought up but since the box is idle) You could simply create the config file in /etc/sysconfig/network, but I thought I'd keep it consistent:

ln -s /etc/sysconfig/networking/devices/ifcfg-eth1 /etc/sysconfig/network/ifcfg-eth1; service network restart

You should now have a vCSA with two NICs with different VLANs, which means that you can configure AD authentication (In essence, change hostname, join domain, restart.):

1. hostname -v vCSA
2. domainjoin-cli join dev.org domainadmin 
3. init 6

Note that before you can login to vcenter using AD credentials you need to add permissions to AD users in vcenter by logging in with the root account first.

Using NFS for ESX/ESXi

Last week I attended a VMware training course and this week I was trying to set up a nfs datastore but I was making no headway, it always failed to mount from vSphere:

Create NFSTest datastore 172.10.121.213
Error during the configuration of the host : NFS Error: Unable to Mount filesystem: Unable to connect to
NFS server

I then remembered the training course. ESX only supports NFS 3, so I mounted the NFS share from a linux box using nfs 3:

mount <host>:/shareddir /localdir -o nfsvers=3

This worked fine from linux but not from ESX.

Network connectivity existed, i.e. I could ping and telnet on port 2049 to the NFS server, yet ESX stubbornly refused to mount the NFS share.

After doing a little bit of reading, I learnt that I was using the wrong ping command, I should have used vmkping instead, which dutifully failed to reach the NFS server.

It turns out that VMKernel needs to be able to connect to the NFS server and due to network segmentation (VLANs) the ESX hosts could not reach the NFS server from the existing VMKernel connections. So I added another VMKernel connection and I was able to mount the share with this following command:

esxcfg-nas -a nfstest -o nfsserver -s /shareddir

Although this was on a ESX 4 host, this command exists on ESXi 5.0 too.

Rescan NICs in CentOS 6.2

Our virtualization environment runs on ESX 4 and Vcenter 4, which means that it is not possible to customize 64bit Linux distros or at least Red Hat distros, following cloning. This is annoying as CentOS does not seem to be able to pick up the new NICs in a clone, even after the ifcfg files have been edited to amend the new MAC addresses.

It turns out there is a very easy solution to this problem:

rm -f /etc/udev/rules.d/70-persistent-net.rules; init 6

This will delete the file containing NIC information and rescan after a reboot.

Do note that the second time I tried this, it created new NIC names, i.e. eth4,eth5 and eth6 for my multiple nic clone.

If you encounter this issue, you could edit the file /etc/udev/rules.d/70-persistent-net.rules or rename the ifcfg files, either should work.

Hopefully, we will move to ESXi 5 soon.

SSH Single Sign On for CentOS 6.2 or RHEL 6.0 using a Windows 2008 AD domain

In one of my previous posts I discussed how to join a CentOS 6.2 server to a Windows 2008 AD domain. There was one thing that wasn't working and that really, and I mean, REALLY annoyed me and this was: single sign on, i.e. using SSH to login to another server in the domain without being prompted for your password again.

After a lot of head banging, cursing and wondering why oh why had I decided to embark in such a doomed enterprise, I managed to get it working. I assume that you have followed my previous post on how to join a CentOS 6 (RHEL 6 works too) and that you have two linux machines that have joined the domain. A second machine is only needed for testing purposes, you could use putty instead. I needed the second machine for other purposes, so that is the route I chose. I have also tested it with putty and it does work as well.

Here is the list of steps needed:

1. From the Windows domain controller run the following command, which will create spns and upns. Note that you will need to run it as Administrator:

   ktpass -princ host/adtest.my.org@MY.ORG -mapuser MY\adtest$  -pass Passw0rd123 -ptype KRB5_NT_PRINCIPAL -crypto All -out adtest.keytab


2. Copy adtest.keytab to your linux box, I simply mounted the c drive of the DC on the linux box, but this might not be available to you.
3. If your server doesn't have a keytab file (/etc/krb5.keytab), then you can just move adtest.keytab to /etc/krb5.keytab otherwise you will need to merge it, which you can do with the ktutil tool, see this link for instructions.
4. [Optional] Limit encryption to RC4-HMAC, by editing the kerberos configuration file /etc/krb5.conf and adding the following to the [libdefaults] directive:

default_tkt_enctypes=rc4-hmac
default_tgs_enctypes=rc4-hmac
permitted_enctypes =rc4-hmac

5. Restart the OpenSSH daemon:

   service sshd restart

6. Configure the OpensSSH client. This will limit SSO to hosts in the domain:

Host *.my.org
GSSAPIAuthentication yes
GSSAPIDelegateCredentials yes

7. Repeat steps 1 to 6 for the second server if needed.
8. Login to first server with a domain account that has linux attributes set.
9. Ensure that a Kerberos ticket has been issued: 

klist

Ticket cache: FILE:/tmp/krb5cc_10000_TjT7rk
Default principal: linuxuser@MY.ORG

Valid starting     Expires            Service principal
02/28/12 17:41:06  02/29/12 03:39:31  krbtgt/MY.ORG@MY.ORG
        renew until 02/29/12 03:41:06

10. Open secure shell on second server, which will log you without a prompt for credentials
    

    ssh adtest5.my.org

It is very important that name resolution is working correctly as you could get issues if it doesn't work properly, thus an up to date DNS server is quite useful. If you don't have a DNS server make sure that your hosts files are up to date with all the server names involved.

If you hit any problems, the simplest way to trouble shoot is to open a debug ssh daemon, which you can do like this (you can add a couple more ds for extra debug info but I think debug1 is all you need):

/usr/sbin/sshd -p 31415 -d

You'll need to allow traffic on port 31415 or the port you choose, which you can easily do by stopping iptables. Clearly this should only be done in servers that are not internet facing. If the server is internet facing then just open port 31415, e.g:

iptables -I INPUT -p tcp --dport 31415 -j ACCEPT

You can connect to this server with:

ssh servername -p 31415 -v

This should tell you what the problem is, e.g:

debug1: Unspecified GSS failure.  Minor code may provide more information
Key table entry not found

This was actually caused by a name resolution problem.

A few random quotes

In theory, there is no difference between theory and practice; In practice, there is.

Chuck Reid

We cannot renounce the use of force otherwise a peaceful reunification would be impossible 

China's Jhian Xemin on Taiwan

It is hard to believe that a man is telling the truth when you know that you would lie if you were in his place 

H.L. Mencken

It is always the best policy to tell the truth, unless, of course, you are an exceptionally good liar 

 Jerome K. Jerome

Join CentOS 6.2 server to a Windows 2008 Active Directory domain

Following on from my previous post detailing how to join a RHEL6 box to a Windows 2003 AD domain, in this post I discuss how to join to Windows 2008 AD domain (2008 Mode). This time rather than using RHEL 6, I've decided to use Centos 6.2 instead. I have used the standard installation rather than the minimal installation that used in the previous post, so here are the steps needed. 


Before I start though, I'd like to note that I installed the Identity Management for UNIX role in the Windows domain controller, so if you are following these instructions, make sure that you have that role installed in your domain controller. You will also need a binding account that has its Unix attributes set.

Without further ado, here are the instructions:

1. Ensure that name resolution is working. At the very least you should be able to ping your domain controller, in my case pdc1.dev.org. If you can't, have a look at your /etc/resolv.conf file. Sample file:

   search dev.org test.com
   nameserver 10.168.20.203

2. Depending on your installation type, you might have to install several of the packages below (It looks like I went for a base install only):

   yum install pam_krb5 pam_ldap nss-pam-ldapd samba policycoreutils-python -y

3. Run authconfig-tui. Make sure that Kerberos realm is in capitals:
   
   

   
   

   

   
   

   
4. Ensure that Name Service Switch is configured for ldap authentication. In essence, check that /etc/nsswitch.conf has the following values:

passwd:     files ldap

shadow:     files ldap

group:      files ldap

5. Edit the local LDAP name service daemon configuration (/etc/nslcd.conf). A bind account to the Active Directory is needed, so create that account now (I have created binding in the Users OU). The mappings (for Active Directory) need to be modified. Below is a list of changes to /etc/nslcd.conf. In essence uncomment the relevant parts:

binddn cn=binding, cn=Users,dc=dev,dc=org

bindpw mypass 

#The Default search scope

scope sub 

#Customize certain database lookups

base   group  dc=dev,dc=org

base   passwd dc=dev,dc=org

base   shadow dc=dev,dc=org

# Mappings for Active Directory

pagesize 1000
referrals off
filter passwd (&(objectClass=user)(!(objectClass=computer))(uidNumber=*)(unixHomeDirectory=*))
map    passwd uid              sAMAccountName
map    passwd homeDirectory    unixHomeDirectory
map    passwd gecos            displayName
filter shadow (&(objectClass=user)(!(objectClass=computer))(uidNumber=*)(unixHomeDirectory=*))
map    shadow uid              sAMAccountName
map    shadow shadowLastChange pwdLastSet

map    shadow userPassword     unixUserPassword
filter group  (objectClass=group)
map    group  uniqueMember     member

6. Change permissions on /etc/nslcd.conf file so that it is only readable by root:

   chmod 600 /etc/nslcd.conf

7. Restart the local LDAP name service daemon:

   service nslcd restart

8. Ensure that the local LDAP name service daemon (nslcd) is set to start with the server:

   chkconfig nslcd on

9. Edit /etc/samba/smb.conf. Make sure that there is only a security directive active. Comment out all others.

Network Related Options

workgroup =dev

Domain members options

security = ads

realm = DEV.COM

use kerberos keytab = true  #not really sure about this one

password server = pdc1.dev.org

10. Ensure that iptables lets traffic through on port 389:

iptables –I INPUT –p tcp --dport ldap –j ACCEPT; service iptables save

11. Run the following command to join the domain:

net ads join –U domainadmin

12. A DNS record was not created for this server in my DNS server, not sure why, which meant that I had to add the record myself manually. Thus ensure that you do this before moving on if the DNS record is not added automatically, otherwise you might be unable to login.
13. At this point you have successfully joined to the AD domain, you can test this by getting a list of users or group. You should get back the users and/or groups that have linux attributes, at least the binding account.

    getent passwd

    getent group

14. In order to create a user's home directory on first login add this directive to /etc/pam.d/sshd. I only log on using ssh. If you are logging in at the box, rather than remotely, you need to modify /etc/pam.d/logon too, I believe. Note that this will not work if SELinux is on.

    session required pam_mkhomedir.so skel=/etc/skel umask=0022

15. Allow polyinstatiation in SELinux settings:

     setsebool -P allow_polyinstantiation 1

16. Temporarily set SELinux to permissive:

setenforce 0

17. If you login with a domain user (ssh binding@domainadtest, where domainadtest is the server that has just joined the domain), the directory will be created, but you will also have a record of what would've gone wrong on /var/log/audit/audit.conf had SElinux been on, which in my case is this:

type=AVC msg=audit(1329063091.971:160): avc:  denied  { create } for  pid=5510 comm="mkhomedir_helpe" name="binding" scontext=unconfined_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:home_root_t:s0 tclass=dir type=AVC msg=audit(1329063091.973:161): avc:  denied  { create } for  pid=5510 comm="mkhomedir_helpe" name=".bashrc" scontext=unconfined_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:home_root_t:s0 tclass=file type=AVC msg=audit(1329063091.973:161): avc:  denied  { write open } for  pid=5510 comm="mkhomedir_helpe" name=".bashrc" dev=dm-0 ino=263825 scontext=unconfined_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:home_root_t:s0 tclass=file type=AVC msg=audit(1329063091.973:162): avc:  denied  { setattr } for  pid=5510 comm="mkhomedir_helpe" name=".bashrc" dev=dm-0 ino=263825 scontext=unconfined_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:home_root_t:s0 tclass=file type=AVC msg=audit(1329063092.015:163): avc:  denied  { setattr } for  pid=5510 comm="mkhomedir_helpe" name="binding" dev=dm-0 ino=263284 scontext=unconfined_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:home_root_t:s0 tclass=dir

18. Create a SELinux policy module to allow the creation of home directories when the user first logs in:

    less /var/log/audit/audit.log  | grep denied > mkdir.log 

    audit2why < mkdir.log 

    audit2allow -M mkdir -i mkdir.log 

    semodule -i mkdir.pp

19. Renable SELinux:

    setenforce 1

That is it.

See this post to configure openSSH single sign on.

Attach to Debugger shortcut in Visual Studio

We've been doing some bug fixing this week and have ended up creating a new plug-in, so to ease the debugging pain, I have used some post build actions to deploy the plug-in while debugging, see this post for more details. The one thing that I hadn't automated was attaching to the debugger, which while not very time consuming it was starting to grate me a little bit, particularly after I mistyped a property name twice in a row.

So after a bit of googling I found this and this, which I have combined into this post. Note that I've used VS 2008, but this should work in VS 2010, just need to add import EnvDTE100 to the macro, I haven't tried it though.

In Visual Studio go to:

1. Tools |  Macros | Macro Explorer 
2. Right Click MyMacros | New Module
3. Name it AttachDebugger
4. Double Click on Attach Debugger to open the Editor.
5. Paste the code below:

Option Strict Off
Option Explicit Off
Imports System
Imports EnvDTE
Imports EnvDTE80
Imports EnvDTE90
Imports System.Diagnostics
Public Module AttachDebugger
    Sub W3WP()
        Try
            DTE.Debugger.DetachAll()
            For Each proc As EnvDTE.Process In DTE.Debugger.LocalProcesses
              If proc.Name.IndexOf("w3wp.exe") <> -1 Then
                    proc.Attach()
                End If
            Next
        Catch ex As System.Exception
            MsgBox(ex.Message)
        End Try
    End Sub
End Module

6. Save it.
7. To add the Shortcut, now go to Tools | Options | Environment | Keyboard 
8. Type W3WP and enter your shortcut

9. Click Assign and off you go.

If you are deploying asynchronous plugins as well, you could create another Sub that attaches to the asynchronous service  and simply give it another shortcut.

The Windows Security problem or ...

A few weeks ago, I got home, switched my desktop PC, which runs Windows 7 Ultimate, patched to gills, and I was greeted with bucket loads of errors.

It complained that I had some sort of issue writing to disk. One of my hard drives makes a really strange noise sometimes and I thought it had finally given up the ghost. 

A pop-up appeared suggesting that I could fix it now or later, when I clicked later it bounced my desktop. It started up again and the same malarky with the errors occurred, so I decided to fix it this time. This started a program that scanned my desktop and it announced that the c:\ drive was unreadable, which is when the alarm bells should have started ringing, because if it weren't readable then how did the system boot up? 

Anyway, I followed the charade a bit more to see how it develops, after all I wasn't sure what was going on and after a diagnosis telling me that my ram is running at 83 C, which to my eternal shame I believed, at least for a moment; I'm informed that most of my problems will not be fixed by free version, but if I buy the paid version, all will be well. So I follow the link, which launches I.E.

Except that it isn't I.E. but something that looks like I.E., I guess so that people are not freaked out, and this is when the penny finally dropped and I realized that this is one of them fake programs, welcome System Check Virus;a bit slow on the uptake, I know (hangs head in shame).

I have read about these virii before and I know my mother in law actually paid to get rid of it. She got rid of her computer shortly after, so I don't know whether the pop-ups would come back after a while. I suspect that not for a while, until the new super duper version was out and her credit card was not abuse, but I digress.

So I followed this guide to get rid of the little bugger, which can be done with Malwarebytes' Anti Malware. Ironically, I had installed it but because I barely use my desktop at home anymore, I never actually ran a scan with it.

Let that be a lesson for me in the future.

It's simply mind boggling, though, that this kind of virus can run in Microsoft's latest and greatest OS and yet UAC prompts you to approve even something as mundane as opening a management console.

One of these days I will bite the bullet and buy a Mac... well maybe.