Free Virtual San Appliance on ESXi

I recently had the need to use some NFS storage within my ESXi hosts. I had plenty of data store left on the physical disks and wanted to create a VM with a large amount of storage. Then I ran into the size limitation of VMDK files which derailed my idea for a while.

Then I did some thinking about how to get around this issue and was able to create my own virtual san within a VM. Here is a rough guide of how you can create your own Virtual SAN within VMware ESXi. This guide assumes some Linux knowledge and will create a 1.17 terabyte NFS VM.

First create a VM with a single drive large enough for your distro of choice and with enough ram. I used CentOS and gave the VM a 4gb disk and 512mb ram. I did the minimal install of CentOS for my appliance and this included NFS in the default group of packages. You will also need packages for Linux LVM in your distro.

After Linux is installed edit the VM settings and add 6 200gb disks to the VM and reboot the VM. Once your back up and running You can list them out with fdisk -l. The first thing we need to do is partition each one of our disks. You can do this with fdisk /dev/sdX with X corresponding to the device indicated in the fdisk -l output. You have to do this for each of the 6 disks. Here is the "template" that I used for my disk partitioning.

fdisk /dev/sdX

n - new

1 - partition number

default - start of disk

default - end of disk

t - type

8e - hex for linux LVM

w - write

Now that the drives are partitioned we have to create physical volumes, volume groups and do a number of other steps to get LVM going. Here is part of my template for doing this.

pvcreate /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1 /dev/sdg1

vgcreate vol1 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1 /dev/sdg1

lvcreate --name shared --size 1.17T vol1

mkfs.ext3 /dev/vol1/shared

mount /dev/vol1/shared /export/shared

Now that LVM is going we need to mount it to the filesystem. Create the directory /export/shared and mount the filesystem using mount /dev/vol1/shared /export/shared You should be able to see the filesystem mounted in the output of df -h at this point.

Now that we have the filesystem mounted we need to have it mount on boot and export it via NFS to make it useful to us. Here are the steps to do this. In this example vi /filename is followed by the changes needed to the file. You will need to modify the changes to the specific configuration of your network.

vi /etc/fstab

/dev/vol1/shared /export/shared ext3 rw,noatime 0 0

edit /etc/exports

/export/shared 192.168.1.0/255.255.255.0(rw,sync)

Permissions to /export/shared should be changed to fit the needs of your environment. Adjust the permissions on /export/shared to fit the needs of your environment. If this is a well protected test box you can chmod 777 /export/shared for full access. You will also want to make sure NFS is up and running in the VM at this point and that you can mount the filesystems under NFS.

Now you can go in and mount this storage within ESXi. Click on Configuration>Storage>Add Storage and Select Network File System. Put the hostname or IP address of your NFS VM in the Server field and the mount point of /export/shared in the folder field. The Datastore Name field is a label which will be the name that ESXi refers to this NFS storage. This can be set to anything descriptive. I used the hypervisor_hostname-nfs as my name to show which system this NFS storage really lives on. Once you hit next your new data store will come up.

For my needs NFS works fine however one could configure the VM as an iscsi target as well and have an even more usable storage VM.

Vmware Console Repeating Chararacters

Ever notice when your typing on a VMware console that you get repeat characters in the console for no reason? This was a slight annoyance for me for a while that I was able to put up with until I attempted to install the Vsphere Management Assistant. Because VMware let the password nazi's loose on this product I had to select a password within their complexity standards and do this through the console which is known to repeat keys. I tried in vain to do short passwords but the script kept throwing me in loops. My long password was simply impossible to enter because of the repeat bug in the console. Bringing the VM into single user mode and setting a password was not possible either because the VM would launch it's annoying configuration script again demanding I assign a long password.

With a little research I found the Console issue is due to network latency. When you type the latency causes the console to think your holding the key and the repeat function comes into play causing the all to familiar and annoying repeat issue.

To fix this you can enter a single line into your .vmx file that fixes this. VMware if your out there and listening perhaps you could make this standard in the Vsphere Management Assistant Appliance especially if you insist on enforcing your ridiculous password scheme in the setup utility. This fix would have been impossible on an ESXi host unless ssh was enabled or the user knew about the "unsupported" console.

keyboard.typematicMinDelay = "2000000"

The Caveat of course is your keyboard repeat is gone so it makes working within text editors and other applications that require repeat characters difficult to work with. You of course can still hit the key multiple times. For my purposes being able to authenticate is much more important than a few more keystrokes to move around in vi.

VMware ESXi Infrastructure Client Issues revisited

In a previous blog entry I posted about how to restart the VMware management agents in the event that you could not connect via the Infrastructure Client. I have found however this does not always work and sometimes one has to get a bit more involved to regain control of a non responsive host.

After restarting the management agents within dcui from the ssh prompts sometimes you will find that the host is still unresponsive. You may also observe the following.

Before our restart in DCUI

~ # ps aux | grep hostd
62991508 22972707 hostd hostd
30481665 22972707 hostd hostd
22972707 22972707 hostd hostd
42826021 22972707 hostd hostd
42826022 22972707 hostd hostd
42826023 22972707 hostd hostd
62069036 22972707 hostd hostd
62069039 22972707 hostd hostd
45624624 22972707 hostd hostd
58286499 22972707 hostd hostd
63749610 22972707 hostd hostd

After the restart in DCUI

~ # ps aux | grep hostd
62991508 22972707 hostd hostd
30481665 22972707 hostd hostd
22972707 22972707 hostd hostd
42826021 22972707 hostd hostd
42826022 22972707 hostd hostd
42826023 22972707 hostd hostd
62069036 22972707 hostd hostd
62069039 22972707 hostd hostd
45624624 22972707 hostd hostd
58286499 22972707 hostd hostd
63749610 22972707 hostd hostd

Notice none of the PID's changed?

In order to get the host responding again to our Infrastructure client we have to kill off these processes with and restart the hostd process. After killing the processes and restarting hostd you should be able to connect again with the infrastructure client. If you don't have ssh enabled on your host you can also do this from the host console.

kill -9 PID#
/etc/init.d/hostd start

Apple what are you thinking?

Today Google released some of the previously redacted portions of their statement to the FCC regarding Google voice on the iPhone. Here is Google's statement on their blog and an article on techcrunch.com.

One has to wonder exactly what Apple is thinking. Corporate spin is one thing but blatantly lying to the FCC?

I have been a supporter of Apple products for many years now. I am typing this article on an Aluminum Macbook purchased within hours of them being officially released to the public. I don't own an iPhone and refuse to purchase one mostly because the locked platform and lack of tethering make it useless to me.

I have been wondering a lot lately though about how much Apple's other products have been neglected because of the obsession with the mobile platform that Apple has? Is Apple's commitment to make great products or just to make money?

Virtualizing Routers with Dynamips and GNS3

One of the challenges that I run into trying to Virtualize environments is the dependence on physical routers to route traffic. In some cases where there is heavy traffic being utilized and a good network design these routers are needed. In most cases they are not and can easily be virtualized to save power, rack space and make an easier to manage environment.

VMware ESX has a great networking stack and allows for virtual switching and a lot of IP operations can be done in software rather than on physical networking hardware. One thing that VMware ESX is missing is the ability to route traffic. There are a number of different ways around this and any Unix based VM can be used to route traffic around the virtual networks. But sometimes you really need a Cisco only feature such as Netflow or the Cisco SNMP functions.

Dynamips is a software package that sits on top of a Unix system and acts as a hypervisor for Cisco IOS images. You can bring up multiple virtualized routers and access their configuration and actually route traffic with them. They use an actual IOS image and are essentially software based Cisco routers. Dynagen is a front end for Dynamips and GNS3 is a GUI that allows for easier configuration of routers and their setup and connectivity.

One of the big advantages of using Dynamips to do routing is the configuration from the physical Cisco router being Virtualized can be taken directly with some minor modifications and be used directly. Dynamips is also quite handy for people used to doing their router configuration in the popular IOS format and as well for people who want access to the IOS command line without a huge investment in routing gear.

Dev Channel Chromium for OS X

I have been trying out Google's new Chromium browser for OS X and it is quite impressive. Chromium is rather nimble and the rendering engine is rather bug free. I have run actually released web browsers that have mangled web pages beyond recognition and this one I have yet to run into issues. While there are a lot of features that are not working such as flash support and some of the configuration options are missing its is still quite a usable browser. I have found it is even more robust than Safari and Firefox on my mac for crashes.

I really can't wait to see what the finished product is like as right now the browser is quite usable and stable already. Do keep in mind however that if you decide to try Chromium its not "quite" finished yet and you won't be able to use it as a primary browser. See this entry in the Chromium blog for more information. Developer Documentation can be found here. And if you want to put Chromium through its paces check out the Chrome experiments site out.

One of the things that makes Chromium interesting is its multi threaded. Each browser tab is a separate process on your system. If you look at the output of ps or top you will see these processes running. Killing one of these processes will kill the tab and give you a cute sad mac icon in your browser. What is also quite interesting, the process is created only when the new tab starts to load content not during the creation of the tab itself.

Fun With Analog Machines

Today I realized that I had a month long gap of posts on my blog and suddenly realized my distraction. Rather than a machine that fits in a 19 inch rack I have been having fun with a machine that fits into a standard parking space.

I bought my beautiful classic 1989 Porsche 944 2.7L on a whim at a local used car dealer and got a terrific deal on one owner car with full books and records. I have been using it as a daily driver while trying to get most of the bugs out of it and have been having a great time in the process.

While it looks like an impractical sports car its quite the opposite. In addition to getting great gas mileage its quite roomy and comfortable on the inside and is a blast to drive. This car was rated the "best handling car in the world" when it was new and even with the cars age the handling is absolutely amazing. It has a perfect 50/50 weight balance with its rear mounted transmission. There's even cargo space! A friend of mine gave me some servers and the rear hatch gobbled up two full 4u servers and a 1u server with room to spare. Even Jeremy Clarkson from Top Gear loves the 944 and he is known to hate Porsche's. You tube video here.

Speaking of those servers they are still sitting in my basement without an OS on them. Perhaps its time I get back to the word of digital machines.

Veeam Monitor 4.0

One of the more painful aspects of working with ESXi hosts is as you have more and more hypervisors you have it become more difficult to manage them with the VMware Infrastructure client. I posted a previous blog entry about VMware ESXi management with Veeam monitor and conserver. The engineers at Veeam have been busy indeed and have released version 4.0 and have drastically improved what was a great product to begin with.

With Version 4.0 of Veeam monitor the Infrastructure client for each host can be opened from within Veeam monitor. This eliminates the need to log into these hosts seperately when you need to do a configuration change that is only possible through the Infrastructure client or through the CLI. I will still be utilizing the CLI for a lot of my tasks but having this integration within Veeam Monitor is quite impressive. Especially considering the price that they are offering it for. Free of charge.

If you manage a lot of ESXi hosts and you don't already have a copy of Veeam Monitor 4.0 you really should check it out. You can download a copy here.

Sun responds to IBM's incentive program

I wrote a blog article previously about IBM being upset about the Sun Oracle deal and throwing a bit of a tantrum. Sun apparently has some not so nice things to say about the innovation at IBM. From the article on Sun's website. "Industry experts frequently talk about the "Brick Wall" of Computing Performance. Well, everyone except IBM. Cranking the clock frequency to scale performance is an evolutionary dead-end; while Sun and the industry as a whole move forward with multi-core, multi-thread processors. IBM is the last holdout and is exhausting the old technique with higher power consumption and higher heat generation."

The article makes quite a few valid points and hits IBM where it really hurts. If you look at the slanted design of the racks for the IBM Blue Gene Supercomputer this was done to house additional air ducts for cooling. IBM is pushing for water cooled designs to further cool these really hot processors which is directly opposite of the industry's air cooled movement. Even Cray moved away from liquid Fluorinert cooling in their later designs.

Unlike IBM with their incentive paid in professional services "blue dollars" Sun is offering a 40% discount on hardware replacing IBM systems.

Rackable Rebranding as SGI

In an unexpected and strange move Rackable announced that they will be re branding their company as SGI. From their press release “The Rackable name will become the brand for the SGI x86 cluster compute products. Rackable will join our other industry-recognized brands – such as ICE Cube, Altix, InfiniteStorage, CloudRack, MicroSlice, Origin, and VUE – to comprise the new SGI.”

While this may be ancient history to most SGI had a line of x86 products that failed miserably. Systems like the SGI 320 workstation that used UMA technology and their full line of rack mounted servers that never sold well like the SGI 1200. SGI's original x86 servers had real engineering inside them and were not just re branded commodity systems and still did not sell. With cost a major factor in most data center build outs its really questionable if "brand name" really matters anymore. What most companies look for when selecting server hardware is price, performance and service rather than a pretty logo in their racks.

It will be interesting to see how this unfolds in the future. A brand represents much more than what can be purchased in a contract. It's a history of engineering and support. Customers can see through an SGI logo attached to an inferior product. Rackable has bought some really big shoes to fill. Hopefully their commitment is more than wanting pretty logo's for their products. That logo alone did not sell SGI's x86 systems. Hopefully Rackable is committed to the same levels of performance, innovation and engineering as SGI was.

IBM = Sore Loser?

IBM loses out on the Sun Microsystems deal and then gets angry because Sun went with a better offer from Oracle. But in typical red faced angry toddler fashion if they can't have their way they are going to take their ball and go home and not be Sun's friend anymore.

This article describes how IBM has doubled the incentive for customers to dump Sun hardware in favor of their hardware. "IBM will offer customers $8,000 worth of software or services for every Sun Sparc processor ditched in favor of IBM Power servers. IBM previously offered $4,000 per processor for customers that made the switch. " This sounds like a great deal but its limited to their already overinflated software and professional services. How about hardware for hardware? Oh thats right. Hardware costs money. I wonder how much of those professional services will be serviced out of Banglore India?

Just Last week IBM announced plans to parner with Brocade because they are angry about Cisco selling their Unified Computing System. Article here in case you missed out. Seems like a pattern is emerging here.

If Dr Watson was alive today I wonder what he would say? I have a feeling that he would not approve of this behavior.

VMware Introduces vSphere 4

VMware released their new product vSphere 4. From their press release they state the following "VMware, Inc. (NYSE: VMW), the global leader in virtualization solutions from the desktop to the datacenter, today announced VMware vSphere™ 4, the industry’s first operating system for building the internal cloud, enabling the delivery of efficient, flexible and reliable IT as a service."

Of all my favorite overused buzzwords "cloud computing" is my favorite because of the wide variety of different computing models it describes. Many people use the term having no real idea what it means just because its the popular buzzword and have no concept that even though your running "on the cloud" your connected to a physical server somewhere with the same physical resources as a standard server. A lot of people fail to do the math of the real costs and think they are saving a bundle of money when in reality their costs have increased using an external provider.

While VMware is guilty of abusing the "cloud" buzzword their new product vSphere is actually rather innovative. From their press release again. "Up to 20 percent additional power and cooling savings with VMware Distributed Power Management which uses VMware VMotion to automatically place all virtual machines on as few physical servers as possible without compromising service levels, and power down physical servers that are not needed. The power savings with VMware Distributed Power Management across all VMware vSphere 4 customers over one year could power a country the size of Denmark for 10 days"

Essentially one could have a cluster of hardware ready to go powered off and have vSphere actively powering on and off servers as needed and using vmotion to move VM's around the cluster. Not only are you not getting wear and tear on your hardware from being powered 24/7 there is an energy and cooling savings as well. Not to mention saving a lot of time with provisioning and capacity calculation. Resources as they are needed power up and down by themselves. Because there is less stress on the hardware it could also last longer as well providing more reliable service over more time. A whole cluster could be installed with more capacity than needed and then left unattended for many years. When the hardware fails its just taken out of the pools of machines. As the compute resources increase more machines in the cluster power on to be utilized providing an almost hands off administration experience for the entire lifespan of the installation.

Now if it were only priced reasonably. The Full featured top of the line version of vSphere costs $3,495 per processor. Their stripped out bottom of the line version is $795 per processor. Depending on your hardware the OS could easily excede the costs of your hardware investment. At these prices power and administrative staff seem really cheap.

ESXtop on ESXi

If you have ssh enabled on ESXi one of the more useful commands for determining system utilization is esxtop. When you run esxtop you get a display of all the processes on the system which is not exactly the most useful information. The program is capable of displaying other information as well. The most useful is the capital V option which shows per VM usage. You can also display disk usage memory usage and network usage as well with esxtop.

Here is a snippet of the help page for esxtop.

Esxtop version 3.1.0
Secure mode Off
Esxtop: top for ESX

These single-character commands are available:

^L - redraw screen
space - update display
h or ? - help; show this text
q - quit

Interactive commands are:

fF Add or remove fields
oO Change the order of displayed fields
s Set the delay in seconds between updates
# Set the number of instances to display
W Write configuration file ~/.esxtop3rc
e Expand/Rollup Cpu Statistics
V View only VM instances

Oracle to buy Sun for $9.50 a Share

Oracle is now going to buy Sun at $9.50 a share in cash. From the article on Sun's front page "Sun and Oracle today announced a definitive agreement for Oracle to acquire Sun for $9.50 per share in cash. The Sun Board of Directors has unanimously approved the transaction. It is anticipated to close this summer."

I think that the Oracle purchase is a more natural transition for Sun. Oracle has a large amount of cash which Sun needs for research and growth. Oracle also seems to be rather hands off with their purchases. This will also be beneficial to Oracle as well having slightly more say as to some of the internals of Solaris so that perhaps there may be more integration into the way the database works with the OS.

The IBM deal would have been troublesome for both Sun and IBM. IBM's own UNIX variant AIX would have come into question in addition to lots of blockers being thrown in Sun's path budget wise for by IBM for innovation and R&D.

I hope IBM has learned their lesson as well from this. When you have the deal of century on your desk it does not pay to be cheap and play hardball in negotiations. Someone like Oracle might be around the corner willing to pay more than the asking price and snatch the deal right out of your hands. I suppose the bean counters and negotiatiors at IBM are kicking themselves right about now.

IBM Withdraws $9M Offer to Buy Sun

This weekend IBM withdrew its $9M offer for Sun after the Sun board balked at their lower offer. From the NY Times article.

"After the legal review, I.B.M. shaved its offer Saturday from $9.55 a share, the proposal on the table late last week, to $9.40 a share, said one person familiar with the talks. The offer was presented to Sun’s board on Saturday, and the board balked. The Sun board did not reject the offer outright, but wanted certain guarantees that the I.B.M. side considered “onerous,” according to that person. Sun then said it would no longer abide by its exclusive negotiating agreement with I.B.M., a second person familiar with the discussions said. On Sunday, I.B.M.’s board decided to withdraw the offer."

Sun is now on their own to make their business model work which I think is a good thing. This gives more choice in the Unix market. Doing business with IBM would have ultimately failed anyway due to their drastically different way of doing business. Its good to see Sun able to walk away from the deal mostly unscathed. Now its time for the board and the executives to sit down and do their jobs to make Sun more profitable rather than looking for a quick payout.

Farewell Old Friend

In September 2006 many people marked the "death" of one of the biggest innovators in the computing world Silicon Graphics when they killed off their operating system IRIX and MIPS technology to produce systems made up of commodity processors and Linux. Today Rackable Systems announced their purchase of SGI for 25 million. If you did not consider SGI to be dead in 2006 they certainly are now being bought in bankruptcy by Rackable Systems. Farewell old friend!

The cray-cyber.org Computer Museum

Occasionally I feel nostalgic and want to log into an old supercomputer. The guys at cray-cyber.org have a great collection of older machines that they keep powered up for the general public to explore. Not all machines are kept powered up 24/7 but its a rare chance to get access to some very uncommon machines.

Among their collection is a Cray Y-MP EL, a SGI Origin 2000 and a Sun Enterprise 10,000. They also have a number of other Control Data, NEC, Cray supercomputers and mainframes. In addition to their extensive collection of working older machines connected directly to the internet is a wealth of documentation and photographs of these historic machines.

These guys have saved a number of machines from the recyclers and go through a lot to get them moved and to restore them to operational condition when they arrive to the museum. In addition to moving them and restoring them to operational condition there is power and space requirements for such systems. The passion that they have for preserving these parts of history is amazing for our day and age. Tours and public access to these systems are free of charge and they do accept donations for power and operational costs.

IBM to buy Sun??????

Reading the news today I saw a rather interesting and shocking article regarding IBM wanting to buy Sun. You can read the article here.

I used to work for IBM and I am quite familiar with their corporate culture. There are pockets of innovation but its mostly a big stable corporation where procedure and chain of command is very important. Business groups are very seperate an competitive and rarely collaborate. Sucess is based off of metrics and numbers rather than innovation and solutions. This made IBM very efficient but also allowed customers to fall through the cracks being bounced between business groups. Sun is somewhat the opposite and is a very innovative company that produces hardware and software that is much more cutting edge. Sun's workforce is really mobile with Sunray thin clients and generous work from home arrangments. IBM's workforce is more rooted to drab office space and more rigid work from home arrangements.

When I worked for SGI I saw this exact screen play of culture shock come together when SGI bought Cray. Cray was very much like IBM. All Cray employee's had offices and things were more formal with scheduled meetings. SGI was very much like Sun a very "hip" and relaxed work force that was very innovative. SGI employees worked in purple cubes decorated in unique ways and meetings usually occoured in front of the many esspresso machines in the same 1600 Ampitheatre Parkway building that Google now occupies. Ultimately this combination failed because they tried to assimilate two very different ways of approaching the solution to the same problem. It cost SGI more than they could ever anticipate. SGI now is selling commodity based Linux systems a mere shell of their former glory.

If IBM can buy Sun and then "stay out of the kitchen" it could work. I just don't see IBM doing this once the metrics are set they will step in and try to make things more efficent and fail in an epic way. While IBM's way of doing business is far from wrong its just too different from Sun's for them to sucessfully combine things. It would be like the "Hi im a mac" guy and the "Hi im a PC" guy collaborating on the same project.

Veeam "Fast" SCP for ESXi

I got an e-mail from the marketing people at Veeam inviting me to try their new Fast SCP for ESXi application "before everyone else". Needless to say I was really excited about the prospect of having something that was faster than the method that I was currently using with the manual SCP on the hypervisors and to try out some new software before its general release.

I did my comparison test with an 8gb VM transferred between two Dell 2950's with gigabit nics both connected to a Foundry x448 gigabit switch. With regular machine to machine SCP I get about 10mb/sec out of my transfers which is not great but works. Now to see the Fast SCP software in action. I start my transfer and hmmm we are only getting 5mb/sec out of the transfers. That's about half the speed I see out of command line SCP and WinSCP. Not really impressive at all.

If you look at this page Veeam clearly claims to be 6 times faster than regular SCP. I e-mailed the company bringing this to their attention and got a canned response back and then no response from a second email. I have no idea why a company would ask a someone to try software and get a valid user story back and let their support department blow them off with a canned response. I really love Veeam Monitor for my ESXi management but honestly Fast SCP is no better than WinSCP for ESXi machines.

Fun With New Hardware

It's been a long time since I have bought any new hardware. This is mostly because I am having so much fun in the virtual hardware world. Occasionally a really good deal comes your way and you just have to give in.

I picked up a IBM RS/6000 7044-170 and got quite a good deal. Its an older RS/6000 machine but is still a 400mhz Power3 with 2gb ram. It's been while since I have worked with AIX so it should be a lot of fun to tinker with. Whats most impressive about this machine is for a workstation class machine it has a full Service Processor with remote access capabilities so it can phone home in the event of issues. Its extremely well made and weighs a ton.

At the moment I have it hanging off the back of my Ultra 60's serial port doing some initial configuration. This is a workstation and has DVI graphics but from what I have read run's a really high single resolution output. For my purposes serial and ssh is fine for now.

Blastwave Open Source Package Manager for Solaris

In every operating system there are things to be loved and hated. I have always loved the rock solid hardware and software platform that Sun has sold for years. One of the things I have not liked for quite some time is having to install software on Solaris. If your lucky enough to find an actual package then you have to deal with the sometimes nightmarish dependencies not included with the package. Once you get all those packages on the machine (usually without wget unless you installed it) you have to install the packages individually and hope things don't break in the process. Especially if you are dealing with non Sun open source packages that have been put together by someone not very familiar with Solaris. And of course we won't even talk about getting compilers working and building from source.

Downloading this simple package from Blastwave.org may be the last time you have to type pkgadd -d on your Solaris host. Once the package is installed you have a powerful package management system that can download and install packages just like apt-get or yum on a Linux host. Instructions here describe the installation and getting things setup correctly which I would recommend taking your time to go through.

One of the things I really like about Linux machines is the ability to quickly download and evaluate software to determine their suitability for the task at hand. Now I can have my cake and eat it too on my favorite rock solid OS and hardware platform and an installer that does not eat up tons of my time.

Citrix giving away XenServer for free

Citrix is now providing XenServer as a free product in a seemingly desperate way to grab market share from VMware. While it might be worth a look for many Enterprises just exploring the server virtualization world, companies with established VMware infrastructure may be a much much harder sell.

VMware being a more established product and with the first to market advantage is a tough act to follow especially with a "me too" product like XenServer. I think its a wise move on their part to compete with the free version of VMware ESXi and to counter some of the generally bad experiences that many users have had with earlier versions of open source Xen.

I am in the process of downloading my own copy of XenServer to take a look at what has changed since the last time I tinkered with Xen. Unfortunately it seems that our friends at Citrix under estimated the demand for their free product. I am getting a whopping 36.7KB/sec downloading from my connection at the ISP I work for. You can read more about their announcement on their blog.

Free and Easy Vmware ESXi management with Veeam Monitor and Conserver

One of the drawbacks of VMware ESXi is when the number of servers in your infrastructure grows the free version becomes less and less attractive. Logging into multiple versions of the VMware Infrastructure client is not only tedious but gets confiusing trying to figure out which VM is installed where. Veeam Monitor is a great utility to solve that issue. It can collect performance data, do monitoring, send alarms, and act as a console over multiple VMware ESXi hosts. It is somewhat limited in its ability to create and manage VM's and control the state of VM's. You can download a free version of Veeam Monitor here. In many ways its a lightweight and less scaleable version of VMware Virtual Center.

In one of my previous blog entries I provided a number of different ssh commands to control and manage a ESXi server via ssh. The commands via SSH fill in some of the features not included in Veeam Monitor such as the ability to reboot hosts and perform other tasks such as bringing VM's into inventory. A great application named conserver allows quick access to multiple VMware ESXi command prompts with a minimum of fuss without having to log into them each time. You can leave the hosts logged in and conserver can monitor and log access to the console and provide a single sign on for multiple hypervisors without having to type passwords to jump from host to host. By adding the following to your conserver.cf file you can ssh to a host named vm1 with the command console vm1. [ctrl]E c. allows you to disconnect and jump to another vm by typing console and its console name. The command console -u allows you to see what consoles are configured on your conserver host.

console vm1 {
type exec;
exec ssh root@long-and-ugly-hostname.subdomain.domain.com;
}

While Veeam Monitor does not do everything and can't really be used for a lot of creation and configuration it is easier to deal with than the VMware Infrastructure client when you start to get more than a few ESXi hosts on your network. Between Veeam Monitor and Conserver most daily operations can be handled without logging into the cumbersome VMware Infrastructure Client unless configuration changes are needed.

An update. Veeam has relased an even more powerful version of Monitor. See my blog article here.

Sun xVM Server Future VMware Killer????

If you look at some of the articles on my blog you see that I work a great deal with Virtual hosts on a day to day basis. VM's are by far not a magic bullet in all cases but are great for making the best use of available resources. So far the leader in the game has been VMware with their bare metal hypervisor offering ESX and ESXi. With a really mature easy to use product they have been an industry leader that has been hard to compete with. That is until now.

Sun has been working on their new product xVM Server which is also a bare metal hypervisor that will be open source. Using a similar model to VMmware ESXi the hypervisor itself will be open source and free and the more enterprise add on features will be pay for play. Being open source I see this really having the potential to take off for Sun. If you have used VMware ESXi for any production systems you know a lot of the "gotchas" that you run into with ESXi that were engineered in to push users to the paid version. Looking at some of the screen shots on http://xvmserver.org/ it appears that Sun's product will be fully managable from a web page making it more attractive to enviroments that are not windows centric.

You can read more about Sun's xVM server product at the following URL. While it has not yet been "fully" released yet its a very exciting product. http://www.sun.com/software/products/xvmserver/index.xml

While xVM server has not yet been released Virtualbox has. I have been working with it on my Macbook and have been amazed at its speed. It blows the doors off of VMware Fusion and its a free product unlike the $79.99 that VMware wants for their product. You can download it here. http://www.sun.com/software/products/virtualbox/get.jsp

Update: This product was killed right after the Oracle purchase of Sun. More information here. RIP Sun xVM Server.

Open Solaris........ Hmmmmm......

I have been a big fan of the Solaris OS for many years now. A rock solid operating system combined with rock solid hardware is always a winning combination.

Being an ex SGI employee I have seen what a "disruptive" technology Open Source has been. IRIX SGI's rather well developed OS was killed in 2006 when the company dropped support for MIPS processors and started using Linux on their machines. The company that created OpenGL fell on its own sword financially killed mostly by "cheap" commodity hardware and the Linux OS. Why shell out big bucks on an Origin 3000 supercomputer when you can just link a bunch of cheap Linux boxes together? If your a bean counter your not going to be able to get past the free part in comparison to the outlay for a high end system.

While SGI did some work "embracing" Open Source they did not do a good enough job at it. Sun is taking a much more open and different approach that seems to be working well for them. Solaris has at least since 5.8 been a free download on the Sun Site for all versions (you need a licence for the OS if your going to use it commercially). A far cry from the $500 IRIX CD kit from SGI if you had a system you were willing to buy a service contract for.

Enter now the Open Solaris OS. Several of my Sun friends have suggested I check it out and I have resisted until recently. If you want a "preview" of what the next version of Solaris is going to be like this is probably where your going to see it. The new package manager pkg can download and install packages like apt-get or yum on a Linux system. A far cry from manually downloading (or wget if your lucky enough to have it installed) and then doing a pkgadd -d to the package and crossing your fingers not to run into dependency hell. While not "everything" is open source you do get some rather powerful toys such as ZFS and dtrace.

I am still in my evaluation process and the jury is still out if Open Solaris will see its way outside the Virtual Machine on my Macbook. It does however make me look forward to the Sun Solaris 11 release rather than look at it as another possibly painful event because of the "change for changes sake" that is so popular with the Open Source community.

VMware ESXi Infrastructure Client Issues

Because of the appliance like nature of VMware ESXi there is not a lot that can be done when something goes wrong. In a lot of cases if something stops functioning correctly the only choice is to reboot the system with the risk of loosing settings and suffering downtime.

If you have enabled SSH on your VMware ESXi hypervisor and are having issues with the Infractructure client you can restart the management agents through the CLI to regain "control" of the system. If your Infrastructure client is unable to login or if your getting strange activities or error messages you probably need to restart the management agents. The following steps can be used to do this.

Login to system using SSH

Run the dcui program (if your terminal is not big enough it WILL segfault)

Authenticate to the dcui console program

Select restart management agents

You should now be able to log in and any issues that you were seeing before with the Infrastructure Client should go away.

UPDATE: On rare occasions this process does not work. In those cases these instructions are somewhat helpful on how to regain control of your VMware ESXi host

VMware ESXi SSH CLI commands

One of the biggest challenges I run into working with ESXi hosts is the lack of a real usable CLI. The remote CLI provided by VMware is clunky at best and makes administration a bit of a pain. I did some poking around and managed to find some interesting CLI commands that work for ESXi.

Most of these have no manual page but will output a help message if run by themselves. I would recommend using extreme caution and using a test host to determine the usefulness in your environment before using them on an important system. Also keep in mind that the examples that I give are not the "only" way the commands can be used.

vim-cmd vmsvc/getallvms
Lists all vm's running on hypervisor and provides vmid

vim-cmd vmsvc/power.off vmid
Powers off vmid referenced from getallvms command

vim-cmd vmsvc/power.on vmid
Powers off vmid referenced from getallvms command

vim-cmd vmsvc/power.reboot vmid
Reboots vmid referenced from getallvms command

vim-cmd vmsvc/destroy vmid
Deletes the vmdk and vmx files from disk

vim-cmd hostsvc/maintenance_mode_enter
Puts hypervisor into maintenance mode

vim-cmd hostsvc/maintenance_mode_exit
Takes hypervisor out of maintenance mode

vim-cmd solo/registervm /vmfs/vol/datastore/dir/vm.vmx
Registers vm in hypervisor inventory

vim-cmd vmsvc/unregister vmid
Unregisters vm with hypervisor

vim-cmd vmsvc/tools.install vmid
Starts vmware tools installation for VM

vim-cmd hostsvc/net/info
Provides information about hypervisor networking

chkconfig -l
Shows daemons running on hypervisor. Can also be used for configuration.

esxtop
Same as linux top for vmware

vmkerrcode -l
List of vmkernel errors

esxcfg-info
Lists a LOT of information about the esx host

esxcfg-nics -l
Lists information about NIC's. Can also be used for configuration.

esxcfg-vswitch -l
Lists information about virtual switching. Can also be used for configuration.

dcui
Provides console screen to ssh session

vsish
Vmware interactive shell

decodeSel /var/log/ipmi_sel.raw
Read System Event Log of server

VMware ESXi Loosing Network Settings

Due to the appliance like nature of VMware ESXi there are very few things that can be accomplished troubleshooting wise if there is a problem. Rebooting is often the only choice. Caution must be taken when rebooting a VMware host as sometimes the system will loose its IP settings. When the system is rebooted it will no longer be pingable on the network and the remote console will not function requiring manual intervention on the console. I have seen this happen several times and recovery is sometimes painful especially if there is limited remote access to the system console.

When you go to the console of the system you will notice that the ip address is set to 0.0.0.0 and that you will not be able to edit the network settings within the console. The system will not be able to reach the network at all. I have observed that one sees "odd" behavior on some of the administrative functionality of the console. If your not able to add users or make changes to the systems settings or see odd behavior you may want to schedule a downtime and have someone ready in the datacenter to assist.

At this point the only choice for recovery is to choose the "reset to factory defaults" setting on the console which blows away ALL of your settings including resource pools, networking information, users, and machine inventory. You will need the IP address information for the host and will need to set the root password on the system again. At this point the host is back up on the network and you can start reconfiguring and bringing the VM's back into inventory (they are still on the disk). Since vicfg-cfgbackup is missing out of esxi all of the configuration has to be done by hand.

New Aluminum and Glass MacBook

Normally I am not one to upgrade computer hardware until it's time but occasionally a system come's along that you just have to have. When the original "icebook" ibook came out I just had to have one and the same thing just happened with the new MacBook. Ill be retiring my perfectly good 2.2ghz polycarbonate MacBook and giving it to a friend for it to provide them many more years of reliable service.

Getting my new MacBook was not as easy as I had hoped. After the "official release" I called around to a couple stores and no one had any stock or could provide me any informtion at all about when I could expect to see them in the store. I figured that most stores would have to "reset" the shelves and get the product out in the morning. I stopped into the Lenox Apple store in person shortly after they opened and there was no stock and again no estimate on when I could buy one. Ok this is getting slightly annoying. I called Apple directly no real informaton about stock. They would be glad to place an order for me but they were unable to comment about their store locations inability to provide any information about their stock. I waited until about noon today and started calling around again. I called the Perimeter mall location and they had their stock ready to sell. I called the Lenox location again and asked if they had theirs in yet (they were closer) and they did not yet have anything in. When I informed them that Perimeter had some in I was sarcastically told "go buy it from them then". Hmmm. Unhelpful and rude. How charming.

The Permieter store had the MacBook in stock and ready to go. The transaction took just a few minutes and I was walking back to the car again with my tiny box that cost me $1700 and some change. They were even polite and did not tell me to pound sand or act like I was putting them out by trying to buy something from them like the Lenox mall store. Amazing!!!

I had to forgo the tradition of maxing out the ram on the system as it would have been a 45 minute wait for me to get them to have the officially blessed deciple of Apple pop a new DIMM in for me. I will revisit the ram issue later. While I used every bit of the 4gb in my polycarbonate MacBook with VMware Fusion I will be able to scrape by. My VM's will have to inhabit a little less silicon for now.

I bought this system before there were any demo units out on the tables at the Apple store and was really afraid about being dissapointed once I opened the box. The new MacBook is simply amazing in every way. Its lighter, and has an amazing feel and build quality to it. The screen is amazingly bright and the unit is much smaller than the original polycarbonate design. Essentially the 13 inch MacBook is a "mini" MacBook Pro and reminds me of the old 12 inch Powerbook. The photos on the site don't do the new MacBook justice at all. There is a lot of attention to detail that went into this system. The new glass trackpad and glass screen are amazing. Even the bottom of this machine is now curved better so it fits in your lap more comfortably.

Resolving MAC Address Conflicts in VMware ESXi

When cloning virtual machines the VMX file stores information about the virtual MAC address of the virtual NIC installed within VM. I recently ran into an issue where machines had been cloned on a VMware ESXi installation and new UUID's had not been generated. This caused duplicate MAC addresses to be used on two different VMware servers which caused some connectivity issues between them because of some general confusion in ARP resolution.

From VMware's documentation.

"The UUID is a 128-bit integer. The 16 bytes of this value are separated by spaces, except for a dash between the eighth and ninth hexadecimal pairs. So a sample UUID looks like this: 00 11 22 33 44 55 66 77-88 99 aa bb cc dd ee ff The UUID is based on the physical computer's identifier and the path to the virtual machine's configuration file."

To generate a new UUID do the following within VMware ESXi. Remove the VM from your inventory. SSH into the hypervisor and go to /vmfs/volumes/datastore# Rename the directory that your VM lives in with mv vmdir newvmdir. Go into your VM directory and edit the VMX files and remove these the lines that start with this.

uuid.location=
uuid.bios=
ethernet0.GeneratedAddress=
uuid Action

Add the VM to inventory and do your initial power up. The new UUID will generate a new and unique MAC address and all the layers of your network will be happy once again. This will work with VMware ESX and Server / GSX as well.

Moving VM's in VMware ESXi via SSH

VMware ESXi is a great product but it present's a number of challenges because of its highly streamlined feature set. Lets say for example you need to move a VM from one VMware server to another and don't have an NFS server or iSCSI target. And of course you don't have any of the licensed features like Backup or Vmotion either. Lets add another challenge that you only have SSH access to the system. Sounds like a fun challenge already.

Our first challenge of course is SSH access to VMware ESXi. ESXi is considered an "appliance" by VMware and SSH and their CLI have been removed. Enabling SSH is trivial and can be done by pressing ALT F1 at the console and typing unsupported and then the root password of the machine. From there uncommenting ssh in /etc/inetd.conf and restarting services with a /sbin/services.sh restart will get inbound SSH to the box. You will also need to do this on the system you are trying to move your VM to. Word of caution VMware does not support this and there are security implications to enabling SSH on ESXi. You should evaluate the risks in your environment beforehand. ESX has a SSH interface by default but there is the "warning" message within ESXi.

From there you can navigate to the vmx and vmdk files you wish to migrate under the /vmfs directory. Your VM's should be under /vmfs/volume/yourdatastorename. Once you are in the directory you wish to migrate you can use SCP to copy the physical disk files to the other machine. With a command similar to scp * root@hostname:/vmfs/volumes/datastorename/vm-dir.

On VMware ESXi the command line features of regular ESX have been removed. You will need to install the "Remote Command Line Tools" either on a linux host somewhere on your network. You will need the specific command vmware-cmd for the registration of the VM with the hypervisor and to power on the VM. Download the Remote Command Line reference guide for more information on the CLI. A command similar to vmware-cmd -H hostname_of_hypervisor -s register /vmfs/volumes/datastorename/vm-dir/vm.vmx datacenter-name resource-pool.

Now that our VM has been moved and registered we are ready to boot it. You can do this from the remote command line with vmware-cmd -H hostname_of_hypervisor /vmfs/volumes/datastorename/vm-dir/vm.vmx start

Fun With Fibre Revisited

Today my new toy arrived. I ordered an EMC Connectrix DS-8 fibre switch to retire my dual Sun Vixel Rapport FC-AL hubs. Under the covers its really a Brocade Silkworm 2400 with EMC badging. Brocade sells their switches rebadged for a number of other vendors including IBM and you can snap the "badged" switches up for next to nothing. I initially planned on a Silkworm 2800 or EMC DS-16 for more ports but got a steal of a deal on the smaller configuration. For my home tinkering it will do just fine.

Configuration was amazingly simple and can be done through the Java driven web gui or the command line. The command line is straight forward but a bit "funky" for people who are more used to Cisco or Foundry style CLI's. As with any CLI the help function is quite useful for figuring out the syntax of commands. The Java driven GUI as expected only works well with IE on windows which was not unexpected. The GUI is actually pretty useful as well and is good for quick status info and keeping tabs on the switch.

Impressively enough OpenNMS found the swtich via SNMP and is able to pull graph data from it without having to do any extra configuration. It will be great to be able to keep tabs on what's up on the SAN without having to log into the switch directly.

Network Management with OpenNMS

Over a period of time a small network grows into a larger network. Tracking hosts for availability is constant challenge for an admin on a large network.



For my home network I decided to implement a network management system because the number of hosts on my network had grown to a larger number than I could keep track of. I wanted something simple that would give me the functionality of up or down and have the ability to save notes on a host and get basic trending data regarding availability.

I installed 3 Virtual machines and installed Cacti, Nagios and OpenNMS within them. Out of the 3 packages I evaluated OpenNMS was surprisingly the easiest to configure and work with and provided the most functionality.

With a basic configuration OpenNMS will seek out all of the hosts in the network range you specify and list all the services that are running on those hosts. I honestly wish I had installed OpenNMS first as I spent a lot of time editing the overly complicated Nagios configuration file to even get one host monitored. In the time that it took me to get Nagios working for one host OpenNMS would have been completely setup with all the hosts in the configuration.

http://www.opennms.org/

Vmware Server CLI

If you use a Mac pretty often like I do your limited when it comes to managing a system running VMware because there is not a native VMware console that runs on OS X. And of course the GUI is kind of limiting when you have to traverse a few firewalls. On Solaris the Zone subsystem is run completely by the CLI which I find to be quite useful.

There is a rather primitive but functional CLI for VMware. If you type in vmware-cmd -h the program will display a help file. Here are a few useful examples of things you can do with vmware-cmd.

Listing your VM's is vmware-cmd -l

rchase@darthfinity:~$ vmware-cmd -l
/vmfs/Ubuntu 8.04/Ubuntu.vmx
/vmfs/Ubuntu 8.04 VM2/Ubuntu.vmx
/vmfs/Ubuntu 8.04 VM3/Ubuntu.vmx
/vmfs/CentOS VM1/Red Hat Enterprise Linux 4.vmx

Of course this just lists the VM's and does not really tell us much about them. Here they are. Are they on? Are the off? vmware-cmd getstate will help answer some of those questions. And as you can see theres something to be said about short VM names without spaces.

rchase@darthfinity:~$ vmware-cmd /vmfs/Ubuntu\ 8.04/Ubuntu.vmx getstate
getstate() = on

Lets say for example we wanted to start or stop a VM. The command vmware-cmd start or vmware-cmd stop is quite useful.

Here is some more information on the VMware website.

http://www.vmware.com/support/esx21/doc/vmware-cmd.html

Foundry Switching in the Lab

Faced with a failure of a bank of ports on my Cisco 2924 upstairs and running out of ports on my 2924 in my lab I decided to get an additional switch. I decided on a Foundry EdgeIron 4802CF for my needs. The 4802CF is equipped with 48 standard copper ports and and additional two optical ports for LC fibre.

I have been adding a few QFE cards into my Sun systems to play around with IPMP and faced running out of ports completely in the near future during the deployment of my workstation lab. Additionally the two optical ports will come in handy for getting the optical network cards in my Enterprise 450 on the network.

The network guru's seem to really like Foundry gear a lot and now I know why. Setup was really straight forwards with a lot of the setup commands being very similar to Cisco IOS commands. Not to mention this switch is not taxed at all by my home network traffic which has a lot of large CIFS transfers. The 2924-xl that was removed from service is now going upstairs into my office to replace the switch with the bad ports and the old switch can be a backup in the event I need an extra switch.

Now onto installing more QFE cards and tinkering with IPMP.

OS X the ultimate hypervisor OS?

Ok. I will start off and admit that until I actually bought an Intel based mac I was not a big fan of them. In my opinion Apple was going "backwards" technology wise rather than forwards. Cisc processor technology is straight out of the 1950's and yet it refuses to die. But the simple factor of economics and numbers comes into play here. The simple fact of the matter is there are a LOT more cisc based x86 processors around than PowerPC, mips and sparc based systems. That gives the makers of these chips a lot more money for R&D to make their ancient technology faster and faster. I saw this when I worked at SGI where an off the shelf Intel chip would rival the performance of a small desk side Origin Supercomputer. When you got into the larger configurations where a system could stretch the legs of the Cray link interconnect fabric would you see the SGI system "blow away" even fast intel clustered systems. The SGI R&D people just could not keep up because making chips was just a "part" of our business. Intel just makes chips all day long and has nothing else to worry about.

Virtual Computing is an area of morbid fascination of mine. When implemented correctly it can save cost and add to efficiency and generally do good. When implemented incorrectly it can be a complicated expensive quagmire of downtime and really cause a lot of headaches. When I was at IBM I saw it in its most negative forms. A company would implement a VM server and then overload it with too many VM's and have a hardware failure. One midrange server then had the priority of 20 to 30 downed systems. Capacity planning and the ability to get your VM's off the sinking ship of sick and dying hardware with a vmotion or san type solution avoids these problems but many companies try to cut "too many" corners in order to get into a solution they they could not afford to begin with.

I recently installed VMware Fusion on my Macbook and am AMAZED at how well it performs. The Windows and Linux VMware Server products which are free (unlike Fusion) always seem to show that they are taxing the host system somewhat with their "jumpy" mouse cursor and occasional freezes and hangs during CPU intensive tasks. Fusion does not do any of this at all and in full screen mode you would be convinced that you were not in a VM at all. I can run multiple VM's at the same time without any real performance degridation on my host machine.

The reason for my post title is I think Vmware is on to something and Fusion has some limitations. It would be interesting to see if Vmware would offer a VMware Fusion Server for OS X server allowing their VM product to ride on top of OS X server offering the same seperation of console and VM that ESX server does. I often for convenience launch windows 2003 server and then minimize the console and use my session through RDP to make the integration of the GUI seamless to my host OS. Offering this type of console separation would allow power users that just want a Linux command line or an RDP session into a VM to have this without the clutter of an OS console in their face. Vmware already has a product that runs on OS X. Bringing VM's to OS X server would be an entirely different market for VMware. Apple has had some problems marketing their Xserve and Xsan products mostly because of the major competition in the x86 marketplace and the stigma that "Macs" have in the business market place. An Enteprise VM product that runs intel OS software better than x86 bare metal is a "killer app" that Vmware and Apple might want to look into.

Vmware Fusion on top of Mac OS X is great because you have the rock solid operation of os X that "just works" with the ability to run a multitude of different virtual OS's.

Battery Backup Power

I have been running my servers at my house for a while and have not really had an issue with power except for the occasional outage. Most of my systems don't miss a beat and just power right back up when the power returns. The most notable exception is the my most recent addition the IBM Netfinity 7600 with its rather temperamental 4L ServRaid Controller. The 4L has no battery backup cache for the controller and gets rather upset when there is an abrupt shutdown before it can sync its metadata to the array. I used to support these evil little controllers at IBM and we used to refer to the recovery as "reset to defaults and copy config". Of course you can't really do that from an ssh session when your not in front of the machine so backup power is the way I needed to go.

I decided on an APC brand UPS mostly because they have been in the business for so long. I also wanted to know that I could get replacement batteries when the time comes in about 3 years. With the stories about the exploding laptop batteries the last thing I want is some Chinese made batteries doing something crazy and causing lots of problems.

One of the really cool side things of going with the APC ups was the software apcupsd. I run Debian 4.0 on darthfinity and installing apcupsd was just a simple "apt-get install apcupsd". A few minutes editing the configuration file for my specific setup and now my server automatically gracefully shuts down when the batteries are running low. I also get notified of power fluctuations and outages on my Blackberry. Apcupsd also has a cool monitoring program that allows you to see the status of your ups the command "apcaccess status" tells me how much battery I have and the load capacity I have on the UPS.

Now on to having some reasonable uptimes. :)

Neat Older Hardware

Ever since I saw the massive 8U Netfinity 7600 I knew I wanted my own. Now that they are getting much older one can pick one up for next to nothing on the used market.


I located a local Netfinity 7600 and and decided to buy one. Here is a photo of "darthfinity" mounted in my Sun rack. My Netfinity 7600 is a dual PIII xenon's with 2.5gb ram and 8 drives in a raid 5 on a ServRaid 4L controller running Debian 4.0.

I tend to gravitate more towards Redhat for my machines but was really impressed with how easily Debian installed and configured. It's certainly an improvement over my last Debian experience and quite honestly a reason to reevaluate my affinity for Redhat.

What is also very interesting is this machine is not really as slow as I was expecting. Debian is quite quick on this machine. If one did not look at the hardware and did not have really CPU intensive applications it would be possible to not even realize you were on an older system.

IBM Management Module CLI Reference Guide

The IBM Blade Center hardware has a rather under documented SSH and Telnet based CLI that offers the same level of control that the web based interface allows.

While I was working at IBM I ran into many scenarios where a remote administrator did not have access to a system on port 80 for the web interface or the web server within the management module failed to respond. I wrote a document providing some examples of the commands one might use to administer the blade center via the CLI. The existing guide written by IBM provides the same information in a less hands on way. You can download the guide I wrote here.

Here is a simple example where we would give blade 1 control of the media tray and set the boot sequence of blade 1 to the cdrom and then power on the blade.

mt -b 1
bootseq cd -T system:blade[1]
power -on -T system:blade[1]

Solaris 10 Zones Survival Guide

Since I was dealing with so many more machines at work with Solaris Zones I decided to setup a zone lab on my home Enterprise 450 so that I could be a bit more invasive in working with them and do things that I would never do on a production system (destructive testing is a lot more fun too!). In setting up this lab I wrote this white paper for some of the other Administrators at work.

Not only are the command line tools really simple and work really well the Zone's themselves take very little in the area of CPU and memory resources when they are not in use. In addition the disk resources are shared with that of the host system so you can move through the zone's file system from within the global zone without having to mount a disk image like one has to do with Vmware ESX.

A little rewiring never hurt

The old saying of "if it's not broken don't fix it" is good advice in most cases. Except when you have a small group of servers that suddenly turns into a large rack full of servers. The rats nest of ethernet cables was beginning to be a bit of a problem so I decided to rewire my rack.

They have been doing the wiring in a new frame at work and I saw the way "the real wiring guys" put in wiring. Although my results are not quite as professional as theirs I did make a large improvement. To be honest the professional wiring people make cable look amazing. I now understand how the network engineers can get so excited about ethernet cable.

I even added some additional connections for some of the nic's I was not using at the time for future expansion and flexibility. Now if I need an additional IP address or network connection its a matter of a few ifconfig commands rather than going down and finding a cable long enough and getting the cable connected.

On the optical side of the house I was able to organize some of my fibre cables a little more as well. I have dual fibre hubs and a few of the server's hooked up to the SAN so the optical cables were getting a bit tangled too.

Fun in Downtown Atlanta

We had a tornado downtown as most everyone know's by now due to the media coverage. I guess nothing will get media attention than a CNN building having most of its windows blown out by a tornado. The tornado itself is nothing compared to the aftermath.

Since it has been a week before they started any work boarding up and removing remaining window glass from the broken windows in the Westin Peachtree Hotel high winds have caused pressure issues inside the hotel which has "popped out" many full panes of glass sending them raining down on Peachtree st below. Peachtree street is now completely blocked off to both vehicle and pedestrian traffic. Supposedly the entire building moved a few feet as well but thats only what I have heard and I have not really seen anything to confirm this.

Getting anywhere on Peachtree is somewhat of a challenge as they also decided to close the Peachtree Center Marta station and are sending commuters down to the 5 points station. To get to the parking deck that I park in I have to redirect down at least 10 side streets many of which close and open for no real logical reason at all other than the whim of construction crews and Atlanta Police. I consider myself lucky that my commute is during off hours. Several of my co workers have been stuck in nightmare bumper to bumper traffic on some of the small back streets in an attempt to get to and from the office. Hopefully things are going to get wrapped up soon so that the city can get back to business as soon as possible. The Peachtree Center mall has completely been shut down due to its proximity to the Westin and the America's Mart may have to cancel some very profitable conventions because of the streets being blocked.

Its ALIVE !!!! [evil mad scientist cackling in the background]

Ok so I admit I have a Frankenstein monster in my basement but now finally that monster is alive. I finally got my StoreEdge A3500FC up and running and configured. For a while I was having issues with the fibre connectivity where my fibre cards could not see the interfaces on the two controllers. I finally got those resolved and was really thrilled when Raid Manager was able to communicate with the controllers.

Here is a photo of Raid Manager communicating with the array. Right now I only have one D1000 powered up and the unit seems fine with the other unit's powered down. My biggest problem now is throughput. Initially it was difficult finding "older" fibre cards that would work with this system. I read a lot about "issues" with this array and wanted to avoid as many headaches as possible so I wanted to stick to "vintage" 1gb cards. I ended up finding an older Sbus card and stuck it in an Ultra 2.

The Ultra 2 is a dual 300mhz with 384mb ram at the moment. Needless to say NFS and CIFS network activities are a bit slower than I expected due to the CPU and memory restrictions with the host machine. I have some additional memory and processor's sitting around and I am planning an upgrade really soon but even then the Ultra 2 will still be a bottleneck due to its older architecture. I may also add some additional hosts to the san and repurpose the Ultra 2 for something less taxing.

Fun with Fibre

One of my long term projects is to get my A3500FC array functional. Its quite ancient as its a first gen type of device that is merely a front end to some High Voltage Differential arrays but I have a soft spot for obsolete once tip top of the line cutting edge hardware. My unit has 5 D1000 High Voltage Differential arrays connected to the back of the A3500FC. The A3500FC has dual SCSI controllers each with their own really heavy cache battery. There are two Fibre Channel connections on the front of the A3500FC controller (why they did not put them on the back with the rest of the connections is beyond me). Each one of the Fibre connections is a blazing 1GB FCAL. But back in the day it was fast and its more than adequate for a "home SAN".

Sun officially dropped support for the A3500FC and seemed to go out of their way to EOL it by not supporting it with Solaris 10. Here is a photo of me downgrading one of my Netra's to Solaris 9 so that I can run raid manager and support the array. I have read about people being able to run A3500FC's off of Solaris 10 by doing a few changes to the OS but it seems like too much trouble with the potential for unexpected results.

TIP OF THE DAY: Never name your terminal server blinkey. Its bad luck! I named my Ultra 5 that I use as a terminal server blinkey and it has power problems (loose IEC plug) and is now having issues with its serial ports so I am stuck in my basement with a terminal arghhhh!

One of the big challenges for running a piece of hardware like this in a house is power. Just the array itself wants 12 110v IEC plugs. Thats not including power for the 6 systems that I have installed in the rack as well or the dual fibre hubs (yes hubs its FCAL). The rack itself is wired for three phase power with a really cool power switching module that supplies power to each one of the arrays. Since I have no real needs for redundancy or major capacity I plan on only running a couple of the D1000's and at that not in redundant power mode. This will probably be ok but as I buy more hardware I will probably have to look into having a more dedicated power source for the basement. So far I have yet to pop a breaker or even get close for that matter. :)

Snow Day. Well Sorta :)

Atlanta got a light dusting of snow today. Not really much of a snow storm but the airport canceled a lot of flights. The roads stayed clear and it was a great day to get lots of errands done. Went to get some groceries and of course the bread and milk were all gone. What exactly does one do with bread and milk when they are snowed in with less than an inch of snow? Went out and had a wonderful late night dinner at Landmark Diner where it was quiet and devoid of people.

2008 The Future is now

Its very difficult to keep a positive view of the direction of the technology market for many people. There are simply a lot of negative things happening in the market. This has never really changed over time. Take for example the factory workers in the industrial revolution afraid that they were going to be replaced by machines. There of course needed to be someone to operate the machines and deal with the increased efficiency of the machines. Factory jobs actually grew as the demands for the products increased.

The same is happening in a way in the technology market. While there are a lot of negative things going on in the market which I won't touch on since the internet is flooded with them there are a lot of good things going on. Social networking technology is being adopted by people who have never owned a computer before and spaces like Myspace, Youtube and Google are reasons for people to own computers and get on the internet. With blogs and video sites like Youtube the internet is also becoming a source of news without the predictable spin that many news outlets have. Devices like the iphone are giving regular people the ability to do more in the palm of their hand than many science fiction shows allowed. Did the Star Trek Communicator have Google maps? Consumer demand for the internet and new technology is growing like wildfire!

All of these technologies require people on the back end to keep them going. While many of the classical computer jobs of the past have been reduced in the quest to reduce operating costs so many more are popping up to replace them as technologies grow.

The news seems to be full of gloom and doom about how the economy is going to go into a recession but I am not sure I am really afraid yet. During the .com blowout that many of us all still have horrible memories about what happened to the big banks that are having problems now? Not a lot as they just adapted to the market. The technology market is filled with a lot of smart people who have the rug pulled out from them before. I think we learned from our past mistakes and as an industry will continue growing.

Fedora 8 is Amazing

I have been tinkering with Linux for several years now and have mostly considered the "quest for the desktop" that linux has been moving towards to be a minor annoyance as things that are supposed to make it easier for the end user lead to bloat and lost performance. There is also a fine line between making things easier for the end user and irritating more experienced users and I think Fedora has finally found that balance.

I installed Fedora 7 on my Thinkpad T23 several months ago and was really impressed with the ability the OS had to support all of my hardware out of the box. My wireless worked the with zero tinkering and so did the sound and X server.

I have been reading about Fedora 8 for a while now and usually always wait to let other people do the beta testing and find all the bugs before upgrading. Today I finally decided to upgrade from Fedora 7 to Fedora 8 and the process went flawlessly with a Yum update.

Fedora's new look is quite striking as well. Finally someone in the linux world understands the need for an attractive and usable enviroment. While the argument is made that most users customize their UI there is really no excuse for an ugly interface. The default gnome interface has been pretty ugly for years now and it takes a lot of tinkering to make it stop hurting your eyes. Fedora 7 had some attractive UI but Fedora 8 is even more attractive with a clean and professional look.

With a professional UI like this I could actually see in the near future replacing my mac with a Linux Machine. I have not liked the direction Apple has been going in since their switch to Intel and its nice to know that their really is a viable non windows alternative if Steve keeps golfing with Michael and Bill.

Mainframe History

Was stumbling around the IBM site today looking for some information regarding a M/T 7017 and found this really interesting article with historical photographs of the entire IBM Mainframe line.

http://www-03.ibm.com/ibm/history/exhibits/mainframe/mainframe_album.html

Each older system has an entire section devoted to it with lots of really neat photographs of all the systems and interesting hairstyles and clothing of the operators as well

New Blog

So I have been dragged kicking and screaming into the whole web 2.0 craze. Here are some of my personal rantings and notes in no order whatsoever. More semi interesting things to come.