Cheshire Cat Computing

I think the bug lies not in the SDK, but in the virtualcentre. If you go to the virtualcentre api web page (the API runs over http) you can see that the virtualcentre is returning this invalid data to the SDK. So it looks like we'll have to wait for the next virtualcentre upgrade... I was able to duplicate it when we had a test verison but we dont have the problem currently.

Our virtualcentre is v4.0.0, build 162856 (get this from the help menu on the VC client)

Can you let me know what version/build your experiencing the problem on?

Steve

_________________
Steve Shipway
UNIX Systems, ITSS, University of Auckland, NZ
Woe unto them that rise up early in the morning... -- Isaiah 5:11

Hi There,

I am seeing the same issue as this when I do this command;

/check_vmware.pl --config=/usr/local/nagios/etc/vmconfig --server vm3 --report=cpu
CRIT: CPU usage at 48822148.28% (need more CPU allocation?)<BR> (No CPU fairness data)|cpu=48822148.28%;80;90;0;100

To my knowledge I am running the latest of everything

Check-vmware.pl = 0.12
SDK = VMware-vSphere-Perl-SDK-4.1.0-254719.x86_64.tar.gz
Running VMware ESXi Version 4.1.0 Build 260247

I realise after skimming the forums that you do not run a similar system but I was wondering if you had any suggestions on how I could figure out what was wrong with the calculations of CPU?

Cheers

Someone was helpful enough to post the debug output of check_vmware (run with -d) and it shows what the problem is.

The cpu statistic pulled form virtualcentre, which is supposed to be in percent, is instead returning the value in MHz -- although the corresponding memory percentage is returned correctly. All other versions of virtualcentre return the value correctly as a percentage.

There is nothing I can do about this! The VC API is returning incorrect data, and the correct data is not available anywhere else.

_________________
Steve Shipway
UNIX Systems, ITSS, University of Auckland, NZ
Woe unto them that rise up early in the morning... -- Isaiah 5:11

Hey Stevesh thanks for coming back to me!

I think it maybe possible to get good enough cpu % using a different calculation hower I am not all that familiar with perl so it could be in the too hard basket for me.

Inside the SDK is this file /vmware-vsphere-cli-distrib/doc/samples/performance/utilization_report.pl

If you do a --countertype cpu when pointed at a server then the 3 usage lines appear;

Usage : 3.17 Percent
Usage : 1.9 Percent
Usage : 5.21 Percent

Fairly sure these are

Usage : 3.17 Percent (Latest)
Usage : 1.96 Percent (Average)
Usage : 5.21 Percent (Maximum)


I gather there is a logical explanation why it does not make sense to use these counters?

Does this help?


Thank You

The value the script uses is the 5-min average as this is the best indication -- 'latest' will make you vulnerable to spikes in the data and 'max' will give artificially high values.

Programmatically calculating the percentage from the MHz value would be technically possible, but would need to store the current value and calculate the differrence and time between invocations, which is awkward and inconsistent.

The problem seems to be in the API on the esx *servers* rather than *virtualcentre*, so maybe you can point the check_vmware script at the virtualcentre rather than at the ESX server itself?

_________________
Steve Shipway
UNIX Systems, ITSS, University of Auckland, NZ
Woe unto them that rise up early in the morning... -- Isaiah 5:11

Hello, I'm using check_vmware.pl 1.13 and am still getting the CPU utilization calculation error as mentioned above.

[root@lib-systems plugins]# ./check_vmware.pl --config=XXXXX --report=cpu --debug
Starting.
Connecting
Connected
Server Time : 2011-07-26T23:21:23.342901Z
Report type requested is [cpu]
Base is ha-folder-root
Retrieving PerfMgr data
Selected interval is: 0
Retrieving list of hosts...
Making new propertyspec
Making new filterspec from
Retrieving vim_service
Retrieving properties from VimService
Checking faults on SoapResponse
Processing entities:
ha-host
Creating query for MORef ha-host
Start time: 2011-07-26T23:16:00Z
End time : 2011-07-26T23:21:00Z
Retrieving data...
Perfstats retrieved...
Results for value ha-host
Perfdata object: 2011-07-26T23:21:20Z
Disconnecting...
Exiting with status (2)
CRIT: CPU usage at 7825528.47% (need more CPU allocation?)|cpu=7825528.47%;80;90;0;100

(I realize that it's an error in the API)

And oddly, I'm getting a similar error for 'net'.

[root@lib-systems plugins]# ./check_vmware.pl --config=XXXXX --report=net --debug
Starting.
Connecting
Connected
Server Time : 2011-07-26T23:24:11.544703Z
Report type requested is [net]
Base is ha-folder-root
Running network report
Retrieving PerfMgr data
Selected interval is: 0
Retrieving list of hosts...
Making new propertyspec
Making new filterspec from
Retrieving vim_service
Retrieving properties from VimService
Checking faults on SoapResponse
Processing entities:
ha-host
Creating query for MORef ha-host
Start time: 2011-07-26T23:19:00Z
End time : 2011-07-26T23:24:00Z
Retrieving data...
Perfstats retrieved...
Results for value ha-host
Perfdata object: 2011-07-26T23:24:00Z
Disconnecting...
Exiting with status (2)
CRIT: Network usage at 242144808Kbps (need more resource shares or physical interfaces?)|net=242144808;7680;10240;0;


The other parameters (memory, disk) produce meaningful output and jibe with my understanding.

The CPU error has been tracked down to definitely being an error in the API implementation on the VirtualCentre, so I'm unable to fix it. The API is returing a value in MHz (actually the value of the cpu:usagemhz counter) for the counter value cpu:usage where it is supposed to be returning a percentage. Debug output from the plugin has confirmed this as the problem. It only seems to affect certain versions of the VirtualCentre software, though, and I can't confirm which though it doesn't affect the one we're running here (making testing more difficult).

I suspect the Network problem you report (which I've not had reported before) is related to the same issue.

_________________
Steve Shipway
UNIX Systems, ITSS, University of Auckland, NZ
Woe unto them that rise up early in the morning... -- Isaiah 5:11

Thanks. I guess then you could convert it to a percentage if you knew the theoretical maximum.

At a guess, I wonder if I could use the API to scrape the total number and speed of the CPUs, and sum them up to create a "maximum possible cycles per second" number, for which the CPU tally returned by check_vmware is a percentage (since the unit is probably Hz).

CPU is probably a naive metric anyway, do you know offhand if the API provides system load? That's probably more reliable.

More work than I'm really authorized to do, since I'm only supposed to be evaluating Nagios and my manager is pushing for Hitachi Network Analyzer. Maybe I can do it in my off hours if I get approval.

The CPU as reported from the API is the CPU usage by the guest; this should be a percentage of allocated resources.

Load is not relevant to VMWare, it is a function of the underlying OS so you need to get it there. The Load average is the number of processes in the guest OS scheduled to run at an average time; this is not equivalent to CPU usage since processes can be in run state but not heavy users of CPU.

It is worth noting that CPU usage by guest is not equivalent to CPU usage as reported by the guest OS itself; and the guest OS will LIE about CPU and Memory usage as it gets tricked by the virtualisation layer.

For a guest, the API reports the CPU usage (how many MHz are used), the Ready time (how many cycles it wants to run, but cannot due to VM resources being unavailable), the Wait time (how many MHz it doesnt want and so are used elsewhere). What matters to you is the Ready Time (if >10% you have serious resource problems) and the Usage (if too high then consider adding a vCPU to the guest). Note that Wait time also includes time waiting for virtualised IO.

_________________
Steve Shipway
UNIX Systems, ITSS, University of Auckland, NZ
Woe unto them that rise up early in the morning... -- Isaiah 5:11