I had the same issue, with that exact error that you posted in the screenshot (Windows 2008 R2). The 'cause 'was the system variable HQ_JAVA_HOME being set to a bogus directory from a previous Hyperic agent installation. I deleted this system variable, then installed the Hyperic agent whch recreated this variable with the correct value. The installation finished successfully and the agent immediately checked in to the Hyperic Server.
# HQ version 5.8.4, upgrading agents from version 5.8.0
