Home > Case Post Mortems > Patching or Upgrading ESXi fails with “insufficient memory” error

Patching or Upgrading ESXi fails with “insufficient memory” error

A couple of days ago I helped out a junior Engineer who had a customer on the line whose hosts could not be scanned, patched or upgraded using Update Manager 5.5. They simply would get the following error on scanning the hosts.

Capture

The same could be seen in the recent tasks and events.

Capture2

 

On trying to simply scan and patch the host against the default VMware delivered baslines we got the following.

Capture3

 

The customer had already uploaded a log bundle of the ESXi and we could not see anything suspicious in the /var/log/esxupdate.log. But the vua.log had an important hint.

Capture4

 

INFO:root:Running /sbin/esxcli system visorfs ramdisk add -M 332 -m 332 -n upgradescratch -t /upgrade_scratch -p 01777

We can see that root tries to do something, specifically creating and mounting a new ram disk. But why does that fail? The customer also could not connect to the host using putty with “Access denied” errors. In the end the solution was rather simple but seems not to be very well documented as a caveat of one of the major security features of ESXi.

Capture5

At first I thought this might have been an issue with permissions on the host for vpxuser not having full root rights as it is officially documented. But taking a more careful look at the log messages Update Manager does not seem to be using the vpxuser at all but still tries to access the host using “root” which indeed is blocked from logging in when lockdown mode is enabled.

Disabling lockdown mode immediately solved the issue for the customer.

 

 

Categories: Case Post Mortems Tags: