During a DR Scenario executed for a client, we encountered a rather classical error, several dismounted datastores that couldn’t be removed from vCenter, because the Storage team was eager to un-present them, without the Systems team to properly dismount them first.
Being unable to remove or dismount the datastores, or them, not able to be removed by a simple vCenter reboot, we decided to remove them from within the vCenter POSTGRES database by using the following steps.
Please note here that prior to altering things, we had verified that a proper configuration backup., a vCenter backup (VM) and a snapshot were properly taken, in order to eliminate possible risks, as altering system components within the vCenter should be always executed under personal responsibility.
First, we needed to connect to vCenter via SSH and enable the shell.
Next, we had to access the psql tool, in order to run some select commands against the database, so we typed the below command:
/opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres
At this point, we had enabled the VCDB prompt, and we were able to execute commands against tables. Typing help there would help you to understand the commands you could use.
Next, we typed the below command, in order to find the specific Datastore’s ID within the database’s tables.
SELECT id FROM vpx_entity WHERE name = 'Your Datastore's name here';
So, this datastore has been granted 2 different IDs within vCenter’s database. This is because it is a replicated datastore that exists (2 different naas) in two different sites.
There is an easy way to verify which of the numbers is the one you need to note down, in order to delete it afterwards.
Do a search for each ID in the specific table (vpx_ds_assignment). This will show you, how many hypervisors have this host mounted on.
In our case, we wanted ID 120, as the DR site has significantly less hypervisors, and we had verified each mount path previously.
Next steps were quite easy to follow. We needed to delete the ID from all relative tables and not only the vpx_entity one (It will prompt you an error message, indicating that the ID exists in some other tables).
So start by executing the commands one by one:
DELETE FROM vpx_ds_assignment WHERE ds_id=120; (where 120 should be your ID)
DELETE FROM vpx_datastore WHERE id=120;
DELETE FROM vpx_vm_ds_space WHERE ds_id=120;
DELETE FROM vpx_entity WHERE id='120';
I’d recommend to do a notepad and write down everything, in order for you to know exactly what have you executed and which IDs have you removed (see below example).
After you have verified that there is no inaccessible datastores, you should perform a vCenter restart.