KVM: resizing an RBD volume fails with "Storage volume not found" until libvirt refreshes the pool #14622

problem

On KVM, resizing a volume on RBD primary storage fails in the agent if this host's libvirt has not yet cached the image in its view of the pool. A freshly deployed VM's ROOT volume is the usual case:

resizeVolume id=<ROOT> size=12
  -> Failed to resize volume operation of volume UUID: [<uuid>] due to -
     org.libvirt.LibvirtException: Storage volume not found: no storage vol with matching path '<ceph pool>/<uuid>'

resizeVolume with a size or a disk offering, changeOfferingForVolume and scaleVirtualMachine all fail the same way, since they all end in the same ResizeVolumeCommand. A size-only grow of a ROOT volume fails too. The resize works again once something else on that host has made libvirt refresh the pool (or after a virsh pool-refresh by hand).

Cause

LibvirtResizeVolumeCommandWrapper gets the disk with pool.getPhysicalDisk() and then looks the volume up through libvirt with conn.storageVolLookupByPath(path) to call StorageVol.resize().

• Before kvm: look RBD volumes up through librbd instead of refreshing the whole pool #14188, getPhysicalDisk() went through LibvirtStorageAdaptor.getVolume(), which refreshes the libvirt pool on a miss. So by the time the wrapper called storageVolLookupByPath(), the volume was in libvirt's cache.
• kvm: look RBD volumes up through librbd instead of refreshing the whole pool #14188 made getPhysicalDisk() stat RBD images through librbd, on purpose, so that a miss no longer refreshes the whole pool. The wrapper's storageVolLookupByPath() now runs against libvirt's cache cold.
• CloudStack creates RBD volumes through librbd (the clone from the template), not through libvirt, so a new volume is not in libvirt's cache on any host until something refreshes the pool there.

The other libvirt volume lookups in the KVM plugin don't depend on getPhysicalDisk(): deletePhysicalDisk() uses getVolume(), which still refreshes on a miss, and LibvirtMigrateCommandWrapper.deleteLocalVolume() only handles local files.

The unit tests mock Connect.storageVolLookupByPath, so they did not see this.

versions

The 4.22 branch since #14188, and main. Released versions without #14188 are not affected. Found while testing #14602 in a lab: KVM, Ceph RBD primary storage, a running Ubuntu VM with its ROOT on RBD.

The steps to reproduce the bug

1. Deploy a VM on KVM with its ROOT volume on RBD primary storage.
2. Before anything refreshes the libvirt pool on its host (no virsh pool-refresh, no storage stats poll yet), resize its ROOT volume, for example resizeVolume id=<ROOT> size=<larger>.
3. The agent fails with "Storage volume not found: no storage vol with matching path ...".

What to do about it?

In the resize wrapper, when the lookup by path fails with "not found" (VIR_ERR_NO_STORAGE_VOL) on an RBD pool, refresh that pool and look the volume up once more. That is what getVolume() did for this path before #14188, and it only costs a refresh when the volume is not cached. Other pool types are unchanged.