Skip to content

Refuse to resize the volume of a running system VM - #14647

Open
bhouse-nexthop wants to merge 2 commits into
apache:4.22from
bhouse-nexthop:fix-resize-running-systemvm-root
Open

bhouse-nexthop wants to merge 2 commits into
apache:4.22from
bhouse-nexthop:fix-resize-running-systemvm-root

Conversation

@bhouse-nexthop

@bhouse-nexthop bhouse-nexthop commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

Description

A size-only resizeVolume of a running system VM's volume, for example a virtual router's ROOT, was accepted. The storage and the database got the new size, but the running VM was never told, so the guest kept the old size until it next started. This PR refuses that resize unless the system VM is Stopped.

Fixes #14645

Why it happens

orchestrateResizeVolume looks the VM up only with _userVmDao.findById(). For a system VM that is null, so:

  • the resize is sent with the instance name none and no host. On KVM the image is grown, but virsh blockresize is never run for the running domain;
  • the checks in the same method are skipped: XenServer resizing only a stopped VM's disk, and on KVM the VM's host being Up and Enabled.

A ROOT resize through a new disk offering, in resizeVolume or changeOfferingForVolume, is already refused unless the VM is Stopped (checkIfVolumeIsRootAndVmIsRunning). The size-only resizeVolume path had no such check.

Fix

validateVolumeResizeWithSize, which both resizeVolume and changeOfferingForVolume call before any size change, now refuses one more case. If the volume is attached to a system VM (VirtualMachine.Type.isUsedBySystem()) that is not Stopped, the call fails with:

Cannot resize volume [ROOT-N] of system VM r-N-VM while it is in state Running. Stop the system VM before resizing its volume.
  • A stopped system VM's volume can still be resized, and the VM sees the new size when it starts.
  • User VMs are unaffected, and so are IOPS-only changes, since the check is in the size-change branch.

I chose to refuse rather than do the resize live. A live resize would need orchestrateResizeVolume to handle system VMs on every hypervisor, through the VM work queue and the host checks, and that needs testing on each one. Refusing matches how other volume operations treat system VMs: snapshots of their volumes are refused, and attach and detach accept only user VMs.

This touches a different part of VolumeApiServiceImpl from #14602, and the two merge cleanly.

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)
  • Build/CI
  • Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Bug Severity

  • BLOCKER
  • Critical
  • Major
  • Minor
  • Trivial

How Has This Been Tested?

The bug was seen on a KVM + Ceph RBD lab while testing #14602. A size-only resizeVolume size=20 on a running router's ROOT succeeded. rbd info and the API showed 20 GiB, but virsh domblkinfo still showed the old size until the router restarted. The fix itself is covered by unit tests; it has not been run on the lab yet.

New tests in VolumeApiServiceImplTest:

  • validateSystemVmIsStoppedForResizeTestNotStoppedSystemVms: a router, console proxy, secondary storage VM and internal LB VM, each in Running, Starting, Stopping and Migrating, are all refused.
  • validateSystemVmIsStoppedForResizeTestStoppedRouter, ...TestRunningUserVm and ...TestDetachedVolume: allowed. A detached volume needs no VM lookup.
  • resizeVolumeSizeOnlyOnRunningRouterIsRefused: a size-only grow through resizeVolume on a running router's volume is refused, and VolumeService.resize is never called. Without the new check, this request gets past validation.

VolumeApiServiceImplTest passes (150 tests). Checkstyle is clean for server.

How did you try to break this feature and the system with this change?

  • I looked for internal callers of resizeVolume / changeDiskOfferingForVolumeInternal that could resize a system VM's volume. There are two, Shared FileSystem and CKS, and both work on user VMs.
  • The check uses isUsedBySystem(), so user, CKS and Shared FileSystem VMs (all Type.User) are unaffected.

A size-only resizeVolume of a running system VM's volume, such as a
virtual router's ROOT, was accepted. orchestrateResizeVolume only looks
the VM up as a user VM, so for a system VM the resize was sent with no
instance name and no host. The storage and the database got the new
size, but the hypervisor was never told and the running guest kept the
old size until its next start. The XenServer and KVM host checks in the
same method were skipped for the same reason.

The resize validation now refuses to change the size of a volume
attached to a system VM that is not Stopped. A stopped system VM's
volume can still be resized. A ROOT resize through a new disk offering
was already refused unless the VM was stopped.

Fixes apache#14645

Signed-off-by: Brad House <bhouse@nexthop.ai>
@codecov

codecov Bot commented Nov 9, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 88.88889% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 18.32%. Comparing base (d02adcd) to head (f4a2abf).
⚠️ Report is 1 commits behind head on 4.22.

Files with missing lines Patch % Lines
...n/java/com/cloud/storage/VolumeApiServiceImpl.java 88.88% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               4.22   #14647      +/-   ##
============================================
- Coverage     18.32%   18.32%   -0.01%     
- Complexity    16644    16653       +9     
============================================
  Files          5939     5939              
  Lines        537768   537796      +28     
  Branches      65929    65933       +4     
============================================
+ Hits          98551    98553       +2     
- Misses       428008   428025      +17     
- Partials      11209    11218       +9     
Flag Coverage Δ
uitests 4.08% <ø> (ø)
unittests 19.41% <88.88%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@bhouse-nexthop

Copy link
Copy Markdown
Collaborator Author

Test report: resizing a virtual router's ROOT volume

Summary: a size-only resizeVolume of a running router's ROOT volume is now refused, and neither the DB nor the RBD image changes. With the router stopped, the same resize is accepted.

Environment

  • A CloudStack 24.0 (main) build with this PR, alongside some of my other open PRs. Three KVM hosts, Ceph RBD primary storage, one management server.
  • The PR targets 4.22; it wasn't run on a 4.22 build. 4.22's orchestrateResizeVolume also looks the VM up as a user VM only, and its validateVolumeResizeWithSize has no system VM check.
  • The router of a throwaway isolated network. Its ROOT volume was about 4.9 GiB on RBD (the system VM template's size).

Steps and results

Step Result
resizeVolume id=<router ROOT> size=5, router Running refused, 432
Size after the refusal unchanged in the DB; rbd info still 4.9 GiB
stopRouter OK
resizeVolume id=<router ROOT> size=5, router Stopped accepted; the DB shows 5 GiB

Refusal message:

Cannot resize volume [ROOT-477] of system VM r-477-VM while it is in state Running. Stop the system VM before resizing its volume.

Before this PR, the same call on a running router succeeded and grew the RBD image, while the router's disk kept its old size until it restarted.

The resized router was never started again. It was destroyed with its network at the end of the test.

Result: pass.

Not covered

  • Starting the router after the stopped resize, to see the guest use the new size.
  • The SSVM, CPVM and internal LB VMs: same check, covered by the unit tests.
  • changeOfferingForVolume, which goes through the same validation.
  • Other hypervisors.

…pedForResize

The check requires the system VM to be Stopped, not merely not Running,
so name it after what it enforces.

Signed-off-by: Brad House <bhouse@nexthop.ai>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants