CKS: scaling up a cluster with imported nodes fails on the SSH firewall rules #14673

problem

When a CKS cluster has nodes added with addNodesToKubernetesCluster, a later scale-up that
increases the number of managed nodes fails while rebuilding the SSH firewall rules, because the
new rule range overlaps the existing rule of an imported node. By the time it fails, the old SSH
rule and the SSH port forwarding rules of the managed nodes are already removed.

From the code:

1. addFirewallRulesForNodes opens one range for the managed nodes, 2222 to 2222+M-1 (M =
   managed non-etcd nodes), and one single-port rule per imported node after that range
   (KubernetesClusterActionWorker.java#L1116-L1146).
   KubernetesClusterAddWorker creates the same single-port rule when a node is imported
   (KubernetesClusterAddWorker.java#L229).
   All rules use the source CIDR 0.0.0.0/0.
2. On scale-up, scaleKubernetesClusterIsolatedNetworkRules revokes a single rule with
   removeSshFirewallRule (the one starting at 2222), removes the port forwarding rules of the
   old range, and calls setupKubernetesClusterIsolatedNetworkRules, which calls
   addFirewallRulesForNodes again
   (KubernetesClusterScaleWorker.java#L128-L149).
   The single-port rules of the imported nodes are not revoked.
3. With one more managed node the new range ends on the port of the first imported node.
   FirewallManagerImpl.detectRulesConflict rejects two firewall rules with the same protocol,
   the same source CIDR and overlapping ports, so addFirewallRulesForNodes fails with "Failed to
   provision firewall rules for SSH access", and the scale-up fails with "unable to update network
   rules"
   (KubernetesClusterScaleWorker.java#L518-L525).
   The new VMs are cleaned up and the cluster goes back to Running (Scaling + OperationFailed
   → Running), but the revoked SSH rule and the removed port forwarding rules of the managed
   nodes are not restored. The cluster looks healthy while the management server can no longer
   reach the managed nodes over SSH, which later operations such as an upgrade need.

Example: 1 control node and 1 worker (rule 2222-2223), one imported node (rule 2224), scale to 2
workers: the new range 2222-2224 conflicts with the rule for 2224.

This affects isolated networks today. With #14670, which adds the same firewall
rules on VPC tiers whose VPC has the Firewall service, it affects those VPC tiers as well. VPC
tiers without the Firewall service have no firewall rules and are not affected by this part.

versions

• Apache CloudStack main (a7be64a), 4.23.0.0 and the 4.22 branch: the code paths above
  are the same in all three
• Isolated network with the virtual router; CKS cluster with at least one node added via
  addNodesToKubernetesCluster

Found by reading the code during the review of #14670, not reproduced on a live system yet.

The steps to reproduce the bug

1. Create a CKS cluster on an isolated network with 1 control node and 1 worker. CKS creates the
   firewall rule TCP 2222-2223 on the source NAT IP.
2. Add a prepared VM with addNodesToKubernetesCluster. CKS creates a firewall rule and a port
   forwarding rule for TCP 2224.
3. Scale the cluster to 2 workers with scaleKubernetesCluster.
4. Expected: the cluster is Running with 2 managed workers plus the imported node. From the code,
   the scale-up fails instead and the cluster is back in Running with 1 worker, but the firewall
   rule for 2222-2223 and the port forwarding rules 2222-2223 are gone.

What to do about it?

When the SSH rules are rebuilt for a new node count, revoke the single-port rules of the imported
nodes together with the range rule before calling addFirewallRulesForNodes, which recreates them
after the new range (and move their port forwarding rules accordingly). A regression test that
imports a node and then scales up would cover it.

removeSshFirewallRule is also being changed in #14165 (scale-down with separate etcd nodes), so
a fix should be coordinated with that PR. Related: #14670 (VPC tiers with the Firewall service).