Repository navigation
CKS: open the cluster public IP on VPC tiers when the VPC has the Firewall service - #14670
adestis-mf wants to merge 1 commit into
Conversation
…ewall service Since apache#12706 a VPC offering can provide the Firewall service for public IPs. The VPC virtual router then drops ingress traffic to a public IP that has no firewall rule. CKS only created network ACL items, port forwarding and the API load balancer rule on VPC tiers, so the management server could not reach the nodes (SSH 2222+, API 6443), the control node could not reach its own API through the public IP, and cluster creation failed after the start timeout. When the VPC virtual router provides the Firewall service of the cluster's VPC (the check that sets vpc_firewall_enabled on the VR), CKS now creates the same ingress firewall rules as on isolated networks on create (including etcd nodes), scale and node add/remove. In a VPC the firewall rules are created after the load balancing and port forwarding rules: applying firewall rules on a VPC public IP does not associate the IP with the VR, and if they come first the IP is never associated. Firewall rules on VPC public IPs have no network_id, so per-node rules are looked up by public IP and ports. On delete the rules go with the release of the cluster public IP. Isolated networks, VPCs without the Firewall service and the NSX/Netris paths are unchanged.
|
Congratulations on your first Pull Request and welcome to the Apache CloudStack community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/cloudstack/blob/main/CONTRIBUTING.md)
|
There was a problem hiding this comment.
🟡 Changes recommended
Scaling a firewall-enabled VPC cluster after importing a worker can fail because unreconciled per-node SSH rules conflict with the replacement port range.
1 open issue
What changed in this PR
This PR fixes CKS connectivity on VPC tiers whose virtual router provides the Firewall service, extending public-IP firewall handling across cluster lifecycle operations.
Changes:
- Opens API and node SSH ports after load-balancing and port-forwarding rules establish VPC IP association.
- Updates scaling and node add/remove paths, with shared firewall lookup by public IP for VPC tiers.
- Adds tests for firewall-service detection, rule lookup, and provisioning order.
| File | Description |
|---|---|
| plugins/integrations/kubernetes-service/src/test/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterResourceModifierActionWorkerTest.java | Tests conditional VPC firewall creation and rule ordering. |
| plugins/integrations/kubernetes-service/src/test/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterActionWorkerTest.java | Tests firewall detection and network/public-IP lookups. |
| plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterStartWorker.java | Opens etcd SSH ports after VPC port forwarding. |
| plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterScaleWorker.java | Rebuilds VPC SSH firewall rules during scaling. |
| plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterResourceModifierActionWorker.java | Adds API and SSH firewall rules during VPC setup. |
| plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterRemoveWorker.java | Revokes per-node VPC firewall rules with missing-rule handling. |
| plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterAddWorker.java | Creates and rolls back per-node VPC firewall rules. |
| plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterActionWorker.java | Centralizes firewall DAO injection, detection, and lookup helpers. |
Review effort: Balanced
| } | ||
| if (vpcFirewallEnabled) { | ||
| // After the port forwarding rules, as when the cluster is created | ||
| addFirewallRulesForNodes(publicIp, clusterVMIds.size()); |
There was a problem hiding this comment.
Thanks, the scenario is valid: the imported node's single-port SSH rule overlaps the rebuilt
range, and detectRulesConflict rejects it (same protocol and source CIDR).
Reading the code, isolated networks on main behave the same way: the add-node path creates the
same per-node rule
(KubernetesClusterAddWorker.java#L229),
and the isolated scale path also revokes a single rule before recreating the range
(KubernetesClusterScaleWorker.java#L135-L149).
This PR mirrors that handling for VPC tiers, so the case is not new, it just becomes reachable on
VPC tiers with the Firewall service.
I'd prefer to keep this PR to the missing rules and fix the import-then-scale case for both
network types in #14673, since it also touches removeSshFirewallRule, which
#14165 is changing. If you'd rather have it here, I'll add it to this PR.
Description
This PR fixes CKS clusters on VPC tiers whose VPC uses the Firewall service for public IPs
(added in #12706). In such a VPC the VPC virtual router drops ingress traffic to a public IP
unless a firewall rule allows it. CKS only created network ACL items, port forwarding and the
API load balancer rule on VPC tiers. The management server could not reach the nodes (SSH 2222+,
API 6443), the control node could not reach its own API through the public IP
(
controlPlaneEndpoint), and cluster creation failed after the start timeout.When the VPC virtual router provides the Firewall service of the cluster's VPC, CKS now creates
the same ingress firewall rules as on isolated networks:
setupKubernetesClusterVpcTierRules),and the etcd SSH ports (
setupKubernetesEtcdNetworkRules);(
scaleKubernetesClusterVpcTierRules), as for isolated networks;(
KubernetesClusterAddWorker,KubernetesClusterRemoveWorker).In a VPC the firewall rules are created after the load balancing and port forwarding
rules. Applying firewall rules on a VPC public IP does not associate the IP with the VR
(
IpAddressManagerImpl.applyRulesskips it when the rule has no network). If the firewall rulescome first, the later LB/PF rules do not associate it either and the IP never reaches the VR
(#14669). On isolated networks the order stays as it was.
Firewall rules on VPC public IPs are stored with
vpc_idand withoutnetwork_id, so theper-node lookup now goes by public IP and ports for VPC tiers
(
findIngressFirewallRuleForPorts); isolated networks keep usingfindByNetworkIdAndPorts.On cluster deletion the rules are revoked together with the release of the cluster public IP
(
cleanupIpResources→revokeFirewallRulesForIp), so the destroy worker is unchanged.The check (
isVpcFirewallEnabled) is the one that setsvpc_firewall_enabledon the VR:VpcServiceMapDao.canProviderSupportServiceInVpc(vpcId, Firewall, VPCVirtualRouter). VPCswithout the Firewall service, isolated networks and the NSX/Netris paths behave as before.
FirewallRulesDaois now injected once inKubernetesClusterActionWorkerinstead of separatelyin three subclasses, so that the shared lookup can use it.
This targets
main: the Firewall service for VPC public IPs is not in4.22, and there is no4.23branch. The commit also applies cleanly on top of the 4.23.0.0 tag.Fixes: #14668
Types of changes
Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale
Bug Severity
Screenshots (if appropriate):
How Has This Been Tested?
Unit tests (new):
KubernetesClusterActionWorkerTestcoversisVpcFirewallEnabled,isPublicIpFirewallRequiredandfindIngressFirewallRuleForPortsfor isolated networks and VPCtiers with and without the Firewall service;
KubernetesClusterResourceModifierActionWorkerTestchecks that
setupKubernetesClusterVpcTierRulescreates the firewall rules only when the VPCfirewall is enabled, and only after the LB and SSH port forwarding rules.
Result on JDK 11 / Maven 3.10: BUILD SUCCESS, 141 tests in the kubernetes-service module,
0 failures, 0 errors; checkstyle 0 violations.
Live, on 4.23.0.0 (KVM) without the patch, adding by hand the firewall rules the patch creates
(same VPC with the Firewall service, same tier, v1.35.4, 1 control + 1 worker):
Errorafter 3600 s: firstkubeadm inithad failed, control node never recoversError: IP never configured on the VR (#14669)Runningin under 3 min; scale-up to 2 workersRunningDeleting each cluster released the public IP and left no firewall rule behind. The patched
build itself was not deployed to a live management server.
How did you try to break this feature and the system with this change?
findByNetworkIdAndPorts.patch creates them afterwards; a unit test pins the order.
NullPointerException.0.0.0.0/0as on isolated networks(CKS: make the source CIDR of the auto-provisioned node SSH firewall rules configurable #13970, CKS: add an editable per-cluster management-access CIDR policy #14212);
removeSshFirewallRulestill matches by port, as before. On scale the VPCpath goes through the same
removeSshFirewallRule, so the fix for clusters with separate etcdnodes in kubernetes: prefer the node SSH rule when scaling a cluster with etcd nodes #14165 (CKS: scale-down with separate etcd revokes wrong SSH rules and fails (4.22.1.1) #14159) applies to VPC tiers as well once it is merged forward.