Pulumi SDK requires master_ipv4_cidr_block for a private GKE (1.29+) cluster

[rmishgoog]

Describe what happened

Creating a private GKE cluster (enable_private_nodes as true) should not necessitate master_ipv4_cidr_block.

Starting GKE 1.29, Google switched to Private Service Connect (PSC) over VPC peering for establishing the connectivity between the worker nodes and the Google managed control plane. The distinction of public vs. private clusters as it relates to the controlplane connectivity has been eliminated and it is always PSC based, regardless of nodes being assigned a public IP address or not. Refer to the below links:

https://cloud.google.com/blog/products/containers-kubernetes/understanding-gkes-new-control-plane-connectivity?e=48754805

https://cloud.google.com/kubernetes-engine/docs/concepts/private-service-connect#architecture

Pulumi (using Go SDK) seems to be mandating that one must provide an IPV4 block for provisioning the hosted controplane infrastructure.

Sample program

func StandardCluster(ctx *pulumi.Context, clusterConfig *StandardClusterConfig) error {

primary, err := container.NewCluster(ctx, clusterConfig.Name, &container.ClusterArgs{
	Name:                  pulumi.String(clusterConfig.Name),
	Location:              pulumi.String(clusterConfig.Location),
	Project:               pulumi.String(clusterConfig.Project),
	RemoveDefaultNodePool: pulumi.Bool(true),
	DeletionProtection:    pulumi.Bool(false),
	InitialNodeCount:      pulumi.Int(1),
	Network:               pulumi.String(clusterConfig.Network),
	Subnetwork:            pulumi.String(clusterConfig.Subnetwork),
	ReleaseChannel: &container.ClusterReleaseChannelArgs{
		Channel: pulumi.String(clusterConfig.Channel),
	},
	PrivateClusterConfig: &container.ClusterPrivateClusterConfigArgs{
		EnablePrivateNodes:    pulumi.Bool(true), // This configuration is not exposed, all the nodes must be private.
		EnablePrivateEndpoint: pulumi.Bool(true), // This configuration is not exposed, all master endpoinst must be private.
		MasterGlobalAccessConfig: container.ClusterPrivateClusterConfigMasterGlobalAccessConfigArgs{
			Enabled: pulumi.Bool(false),
		},
	},
	MasterAuthorizedNetworksConfig: &container.ClusterMasterAuthorizedNetworksConfigArgs{},
	IpAllocationPolicy: &container.ClusterIpAllocationPolicyArgs{
		ClusterSecondaryRangeName:  pulumi.String(clusterConfig.PodRange),
		ServicesSecondaryRangeName: pulumi.String(clusterConfig.ServiceRange),
	},
})

if err != nil {
	return err
}

for _, np := range clusterConfig.NodePools {

	_, err = container.NewNodePool(ctx, np.Name, &container.NodePoolArgs{
		Name:      pulumi.String(np.Name),
		Cluster:   primary.Name,
		NodeCount: pulumi.Int(np.NodeCount),
		NodeConfig: &container.NodePoolNodeConfigArgs{
			Preemptible: pulumi.Bool(np.Preemptible),
			DiskSizeGb:  pulumi.Int(np.DiskSizeGb),
			MachineType: pulumi.String(np.MachineType),
			ShieldedInstanceConfig: &container.NodePoolNodeConfigShieldedInstanceConfigArgs{
				EnableIntegrityMonitoring: pulumi.Bool(true),
				EnableSecureBoot:          pulumi.Bool(true),
			},
			Labels:         pulumi.ToStringMap(np.Labels),
			ServiceAccount: pulumi.String(clusterConfig.ServiceAccount),
			OauthScopes: pulumi.StringArray{
				pulumi.String("https://www.googleapis.com/auth/cloud-platform"),
			},
		},
	})
	if err != nil {
		return err
	}
}

return nil

}

Log output

Updating (gke-nam-alto-core-std-dev):
Type Name Status Info

• pulumi:pulumi:Stack cloud-platforms-clusters-gke-nam-alto-core-std-dev creating failed (5s) 1 error
• └─ gcp:container:Cluster nam-central1-dev-std-01 creating failed 2 errors

Diagnostics:
pulumi:pulumi:Stack (cloud-platforms-clusters-gke-nam-alto-core-std-dev):
error: update failed

gcp:container:Cluster (nam-central1-dev-std-01):
error: sdk-v2/provider2.go:385: sdk.helper_schema: master_ipv4_cidr_block must be set if enable_private_nodes is true: [email protected]
error: 1 error occurred:
* master_ipv4_cidr_block must be set if enable_private_nodes is true

Affected Resource(s)

No response

Output of pulumi about

CLI
Version 3.142.0
Go Version go1.23.3
Go Compiler gc

Plugins
KIND NAME VERSION
resource gcp 7.38.0
language go unknown

Host
OS debian
Version 11.7
Arch x86_64

This project is written in go: executable='/usr/local/go/bin/go' version='go version go1.23.3 linux/amd64'

Current Stack: organization/cloud-platforms-clusters/gke-nam-alto-core-std-dev

TYPE URN
pulumi:pulumi:Stack urn:pulumi:gke-nam-alto-core-std-dev::cloud-platforms-clusters::pulumi:pulumi:Stack::cloud-platforms-clusters-gke-nam-alto-core-std-dev
pulumi:providers:gcp urn:pulumi:gke-nam-alto-core-std-dev::cloud-platforms-clusters::pulumi:providers:gcp::default_7_38_0

Found no pending operations associated with gke-nam-alto-core-std-dev

Backend
Name penguin
URL gs://foundations-pulumi-back-end-kubernetes-clusters
User ******
Organizations
Token type personal

Dependencies:
NAME VERSION
github.com/pulumi/pulumi-gcp/sdk/v7 v7.38.0
github.com/pulumi/pulumi/sdk/v3 v3.137.0

Pulumi locates its logs in /tmp by default

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[rmishgoog]

Additional note: This actually works for a GKE Autopilot cluster created on version 1.29+ but not for a standard GKE cluster. I have not tried using Terraform but it's likely that this behavior is imposed via the underlying provider.

https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/container_cluster#nested_private_cluster_config

[guineveresaenger]

Hi @rmishgoog, thank you for filing this issue.

I'm seeing from your output that you seem to be on v7 of this provider, which is over a year out of date.
I'm fairly certain that this issue has recently been addressed in the upstream Terraform provider in hashicorp/terraform-provider-google#20191.
Could you try again with the most recent version of this provider and see if your issue persists?

Thank you!

[rmishgoog]

Hi @rmishgoog, thank you for filing this issue.

I'm seeing from your output that you seem to be on v7 of this provider, which is over a year out of date. I'm fairly certain that this issue has recently been addressed in the upstream Terraform provider in hashicorp/terraform-provider-google#20191. Could you try again with the most recent version of this provider and see if your issue persists?

Thank you!

It indeed seems to work with v8! Thank you so much. Closing the issue.