Service Account fails to create with non-nil error with nil state error

[guineveresaenger]

Describe what happened

The service account resource occasionally fails to create with non-nil error with nil state. The account is actually created so further attempts to re-create it cause conflicts. This was reported by users and also affects our tests.

I've come across this particular flake a few times now:
#2683
#2530

but those are just the two issues outlining the exact failure. There could be more where the error wasn't noted. We should investigate.

Sample program

See run output in those links.

Log output

No response

Affected Resource(s)

No response

Output of pulumi about

n/a

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[wvanderdeijl]

We suffer from this on an almost daily basis since a couple of weeks.
The first attempt to create a service account frequently fails with:

gcp:serviceAccount:Account (*****):
      error: expected non-nil error with nil state during Create of urn:pulumi:***

A retry of the pulumi up than fails with:

    gcp:serviceAccount:Account (*****):
      error: 1 error occurred:
      	* Error creating service account: googleapi: Error 409: Service account sa-**** already exists within project projects/*****.
      Details:
      [
        {
          "@type": "type.googleapis.com/google.rpc.ResourceInfo",
          "resourceName": "projects/*****serviceAccounts/*****@*****.iam.gserviceaccount.com"
        }
      ]
      , alreadyExists

Only remedy we have is to manually delete the service account in GCP and then do a pulumi retry.

It feels like this just started happening one day without us upgrading the pulumi libraries. So it might be related to a server-side change at GCP.

[VenelinMartinov]

Thanks for reporting @wvanderdeijl. Are you also running this in tests or are you seeing these errors in production workflows? I strongly suspect this is indeed GCP API doing something but we might be able to mitigate it.

[wvanderdeijl]

This is in production workflows. We do multiple deployments a day. Not all run into this issue, but a significant percentage does.

[wvanderdeijl]

This might be version related. I tried with the following simple pulumi (clean) project:

import * as gcp from "@pulumi/gcp";

const prefix = 'a';
for (let i =0; i<=75; i++ ) {
    new gcp.serviceaccount.Account(`${prefix}account${i}`, { accountId: `${prefix}account${i}` });
}

With the freshly installed npm install @pulumi/[email protected] @pulumi/[email protected] I could run this script without issues and update the prefix from a, to b, to c, so effectively running it three times without issues.

But when downgrading to npm install @pulumi/[email protected] @pulumi/[email protected] it failed on the first attempt with 6 service accounts failing with this error.

Not sure if this is concidence. I haven't tested any of the intermediate versions.

[VenelinMartinov]

Interesting, thank you for reporting back @wvanderdeijl. It would be great if this is indeed fixed!