Permission denied in Google Cloud Scheduler

ZHIHANG · 01-24-2023 07:06 PM

I can't seemed to trigger my google cloud scheduler job. The error shown is that permission denied. I have already enable my service account with the cloud run invoker, cloud scheduler admin and the service usage account ActAs permission already as well.

kolban

I think we'd like to hear more about the scenario. Have you created a scheduled job? Is the issue that the job is not running? What Cloud Logging errors are you finding? How do you know the failure conditions?

ZHIHANG

Yes, I have already created a scheduled job. It is meant to trigger the workflows to execute my dataform script. I found the error in the log explorer tab. The only thing I can see in the log is PERMISSION DENIED ( I will attach the photo below for you to see what I mean here). The error message never shows what permission I am still lacking for the service account. I gave the service account different roles like workflow invoker, cloud run invoker, cloud scheduler job runner, and service account user but it is still not working.

kolban

Looking at the log, my immediate interpretation is that your scheduler job is running and is trying to invoke Google Workflows. The URL/API that appears to be being called is this. The Workflow that appears to being called is dataform-scheduler. To my eyes, it isn't that Cloud Scheduler is failing ... but instead it is the invocation that Cloud Scheduler is making. I am imaging that a "time" has been reached and Cloud Scheduler is saying "Ok ... time to invoke/do something" and is trying to call Workflows. Scheduler calls Workflows and Workflows says "Nope! You are not authorized". Looking here I get the impression that we need workflows.executions.create permission and that is supplied by the role Workflows Invoker. Now, in your original post, you said that you had already associated that role with the service account that is associated with the scheduled job. However, I think I now want to look in depth at that ... let's super validate that the Scheduled Job is INDEED running as that service account AND that the service account does indeed have the role. I'm thinking that we probably want to post a redacted version of:

1. The JSON format of the log record that you showed in your screen shot

2. The output of the gcloud command that logs the configuration of your scheduler job ... maybe gcloud scheduler jobs describe.

ZHIHANG

The service account can be used to run the workflow with no problem when I tried running it manually. I will attach the JSON format of the log record and the configuration of the scheduler job below.

1. Log Record

{
  "insertId": {inject_id},
  "jsonPayload": {
    "status": "PERMISSION_DENIED",
    "targetType": "HTTP",
    "url": "https://workflowexecutions.googleapis.com/v1/projects/{project_name}/locations/{project location}/workflows/{workflows_name}/executions",
    "@type": "type.googleapis.com/google.cloud.scheduler.logging.AttemptFinished",
    "jobName": "projects/{project_name}/locations/{project location}/jobs/Test"
  },
  "httpRequest": {},
  "resource": {
    "type": "cloud_scheduler_job",
    "labels": {
      "job_id": {job_id},
      "location": {location_name},
      "project_id": {project_id}
    }
  },
  "timestamp": "2023-01-27T03:58:52.714696558Z",
  "severity": "ERROR",
  "logName": "projects/{project_id}/logs/cloudscheduler.googleapis.com%2Fexecutions",
  "receiveTimestamp": "2023-01-27T03:58:52.714696558Z"
}

2. Configuration of the scheduler job

Since I did not enable the google cloud shell, I will just type out the configuration here for your reference

# Configure the execution

### Target type
-> HTTP 

### URL
-> https://workflowexecutions.googleapis.com/v1/projects/{project_id}/locations/{project_location}/workflows/{workflow_name}/executions

### HTTP method
-> POST

###HTTP headers
-> Name 1 = Content-Type, Value 1 = application/octet-stream
-> Name 2 = User-Agent, Value 2 = Google-Cloud-Scheduler

### Body
{"argument":"{}","callLogLevel":"LOG_ALL_CALLS"}

### Auth header
-> Add OAuth token

### Service account 
-> {service account that I created for this use case} 

### Scope
-> https://www.googleapis.com/auth/cloud-platform

kolban

Thank you for the quality posts ... VERY appreciated. I'm interpreting from your response that the Cloud Scheduler job is indeed trying to kick off a Workflow job (as opposed to calling some other REST based service). Is the Cloud Scheduler Job defined in the same Google Cloud project as the Workflow job? A notion going through my mind is as follows:

Imagine TWO GCP projects ... call them A and B. In A we define a service account and a Scheduler Job and in B we create a Workflow definition. The scheduler job in A points to the Workflow definition in B. If we have now used IAM to associate the role Workflow Invoker ... in which project did we make that IAM role assignment? My mind is saying that it should be in project B ... the same project that owns the Workflow. However, I'm also thinking that Google Cloud would simply "allow" us to define the role assignment in project A without objection but when we run, we would get the error described here. So ... my next question to you is to drill down and describe WHERE and HOW you assigned the Workflow Invoker role. I also heard you say that you invoked the Workflow manually using the service account identity. Could you describe how you did that? I want to virtually look over your shoulder and double check your workings (to the best of my ability).

ZHIHANG

I have just double checked, both the scheduler job and the workflow are created in the same project. As for the workflow invoker role, I assign it to the service account using the IAM GUI. I believe that the service account is also created in the same project. Where as for the Google Workflows, I navigated to the Google Workflows GUI and executed the particular workflow. It ran without any issue.

GopiKiran

I am also facing same issue, when I execute manually Workflows everything fine.

But when I execute using cloud scheduler , it says

{
  "insertId": "1eg3jlf",
  "jsonPayload": {
    "status": "PERMISSION_DENIED",
    "targetType": "HTTP",
    "@type": "type.googleapis.com/google.cloud.scheduler.logging.AttemptFinished",
    "jobName": "projects/project/locations/europe-west3/jobs/standard_",
    "url": "https://workflowexecutions.googleapis.com/v1/projects/project/locations/europe- 
 west4/workflows/standard/executions"
  },
  "httpRequest": {},
  "resource": {
    "type": "cloud_scheduler_job",
    "labels": {
      "location": "europe-west3",
      "job_id": "standard",
      "project_id": "project"
    }
  },
  "timestamp": "2023-02-07",
  "severity": "ERROR",
  "logName": "projects/<project>/logs/cloudscheduler.googleapis.com%2Fexecutions",
  "receiveTimestamp": "2023-02-07"
}

kolban

When you submit a request manually, how are you doing that? Are you using gcloud? If yes, then the request is submitted as your current gcloud authenticated identity. When you run through Cloud Scheduler, what identity are you asking your job to run as? How does that identity differ from the one you manually ran as.

GopiKiran

Bellow is the screen shot for manual workflow trigger

If I run this workflow using cloud scheduler I am getting above mentioned error.

Note : I am using service account which has workflow admin and scheduler admin access

kolban

From your last post, I think I sense that you are manually testing the Workflow by running it from the console. This means you are running it AS your currently logged in Google Identity. Since it runs, that tells me that your current Google Identity has permissions to run the Workflow. All good. I am also sensing that when Cloud Scheduler tries to run the workflow, it fails because it is not authorized. This is likely caused by the fact that when Cloud Scheduler calls the Workflow engine to start a workflow job, the identity submitted to run the workflow is NOT your console login but is instead an identity which does NOT have permissions to run the Workflow. Let me suggest that you read the following article

Use authentication with HTTP targets

This talks about how to setup Cloud Scheduler such that when scheduler calls a service provider (eg. Workflow) it passes an OAuth identity. That identity should be granted authority to run the workflow.

GopiKiran

I do not know what I did , I disabled and enabled cloud scheduler then it is working now .

Just for INFO role bellow role deprecated

kolban

Awesome news!!! I think if we look at the header of the screen shot you posted it says "This is necessary ONLY if you enabled prior to 2019" ... since that isn't true for you, then no need to perform that step.

amurmann

I just also wasted many hours on this and it turns out disabling and enabling the scheduler service fixed it. This is really unacceptable for a major IaaS!

IMO, I shouldn't have to care about any of this in the first place. Cloud Build triggers should just have a schedule option that takes a cron setting. I shouldn't have to worry about auth header or any such stuff. Those are implementation details being leaked that are below the value line for users trying to just run on a schedule.

lahpdamo

Disabling and enabling the cloud scheduler also worked for us.
Thank you!

anthonydmays

Ran into this problem as well scheduling a workflow. Updated the service account with the Workflow Invoker role didn't fix the problem immediately. Then I paused/resumed the scheduler job and the problem went away.

ZHIHANG

Maybe you can try to reenable the google workflows api and google cloud api again. IT works for me for some reason🤣

mohitkale

Hi, I am facing similar issue as mentioned above. The below is from the Error Log

ERROR 2023-03-01T00:10:00.279780163Z {"@type":"type.googleapis.com/google.cloud.scheduler.logging.AttemptFinished", "jobName":"projects/<project name>/locations/us-central1/jobs/Trigger_CopyCaseImage", "status":"PERMISSION_DENIED", "targetType":"HTTP", "url":"https://storagetransfer.googleapis.com/v1/transferJobs/<job name>:run?alt=json"}
{
"insertId": "10wui36f7wmjm1",
"jsonPayload": {
"status": "PERMISSION_DENIED",
"url": "https://storagetransfer.googleapis.com/v1/transferJobs/<Job Name>:run?alt=json",
"jobName": "projects/<project name>/locations/us-central1/jobs/Trigger_CopyCaseImage",
"targetType": "HTTP",
"@type": "type.googleapis.com/google.cloud.scheduler.logging.AttemptFinished"
},
"httpRequest": {
"status": 403
},
"resource": {
"type": "cloud_scheduler_job",
"labels": {
"location": "us-central1",
"project_id": "<project name>",
"job_id": "Trigger_CopyCaseImage"
}
},
"timestamp": "2023-03-01T00:10:00.279780163Z",
"severity": "ERROR",
"logName": "projects/<project name>/logs/cloudscheduler.googleapis.com%2Fexecutions",
"receiveTimestamp": "2023-03-01T00:10:00.279780163Z"
}

kolban

When your Cloud Scheduler job runs, I think we will find that it runs with the identity of a Service Account. In your environment, your Service Account will have been granted access to a set of Google Cloud IAM roles. First thing I'd look at is "What is the set of roles granted to the service account?". Following that, we want to look at what is the request you are submitting? It looks like a request to run a Cloud Storage Transfer Service job. Now we need to ask ourselves "What roles are required to run such a job?". From there, it becomes a case of pairing the two together. Marry "What roles does my service account posses?" against "What roles does my service account require?" ... and match the two.

See also:
https://cloud.google.com/storage-transfer/docs/iam-cloud

mohitkale

Hi Kolban - i did setup an appropriate data transfer service account, under the same project with permissions like including Cloud Run invoker, workflows invoker, cloud scheduler job runner, cloud scheduler service agent. But still not go.

kolban

Howdy mohitkale,

Lets take another approach. The request I believe your are running is Method: transferJobs.run. Google is coming back and saying "Permission Denied". Now we need to drill into this. My mind is taking me to two possible thoughts:

1. When you invoke the service, you aren't authorized to run that command

or

2. When the command runs, based on its nature (transferring data), the job is being denied accessing either the source or destination.

For (1), let's take a list of the roles you have given your service account and see if it contains a role that includes the ability to run "Storage Transfer Service - run" which I believe will need to include the permission "storagetransfer.jobs.run".

For (2), lets "become" the service account at the Cloud Shell command line and execute

gcloud transfer jobs run <NAME>

from the command line. If THAT fails, then we need to fix that first before trying to invoke it at a scheduled rate.

ChrisBI

Hi Kolban,
I have an similar case and can´t find a solution. I try to load a CSV from GCS.
The Service Account is "Worker" already. But it wont work...

{
httpRequest: {1}
insertId: "gxg43mfdn3r6i"
jsonPayload: {
@type: "type.googleapis.com/google.cloud.scheduler.logging.AttemptFinished"
jobName: "projects/bi-core-data-warehouse/locations/europe-west3/jobs/adobe_analytics_traffic_exports_bi001"
status: "PERMISSION_DENIED"
targetType: "HTTP"
url: "https://europe-west3-bi-core-data-warehouse.cloudfunctions.net/adobe_analytics_traffic_exports_bi001_getZipFile"
}
logName: "projects/bi-core-data-warehouse/logs/cloudscheduler.googleapis.com%2Fexecutions"
receiveTimestamp: "2023-08-22T10:15:16.248595256Z"
resource: {2}
severity: "ERROR"
timestamp: "2023-08-22T10:15:16.248595256Z"
}

mahendren

For me,

service-{project.number}@@gcp-sa-cloudscheduler.iam.gserviceaccount.com
added with the below role fixes the issue. Found this while scratching my head on Google documentation.

roles/cloudscheduler.serviceAgent

burakuyar

Disabling and re-enabling the Cloud Scheduler service just worked for me, as well.