Cloud Build Operator

[morpheus]

w.r.t. this problem: We have an application that has a dependency DAG. To build the application we need to build the DAG in topological order. Each element in the DAG has its own Git repository with a cloudbuild.yaml file. How can we build the overall application?

one way to solve it is through cloud composer by creating a workflow that has cloud build as its steps. refer this:

create_build_from_repo_body: dict[str, Any] = {
"source": {"repo_source": {"repo_name": GCP_SOURCE_REPOSITORY_NAME, "branch_name": "master"}},
"steps": [{"name": "ubuntu", "args": ["echo", "Hello world"]}],
}

question: we have a complex cloudbuild whose config is stored in a cloudbuild.yaml file. how do we tell airflow or cloud composer to use the cloudbuild.yaml in the source repository instead of duplicating the steps in python code?

[ms4446]

To use the cloudbuild.yaml file in the source repository instead of duplicating the steps in Python code, you can define a Cloud Composer workflow that executes a Cloud Build step with the necessary configuration. Here's how you can do it:

1. Define a Cloud Composer DAG (Directed Acyclic Graph) that represents the dependency order of your application components. Each node in the DAG corresponds to a separate Git repository with its own cloudbuild.yaml file.

2. In the DAG, for each node (repository) that needs to be built, create a Cloud Build step that references the cloudbuild.yaml file in that repository. You can use the gcp_build_operator in the Google Cloud Provider package of Airflow.

   Here's an example of how to create a Cloud Build step in a DAG:

   from airflow import DAG
   from airflow.providers.google.cloud.operators.cloud_build import CloudBuildCreateBuildOperator
   from datetime import datetime

   dag = DAG(
   'build_application',
   schedule_interval=None,
   start_date=datetime(2023, 5, 1),
   )

   build_step = CloudBuildCreateBuildOperator(
   task_id='build_component_a',
   source={
   'repoSource': {
   'repoName': 'YOUR_GIT_REPO_A',
   'branchName': 'master'
   }
   },
   build={
   'steps': [
   {
   'name': 'gcr.io/cloud-builders/gcloud',
   'args': ['builds', 'submit', '--config', 'cloudbuild.yaml', '.']
   }
   ]
   },
   dag=dag
   )

3. Define the dependencies between the nodes in the DAG to represent the order in which they need to be built.

   For example, if component A depends on component B, you can set the dependency as follows:

build_step_a >> build_step_b

This ensures that component B is built before component A.

By defining the workflow in this way, you can leverage the cloudbuild.yaml files in your source repositories without duplicating the steps in Python code. Airflow or Cloud Composer will execute the Cloud Build steps based on the configuration provided, allowing you to build your application in the required topological order.

[morpheus]

thanks. we tried it but get this error:

[2023-05-04, 18:19:24 UTC] {base.py:71} INFO - Using connection ID 'google_cloud_default' for task execution.
[2023-05-04, 18:19:24 UTC] {credentials_provider.py:323} INFO - Getting connection using `google.auth.default()` since no key file is defined for hook.
[2023-05-04, 18:19:24 UTC] {cloud_build.py:164} INFO - Start creating build...
[2023-05-04, 18:19:25 UTC] {taskinstance.py:1853} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/opt/python3.8/lib/python3.8/site-packages/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/opt/python3.8/lib/python3.8/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/opt/python3.8/lib/python3.8/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.NOT_FOUND
	details = "Error determining source provenence"
	debug_error_string = "UNKNOWN:Error received from peer ipv4::443 {created_time:"2023-05-04T18:19:25.111589105+00:00", grpc_status:5, grpc_message:"Error determining source provenence"}"
>

[ms4446]

Based on the error message your are getting it appears there might be an issue with entitlements with the Cloud Build service account. Please verify this account has cloudbuild.builds.create IAM permission.

[morpheus]

thanks. when you say Please verify this account has cloudbuild.builds.create IAM permission. WHAT ACCOUNT ARE WE TALKING ABOUT?

1. we have verified that we can trigger the build in cloud build console ourselves

2. the account that performs the build also has necessary permissions. this account is called PROJECT_ID@cloudbuild.gserviceaccount.com and we have verified that manually triggered builds complete successfully

3. cloud composer is running under a user created SA. composer@PROJECT_ID.iam.gserviceaccount.com. this SA has been given Composer Worker role

4. service-PROJECT_ID@cloudcomposer-accounts.iam.gserviceaccount.com has been given Cloud Composer v2 API Service Agent Extension role

[morpheus]

also we are seeing more than 10k errors in span of a minute from k8s. is this normal?

[ms4446]

Experiencing more than 10,000 errors in a minute from Kubernetes is not normal and indicates an underlying issue. Is it the same error message repeated or multiple messages?