Re: how I can get the last compilation result id d...

chawki · 11-26-2023 09:18 AM

Hello,

We are migrating from legacy dataform to GCP dataform. Our existing scheduling (in Legacy dataform) is based on call rest api app.dataform.co, so we want to use the same approach for GCP dataform.

we created a releaseConfig per environnement : releaseConfig dev will use dev git branch ,
releaseConfig int will use git int branch and releaseConfig prd will use prd git branch.

All these release config are not automatically scheduled

Then we developed a git CI that will launch Compilation of releaseConfig when new commit is added in dev, int and prd branch

The problem that we can't get the last compilation_result_id when we execute this

GET https://dataform.googleapis.com/v1beta1/projects/project_id/locations/region/repositories/repo-name/compilationResults

it returns all the compilationResults but without release time so we can't know witch compilation result we must use and witch one is the last ..
when we lanch this one:

GET https://dataform.googleapis.com/v1beta1/projects/project_id/locations/region/repositories/repo-name/releaseConfigs/releaseConfig-name

it returns the initial releaseCompilationResult or the releaseCompilationResult that we updated in the UI, but not the one that we updated with our Git CI
when we schedule the compilation it returns all recentScheduledReleaseRecords , how I can get that but without scheduling compilation because we don't want to schedule the compilation...

ms4446

You're encountering a challenge in identifying the latest compilation result ID when executing the GET request to the Dataform API, primarily due to the lack of inherent sorting by release time in the API response.

Proposed Solution:

Retrieve Compilation Results: Execute the GET request to fetch the list of compilation results. Ensure robust error handling to manage any issues during the API call.
Parse and Sort Results: Extract timestamps or other relevant data from the API response. Implement an efficient sorting algorithm to organize the results in descending order based on their creation or update time.
Identify the Latest Result: The first element in the sorted list should represent the latest compilation result ID.

By incorporating these strategies, you can effectively identify the latest compilation result in the Dataform API

chawki

Thanks for you response , the data retrieved from the API does not contain any timestamp or date , look at an example of response when calling the api :

        {
            "name": "projects/project-name/locations/region/repositories/repo-name/compilationResults/compilation_result_id",
            "codeCompilationConfig": {
                "defaultDatabase": "defaultDatabase",
                "defaultSchema": "defaultSchema",
                "assertionSchema": "assertionSchema",
                "vars": {
                    "var1": "var1-dev",
                    "var2": "var2-dev",
                    "var3": "var3-dev",
                    "var4": "var4-dev",
                    "var5": "var5-dev",
                    "var6": "var6-dev"
                },
                "defaultLocation": "region"
            },
            "dataformCoreVersion": "2.7.0",
            "releaseConfig": "projects/project-name/locations/region/repositories/repo-name//releaseConfigName",
            "resolvedGitCommitSha": "87fb9921a51838d3ee038d3ee8dfb992be"
        },

we don't found any release time...

ms4446

Yes, the Dataform API response does not include timestamps, presenting a challenge in identifying the latest compilation result. To address this, consider the following approaches:

External Timestamp Tracking: Implement a system to track compilation timestamps externally. This could involve a database or a file system that logs timestamps whenever a compilation occurs. Ensure this system is robust, integrates seamlessly with your workflow, and is updated in real-time to maintain accuracy.
Utilizing resolvedGitCommitSha: Leverage the resolvedGitCommitSha property from the API response. This requires a disciplined approach to Git commits, ensuring that the commit history accurately reflects the sequence of compilations. You can then correlate these Git commit hashes with the compilation results to determine their order.
Automation and Integration: Automate the process of updating your timestamp records or Git commit tracking. This can be integrated into your CI/CD pipeline, reducing manual effort and minimizing errors.
Error Handling and Synchronization: Implement comprehensive error handling strategies, especially for scenarios where your external tracking system might fail or become desynchronized with the Dataform API data.
Scalability and Maintenance: Consider the scalability of your chosen approach, especially if your project size and frequency of compilations are expected to grow. Regular maintenance of your tracking system will be crucial to ensure its effectiveness over time.

KasperL

To me this looks like an oversight not being able to fetch the latest compilation in a standardized way. Is it possible to request the functionality somehow?

ms4446

You might want to file a feature request through official Google Cloud Support channels. Find the appropriate place to raise a feature request within your Google Cloud console or through their support documentation (https://cloud.google.com/support/docs/issue-trackers

how I can get the last compilation result id dataform?