About francisatoyebi

francisatoyebi

Hi All,I am trying to set up my teams' infrastructure in BQ and currently, we are looking to have 2 projects 1 for prod and 1 for dev. and in each of the projects we hope to set up our BQ with a medallion structure. we hope to have the prod to be a c...

francisatoyebi · 04-09-2024

I am currently managing a dataform project with numerous transformations. These transformations are executed through a combination of workflow configuration and release configuration for orchestration, scheduled to run daily, weekly, or monthly, depe...

francisatoyebi · 10-18-2023

Hi all,I have the configuration below in my dataform.json file{ "defaultSchema": "datalake_analytics", "assertionSchema": "dataform_assertions", "warehouse": "bigquery", "defaultDatabase": "cfc-2017-site-login", "defaultLocation": "EU"} the "defaultL...

francisatoyebi · 08-04-2023

I created an incremental Dataform code that when I run as not incremental, it runs so smooth and perfect but when I run it as incremental, I get errors. Please see code below and what could be the cause of the errorconfig { type: "incremental", schem...

francisatoyebi · 07-24-2023

Hi guys,I am trying to create an incremental table for a web and app event tracker but because I have a limit on my Big Query, I run into the error below"Resources exceeded during query execution: The query could not be executed in the allotted memor...

francisatoyebi · 04-09-2024

Hello @novas and @ms4446 ,What does the Dag_ID stand for? is it the same as a tag defined in the config file? So currently I have a dataform project with multiple transformations, I currently run these transformations using the workflow configuration...

francisatoyebi · 10-20-2023

Thanks mate!!!

francisatoyebi · 10-20-2023

So how do I handle this if I want to work with datasets stored in multiple locations in GCP? Cos I noticed that dataform only runs for datasets that are saved in the same location as the default location.

francisatoyebi · 10-19-2023

I have tried that before and when I did, it throws an error that says location is not a parameter.I am currently thinking this feature is not yet live in dataform.

francisatoyebi · 07-25-2023

@ms4446 wrote:config {type: "incremental",bigquery: {partitionBy: "DATE(timestamp)",clusterBy: ["user_id", "event_id"]}}SELECT*FROM${ref("your_source_table")}WHEREDATE(timestamp) > (SELECT MAX(DATE(timestamp)) FROM ${self()}) So I must create the tab...

My Stats

francisatoyebi's Bio

Badges francisatoyebi Earned

Recent Activity

Dataform Development

Dataform<>Airflow Run Using Tags

Dataform Location Error

Dataform Incremental Errors

Dataform Incremental Backfill

Re: Dataform workflow orchestration

Re: Dataform Location Error

Re: Dataform Location Error

Re: Dataform Location Error

Re: Dataform Incremental Backfill