Dataform - Documentation Generation

[fansari]

Hi, 

Does any know how we can generate documentation i.e. our model's documentation? 

Like in DBT, we can generate docs there.

Thanks

[ms4446 · Accepted solution]

Hi @davidregalado25 ,

Here are a few things to check:

1. Check Dataform Compilation:

   • Compile After Changes: Did you compile your Dataform project after updating the descriptions in test_david_ILS_categories.sqlx? Changes won't show up until you do.
   • Review Compilation Logs: Look for any errors in the logs that might be related to descriptions.

2. Permissions:

   • Verify Permissions: Make sure the Dataform service account has permissions to edit table metadata in BigQuery (roles like "BigQuery Data Editor" or "BigQuery Job User").

3. BigQuery Metadata:

   • Allow Time for Updates: BigQuery can take a little while to reflect changes. Give it some time, then check the descriptions again in the BigQuery UI or query INFORMATION_SCHEMA.COLUMNS.

4. Syntax or Formatting Issues:

   • Check Indentation and Typos: Double-check the indentation in your config block (in test_david_ILS_categories.sqlx) and ensure there are no typos in your column names.

[davidregalado25 · Accepted solution]

---

1. Check Dataform Compilation:

   • Compile After Changes: Did you compile your Dataform project after updating the descriptions in test_david_ILS_categories.sqlx? Changes won't show up until you do.
   • Review Compilation Logs: Look for any errors in the logs that might be related to descriptions.

The UI says COMPILED. I'm using the UI, if there are more functionality in the CLI, I think it should be included in the UI too.

---

Permissions:

1. • Verify Permissions: Make sure the Dataform service account has permissions to edit table metadata in BigQuery (roles like "BigQuery Data Editor" or "BigQuery Job User")

1.  

---

It can create the tables. Only the descriptions are missing.

 BigQuery Metadata:

• Allow Time for Updates: BigQuery can take a little while to reflect changes. Give it some time, then check the descriptions again in the BigQuery UI or query INFORMATION_SCHEMA.COLUMNS.

There's anything about descriptions when running

select * from ILS.INFORMATION_SCHEMA.COLUMNS

WHERE table_name = 'test_david_ILS_categories'

1. Syntax or Formatting Issues:

   • Check Indentation and Typos: Double-check the indentation in your config block (in test_david_ILS_categories.sqlx) and ensure there are no typos in your column names.

Is indentation important? I clicked on the FORMAT button. No luck.

--
Best regards
David Regalado
Web | Linkedin | Cloudskillsboost

[ms4446 · Accepted solution]

Hi @davidregalado25 ,

You mentioned that the Dataform service account can create tables, which is a positive sign. However, there might still be a permissions issue specifically related to modifying table metadata. Please double-check that the service account has the "BigQuery Data Editor" role at the dataset or project level without any restrictions that might prevent metadata changes.

BigQuery sometimes delays reflecting metadata changes in both its UI and the INFORMATION_SCHEMA queries. Although it's frustrating, giving it a bit more time might resolve the issue. If, after waiting, the descriptions still don't appear, the problem likely lies elsewhere.

You've done the right thing by using the FORMAT button in Dataform's UI. However, manual changes after formatting can introduce errors. Indentation is crucial in YAML-like configurations, as it helps define the structure. Here's the correct structure for your SQLX file's config block:

config {
  type: "table",
  database: "your_project_id",
  schema: "ILS",
  name: "test_david_ILS_categories",
  description: "A table description here",
  columns: {
    category_id: {
      description: "Description of category_id",
      type: "STRING"
    },
    category_name: {
      description: "Description of category_name",
      type: "STRING"
    }
  }
}

[davidregalado25 · Accepted solution]

I got this error:

Unexpected property "category_id" in columns column descriptor. Supported properties are: ["description","columns","displayName","dimension","aggregator","expression","tags","bigqueryPolicyTags"]

--
Best regards
David Regalado
Web | Linkedin | Cloudskillsboost

[ms4446 · Accepted solution]

Hi @davidregalado25 ,

The error message you received suggests that the way you're attempting to define the column descriptions and types is not recognized by Dataform. Specifically, the properties you should be focusing on within the columns section are those listed in the error message, such as description, displayName, and tags.

You should adjust the configuration to correctly define the column descriptions. Here's how you can structure your table and column descriptions in the SQLX file, focusing on incorporating the supported properties correctly:

config {
    type: "table",
    database: "your_project_id",
    schema: "ILS",
    name: "test_david_ILS_categories",
    description: "A table description here",
    columns: {
        category_id: {
            description: "Description of category_id",
            tags: ["example_tag"] // Note: Remove the type property as it's not supported here
        },
        category_name: {
            description: "Description of category_name",
            tags: ["example_tag"] // Note: Similarly, remove the type property
        }
    }
}

In this corrected format, I've removed the unsupported type property from the column descriptors and ensured that only supported properties are included. If you were attempting to specify data types for these columns directly in the SQLX file, it's important to note that Dataform uses this configuration primarily for documentation and metadata purposes, rather than for defining the schema of the table directly in BigQuery.

Additional Steps

1. Apply the Corrected Configuration: Adjust your SQLX file according to the corrected format and redeploy your Dataform project.
2. Verify Changes in BigQuery: After redeployment, check again in BigQuery to see if the column descriptions are now appearing as expected.
3. Review Dataform Documentation: For further clarification on defining column properties and any additional configuration options, reviewing the Dataform documentation can be very helpful.