Requested column family not found while exporting data from BigQuery to Bigtable

[piyush_zlai]

Hi folks,

I'm trying to test out some really simple exports from BigQuery to Bigtable and I'm hitting a persistent error while trying to do so.

I have a BigTable that looks like:
gcloud bigtable instances tables create TEST_EXPORT_NEW \
> --instance=my-instance \
> --column-families=cf
Created table [TEST_EXPORT_NEW].

And I'm trying to export using this query:
EXPORT DATA OPTIONS (
  format='CLOUD_BIGTABLE',
  uri="https://bigtable.googleapis.com/projects/canary-443022/instances/my-instance/appProfiles/MY_APP_PROF...",
  bigtable_options="""{
  "columnFamilies": [
  {
    "familyId": "cf",
    "encoding": "BINARY",
    "columns": [
    {
        "qualifierString": "value",
        "fieldName": "value_bytes"
     }
   ]
}
]
}"""
) AS
SELECT
CAST('test' AS BYTES) as rowkey,
CAST('test' AS BYTES) as value_bytes
FROM (SELECT 1)

I seem to be constantly hitting issues:
Error while mutating the row 'test' (projects/canary-443022/instances/my-instance/tables/TEST_EXPORT_NEW) : Requested column family not found.

I've been able to confirm that I can manually write Rows / retrieve rows to the 'cf' via cbt just fine. I tried changing the name of the column family from 'cf' to 'test_cf' but that doesn't help either. 
I have confirmed I've added the IAM roles called out - https://cloud.google.com/bigquery/docs/export-to-bigtable#required_roles. I'm not sure what I'm doing wrong here so any suggestions / ideas would be welcome.

Thanks,
Piyush

[mcbsalceda]

Hi @piyush_zlai,

Welcome to Google Cloud Community!

This error message indicates that the column family cf which you are trying to access does not exist in your Bigtable TEST_EXPORT_NEW. You can use the cbt ls <table-id> to list a table's column families and garbage collection policies. If the column family you are looking for is not in the list, you will need to create it using the cbt createfamily <table-id> <family>.

It is also worth checking the full documentation on Export data to Bigtable to understand its limitations. If the issue persists, reaching out to Google Cloud Support can provide further assistance and clarification.

Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.

[piyush_zlai]

Thanks for getting back @mcbsalceda - I did have the column family created which is why I was puzzled about the error. My table had it as part of creation + I was able to see it using the cbt commands as well. 

What worked though was using this export setting - 

auto_create_column_families=true