Rename a Datastream destination table

[lorenh]

I recently got an alert that several events in a Datastream Stream were discarded because streaming to a BQ table partition more than 5 years in the past is not supported. I paused the stream and created non-partitioned BQ tables with the same schema, and then loaded the original tables' data into them. However, now I'm not able to rename the old tables so I can use the original names for the non-partitioned tables and restart the steam. When I attempt an `ALTER TABLE ... RENAME TO` command, I get this error:
```
Invalid value: Cannot rename project:dataset.tablename because it has streaming data. at [1:1]
```
I paused the Stream 3 days ago, and I'm still getting this message. What is the right way to rename a table Datastream has streamed to historically, or alert BQ that there is no more data in the streaming buffer? Do I need to delete the Datastream info columns? I'm wary of creating a new stream and potentially losing the record of progress for what has already been written to BQ from the current Stream. 

[-Rhett]

Hi @lorenh,

The error Cannot rename <dataset.table> because it has streaming data happens due to a known limitation when renaming BigQuery tables which have data streaming into them. To troubleshoot, follow these steps:

• Confirm that your stream follows the known limitations when using BigQuery as a stream destination.
• Pause the stream, complete any pending backfills, and wait for BigQuery to indicate that streaming is not in use.
• It normally takes around 72 hours after the last streaming operation to be able to rename the table, however there are some cases where it may take up to 7 days. The Datastream team is aware of this delay and is actively working on reducing the time needed.
• To confirm if there’s no more data in the stream, the status should change from Draining to Paused.

If these steps didn’t resolve the issue, you can also try the following workaround instead of renaming a table:

• Copy your BigQuery table with your intended table name.
• Verify that the numbers of rows of the new table and the source table are equal (this might also take a few seconds to propagate into the UI)
• Drop your source BigQuery table.

For now, I recommend to +1 yourself to the public tracker issue, subscribe for updates, and add a comment with a summary of the issue, screenshots of your logs, and other errors you might have received. Our team may sometimes ask for more details, clarification, or a follow up. Since this is a developing issue, there isn’t a specific time-frame for its resolution. However, once we've fixed the issue, we'll indicate this on the bug and provide a resolution or workarounds if available.

Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.