I am trying to transfer data from on premise Postgres dbs into big query. This works as expected for databases with standard naming conventions. However I have a number of dbs with hyphens in the name app-customer for example. I have tried to escape the db name with "" '' /s but not joy. Is there anything that can be done rather than renaming all my dbs?
For databases with hyphens in the name like app-customer, instead of trying to query them directly, use standard PostgreSQL tools such as the COPY command or pg_dump, which correctly handle quoted identifiers.
Export to a neutral format (CSV or Avro):
SQL (inside psql):
-- Export table to stdout
COPY your_schema."your-table" TO STDOUT (FORMAT CSV, HEADER);
-- Or use \copy to write directly to a local file
\copy your_schema."your-table" TO 'local_file.csv' WITH (FORMAT CSV, HEADER);
Bash:
# Export schema or data using pg_dump
pg_dump -h your_host -U your_user -d "app-customer" -t 'your_schema."your-table"' > backup.sql
# Or dump to CSV using COPY/\copy
Then:
Upload the exported files to GCS using gsutil or a client library.
Load the data into BigQuery using the web UI, bq, CLI, or a script.
BigQuery handles ingestion from GCS very well. This avoids the need to rename your databases and keeps your current naming intact, even though it adds a step to the process.