Get hands-on experience with 20+ free Google Cloud products and $300 in free credit for new customers.

GCP Workflows - How to convert JSON to NDJSON/JSONL

We are developing a workflow that will store the results obtained from a web API into files in a bucket. 

The workflow ends up generating a JSON with this format:
[{entry},{entry},{entry}...]

Each entry has multiple keys with corresponding values.

Here is the challenge, we want to load these JSON files into BigQuery and it does not like the format, it wants a NDJSON instead, something like this should be the format:

  • {entry}
  • {entry}
  • {entry}

The question is: How can I convert my JSON to NDJSON in a Workflow? Is it even possible?

If its not, can we call a Function, send the //bucket/file and have the function convert it to NDJSON? How would it work?

0 2 624
2 REPLIES 2