Dataflow job returns error Invalid TIMESTAMP -292277022657-01-27T00:29:52-08:00

[adityaz7]

Hi All,

I got this error on just one day i.e. 27th January. Is 

{
"why": "400 Bad Request GET https://dataflow.googleapis.com/v1b3/projects/*/locations/*/jobs/2023-09-21_05_27_48-549294681498480...",
"code": 400,
"errors": [
{
"domain": "global",
"message": "Invalid TIMESTAMP -292277022657-01-27T00:29:52-08:00. The valid range is from 0-12-31T16:00:00-08:00 to 9999-12-31T15:59:59.999999999-08:00",
"reason": "outOfRange"
}
],
"message": "Invalid TIMESTAMP -292277022657-01-27T00:29:52-08:00. The valid range is from 0-12-31T16:00:00-08:00 to 9999-12-31T15:59:59.999999999-08:00",
"status": "OUT_OF_RANGE"
}

[ms4446]

This error can occur for a number of reasons, such as:

• A bug in your Dataflow pipeline code.
• A data quality issue in your input data.

If you are getting this error on only one day, it is likely due to a data quality issue in your input data. You should check your input data for any invalid timestamp values.

To troubleshoot this error, you can:

• Check your Dataflow pipeline code for any bugs.
• Examine your input data for any invalid timestamp values.
• Once you've identified and addressed the root cause, consider restarting your Dataflow pipeline.
• If the error persists, contact Google Cloud support.

Additional tips for preventing timestamp errors in Dataflow:

• Ensure your input data is in the correct format. Dataflow expects timestamps to be in the ISO 8601 format.
• If you're working with Dataflow SQL, use the ParseTimestamp function to parse timestamp values from your input data. This function will validate the timestamp values and convert them to the correct format.
• Also, in Dataflow SQL, use the FormatTimestamp function to format timestamp values in your output data, ensuring they are in the correct format for your output destination.