This website uses Cookies. Click Accept to agree to our website's cookie use as described in our Privacy Policy. Click Preferences to customize your cookie settings.
Accept
Reject
Preferences
Log in to ask a question
Get hands-on experience with 20+ free Google Cloud products and $300 in free credit for new customers.
Log in to ask a question

DataFlow not able to connect to the JSON file stored in S3

Posted on 01-20-2023 03:19 AM
rajkumarjai1
Bronze 2
Bronze 2
Reply posted on --/--/---- --:-- AM

Getting the below error when DataFlow is running. It is not able to connect to the JSON file stored in S3 for some reason. I have configured aws , credentials, region and installed boto3. Still couldn't find out the exact reason of this failure. Need help on the below error,

raise BeamIOError("Match operation failed", exceptions)
apache_beam.io.filesystem.BeamIOError: Match operation failed with exceptions {'s3://raj-table/600568ae88e017745e/61b081877b2cc3ab772/11675ss.json': BeamIOError("exists() operation failed with exceptions {'s3://raj-table/600568ae88e017745e/61b081877b2cc3ab772/11675ss.json': AssertionError('Missing boto3 requirement')}")} [while running 'ref_AppliedPTransform_ReadFromText-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_7/PairWithRestriction-ptransform-37'] with exceptions None

 

0 0 256
Topic Labels
0 Likes
0 REPLIES 0
Top Solution Authors
View all