LAB BUG - Dataflow job failing

[Abhishek213]

Offloading Financial Mainframe Data into BigQuery and Elastic Search

please fix this lab due to this lab i am unable to get the triviabadge 

@Yugali @carrie 

[omarfaruk016]

what is the update regarding this  @carrie @Yugali  

[AvikDeb]

Same here too

[vj151655]

Here’s a summary of the process for offloading financial mainframe data into BigQuery and ElasticSearch:

1. Data Extraction:

   • Extract data from mainframe systems (e.g., DB2, flat files) using tools like Informatica, IBM DataStage, or Apache Nifi.

2. Data Transformation:

   • Convert data into structured formats suitable for BigQuery (e.g., CSV, JSON, Avro) and ElasticSearch (typically JSON).
   • Use ETL tools like Apache Beam or Google Cloud Dataflow for transformation.

3. Loading Data into BigQuery:

   • Load transformed data into BigQuery using BigQuery Load Command or the Data Transfer Service.
   • Ensure proper schema mapping in BigQuery tables.

4. Loading Data into ElasticSearch:

   • Index the transformed data into ElasticSearch using tools like Logstash, Beats, or the ElasticSearch Bulk API.

5. Security & Compliance:

   • Encrypt data during transfer and follow security standards like GDPR and PCI-DSS.

6. Monitoring & Optimization:

   • Monitor performance in both systems (BigQuery and ElasticSearch), using Query Execution Plans (BigQuery) and Kibana (ElasticSearch) for analysis.

By following these steps, you can efficiently offload and manage financial data in BigQuery and ElasticSearch.

[RAOKS]

There is no problem with the lab, if followed instructions correctly one by one it is working properly.

Only youtube scripts not working as most of them are missing important information.

[RAOKS]

I have checked again and there is no problem in the lab .If instructions are followed, it's working .

[madushaka]

same as here

[madushaka]

Guys I did this just now. successfully completed. Please follow the guide. then you can complete successfully.

[A11an]

Guys did they return you credits for this or you just pay them for start broken labs again and again? Is it only me here thinking about how much it cost?