Multiple Database Tables from Oracle in DataFusion

[giulio1]

I want to load multiple tables from an Oracle db using DataFusion, I am using "Multiple Database Tables".

If I use normal Oracle source everything works fine but when I use the Multiple Database I get an error "Output records have not been generated. Please verify your logic or try sending more data.". I think there is an error in my configuration but I can't find it. I set the jdbc string and plugin and I am using Table Allow List leaving all the fields empty. I can not find any example of using Multiple Database to load Oracle, is there an issue with Oracle?

[ms4446 · Accepted solution]

There is no inherent issue with using the "Multiple Database Tables" source to load Oracle data in Cloud Data Fusion. The error message you're encountering suggests that the source isn't generating any data. This could be attributed to several reasons:

• The targeted tables might not exist.
• The tables you're attempting to load might be empty.
• There might be permission restrictions preventing you from reading the tables.
• The JDBC connection string might be incorrect.
• The plugin might not be correctly installed or configured.

To address this, consider the following steps:

1. Table Verification: Ensure the tables you're trying to access exist and contain data.
2. Permission Check: Confirm that you have the necessary permissions to read the tables.
3. JDBC String: Double-check the JDBC connection string. The correct format can be found in Oracle's official documentation.
4. Plugin Configuration: Ensure the plugin is both installed and correctly configured. The Cloud Data Fusion documentation provides guidelines for installing and setting up the Oracle plugin.

Additional troubleshooting steps:

• Preview Mode: Use Cloud Data Fusion's Preview mode to test your source's configuration. This lets you view the data generated by the source without executing the entire pipeline.
• Debug Mode: Activate Debug mode in Cloud Data Fusion to step through your pipeline's execution. This provides insights into logs and potential errors from the source.
• Log Review: Examine the Cloud Data Fusion logs for any reported errors or issues from the source.

[ms4446 · Accepted solution]

There is no inherent issue with using the "Multiple Database Tables" source to load Oracle data in Cloud Data Fusion. The error message you're encountering suggests that the source isn't generating any data. This could be attributed to several reasons:

• The targeted tables might not exist.
• The tables you're attempting to load might be empty.
• There might be permission restrictions preventing you from reading the tables.
• The JDBC connection string might be incorrect.
• The plugin might not be correctly installed or configured.

To address this, consider the following steps:

1. Table Verification: Ensure the tables you're trying to access exist and contain data.
2. Permission Check: Confirm that you have the necessary permissions to read the tables.
3. JDBC String: Double-check the JDBC connection string. The correct format can be found in Oracle's official documentation.
4. Plugin Configuration: Ensure the plugin is both installed and correctly configured. The Cloud Data Fusion documentation provides guidelines for installing and setting up the Oracle plugin.

Additional troubleshooting steps:

• Preview Mode: Use Cloud Data Fusion's Preview mode to test your source's configuration. This lets you view the data generated by the source without executing the entire pipeline.
• Debug Mode: Activate Debug mode in Cloud Data Fusion to step through your pipeline's execution. This provides insights into logs and potential errors from the source.
• Log Review: Examine the Cloud Data Fusion logs for any reported errors or issues from the source.

[giulio1 · Accepted solution]

Enabling "Allow flexible schemas in Output" in my sink "BigQuery multi table" solved the issue.

[gpaspot13]

I'm encountering an issue while attempting to fetch data from multiple tables within an Oracle database using DataFusion's 'Multiple Database Tables' feature. Strangely, when I employ a standard Oracle data source, everything functions as expected. However, as soon as I switch to the 'Multiple Database' approach, an error crops up stating: 'Output records have not been generated. Kindly double-check your logic or attempt to send more data.' I'm inclined to believe that the error might stem from a configuration misstep on my part, though I'm struggling to pinpoint it. In my setup, I've configured the JDBC string and plugin appropriately, and I've opted for a Table Allow List configuration with all fields left empty. My attempts to locate an example of using 'Multiple Database' for loading Oracle databases have been fruitless so far. Could there possibly be an inherent compatibility issue between Oracle and the 'Multiple Database' approach?

[ms4446]

It's improbable that there's an inherent compatibility issue between Oracle and Data Fusion's Multiple Database approach. The error message "Output records have not been generated. Please verify your logic or try sending more data." indicates that the Multiple Database Tables source isn't producing any data, which could stem from various factors.

1. Table Allow List: If the Table Allow List field is left empty, Data Fusion might attempt to load all tables from the database, which might not be your intention. To address this, specify the desired tables by entering their names in the Table Allow List field.

2. Query Field: An empty Query field means Data Fusion will resort to a default query, which might not align with your requirements. Ensure you provide a specific query in the Query field to fetch the desired data.

3. Validation: Utilize the Validate button within the Data Fusion UI to ascertain the syntax correctness of your query.

If, after these checks, the issue persists, it's plausible that there's a configuration mishap with the Multiple Database Tables source. Consider the following steps:

• Review the official documentation for the Multiple Database Tables source to ensure proper configuration.
• As a fresh approach, create a new pipeline and integrate the Multiple Database Tables source.