Application Integration | Best Practices

[melfeqy03]

Hello

Is there any best practice guide for application integration development  for examples like below

• Using Data Mapper Task  over Java script Task for mapping if applicable
•  Using parallel processing 
• async logging over sync fro performance

[desireejoy]

Hi @melfeqy03 ,

Welcome to Google Cloud Community!

Here are some of the best practices for application integration development specifically focusing on data mapping, parallelism, and asynchronous logging.

1. Data Mapping

Using the Data Mapping task can be a powerful way to transform and pass key variables to various tasks in your integration. Here are a few tips to keep in mind as you build your integration:

• Mappings are run in sequence from top to bottom. That is, if input variable A is mapped to an output variable B in the first row, the variable B is available for mapping in the subsequent rows.
• In each row, the data type of the Input row must match the data type of the Output row. To cast between types, use transformation functions such as TO_STRING and TO_INT.
• There are no limitations on the length of transformation chaining. However, debugging large chained transformations can be difficult. We recommend keeping input transformations readable and splitting complex transformations into multiple mappings.
• If a mapping requires a fallback value, set a fallback value for the mapping. If you do not provide a fallback value, the mapping returns an error when the input value or transformation returns null.
• When deleting a variable, make sure to delete any mapping that contains it.

For information about the usage limits that apply to the Data Mapping task, see Usage limits.

2. Parallel Processing

We recommend the following best practices when using the For Each Parallel task:

• Set your cohort size based on any quota limitations imposed by your sub-integrations.
• Use a SYNC execution strategy when you want to collect the output of the sub-integration for later use.
• Use an ASYNC execution strategy when your sub-integration might run longer than 2 minutes.

For information about the usage limits that apply to the For Each Parallel task, see Usage limits.

Considerations

In general, do not use a SYNC execution strategy if your sub-integration takes longer than 2 minutes to run or you plan to use this task in conjunction with naturally asynchronous tasks like the Approval task.

3. Asynchronous Logging (Over Synchronous)

Why Asynchronous Logging:

• Performance Improvement: Synchronous logging can block the main execution thread, especially when dealing with heavy logging volumes. Async logging offloads logging to a separate thread, minimizing performance impact.
• Increased Responsiveness: Your application remains responsive, even during logging operations.

How to Implement:

• Logging Frameworks: Most popular logging frameworks like Log4j (Java) and Winston (Node.js) support asynchronous logging.
• Custom Async Logging: You can build custom asynchronous logging mechanisms using queueing or asynchronous communication patterns.

Best Practices:

• Use a dedicated logging queue: A queue helps manage the flow of logging messages and prevents blocking.
• Handle logging exceptions: Implement error handling to ensure logging failures don't crash the application.
• Monitor queue size: Keep track of the logging queue size to identify potential issues or bottlenecks.

Additional Best Practices for Integration Development:

• Use a dedicated integration framework: Frameworks like Spring Integration (Java) or Mule (Java/Anypoint Platform) provide tools and abstractions for building robust and reusable integrations.
• Implement robust error handling: Handle potential errors gracefully, log them effectively, and provide mechanisms for retries or notifications.
• Use monitoring tools: Monitor integration performance, data flow, and error rates to identify issues and optimize performance.
• Document thoroughly: Document integration components, data formats, error codes, and other relevant information for maintainability and troubleshooting.

By incorporating these best practices, you can create integrations that are performant, reliable, and maintainable.

I hope the above information is helpful.

[melfeqy03]

Thanks for your answers.

When you mentioned "Logging Frameworks and Custom Async Logging," wouldn't it be sufficient to enable cloud logging in async mode for application integration? In what cases would you use dedicated Logging Frameworks and Custom Async Logging instead?

Additionally, when you mentioned "Use a dedicated integration framework," I didn't fully understand your point. Could you explain that in more detail?

[desireejoy]

You are right, enabling cloud logging in async mode can be a sufficient solution for many applications. However, there are cases where dedicated logging frameworks and custom async logging might be preferred:

When to use Logging Frameworks:

• Complex Logging Needs: If you require advanced features like structured logging, custom log levels, filtering, and routing, a logging framework provides a robust and flexible solution. Frameworks like Logback, Log4j, and Serilog offer rich functionalities that simplify complex logging scenarios.

• Centralized Logging: Frameworks often integrate seamlessly with centralized logging systems like ELK stack or Splunk, making it easier to collect and analyze logs from different applications.

• Performance Optimization: Some frameworks include optimizations for asynchronous logging, minimizing the impact of log writing on application performance.

When to use Custom Async Logging:

• Specific Requirements: When your application has unique logging requirements not met by existing frameworks, custom async logging allows you to tailor the implementation to your specific needs.

• Lightweight Solutions: If you have minimal logging requirements and want to avoid the overhead of a full-fledged framework, custom async logging provides a lightweight solution.

• Tight Control: Custom implementations give you complete control over the logging process, including message formatting, destinations, and error handling.

Regarding "Use a dedicated integration framework," here's a more detailed explanation:

This refers to using a dedicated framework specifically designed to simplify integration between your application and a particular service or platform. For example:

• Cloud Platform Integrations: AWS SDK, Google Cloud Client Library, Azure SDKs provide dedicated frameworks for seamless integration with their respective cloud services. These frameworks handle authentication, resource management, and other platform-specific details, simplifying development.

• Third-party Services: Many third-party services, such as payment gateways, email providers, or analytics platforms, offer dedicated integration frameworks. These frameworks provide pre-built libraries and tools for easy interaction with their services.

The key benefit of using a dedicated integration framework is that it reduces boilerplate code, streamlines the integration process, and ensures compatibility with the target service.