LinkedIn
Twitter
Google Cloud's Application Integration is a powerful Integration-Platform-as-a-Service (iPaaS) solution that streamlines data connectivity and management across various applications and services. Integration Connectors in the Application Integration platform lets you connect to various data sources from your integrations. There are over 100 connectors as of today .
These connectors include a set of database connectors such as BigQuery, MySQL, MongoDB, Elastic, Neo4j, etc which play a huge role in enabling seamless integration and data processing by providing an ease mechanism for connectivity in a secure manner with standard (actions/entities) interfaces. The Vertex AI connector allows you to build real time business automation use cases using connectors and data in the databases. This integration also supports Generative AI use cases.
Database connectors:
The Database Connectors enable you to connect to the databases and offer a layer of abstraction for the objects of the connected application. You can access an application's objects only through this abstraction. The abstraction is exposed as entities, operations, and actions.
- Entity: An entity can be thought of as an object, or a collection of properties.. For example, in a database connector, tables are the entities.
- Operation: An operation is the activity that you can perform on an entity.
Examples: List, Get,Create on a database table - Action: An action lets you make changes to an entity or entities and varies from connector to connector.
Examples:
Below is the list of database connectors. You can also refer here to check whether the connector is GA or Public preview.
|
AlloyDB |
Enterprise DB |
SAP Hana |
Redis |
PostgreSQL |
Cockroach DB |
|
Apache Cassandra |
Firestore |
SingleStore |
Cloud spanner |
Redis |
MongoDB |
|
Apache CouchDB |
MariaDB |
Snowflake |
Couchbase |
PostgreSQL |
MySQL |
|
BigQuery |
Teradata |
SQL server |
Neo4j |
CloudSQL-PostgreSQL |
|
|
ElasticSearch |
RedShift |
CloudSQL- MySQL |
Oracle DB |
CloudSQL- SQLserver |
|
Use Case Patterns for Database Integrations:
Application Integration supports various database integration patterns using the database connectors to address various business use cases, here are a few widely used:
- Data Synchronization
- Analytics and Data Processing
- Vertex AI integration for AI use cases
- Real time data streaming into Databases using Pub/Sub
Pattern 1 : Data Synchronization: Database -> APPs
Application Integration can be used to synchronize data from databases to various business applications such as SAP, NetSuite, Salesforce in real-time. This seamless real-time integration ensures data accuracy and consistency across different systems, eliminating the need for manual data entry and reducing the risk of errors. If synchronization is not required in real time, it can be run in regular intervals as hourly or required time intervals using scheduler.
Example: Invoke an integration for a Salesforce Change Data Capture (CDC) event
Use cases:
Salesforce Customer Record update:
When customer or contact information in Salesforce changes, an event is received containing the updated details.This information is then used to update the connected database, ensuring synchronized data on customer or contact. The example provided in the above diagram illustrates this process with CloudSQL , but it is applicable to any other database.
Zendesk/Jira Ticketing Integration:
When a new ticket is raised in Zendesk or Jira, the ticket information is sent to BigQuery for analysis.This enables insights into ticket trends and issues encountered.
Financial Reconciliation:
Transaction data from multiple bank accounts can be consolidated into a central database.This streamlines the reconciliation process and provides a comprehensive view of financial transactions.
These use cases demonstrate the versatility and value of the database connectors for real-time data synchronization, enhancing business efficiency and decision-making with Application Integration
Pattern 2: Analytics and Data Processing:
The combination of the Database Connectors and BigQuery jobs provides a powerful platform for analytics and data processing. Data can be extracted from a transactional or operational database using the Database Connector, and then processed and analyzed using BigQuery jobs. This can be used to gain insights from data, make decisions, and improve business outcomes.
Database Connectors can extract data from a variety of transactional databases as Oracle, SQL Server, MySQL, and PostgreSQL using connector operations as LIST/GET and send to Bigquery for data processing.
BigQuery Jobs:
BigQuery jobs are used to perform data processing and analytics on data in BigQuery. There are a variety of BigQuery jobs available, including:
- Query jobs: Query jobs are used to run SQL queries on data in BigQuery.
- Load jobs: Load jobs are used to load data into BigQuery from a variety of sources, such as Cloud Storage or a database.
BigQuery Connector supports InsertJob action which can run a BigQuery job to execute a query .
Connect BigQuery to external systems :
Connecting BigQuery to external systems can be highly beneficial for various use cases where data needs to be transferred for specialized processing and analytics. There are database connectors like Neo4j connector and Elastic connector for example that can offer some benefits.
Some of the benefits include optimized workflows through specialized systems such as BigQuery for advanced analytics and machine learning, a graph database for relationship-focused analytics, and a search engine for quick search and retrieval. These integrations help with comprehensive data analysis and support better decision-making and strategic planning. They further improve query performance as these systems are optimized for specific query types, such as graph-based queries in Neo4j or unstructured data searches in Elasticsearch.
Here are some use cases of how the Database Connectors and BigQuery can be used:
- Customer analytics: A company can use a Database Connector to extract customer data from their transactional database and load into BigQuery. This data can be processed and analyzed using BigQuery jobs to identify customer trends, preferences, and behaviors. This information can be used to improve marketing campaigns, develop new products and services, and provide better customer service.
- Fraud detection: A financial institution can use a Database Connector to extract transaction data from their transactional database. This data can then be processed and analyzed using BigQuery jobs to identify suspicious transactions. This information can be used to prevent fraud and protect customers.
- Supply chain management: A manufacturing company can use a Database Connector to extract inventory data from their transactional database. This data can then be processed and analyzed using BigQuery jobs to optimize inventory levels and reduce costs.
- BigQuery to Neo4j: Enhance CRM data by analyzing complex relationships between customers, transactions, and interactions. Identify influential customers, uncover hidden connections, and improve targeting for marketing campaigns.
- BigQuery to Elasticsearch: Implement real-time search and analysis of transaction logs to identify and respond to suspicious activities as they happen.
Example to insert records in iterative synchronous manner from Couchbase DB to BigQuery
Pattern 3: Vertex AI integration for AI use cases
Vertex AI is Google cloud’s machine learning (ML) platform that lets you train and deploy ML models and AI applications, and customize large language models (LLMs) for use in your AI-powered applications
In Application Integration, data can be retrieved from a database using native database connectors and sent to AI based platforms/services such as Vertex AI. This enables real-time predictions and recommendations to support various business use cases. This process allows organizations to leverage machine learning models hosted on Vertex AI to make accurate predictions and optimize decision-making. There are also other AI based tasks, such as Document AI to help with various AI use cases.
By seamlessly integrating data from databases with Vertex AI for real-time predictions and recommendations, organizations can gain valuable insights, improve decision-making, and enhance customer experiences.
Important links to using vertex AI for different data types:
Image
Tabular
Get predictions from the following types of tabular AutoML models:
- Tabular classification/regression models
- Online predictions
- Batch predictions
- Tabular forecasting models (batch predictions only)
Text:
To get predictions from the following types of text AutoML models:
Video
To get predictions from the following types of video AutoML models:
- Video action recognition models (batch predictions only)
- Video classification models (batch predictions only)
- Video object tracking models (batch predictions only)
You can easily connect to external databases, prepare the dataset and make it available on either GCS or BigQuery to further train the model. Using these APIs available via actions on the application integration platform, you can do various things depending on the type or format of the data.
Generative AI:
The Vertex AI connector also has support for the actions based on the Vertex AI APIs which can help you send requests to the Gemini API in Vertex AI to begin building your generative AI applications on Google Cloud.
As the Gemini models are multi-modal, this connector can help you send a text-only request, send a request that includes an image and send a request that includes audio and video using the streamGenerateContent for streaming and GenerateContent for non-streaming. Refer this for more information: https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/inference
The connector also provides you a way to configure the tools that the model can use to interact with external systems to perform an action, or set of actions, outside of knowledge and scope of the model. The tools accepted here are functions that are available via the Gemini function calling. Refer this for more information: https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/function-calling.
The integrations to Databases for structured or unstructured applications from the Application integration platform can act as tools for LLMs as part of this API. In addition, there are also APIs to generate embeddings and batch prediction actions. For a full list of actions, refer to this link.
Integrating MySql and Gen AI with Vertex AI for Personalized Campaigns
AI agents:
There are multiple ways to create AI agents on Google Cloud platform, such as using the No-code Agent builder or open-source frameworks like Langchain. In any scenario where there is a need for Vertex AI extension or Gemini function calling to connect to a database, application integration can enable native connectivity with the databases and can help with mediation related tasks to customize the structure of the data .
Example: Build an AI agent using Vertex AI Agent builder and Application Integration platform that interacts with BigQuery using natural language.
Here are some use cases of how the Database Connectors and Vertex AI can be used:
Real-time recommendations for e-commerce:
- Retail companies use database Connectors to extract real-time sales data, streamed to Vertex AI, which trains a machine learning model for product recommendations displayed on the company's website.
Real-time inventory optimization:
- Manufacturing company uses Database Connector and Vertex AI to optimize inventory levels in real time by training a machine learning model on inventory data and recommends when to reorder entry
Real-time customer service:
- Based on customer data and issues reported by customer stored in database, recommendation and customer queries can be responded by building integration using Application Integration and Vertex AI connector
Personalized product recommendations:
- Based on product and sales data in databases , personalized product recommendations can be provided using GenAI capability in vertex connector and marketing campaigns can be targeted
Pattern 4: Real time data streaming into Databases using Pub/Sub
Application Integration facilitates real-time data insertion into databases through the utilization of triggers and schedulers. When new transactions occur in enterprise applications, messages can be transmitted to Pub/Sub. Within Application Integration, a Pub/Sub trigger identifies newly received messages in a Pub/Sub topic and initiates the integration . These messages or new transactions can be integrated into a database using the CREATE operation of the connector.
Example: Real time updates to MongoDB using pub/sub
Use cases examples for real time updates to databases using Pub/sub
Inventory Management System:
A retail company needs to update its inventory levels in real time as products are sold. The company uses an inventory management system that is integrated with Pub/Sub. When a product is sold, a message is published to a Pub/Sub topic. When a message is received ,inventory data can be deleted from the database by building integration with connectors in Application Integration and keeping the inventory data up to date.
Real-time Order Tracking:
An e-commerce company needs to provide customers with real-time updates on the status of their orders. The company uses an order tracking system that is integrated with Pub/Sub. When an order is placed, a message is published to a Pub/Sub topic. Based on the message, the order status will be updated in the database by building integration with connector in Application Integration. The customer can then track the status of their order in real time.
