Vertex AI Extension via Java

[grnadav]

Hey all!

I'm building an Agent, and I'm trying to make it use a custom tool (Extension) that I created, it seems the Java SDK is lacking the ability to register the Tool for use such that it would execute automatically?

The extension is registered (via the experimental UI), the Java SDK I can do:

Tool myTool = Tool.newBuilder()
        .addFunctionDeclarations(FunctionDeclaration.newBuilder()
                .setName("myTool")
                .setDescription("My amazing tool")
                .build())
        .build();

and later do:

// Create the request to Gemini
GenerateContentRequest request = GenerateContentRequest.newBuilder()
        .setModel("gemini-2.0-flash-exp") 
        .setSystemInstruction(systemInstruction)
        .addContents(userMessage)       
        .setGenerationConfig(generationConfig)  
        .addTools(myTool)         // Register tool with automatic execution ??
        .build();

But can't find anywhere to register the Tool's endpoint.
What am I missing here?

[MarvinLlamas]

Hi @grnadav,

Welcome to Google Cloud Community!

It looks like you are facing an issue with linking the `FunctionDeclaration` in your Java code to the URL endpoint of your custom tool for a Vertex AI Agent. 

Here are the potential ways that might help with your use case:

• The Connection in the Vertex AI Extensions UI: Double-check that you have correctly associated your FunctionDeclaration.setName("myTool") with your Extension in the Vertex AI Extensions UI. This is often where errors occur. Ensure the tool name precisely matches what you've defined in your Java code.
• Correct Parameter Definitions in the Extension UI: Ensure that the parameters you've defined in your Extension's schema (often OpenAPI/Swagger) in the UI *match* what your Agent expects based on your tool's description. If there's a mismatch, your Agent might not call the tool at all, or it might pass the wrong information.
• Proper Agent Configuration: Confirm that your Agent is properly configured to use the Extensions you've registered. This might involve enabling specific features or setting appropriate permissions within your Vertex AI Agent configuration.
• Authentication: Make sure that Vertex AI has the necessary permissions to call your tool's endpoint. Using your service accounts is the recommended approach.

You can refer to the following documentation, which provides information on Google Cloud's Vertex AI and API Gateway:

• Extensions overview
• Creating an API config
• What is Vertex AI Agent Builder?
• Authenticate to Vertex AI

Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.