Vertex AI caching only system prompt

[arthurbrcuni]

**Issue:**

I'm using Vertex AI and trying to cache only the system prompt through context caching. However, I'm encountering a `TypeError` that states, "value must not be empty."

**Code Snippet:**

```python
system_instruction_content = Content(
role="system",
parts=[Part.from_text(system_instruction)]
)

cached_content: caching.CachedContent = caching.CachedContent.create(
model_name=self.model,
system_instruction=system_instruction_content,
contents=[],
ttl=ttl,
display_name=cache_key,
)
```

**Error Log:**

```
/workspaces/CORTEX/.venv/lib/python3.10/site-packages/vertexai/caching/_caching.py:74 in _prepare_create_request
_generative_models._validate_tool_config_type(tool_config)
# contents can either be a list of Content objects (most generic case)
contents = _generative_models._content_types_to_gapic_contents(contents)
gapic_system_instruction: Optional[gapic_content_types.Content] = None
if system_instruction:

/workspaces/CORTEX/.venv/lib/python3.10/site-packages/vertexai/generative_models/_generative_models.py:262 in _content_types_to_gapic_contents
return [_to_content(contents)]

/workspaces/CORTEX/.venv/lib/python3.10/site-packages/vertexai/generative_models/_generative_models.py:2628 in _to_content
raise TypeError("value must not be empty")

TypeError: value must not be empty
```

**Explanation:**

The error occurs because the `contents` list is empty when attempting to create `CachedContent`. The `Vertex AI` caching mechanism expects at least one `Content` object in the `contents` parameter. I've tried to just use the same "system_prompt_content" object in the "contents" parameter, but I got another error telling me that the role "system" cannot be used in contents.
```
_InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.INVALID_ARGUMENT
details = "Content with system role is not supported."
debug_error_string = "UNKNOWN:Error received from peer ipv4:172.217.30.42:443 {grpc_message:"Content with system role is not supported.", grpc_status:3,
created_time:"2024-10-16T11:53:31.200749119+00:00"}"
>
```

Can someone clarify this for me?

[dawnberdan · Accepted solution]

Hi @arthurbrcuni,

Welcome to Google Cloud Community!

The problem you're facing with Vertex AI caching is due to the contents parameter in caching.CachedContent.create requiring at least one user-generated content object, which means the system prompt can't be added directly to this list. The contents parameter must consist of user-generated content, not system prompts.

While there's no direct way to include the system prompt in the contents list, Here are three workarounds you might explore:

1. Instead of using system_instruction_content in contents, extract the system prompt text (system_instruction).
2. Modify your code to include the extracted text in the request body outside the contents list.
3. Modify your application logic to prepend the system prompt text to user input before creating the Content object for the contents list.

In addition, you may also check these documents which might help you:

• Overview of context caching
• How to get information about a context cache
• Configuring execution caching

Note: Some of these products or features are subject to the "Pre-GA Offerings Terms" outlined in the General Service Terms section. Pre-GA products and features are offered "as is" and may have limited support.

I hope the above information is helpful.

[nil_golakiya]

I'm using the deprecated `google-generativeai` package (model: `gemini-2.0-flash`) and initialize the model as follows:

import google.generativeai as genai

genai.configure(api_key='GOOGLE_API_KEY')

model = genai.GenerativeModel(
model_name='gemini-2.0-flash',
system_instruction=system_instruction,
generation_config=genai.GenerationConfig(temperature=0)
)

Then, I use the model in a function:
def generate_response(user_prompt):
response = model.generate_content(user_prompt)
return response

Questions about google-generativeai:
1. Is `system_instruction` sent with every request when calling the model.generate_content function?
2. Does Gemini cache `system_instruction` by default?

In the new `google-genai` package, the model is used like this:

from google import genai
from google.genai import types

client = genai.Client(api_key="GEMINI_API_KEY")

def generate_response(user_prompt):
response = client.models.generate_content(
model="gemini-2.0-flash",
config=types.GenerateContentConfig(
system_instruction="You are a cat. Your name is Neko."
),
contents=user_prompt
)

return response

Questions about google-genai:
3. The `system_instruction` is passed with every request. Is there a way to set it once, like the deprecated version (google-generativeai)?
4. If not, how can I implement caching for `system_instruction` in the new package (google-genai)?