Consistent response between different prompts

[Gijs1973]

I am trying the run the following code (generated from the sandbox), and I am trying to understand why "Western Europe" is not providing results, but "Eastern Europe" is giving some(times) a list of countries. Other regions have the same problem, so far, I have been unsuccessful in my attempt to create a list of countries in regions. My main problem is that sometimes no results are returned, and when for example, the formatting is removed, the results are inconsistent in the responses, so building a database of countries is not possible (there are too many exceptions).

import vertexai
from vertexai.language_models import TextGenerationModel

vertexai.init(project="XXXX-XXXXXXX-XXXXX", location="us-central1")
parameters = {
    "max_output_tokens": 512,
    "temperature": 0,
    "top_p": 0.8,
    "top_k": 40
}
model = TextGenerationModel.from_pretrained("text-bison@001")
response = model.predict(
    """The United Nations geoscheme is a system which divides 249 countries and territories in the world into six regional, 17 subregional, and nine intermediate regional groups. It was devised by the United Nations Statistics Division (UNSD) based on the M49 coding classification.

input: Geographical Region
output: country=[]

input: Hi Bard, which countries in \"Europe\" are part of the subregion \"Western Europe\", as defined by the UNSD?
output:
""",
    **parameters
)
print(f"Response from Model: {response.text}")

Creating a list of countries by region should be a simple task, and I would like to understand what I am doing wrong. Any insights into this problem are highly appreciated.

[Gijs1973 · Accepted solution]

The problem is solved, it was not easy, but this blog post gives a good insight: https://blog.gopenai.com/fundamental-prompt-engineering-guide-with-vertex-ai-palm-api-c9f307413d85

I used Self-Consistency prompting, directing the code to format the JSON structure to store the answer and be consistent for the next question in the loop. The code was still breaking at several points, but this was because the question didn't give an answer (empty JSON); reformulating the question for the instances that no results were returned gave me a full list of countries for the regions, but not all the countries and duplicates.

Why is this important? It shows (as an indicator) in which regions the model will struggle to find relevant information for questions specifically for that country.

My next steps are: direct prompting for the missed countries, to see if there is information on the country, f.e. in which subregion, as divined by the UNSD, does {country} belong?

For the duplicates, I will have to design a similar test, to see if the correct subregion can be returned.