Adding Google Gemini to AutoGen Studio 0.4

AutoGen Studio supports Gemini out of the box.

Building an agentic team can be daunting but does not need to be. Microsoft’s Multi-Agent Tool AutoGen Studio has one OpenAI model by default. I wrote an article explaining how to add more OpenAI models. In this article, we're going to add a Google Gemini model with a configuration trick. With a little JSON you'll be adding a Google Gemini model in under than 10 minutes.

The AutoGen Studio missing Gemini model problem

When you are looking at the one model in AutoGen Studio (for AutoGen 0.4) you might believe that there is only support for OpenAI. This would be wrong. What you have at your fingertips is the power to use ANY API that supports the OpenAI chat completion. Models that support this include Grok, DeepSeek and for our purposes today? Google's Gemini.

The steps to add Google Gemini are the same as adding an OpenAI model. This is why I went to great length to write the specific nuances and steps in the prior article. I will include those details here, but tailor them for Google Gemini for those who missed the first post.

If you were there for the first post, then you let me give you the short answer. Use the following code and modify the GEMINI_KEY to be your Google Gemini API key that you get from the Gemini AI Studio web portal. Please note the comma. The comma is provided to make this even easier to copy and paste. Remember to paste it into the “models” node.

        ,{
          "model": "gemini-1.5-flash",
          "model_type": "OpenAIChatCompletionClient",
          "base_url": "https://generativelanguage.googleapis.com/v1beta/openai/",
          "api_key": "GEMINI_KEY",
          "api_version": "1.0",
          "component_type": "model",
          "model_capabilities": {
            "vision": false,
            "function_calling": true,
            "json_output": false
          }
        }

That’s it!

OK, but let’s now break this down step by step.

Building an agentic team can be daunting but does not need to be. Microsoft’s Multi-Agent Tool AutoGen received a significant update with 0.4. After installing the platform you may ask yourself how to add a new model to the workflow designer.

You will need mild comfort in editing json. Let's walk thru this step by step.

Navigating to the Gallery

You do need a little passing familarity with the Gallery. I won't be covering the full Component Gallery in deep detail in this article. Follow me for more insight into AutoGen.

In this tutorial it is important to know how to navigate to the gallery. When you have AutoGen Studio open, click on the 'Gallery' link and then you will see this screen.

Our goal today is to add Google’s Gemini to the AutoGen studio. A little light code editing gets you there. As mentioned earlier, you do need a little familiarity with JSON.

How to add more OpenAI compatible models to AutoGen

Adding one or more OpenAI compatible models to AutoGen sounds like a difficult project. In reality the entire process will take you less than two minutes. To make this even easier and provide you a simple reference for the future I have here the list of steps needed.

Quick Guide

If these are a little mysterious then worry not. Here are the steps that I will walk you through. I provide the summary here to make it easy in the future.

1. In the component gallery edit the "models" component

2. The models list is a json array. Find the relevant section and add an extra item(s) for the models you want to use.

3. Save your changed and cascade any updates to your agentic team

Step by Step

What we will do is use the JSON configuration to enable the AutoGen Studio to work with Google’s Gemini. You will not lose support for OpenAI in the process. Your agentic team can now have both OpenAI and Google working together!

Step 1. Navigating the component gallery to edit the models list

With the AutoGen Agent Studio open click on the left navigation where it says "Gallery". This will then open a page where you will see a list of galleries, including the ability to make new ones. You can ignore that for today. We are laser focused on adding one model to your list in AutoGen for your agent team to work with.

Click the 'Gallery' and then scroll to the Default Component. This article assumes you are starting from a new AutoGen installation.

With the Default Component selected now find the edit button on the far right of the screen. Click the “Edit” button and proceed to the next step.

Step 2. Editing the models list and adding your custom json

With the editor open you will see a sprawl of text, brackets, quotes and more. This is the json familiarity that I mentioned earlier is a useful thing to have. Fear not. Json is easy to work with and all we need to do is add a few new lines of it.

Scroll down the file and see if you can find the phrase "models" in quotes. On windows, if you click in the json area and press “ctrl+f” you can open a mini search tool to find this value.

This is a json array structure. What "models" means is that you can place your OpenAI Chat Completion models into this specific place. If you want to add more than one then repeat the process you will now perform. The bold elements of the code block you can change.

Minimally to add an OpenAI model takes the shape of this code below.

Code:

        ,{
          "component_type": "model",
          "description": "Historic ChatGPT model",
          "model": "gpt-3.5-turbo",
          "model_type": "OpenAIChatCompletionClient"
        }

Copy the above code and paste it after the closing space indicated in the image. Doing this extends the json array by one. Note the comma. I included the comma to make it easy to paste this JSON into place.

Let’s add Gemini to AutoGen Studio!

Google Gemini requires a few more configurations options in your JSON. It looks like a lot, but you need only do TWO things. You need your API key. Put your API Key into the string identified GEMINI_KEY. Then the second step is to copy and paste your JSON into the AutoGen Studio editor.

        ,{
          "model": "gemini-1.5-flash",
          "model_type": "OpenAIChatCompletionClient",
          "base_url": "https://generativelanguage.googleapis.com/v1beta/openai/",
          "api_key": "GEMINI_KEY",
          "api_version": "1.0",
          "component_type": "model",
          "model_capabilities": {
            "vision": false,
            "function_calling": true,
            "json_output": false
          }
        }

Your completed setup should look similar to this JSON when complete.

Verify before continuing that you have the same json structure as shown in this example. Refer to the troubleshooting tips at the end of this article if you need further help. If you're stumped, add a comment or send me a message and we can figure it out together.

Step 3. Save the entry and update your team

Once you add and save the json from the prior step you are almost done. The next step depends on how you structured the team. Team construction is beyond the scope of this article. But, it is important to see the full change in action so let's go see where you can find your added model. When loading to edit your team you should see the newly added model in the list like so. Congratulations!

But wait, there’s more.

Save copies of your components.

⚠️There is data loss risk.

If AutoGen Studio crashes or fails to start your file edits may disappear from the gallery. A simple solution to this problem is to maintain manual backups. When you edit a file copy those contents to a trusted location and keep them accessible. Remember that this is a "preview" technology and so a little extra precautions are always in order.

Congratulations!

Following the three steps means you added you took your first step into a new world. The world multi-agent workflows. This article provided the little details that you might not have expected.

I write about engineering and executive topics. I encourage you to subscribe to this substack and we can navigate AI together.

ℹ️Troubleshooting

“Stop reason:” is empty

This can happen if your model name is invalid with Google Gemini. It is one of two possible errors. Double check and ensure the model name is valid. The AutoGen Studio does not provide enough error log in the web interface to troubleshoot. You may need to look carefully at the console logs.

“Team creation failed: Model creation failed: model_info is required when model name is not a valid OpenAI model”

If you see this error it means you either have lost access or typed in a bogus entry. Most likely you typed in the model name wrong. If you lost access to the model (or never had it), you can check out the platform of OpenAI.

The question then becomes what specific steps need to occur next. There are two steps that I recommend. First we need to double check the entry in the component gallery. Inisde the gallery confirm that your json is the same model name as the official OpenAI platform. Click save.

Next you need to also update your team. This is because you changed the component but it is possible the team setup is invalid. Edit your team and ensure that the model assigned to your agent is using the correct model. If not, you might experience the problem of "I changed the autogen model but it didn't save". That's addressed next.

I changed the model in the component gallery but my team session still doesn't work.

Changing a model component does not cascade the change everywhere. The way this version of AutoGen works is that the component gallery is a template factory. Templates copy each component into your team session. This means you need to also edit the model name in the edit tool of the team component.

If you found me while troubleshooting your AutoGen setup then I welcome you to my little corner on the internet. Consider subscribing or tapping that 💖icon if you are reading this on the substack mobile application.