Creating Conversational AI Agents with Azure AI Foundry

Introduction

Microsoft’s Azure AI Foundry simplifies AI development by providing a unified platform for building, deploying, and managing intelligent applications. It serves as your AI development hub, combining language models, data management, and deployment tools in a single environment. It abstracts the complexity of AI development while maintaining the flexibility developers need for sophisticated solutions.

We’ll start with a basic chat interaction to understand the foundational conversation capabilities of a large language model. Then we’ll explore Azure AI Foundry Agent Service, where specialized agents can be created to handle specific tasks, maintain conversation context, and integrate with external data sources.

I’ll show how to quickly create a basic agent that relies solely on its training data, and then an enhanced version incorporating additional knowledge sources for more accurate responses. Finally, we’ll demonstrate a practical Python code example showing how to integrate these agents directly into your applications for scalable, intelligent solutions.

Azure Portal

For this example, we will perform a standard setup of Azure AI Foundry and an Agent Service using the Azure Portal. The prerequisites to get started merely involve a few permissions after you access your Azure subscription.

• Get access to an Azure subscription, and you can start for free.
• Ensure you have the Azure AI Account Owner and Azure AI Project Manager roles at the subscription scope.
• If configuring an Azure AI Foundry standard setup, ensure you have permissions to assign roles to any required resources (Cosmos DB, Azure AI Search, Azure Blob Storage). For more information on RBAC roles, specific to the Azure AI Foundry Agent Service, see Azure AI Foundry Agent Service RBAC roles. The built-in role needed is Role Based Access Administrator, which provides the key permission Microsoft.Authorization/roleAssignments/write
• If you encounter errors about missing registrations or unregistered providers, then you must register the provider mentioned in the error message.

Deploy the Project and basic AI Agent

Use this custom deployment template to provision the necessary Azure AI Foundry project and agent.

This launches in the Azure portal and creates an Azure AI Foundry project where you can work in an independent environment to quickly explore your AI ideas, build prototypes, and more.

Select the subscription and resource group, fill in the other fields, including location, and leave all the model-related fields untouched. This provides the details for creating the project and deploying a GTP-4o model.

Click the ‘Review + create’ button to deploy the template. It will take a few minutes to complete.

Click ‘go to resource group’ when the deployment is done to view the resources.

If you click on the project link, you can go to the Azure AI Foundry portal.

Azure AI Foundry Portal

Chat with the model

Once you access the Azure AI Foundry portal, if it doesn’t take you directly to the Playground for your project, select the project name from the main page and then select Playgrounds. This will select the gpt-4o deployment for you, where you can begin chatting with the model. Try a few prompts and see what you get. And maybe try out different model instructions, or restrict answers to a certain topic, or add a data source to supplement the model with your data.

For example, I instructed the model to only provide answers about earthquakes, and then I asked for a recipe.

Deploy chatbot as a web app

One cool feature is deploying a simple chatbot as a web app. It will take quite a few minutes to deploy, and then you can click the link to open the web app and try it out. Note that there is much to be desired here to make something like this production-ready, but this demonstrates how easy it is to create a basic chat web app.

Next to the ‘View code’ button at the top of the Chat playground interface, select ‘Deploy as web app’.

To launch the web app after it deploys, go to ‘My assets/Web apps’ and click the link for the deployed web app.

If you notice in the Chat playground, when I provided instructions to the model, I restricted its answers to only questions about earthquakes. As you can see in the chat interface in the web app, the model failed to tell me a joke, but it did answer an earthquake question. However, the model was only trained on data up to October 2023, so in the next section, I’ll show you how to supplement the model with a custom data source without any need to retrain it.

Use an agent to chat with the model

Creating a simple agent

Now that you have a good understanding of how chatting with a model works, let’s enhance this with an agent that wraps this functionality. Then, we’ll supplement the model’s knowledge with a custom data source and then call this agent via a simple Python example.

Select ‘Agents/New Agent’ to create a new agent. Provide an agent name and set of instructions. The instructions are required, and they should be specific to what an agent should do, and in this case, it will be an agent that can only answer questions about earthquakes. You can leave the Deployment as the default gpt-4o, and the Temperature and Top P defaults are fine as they are. Agent configuration changes are auto-saved.

If you click the ‘Try in playground’ button, you can test out the agent the same way you chatted directly with the model previously.

Supplementing the agent with additional knowledge

Azure AI Foundry offers multiple options for integrating custom data into your AI applications and services. It is beyond the scope of this article, but more information is available here.

As a simple example, I’ll be adding one PDF file as a data source to provide my agent with more knowledge about earthquakes. I downloaded a PDF copy of the Wikipedia page on the 2025 Kamchatka Peninsula earthquake, which you can download here if you want to follow along and use it in your agent.

Once you have the PDF file, click the ‘+Add button’ for Knowledge in the Agent configuration, then select ‘Files’ to upload the file. You can leave the defaults as-is to create a new vector store, then select ‘Upload and save’.

If you click the ‘Try in playground’ button again, this time you can ask about the July 2025 earthquake.

Python example to call the agent

Now that you have a good understanding of how a basic Agent works in Azure AI Foundry, let’s try out some Python code to hit the model endpoint directly. But first, get a copy of the Azure AI Foundry project endpoint from the Overview page, and the Agent ID from the Agent page.

Copy this source code locally to a file named agent_earthquake_chat.py. It’s a simple, silly Python example that is not production-friendly by any means :)

Here are step-by-step instructions and commands to run your script in an isolated Python environment, ensuring no reliance on global packages.

1. While in the same folder as the source code downloaded, create a virtual Python environment:

2. Activate the environment:

3. Upgrade pip (recommended):

4. Install required Azure dependencies:

5. Set environment variables for Azure AI Foundry Agent with your specific values:

6. Run your script:

Example chat via Python

Clean up Azure Resources

The cost of the resources created in this exercise is very minimal, but you can expect to spend $1 or $2. After creating the project and experimenting with a few dozen prompts via the chat playground and Python example, my costs were under $3.

To clean up all the Azure resources used during the exercise, Azure makes it very simple for you. Back in the Azure Portal, navigate to Resource groups and select the resource group you created for this exercise, then select ‘Delete resource group’ to delete it.