Skip to main content Link Menu Expand (external link) Document Search Copy Copied

Meeting Summary – CS 486-686 Lecture/Lab Spring 2025

Date: March 27, 2025, 08:14 AM (Pacific Time, US and Canada)
Meeting ID: 893 0161 6954

Quick Recap

  • OpenAI Supports MCP: The session began with a discussion about OpenAI’s recent announcement supporting MCP (Model Context Protocol), designed by Anthropic. MCP enables the creation of interconnected tools and servers.
  • Agent Concepts: A new approach was presented in which Large Language Models (LLMs) are given control over the order of operations, rather than being used solely as advanced functions.
  • Tool Calling: The discussion explored how tools can be invoked within a prompt—including autonomous LLM-based processes that interact with external systems—and the complexities involved in JSON-formatted tool calling.
  • SDK and Implementations: There was an in-depth look at function calling within the OpenAI SDK, as well as the use of frameworks like Llama Index to create advanced AI agents.

Next Steps

  • Research Paper: Greg will select and share an agent-related research paper for the class to review before next week’s session.
  • Agent Development: Students are expected to practice building basic agents using Llama Index.
  • MCP Deep Dive: Greg will prepare a detailed explanation of MCP (Model Context Protocol) for the upcoming Tuesday class.
  • Tool Demonstration: An upcoming session will include a demonstration of the Microsoft Playwright MCP tool.
  • Code Review: Students should review the provided code examples for stack-based calculator agents.

Detailed Topics

1. OpenAI Supports the MCP (Model Context Protocol)

Greg discussed OpenAI’s support for MCP, a protocol developed by Anthropic that enables the creation of sophisticated tools and servers. MCP promotes a new way of working with LLMs by allowing them to control the sequence of operations rather than simply acting as a collection of advanced functions. This approach could lead to building specialized agents for diverse tasks and promote a proliferation of MCP-compliant tools.

2. Tool Calling in LLMs Explained

Tool calling is a protocol that allows an LLM to interact with functions or tools within a program. This process empowers the LLM to determine the order of operations dynamically. An example discussed was the “Root Code” tool, designed for writing and modifying code.

Tool Calling Flow Diagram

sequenceDiagram
    participant P as Program
    participant L as LLM
    participant U as User
    P->>L: Provides expression and function details
    L->>U: Requests intermediate computations via tool call
    U->>L: Supplies computed values (JSON response)
    L->>P: Returns final result

3. Tool Functionality and Agent Applications

The session covered how tools can be defined and invoked directly within a prompt—without requiring a specific SDK or framework. An LLM can decide when to call a tool provided in the prompt, though it was noted that compatibility may vary among models. The possibilities of creating agents for various practical applications were discussed, even though their current real-world usage might still be in early stages.

4. Autonomous LLM-Based Processes and Agents

Greg introduced the idea of autonomous LLM processes that interact with external systems. These processes are designed using three core components:

  • A prompt that defines the process.
  • One or more tools that the LLM can invoke.
  • A program that collaborates with the LLM to execute the process.

In this model, an agent acts as a specialized tool that bridges the LLM and external services. Multiple agents could be integrated within a single process, each handling different functions.

Autonomous Process Flow Diagram

flowchart TD
    A[Prompt Definition] --> B[Tool Definitions]
    B --> C[LLM Decision-Making]
    C --> D[Agent Invocation]
    D --> E[Interaction with External Systems]
    E --> F[Process Outcome]

5. Tool Calling and JSON Responses

The discussion explained that in tool calling, the tool (or program) sends an expression for which intermediate values need to be computed. The process involves:

  • Parsing the JSON response.
  • Mapping the response to a specific function.
  • Extracting the necessary arguments.

The system must then perform the function call as indicated by the JSON. Early demonstrations highlighted some confusion around this mechanism, emphasizing the need for clarity in how tool calls are parsed and executed.

6. Tool Calling in JSON Form

A detailed exploration of JSON-based tool calling was provided. Key points include:

  • Conformity: Adherence to specific data types, parameters, function names, and descriptions is crucial.
  • Abstraction: Frameworks like OpenAI hide much of the complexity behind a more verbose process.
  • Minor Issues: There was a note regarding a minor issue where the tool result was printed twice during the demonstration.

7. Function Calling in the OpenAI SDK

The session covered function (or tool) calling within the context of the OpenAI SDK. Although the terms are used interchangeably, “tool calling” is preferred due to its close association with LLM functionalities. Highlights include:

  • Timestamp Integration: The new SDK version integrates the current time and time zone to help track operation times.
  • API Use Cases: It demonstrated potential use cases, such as accessing weather data (noting the challenges in finding suitable weather APIs without API keys).

8. Implementing AI Agents with OpenAI

Greg discussed various methods and tools for implementing AI agents using OpenAI and Llama Index. Key topics included:

  • Creating Function Tools: How to create and integrate tools into an OpenAI agent.
  • Iteration Loop with Llama Index: Llama Index implicitly manages the iteration loop required for agent operations.
  • Examples Provided: A stack-based calculator and a React-based agent implementation were demonstrated.
  • The Promise of MCP Servers: The introduction of MCP (Machine Comprehension Protocol) servers suggests a future where interlinked, highly capable AI tools can be developed.

Conclusion

The meeting highlighted several emerging trends in AI tool and agent development, notably the use of MCP and advanced tool calling protocols to give LLMs greater autonomy and flexibility. Future sessions will further explore these technologies, including detailed explorations of MCP and demonstrations of new tools such as the Microsoft Playwright MCP tool.