Skip to main content Link Menu Expand (external link) Document Search Copy Copied

Meeting Summary for CS 486-686 Lecture/Lab – Spring 2025

Date: January 28, 2025, 08:08 AM Pacific Time (US and Canada)
Meeting ID: 893 0161 6954

Quick Recap

Greg led an in-depth discussion on recent developments in artificial intelligence. The session focused on various large language models and their applications in programming. Key topics included:

  • Technical Aspects: Tokenization, inference, and costs.
  • Practical Demonstrations: Experimentation with different AI tools.
  • Future Possibilities: Multimodal inputs and artificial general intelligence (AGI).

Next Steps

  • Students: Provide proof of tutorial completion to Greg and Stephen at the next lab session.
  • Greg:
    • Distribute the next paper for students to read.
    • Finalize and send out details for the next lab activity.
    • Develop and share information about the first project, emphasizing practical and potentially multimodal aspects.
  • Students: Continue exploring the discussed AI concepts and share experiences on CampusWire.

Summary of Discussion Topics

1. Exploring AI Models and Applications

  • Focus Models: Discussion centered on models such as DeepSeek, Groq, OpenRouter, and others.
  • Demonstrations:
    • Use of Groq for generating a RISC-V emulator.
    • Highlighting fast inference speeds and utilization of coding assistance tools like Cline.
  • Cost and Provider Discussion: Analysis of costs and different AI providers, including aspects related to token counts and context windows.

2. LLMs, Text Prompts, and Tokenization

  • Interaction with LLMs:
    • Emphasis on the role of text prompts for effective communication.
    • Explanation that LLMs operate on token representations rather than raw text.
  • Tokenization Process:
    • Specific to each LLM.
    • Auto-regressive inference computes the next token based on the input sequence.
  • Multimodal Possibilities: Hints at potential future developments involving multimodal inputs and outputs.

3. Exploring the Costs of Large Language Models

  • Cost Drivers:
    • Factors such as training, inference, and potential market subsidies.
  • Technical Details:
    • Discussion of input/output tokens and context windows.
  • Demonstration:
    • Use of LiteLLM to interact with multiple models.
    • Interpreting response objects, including token counts.

4. Harnessing LLMs in Programming

  • Challenges and Benefits:
    • Importance of grammatically correct prompts to optimize LLM performance.
    • The need for predictable behavior and avoidance of issues like prompt injection.
  • Prompt Construction:
    • Use of input, instructions, and examples.
    • Mention of higher-level frameworks and context learning for improved generalization.

5. Enhancing Language Models With Data

  • Limitations of Current Models:
    • Current models (e.g., Anthropic and OpenAI) are limited by point-in-time data and lack of access to proprietary or personal data.
  • Enhancement Strategies:
    • Incorporation of additional inputs (images, Excel data).
    • Customizing language models (“GPTs”) by uploading user-specific knowledge.
    • Utilizing retrieval augmented generation (RAG) and vector databases.
  • Service Providers:
    • Companies like Anthropic and OpenAI offering managed RAG services.

6. Large Language Models and AGI

  • Performance and Efficiency:
    • Questioning whether increasing parameters leads to improved performance.
  • Discussion on AGI:
    • Debates on the definition and potential of AGI.
    • Integration of synthetic data and models from providers like Anthropic and Amazon Bedrock.
  • Demonstrations:
    • Use of streaming mode in LLMs with noted issues (e.g., an unexpected percent symbol).
  • Collaboration Encouraged:
    • Students are encouraged to share experiences on CampusWire.

7. System Prompts in AI Models

  • Influence on Behavior:
    • Role of system prompts in guiding model responses.
    • The possibility that system prompts generate subsequent prompts due to underpinning training data.
  • Efficiency Techniques:
    • Discussion of prompt caching to save processing time.
    • Potential for embedding more system prompts to enable effective prompt engineering.

8. Prompt Structure and Tutorial Progress

  • Tutorial Guidance:
    • Keeping different versions of settings files and outputs for comparison.
    • Evidence of following the tutorial (via git commits or saved inputs/outputs) is required for the lab session.
  • Real-World Applications:
    • Large chat logs may be chunked rather than processed entirely.
  • Upcoming Plans:
    • Distribution of a new paper, a new lab activity, and a potential multimodal LLM project.
  • Continued Experimentation:
    • Students are encouraged to experiment further and share experiences on Campus Wire prior to the next lab.

References