Multimodal LLM Workflows in Vertex AI Training Course

Introduction to Multimodal LLMs in Vertex AI

• Survey of multimodal capabilities available in Vertex AI
• Overview of Gemini models and their supported modalities
• Applications in enterprise and research sectors

Establishing the Development Environment

• Setting up Vertex AI for multimodal workflows
• Managing datasets across different modalities
• Practical lab: configuring the environment and preparing datasets

Long Context Windows and Sophisticated Reasoning

• Concepts behind long-context workflows
• Applications in planning and decision-making processes
• Practical lab: executing long-context analysis tasks

Architecting Cross-Modal Workflows

• Synthesising text, audio, and image analysis
• Linking multimodal steps within pipelines
• Practical lab: constructing a multimodal pipeline

Managing Gemini API Parameters

• Configuring multimodal input and output streams
• Optimising inference speed and operational efficiency
• Practical lab: adjusting Gemini API parameters for optimal results

Sophisticated Applications and System Integrations

• Developing interactive multimodal agents and assistants
• Connecting external APIs and utility tools
• Practical lab: creating a comprehensive multimodal application

Assessment and Iterative Improvement

• Testing the performance of multimodal systems
• Tracking metrics for accuracy, alignment, and model drift
• Practical lab: assessing multimodal workflow performance

Conclusion and Forward Path