To Do #155
Updated by Harikrishnan Murugan 3 months ago
**Task Description** Build Pipecat pipeline with GeminiMultimodalLiveLLMService for real-time audio communication. Implement SmallWebRTC transport for peer-to-peer audio streaming and create a weather assistant demo to verify the complete setup works correctly. This establishes the technical foundation that can later be adapted to the AI Communication Coach by changing the system prompt and tools configuration. **IMPORTANT NOTE:** Use SmallWebRTCTransport specifically for this project. Do NOT use DailyTransport, FastAPIWebSocketTransport, or any other transport implementations. **Model to Use:** gemini-2.5-flash-native-audio-preview-09-2025 **Technical Approach** - Configure GeminiMultimodalLiveLLMService with model: gemini-2.5-flash-native-audio-preview-09-2025 Gemini 2.5 Flash Live API - Implement SmallWebRTCTransport (NOT DailyTransport or FastAPIWebSocketTransport) - Build Pipecat pipeline (audio input → processing → Gemini → audio output) - Set up session management (start/stop/reconnect) - Add error handling and connection recovery - Create weather assistant demo with function calling - Implement weather API tool for real-time tool calls - Create minimal React frontend using Pipecat React SDK - Implement weather UI display component for tool results - Test complete pipeline with voice commands **Acceptance Criteria** - Pipecat pipeline runs successfully - GeminiMultimodalLiveLLMService configured with gemini-2.5-flash-native-audio-preview-09-2025 model and working - SmallWebRTCTransport implemented for audio streaming (not other transports) - Audio input/output streaming works via WebRTC - Error handling covers disconnections - Session lifecycle managed properly - Weather assistant responds to voice queries - Tool calling works (get_weather function) - Frontend displays weather UI when tool is called - Complete setup can be adapted by changing system prompt and tools config **Resources/References** - Pipecat Gemini Multimodal Live Guide: https://docs.pipecat.ai/guides/features/gemini-multimodal-live - Pipecat SmallWebRTC Example: https://github.com/pipecat-ai/pipecat-examples/tree/main/p2p-webrtc - SmallWebRTCTransport Documentation: https://docs.pipecat.ai/server/services/transport/small-webrtc#smallwebrtctransport - Pipecat React Client SDK: https://docs.pipecat.ai/client/react/introduction - Pipecat Pipeline Documentation: https://docs.pipecat.ai/server/introduction