{ "cells": [ { "cell_type": "code", "execution_count": null, "metadata": { "id": "ur8xi4C7S06n" }, "outputs": [], "source": [ "# Copyright 2025-2026 Google LLC\n", "#\n", "# Licensed under the Apache License, Version 2.0 (the \"License\");\n", "# you may not use this file except in compliance with the License.\n", "# You may obtain a copy of the License at\n", "#\n", "# https://www.apache.org/licenses/LICENSE-2.0\n", "#\n", "# Unless required by applicable law or agreed to in writing, software\n", "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", "# See the License for the specific language governing permissions and\n", "# limitations under the License." ] }, { "cell_type": "markdown", "metadata": { "id": "JAPoU8Sm5E6e" }, "source": [ "# Multimodal Sentiment Analysis with Gemini\n", "\n", "\n", " \n", " \n", " \n", " \n", "
\n", " \n", " \"Google
Open in Colab\n", " \n", " 		\n", " \n", " \"Google
Open in Colab Enterprise\n", " \n", " 		\n", " \n", " \"Vertex
Open in Vertex AI Workbench\n", " \n", " 		\n", " \n", " \"GitHub
View on GitHub\n", " \n", "
\n", "\n", "
\n", "\n", "Share to:\n", "\n", "\n", " \"LinkedIn\n", "\n", "\n", "\n", " \"Bluesky\n", "\n", "\n", "\n", " \"X\n", "\n", "\n", "\n", " \"Reddit\n", "\n", "\n", "\n", " \"Facebook\n", "" ] }, { "cell_type": "markdown", "metadata": { "id": "84f0f73a0f76" }, "source": [ "| | |\n", "|-|-|\n", "| Author(s) | [Mohammad Al-Ansari](https://github.com/mansari/) |" ] }, { "cell_type": "markdown", "metadata": { "id": "mzi17id6IXv-" }, "source": [ "## Overview\n", "\n", "This notebook demonstrates multimodal sentiment analysis with Gemini by comparing sentiment analysis performed directly on audio with analysis performed on its text transcript, highlighting the benefits of multimodal analysis, including capturing nuances like tone, inflection, and other non-verbal cues, for a richer understanding of sentiment.\n", "\n", "Gemini is a family of generative AI models developed by [Google DeepMind](https://deepmind.google/) that is designed for multimodal use cases. [Gemini 2.0](https://cloud.google.com/vertex-ai/generative-ai/docs/gemini-v2) is the latest model version.\n", "\n", "### Objectives\n", "\n", "In this notebook, we will explore sentiment analysis using text and audio as two different modalities.\n", "\n", "In summary, you will learn how to use Gemini with the Gen AI SDK for Python to:\n", "\n", " - Load a sample conversation for between two individuals.\n", " - Transcribe the audio content.\n", " - Conduct sentiment analysis using two modalities: the audio content and the transcript.\n", " - Compare the two analyses and provide advantages of each.\n", "\n", "For additional multimodal use cases with Gemini, check out [Gemini: An Overview of Multimodal Use Cases](https://github.com/GoogleCloudPlatform/generative-ai/blob/c00b23b5f4963ce3b452b9318dc4e8e3aff7232e/gemini/use-cases/intro_multimodal_use_cases.ipynb)" ] }, { "cell_type": "markdown", "metadata": { "id": "dSxKV-CIH9oa" }, "source": [ "### Costs\n", "\n", "This tutorial uses billable components of Google Cloud:\n", "\n", "- Vertex AI\n", "\n", "Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing) and use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage.\n" ] }, { "cell_type": "markdown", "metadata": { "id": "61RBz8LLbxCR" }, "source": [ "## Getting Started" ] }, { "cell_type": "markdown", "metadata": { "id": "No17Cw5hgx12" }, "source": [ "### Install Google Gen AI SDK for Python" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "tFy3H3aPgx12" }, "outputs": [], "source": [ "%pip install --upgrade --quiet google-genai" ] }, { "cell_type": "markdown", "metadata": { "id": "R5Xep4W9lq-Z" }, "source": [ "### Restart current runtime\n", "\n", "To use the newly installed packages in this Jupyter runtime, you must restart the runtime. You can do this by running the cell below, which will restart the current kernel." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "XRvKdaPDTznN" }, "outputs": [], "source": [ "# Restart kernel after installs so that your environment can access the new packages\n", "import IPython\n", "\n", "app = IPython.Application.instance()\n", "app.kernel.do_shutdown(True)" ] }, { "cell_type": "markdown", "metadata": { "id": "SbmM4z7FOBpM" }, "source": [ "
\n", "⚠️ The kernel is going to restart. Please wait until it is finished before continuing to the next step. ⚠️\n", "
" ] }, { "cell_type": "markdown", "metadata": { "id": "dmWOrTJ3gx13" }, "source": [ "### Authenticate your notebook environment (Colab only)\n", "\n", "If you are running this notebook on Google Colab, run the following cell to authenticate your environment. This step is not required if you are using [Vertex AI Workbench](https://cloud.google.com/vertex-ai-workbench)." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "NyKGtVQjgx13" }, "outputs": [], "source": [ "import sys\n", "\n", "# Additional authentication is required for Google Colab\n", "if \"google.colab\" in sys.modules:\n", " # Authenticate user to Google Cloud\n", " from google.colab import auth\n", "\n", " auth.authenticate_user()" ] }, { "cell_type": "markdown", "metadata": { "id": "DF4l8DTdWgPY" }, "source": [ "### Set Google Cloud project information and initialize Vertex AI SDK\n", "\n", "To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n", "\n", "Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)." ] }, { "cell_type": "markdown", "metadata": { "id": "nQaM-2IPxsfV" }, "source": [ "#### Create a client for Vertex AI (with all Google Cloud capabilities and services)\n", "\n", "Specify the project ID and location while creating the client." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "LxJ-OHg0xsPp" }, "outputs": [], "source": [ "import os\n", "\n", "from google import genai\n", "\n", "PROJECT_ID = \"[your-project-id]\" # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n", "if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n", " PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n", "\n", "LOCATION = os.environ.get(\"GOOGLE_CLOUD_REGION\", \"us-central1\")\n", "\n", "client = genai.Client(vertexai=True, project=PROJECT_ID, location=LOCATION)" ] }, { "cell_type": "markdown", "metadata": { "id": "5303c05f7aa6" }, "source": [ "### Import libraries" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "6fc324893334" }, "outputs": [], "source": [ "from IPython.display import Markdown, display\n", "from google.genai.types import GenerateContentConfig, Part" ] }, { "cell_type": "markdown", "metadata": { "id": "e43229f3ad4f" }, "source": [ "### Load Gemini 2.5 Flash model\n", "\n", "Learn more about all [Gemini models on Vertex AI](https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#gemini-models)." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "cf93d5f0ce00" }, "outputs": [], "source": [ "MODEL_ID = \"gemini-2.5-flash\" # @param {type: \"string\"}" ] }, { "cell_type": "markdown", "metadata": { "id": "pyrOSJi2WPKp" }, "source": [ "### Load sample audio content\n", "\n", "Run the code below to load a sample audio file that was previously generated with Gemini.\n", "\n", "The sample file includes text that can be interperted differently based on tone, and other audio elements that can impact the results of a sentiment analysis performed on the audio content.\n", "\n", "The file has been uploaded to a Google Cloud Storage bucket, so you can load it using the `Part.from_uri` method. You can listen to the content of the file [here](https://storage.googleapis.com/github-repo/generative-ai/gemini/use-cases/multimodal-sentiment-analysis/sample_conversation.wav)." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "_CK7ikGLhQFq" }, "outputs": [], "source": [ "audio_part = Part.from_uri(\n", " file_uri=\"gs://github-repo/generative-ai/gemini/use-cases/multimodal-sentiment-analysis/sample_conversation.wav\",\n", " mime_type=\"audio/wav\",\n", ")" ] }, { "cell_type": "markdown", "metadata": { "id": "06o-EHhcXy3j" }, "source": [ "### Run sentiment analysis on audio content\n", "\n", "Now you can ask Gemini to run sentiment analysis directly on the audio file content." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "ggBSfVDCWW4U" }, "outputs": [], "source": [ "prompt = \"Provide a sentiment analysis of this conversation. Use speaker A, speaker B, etc to identify speakers.\"\n", "response = client.models.generate_content(\n", " model=MODEL_ID,\n", " contents=[\n", " audio_part,\n", " prompt,\n", " ],\n", ")\n", "audio_analysis = response.text\n", "display(Markdown(audio_analysis))" ] }, { "cell_type": "markdown", "metadata": { "id": "LAApDX8NXmHI" }, "source": [ "### Generate a transcript from the audio content\n", "\n", "You can use Gemini to generate a transcript for the audio file that we can use in our text sentiment analysis." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "FyJoPvWlWWPI" }, "outputs": [], "source": [ "prompt = \"Generate a transcript of this conversation. Use speaker A, speaker B, etc to identify speakers.\"\n", "\n", "response = client.models.generate_content(\n", " model=MODEL_ID,\n", " contents=[\n", " audio_part,\n", " prompt,\n", " ],\n", ")\n", "\n", "transcript = response.text\n", "display(Markdown(transcript))" ] }, { "cell_type": "markdown", "metadata": { "id": "AWy4-5ZYXsM0" }, "source": [ "### Run sentiment analysis on the transcript\n", "\n", "Now you can ask Gemini to run the sentiment analysis, but this time on the transcript we extracted from the audio." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "W1ab9WjuWWi9" }, "outputs": [], "source": [ "prompt = (\n", " \"\"\"Provide a sentiment analysis of this conversation. Use speaker A, speaker B, etc to identify speakers.\n", "Transcript:\"\"\"\n", " + transcript\n", ")\n", "\n", "response = client.models.generate_content(\n", " model=MODEL_ID,\n", " contents=prompt,\n", " config=GenerateContentConfig(\n", " response_modalities=[\"TEXT\"],\n", " ),\n", ")\n", "\n", "text_analysis = response.text\n", "display(Markdown(text_analysis))" ] }, { "cell_type": "markdown", "metadata": { "id": "6yvB8ysKX6hl" }, "source": [ "### Compare the two analyses\n", "\n", "Now you can ask Gemini to compare the two different analyses and show if we gained any insight from either of them." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "SwnuX-bwWXGM" }, "outputs": [], "source": [ "prompt = (\n", " \"\"\"Provide a short comparison of two analyses of an audio conversation.\n", "One was made based on a text transcript of the call, and the other is based on an audio recording.\n", "Transcript analysis:\"\"\"\n", " + text_analysis\n", " + \"\"\"Audio analysis:\"\"\"\n", " + audio_analysis\n", ")\n", "\n", "response = client.models.generate_content(\n", " model=MODEL_ID,\n", " contents=prompt,\n", " config=GenerateContentConfig(\n", " response_modalities=[\"TEXT\"],\n", " ),\n", ")\n", "\n", "comparison = response.text\n", "display(Markdown(comparison))" ] }, { "cell_type": "markdown", "metadata": { "id": "vZ6BAQHVNU9A" }, "source": [ "### Conclusion\n", "\n", " Sentiment analysis of audio offers significant advantages, providing a deeper understanding of nuanced sentiment and delivery. Audio cues, like the enthusiastic tone of a speaker or the subtle inflections in their voice, add crucial context often missed by text alone.\n", "\n", " This richer information is invaluable for applications from understanding customer feedback to gauging emotional responses in communication. We encourage you to explore these benefits by analyzing your own audio data, such as interviews, podcasts, or presentations.\n" ] } ], "metadata": { "colab": { "name": "intro_to_multimodal_sentiment_analysis.ipynb", "toc_visible": true }, "kernelspec": { "display_name": "Python 3", "name": "python3" } }, "nbformat": 4, "nbformat_minor": 0 }