Firebase is back at Google I/O on May 20-21! Register now.

Analyze audio files using the Gemini API
Stay organized with collections Save and categorize content based on your preferences.

You can ask a Gemini model to analyze audio files that you provide either inline (base64-encoded) or via URL. When you use Vertex AI in Firebase, you can make this request directly from your app.

With this capability, you can do things like:

Describe, summarize, or answer questions about audio content
Transcribe audio content
Analyze specific segments of audio using timestamps

Jump to code samples Jump to code for streamed responses

See other guides for additional options for working with audio
Generate structured output Multi-turn chat Bidirectional

Before you begin

If you haven't already, complete the getting started guide, which describes how to set up your Firebase project, connect your app to Firebase, add the SDK, initialize the Vertex AI service, and create a GenerativeModel instance.

For testing and iterating on your prompts and even getting a generated code snippet, we recommend using Vertex AI Studio.

Need a sample audio file?

You can use this publicly available file with a MIME type of audio/mp3 (view or download file). https://storage.googleapis.com/cloud-samples-data/generative-ai/audio/pixel.mp3

Send an audio file (base64-encoded) & receive text

Make sure that you've completed the Before you begin section of this guide before trying this sample.

You can ask a Gemini model to generate text by prompting with text and audio—providing the input file's mimeType and the file itself. Find requirements and recommendations for input files later on this page.

Swift

You can call generateContent() to generate text from multimodal input of text and a single audio file.

import FirebaseVertexAI

// Initialize the Vertex AI service
let vertex = VertexAI.vertexAI()

// Create a `GenerativeModel` instance with a model that supports your use case
let model = vertex.generativeModel(modelName: "gemini-2.0-flash")

// Provide the audio as `Data`
guard let audioData = try? Data(contentsOf: audioURL) else {
    print("Error loading audio data.")
    return // Or handle the error appropriately
}

// Specify the appropriate audio MIME type
let audio = InlineDataPart(data: audioData, mimeType: "audio/mpeg")


// Provide a text prompt to include with the audio
let prompt = "Transcribe what's said in this audio recording."

// To generate text output, call `generateContent` with the audio and text prompt
let response = try await model.generateContent(audio, prompt)

// Print the generated text, handling the case where it might be nil
print(response.text ?? "No text in response.")

Kotlin

You can call generateContent() to generate text from multimodal input of text and a single audio file.

^{For Kotlin, the methods in this SDK are suspend functions and need to be called from a Coroutine scope.}

// Initialize the Vertex AI service and create a `GenerativeModel` instance
// Specify a model that supports your use case
val generativeModel = Firebase.vertexAI.generativeModel("gemini-2.0-flash")

val contentResolver = applicationContext.contentResolver

val inputStream = contentResolver.openInputStream(audioUri)

if (inputStream != null) {  // Check if the audio loaded successfully
    inputStream.use { stream ->
        val bytes = stream.readBytes()

        // Provide a prompt that includes the audio specified above and text
        val prompt = content {
            inlineData(bytes, "audio/mpeg")  // Specify the appropriate audio MIME type
            text("Transcribe what's said in this audio recording.")
        }

        // To generate text output, call `generateContent` with the prompt
        val response = generativeModel.generateContent(prompt)

        // Log the generated text, handling the case where it might be null
        Log.d(TAG, response.text?: "")
    }
} else {
    Log.e(TAG, "Error getting input stream for audio.")
    // Handle the error appropriately
}

Java

You can call generateContent() to generate text from multimodal input of text and a single audio file.

^{For Java, the methods in this SDK return a ListenableFuture.}

// Initialize the Vertex AI service and create a `GenerativeModel` instance
// Specify a model that supports your use case
GenerativeModel gm = FirebaseVertexAI.getInstance()
        .generativeModel("gemini-2.0-flash");
GenerativeModelFutures model = GenerativeModelFutures.from(gm);

ContentResolver resolver = getApplicationContext().getContentResolver();

try (InputStream stream = resolver.openInputStream(audioUri)) {
    File audioFile = new File(new URI(audioUri.toString()));
    int audioSize = (int) audioFile.length();
    byte audioBytes = new byte[audioSize];
    if (stream != null) {
        stream.read(audioBytes, 0, audioBytes.length);
        stream.close();

        // Provide a prompt that includes the audio specified above and text
        Content prompt = new Content.Builder()
              .addInlineData(audioBytes, "audio/mpeg")  // Specify the appropriate audio MIME type
              .addText("Transcribe what's said in this audio recording.")
              .build();

        // To generate text output, call `generateContent` with the prompt
        ListenableFuture<GenerateContentResponse> response = model.generateContent(prompt);
        Futures.addCallback(response, new FutureCallback<GenerateContentResponse>() {
            @Override
            public void onSuccess(GenerateContentResponse result) {
                String text = result.getText();
                Log.d(TAG, (text == null) ? "" : text);
            }
            @Override
            public void onFailure(Throwable t) {
                Log.e(TAG, "Failed to generate a response", t);
            }
        }, executor);
    } else {
        Log.e(TAG, "Error getting input stream for file.");
        // Handle the error appropriately
    }
} catch (IOException e) {
    Log.e(TAG, "Failed to read the audio file", e);
} catch (URISyntaxException e) {
    Log.e(TAG, "Invalid audio file", e);
}

Web

You can call generateContent() to generate text from multimodal input of text and a single audio file.

import { initializeApp } from "firebase/app";
import { getVertexAI, getGenerativeModel } from "firebase/vertexai";

// TODO(developer) Replace the following with your app's Firebase configuration
// See: https://firebase.google.com/docs/web/learn-more#config-object
const firebaseConfig = {
  // ...
};

// Initialize FirebaseApp
const firebaseApp = initializeApp(firebaseConfig);

// Initialize the Vertex AI service
const vertexAI = getVertexAI(firebaseApp);

// Create a `GenerativeModel` instance with a model that supports your use case
const model = getGenerativeModel(vertexAI, { model: "gemini-2.0-flash" });

// Converts a File object to a Part object.
async function fileToGenerativePart(file) {
  const base64EncodedDataPromise = new Promise((resolve) => {
    const reader = new FileReader();
    reader.onloadend = () => resolve(reader.result.split(','));
    reader.readAsDataURL(file);
  });
  return {
    inlineData: { data: await base64EncodedDataPromise, mimeType: file.type },
  };
}

async function run() {
  // Provide a text prompt to include with the audio
  const prompt = "Transcribe what's said in this audio recording.";

  // Prepare audio for input
  const fileInputEl = document.querySelector("input[type=file]");
  const audioPart = await fileToGenerativePart(fileInputEl.files);

  // To generate text output, call `generateContent` with the text and audio
  const result = await model.generateContent([prompt, audioPart]);

  // Log the generated text, handling the case where it might be undefined
  console.log(result.response.text() ?? "No text in response.");
}

run();

Dart

You can call generateContent() to generate text from multimodal input of text and a single audio file.

import 'package:firebase_vertexai/firebase_vertexai.dart';
import 'package:firebase_core/firebase_core.dart';
import 'firebase_options.dart';

await Firebase.initializeApp(
  options: DefaultFirebaseOptions.currentPlatform,
);

// Initialize the Vertex AI service and create a `GenerativeModel` instance
// Specify a model that supports your use case
final model =
      FirebaseVertexAI.instance.generativeModel(model: 'gemini-2.0-flash');

// Provide a text prompt to include with the audio
final prompt = TextPart("Transcribe what's said in this audio recording.");

// Prepare audio for input
final audio = await File('audio0.mp3').readAsBytes();

// Provide the audio as `Data` with the appropriate audio MIME type
final audioPart = InlineDataPart('audio/mpeg', audio);

// To generate text output, call `generateContent` with the text and audio
final response = await model.generateContent([
  Content.multi([prompt,audioPart])
]);

// Print the generated text
print(response.text);

Learn how to choose a model and optionally a location appropriate for your use case and app.

Stream the response

Make sure that you've completed the Before you begin section of this guide before trying this sample.

You can achieve faster interactions by not waiting for the entire result from the model generation, and instead use to handle partial results. To stream the response, call generateContentStream.