PaliGemma prompt and system instructions  |  Google AI for Developers
Skip to main content
Models
Gemini
About
Docs
API reference
Pricing
Imagen
About
Docs
Pricing
Veo
About
Docs
Pricing
Gemma
About
Docs
Gemmaverse
Solutions
Build with Gemini
Gemini API
Google AI Studio
Customize Gemma open models
Gemma open models
Multi-framework with Keras
Fine-tune in Colab
Run on-device
Google AI Edge
Gemini Nano on Android
Chrome built-in web APIs
Build responsibly
Responsible GenAI Toolkit
Secure AI Framework
Code assistance
Android Studio
Chrome DevTools
Colab
Firebase
Google Cloud
JetBrains
Jules
VS Code
Community
Google AI Forum
Gemini for Research
/
English
Deutsch
Español – América Latina
Français
Indonesia
Italiano
Polski
Português – Brasil
Shqip
Tiếng Việt
Türkçe
Русский
עברית
العربيّة
فارسی
हिंदी
বাংলা
ภาษาไทย
中文 – 简体
中文 – 繁體
日本語
한국어
Sign in
Gemma
Gemma
Docs
Models
More
Gemma
Docs
Solutions
More
Code assistance
More
Community
More
Overview
Get started
Releases
Models
Core Gemma
Overview
Gemma 4 model card
Gemma 3 model card
Gemma 2 model card
Gemma 1 model card
Core Variants
Gemma 3n
Overview
Model card
DiffusionGemma
Overview
Model card
Diffusion Explained
Generate Output
FunctionGemma
Overview
Model card
Formatting and best practices
Function calling with Hugging Face Transformers
Full function calling sequence with FunctionGemma
Fine-tune FunctionGemma
EmbeddingGemma
Overview
Model card
Generate embeddings with Sentence Transformers
Fine-tune EmbeddingGemma
PaliGemma
Overview
v2 model card
v1 model card
Generate output with Keras
Fine-tune with JAX and Flax
Prompt and system instructions
ShieldGemma
Overview
ShieldGemma 2 Model card
ShieldGemma 1 Model card
Run Gemma
Fundamentals
Overview
Prompt Formatting
Legacy Gemma setup [Gemma 1, 2, and 3]
Legacy Prompt and system instructions [Gemma 1, 2, and 3]
Run locally with a Chat UI or integrate via API
LM Studio
Ollama
Run efficiently on Edge
LiteRT-LM
Llama.cpp
MLX
Build/Train in Python
Tunix (Tune-in-JAX)
Hugging Face Transformers
Keras
Unsloth
Deploy to Production / Enterprise
Gemini API
Google Cloud
Cloud GKE
Multi-Token Prediction (MTP)
Overview
Hugging Face Transformers
Core Capabilities
Text
Basic and multi-turn chat
Function calling
Visual data
Overview
Image understanding
Video understanding
Audio data
Thinking
Tuning guides
Overview
Tune using Hugging Face Transformers and QLoRA
Vision Tune using Hugging Face Transformers and QLoRA
Full model fine-tune using Hugging Face Transformers
Tune using Gemma library
Research and tools
RecurrentGemma
Overview
Inference using JAX and Flax
Fine-tune using JAX and Flax
Model card
DataGemma
Gemma Scope
Gemma-APS
Community
Gemmaverse
Discord
Legal
Terms of use
Gemma 4 license
Prohibited use
Intended use statement
Gemini
About
Docs
API reference
Pricing
Imagen
About
Docs
Pricing
Veo
About
Docs
Pricing
Gemma
About
Docs
Gemmaverse
Build with Gemini
Gemini API
Google AI Studio
Customize Gemma open models
Gemma open models
Multi-framework with Keras
Fine-tune in Colab
Run on-device
Google AI Edge
Gemini Nano on Android
Chrome built-in web APIs
Build responsibly
Responsible GenAI Toolkit
Secure AI Framework
Android Studio
Chrome DevTools
Colab
Firebase
Google Cloud
JetBrains
Jules
VS Code
Google AI Forum
Gemini for Research
Gemma 4 released with text, audio and image input and long up to 256K context window! Learn more
Home
Gemma
Models
Docs
Send feedback
PaliGemma prompt and system instructions
This page describes prompt formatting and system instructions for PaliGemma
models. These Gemma model variants use the same general formatting as Gemma
foundation models and support a special syntax for specific image-related
tasks.
Prompt format
PaliGemma models use the same prompt formatting as the Gemma foundation models
they are based on. However, PaliGemma models also support a special
task syntax, which is described in the next section. For
more information on Gemma prompt formatting, see
Gemma prompt and system instructions.
Note: although the pre-trained checkpoints support this prompt structure, they
require fine-tuning to your specific use cases.
Image and text data order
When prompting PaliGemma models with text and image data, the image data must
always be provided first, and then the text prompting data after it. Reversing
the order of image and text prompt data, or mixing image and text data will
typically generate unusable responses.
Prompt task syntax
The PaliGemma models are trained with specific prompt patterns and syntax for
tasks such as object identification and image captioning. You can use this
prompt task syntax to request specific behavior from the PaliGemma models as
follows:
"cap {lang}\n": Very raw short caption (only supported by PT)
"caption {lang}\n": Short captions
"describe {lang}\n": Somewhat longer, more descriptive captions (only supported by PT)
"ocr": Optical character recognition (only supported by PT)
"answer {lang} {question}\n": Question answering about the image contents
"question {lang} {answer}\n": Question generation for a given answer (only supported by PT)
"detect {object} ; {object}\n": Locate listed objects in an image and
return the bounding boxes for those objects
"segment {object} ; {object}\n": Locate the area occupied by the
listed objects in an image to create an image segmentation for that object
Note: the "object" for detection and segmentation can also be an expression,
such as "the penguin on the left".
The {lang} options are for language codes. PaliGemma supports language
recognition for 34 different languages for task prompts with this option. You
can find the list of supported languages on
GitHub.
Important: The PaliGemma FT category of
models are trained for specific tasks and only support the syntax for the
target task.
For a detailed code examples showing how to use this syntax, see the
Generate PaliGemma output with Keras
tutorial.
Prompting with natural language
Although the syntax in the previous section is recommended, the mix models also
support natural language for many of the tasks. For example, "describe this
image briefly" or "what is this text" will still work even if not prompted with
the exact syntax.
Batched prompt commands
You can provide more than one prompt command within a single prompt as a batch
of instructions. Each prompt command must end with a \n character. The
following example demonstrates how to structure your prompt text to provide
multiple instructions.
prompts = [
'answer en where is the cow standing?\n',
'answer en what color is the cow?\n',
'describe en\n',
'detect cow\n',
'segment cow\n',
]
images = [cow_image, cow_image, cow_image, cow_image, cow_image]
outputs = paligemma.generate(
inputs={
"images": images,
"prompts": prompts,
}
)
for output in outputs:
print(output)
System instructions
The PaliGemma models don't support any additional system instructions beyond
the Gemma system instructions
from the foundation models they are based on.
Send feedback
Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License. For details, see the Google Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.
Last updated 2025-02-25 UTC.
Need to tell us more?
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Missing the information I need","missingTheInformationINeed","thumb-down"],["Too complicated / too many steps","tooComplicatedTooManySteps","thumb-down"],["Out of date","outOfDate","thumb-down"],["Samples / code issue","samplesCodeIssue","thumb-down"],["Other","otherDown","thumb-down"]],["Last updated 2025-02-25 UTC."],[],[]]
Terms
Privacy
Manage cookies
English
Deutsch
Español – América Latina
Français
Indonesia
Italiano
Polski
Português – Brasil
Shqip
Tiếng Việt
Türkçe
Русский
עברית
العربيّة
فارسی
हिंदी
বাংলা
ภาษาไทย
中文 – 简体
中文 – 繁體
日本語
한국어