How to Write Prompts for Vision and Multi-Modal Models

Step 1: Understand What Multi-Modal Models Can See

Before writing prompts, calibrate your expectations against what current models can actually perceive in images. Modern vision models are strong at identifying objects, reading printed text (OCR), interpreting charts and graphs, describing scenes, recognizing logos and brands, and understanding spatial relationships between objects. They handle photographs, diagrams, screenshots, scanned documents, and hand-drawn sketches.

They are weaker at reading handwritten text (accuracy varies dramatically with handwriting quality), counting objects precisely (they estimate rather than count when there are more than 10-15 items), interpreting very small text in large images, understanding 3D spatial depth from 2D images, and recognizing faces (many models are intentionally restricted from identifying specific individuals). They also struggle with images that contain dense, overlapping information where visual elements are ambiguous.

Image resolution and quality directly affect results. A 4000x3000 pixel product photo produces better analysis than a 200x150 thumbnail of the same image. When the model needs to read text in an image (receipts, labels, screenshots), higher resolution makes a measurable difference. If you can control the input image quality, default to higher resolution. Most APIs accept images up to 20MB, and the additional cost of processing a larger image is negligible compared to the accuracy improvement.

Step 2: Write Clear Visual Task Instructions

The biggest mistake in multi-modal prompting is sending an image with a vague instruction like "What do you see?" or "Describe this image." These prompts produce generic descriptions that rarely answer the question you actually have. Instead, state exactly what you want the model to do with the image.

Compare these prompts for analyzing a product screenshot:

The specific prompt tells the model what type of image it is looking at (checkout page), what to look for (UI problems), what categories of problems to consider (alignment, truncation, size, contrast, labels), and how to format the output (location + severity). This produces a structured, actionable response rather than "I see a web page with a form and some buttons."

For extraction tasks, specify exactly what data to pull and in what format. "Extract all text from this receipt and return it as JSON with keys: store_name, date, items (array of {name, quantity, price}), subtotal, tax, total." The model performs OCR guided by your schema, which is both more accurate and more useful than unstructured text extraction.

Step 3: Provide Visual Context and Reference Points

When your question is about a specific part of an image, direct the model's attention explicitly. Models process the entire image and may focus on the wrong elements without guidance.

"In the top-right quadrant of this dashboard screenshot, there is a line chart showing revenue over time. What trend does it show for the last 6 months, and is there a visible inflection point?" is much more effective than "What does the chart show?" because the image might contain multiple charts, tables, and metrics.

For images with annotations, reference them explicitly. "The red circle in this architectural diagram highlights a component. What role does that component play in the overall system, based on the connections shown?" The model can see colored annotations and use them as spatial references.

When comparing multiple images, label them. "Image 1 is the current design. Image 2 is the proposed redesign. List the specific differences between the two designs." Without labels, the model may confuse which image is which, especially if they are similar.

Step 4: Combine Text and Image Inputs Effectively

The most powerful multi-modal prompts use text context to guide visual analysis. Sending an image alone asks the model to figure out what matters. Sending an image with relevant text context tells the model what you already know and what you need it to determine.

For code review with screenshots: "This screenshot shows a React component rendering in the browser. The expected behavior is a centered card with a blue header and white background. The actual behavior is what you see in the screenshot. What CSS properties are likely wrong based on the visual differences from the expected behavior?"

For medical or scientific images (where the model can assist professionals, not replace them): "This is a soil sample cross-section photo from a construction site assessment. The sample was taken at 2 meters depth in clay-dominant soil. Based on the visible layers, coloring, and moisture patterns, describe the soil composition and any visible anomalies that might affect foundation bearing capacity."

The text context dramatically improves output quality because the model can combine its visual perception with the domain knowledge activated by your description. Without the context that this is a soil sample at 2 meters depth, the model might describe it as "a brown and gray layered substance," which is technically correct but useless.

Step 5: Handle Document and Chart Analysis

Documents, tables, and charts are the most common production use case for multi-modal prompting. These tasks require specific techniques because the model needs to extract structured information from visual layouts.

For tables: "Extract the data from this table as a JSON array. Each row should be an object with keys matching the column headers. Preserve the exact values, do not round numbers or paraphrase text. If a cell is empty, use null." Tables with merged cells, rotated headers, or complex formatting benefit from additional guidance: "The table has merged cells in the first column that span multiple rows. Apply the merged cell's value to all rows it spans."

For charts: Ask for specific data points rather than general descriptions. "What was the value for Q3 2025 in the blue line?" produces a precise answer. "What are the approximate values for each data point on the bar chart?" extracts the actual data the chart represents. For trend analysis: "Describe the trend, identify any outliers, and estimate the growth rate between the first and last data points."

For PDFs and documents: Multi-modal models handle PDFs natively (Claude accepts PDF files directly). For multi-page documents, specify what you need: "This is a 15-page contract. Extract only the payment terms, termination clauses, and liability limitations. For each, quote the exact language and provide the page number." Without scope constraints, the model will try to summarize the entire document, which dilutes the useful information.

For diagrams and flowcharts: "This is an architecture diagram for our microservices platform. List every service shown, the connections between them (including direction), and identify any single points of failure where one service's outage would cascade to others." The model can trace connections in diagrams and reason about system topology, which is especially useful for documentation review and architecture audits.

Step 6: Validate and Iterate on Visual Outputs

Multi-modal tasks have more output variability than text-only tasks because visual perception is inherently less precise than text processing. The same image can produce slightly different descriptions on different runs, and the model may miss details that are obvious to a human viewer.

For critical applications, use verification prompts. After the model extracts data from an image, send the same image again with the extracted data and ask: "Here is the data I extracted from this image: [data]. Verify each value against the image. Flag any values that do not match what you see." This two-pass approach catches extraction errors that a single pass misses.

For tasks where accuracy is essential (financial documents, legal contracts, medical records), treat multi-modal extraction as a first draft that requires human review. The model gets 90-95% of values correct on clean, high-resolution documents, but the remaining 5-10% can include critical errors, especially on handwritten annotations, poor quality scans, and unusual formatting. Build your workflow to flag low-confidence extractions for human verification rather than trusting all outputs equally.

Test with representative samples from your actual data. A prompt that works perfectly on clean stock photos may fail on real-world images with poor lighting, partial occlusion, unusual angles, or unexpected content. Build a test set of 20-30 images that represent the full range of quality and content you expect in production, and evaluate your prompt against all of them before deploying.