Images
Send text and images in the same prompt. Choose a model with image input; libfx doesn't send images to a separate fallback model.
Send an image
A session stores each prompt as JSON before it runs, so a session prompt takes image data as a base64 string. Set IMAGE_MODEL to the id of an image-capable model from your model catalog, then run:
import { readFile } from 'node:fs/promises'
import { createFxAgent } from 'libfx'
const agent = createFxAgent({ model: process.env.IMAGE_MODEL })
const data = (await readFile('screenshot.png')).toString('base64')
const turn = agent.session().prompt([
{ type: 'text', text: 'Describe this screenshot.' },
{ type: 'image', data, mimeType: 'image/png' },
])
for await (const event of turn) {
if (event.type === 'text_delta') process.stdout.write(event.delta)
}
await agent.close()
Use your own screenshot.png. An engine from createFxEngine() also takes raw bytes, a Blob, or a File, such as file from a file input:
const turn = engine.prompt([
{ type: 'text', text: 'Describe this screenshot.' },
{ type: 'image', data: file },
])
A Blob's type must name a supported image format, unless resizeImage converts it. Reading a Blob happens after prompt() returns, and a failed read rejects turn.result.
Supported images
PNG, JPEG, GIF, and WebP are supported, and the file contents must match mimeType or the Blob's type. prompt() throws on invalid base64, an unsupported type, or contents that don't match, in an engine or a session. A durable agent runs this check before it stores a prompt, so a prompt it refuses is never stored. With resizeImage, the type and contents of the image it returns are checked instead, when the turn runs. An engine reads a Blob after prompt() returns, so a Blob whose contents don't match rejects turn.result instead.
Each prompt takes up to 8 images, with at most 3.75 MiB of raw data per image and 6 MiB in total. Text, encoded images, and metadata must fit in an 8 MiB frame.
libfx sends images unchanged and doesn't resize them on its own. A model request takes images up to 8,000 pixels per side, or up to 2,000 when the request carries more than 20 images, including images from earlier turns and from tools. An image over that limit isn't sent, and the model sees a note in its place.
For durable sessions, the input, caller context, and session settings must fit within 8 MiB of JSON before anything is stored. This check happens before resizeImage can reduce an image: prepare oversized input in your app before calling prompt() or steer(). See Input limits.
Prepare images
Pass a resizeImage callback to resize or convert prompt images before they're sent. With sharp in Node.js:
import sharp from 'sharp'
import { createFxAgent } from 'libfx'
const agent = createFxAgent({
model: process.env.IMAGE_MODEL,
async resizeImage({ bytes }) {
const resized = await sharp(bytes)
.resize({ width: 1568, withoutEnlargement: true })
.png().toBuffer()
return { bytes: resized, mimeType: 'image/png' }
},
})
Install sharp in your application to use this callback. In a browser, use your own image code and return an ArrayBuffer or typed array. The limits apply to the callback's output. If the callback fails, a session's turn ends with stopReason: 'error' and the reason in error, and an engine turn's result rejects. The callback runs on prompt images only, not tool results.
Keep an original reference
Add a sourceRef, up to 512 bytes of text, to name an original your application stores:
{ type: 'image', data, mimeType: 'image/png', sourceRef: 'uploads:screenshot-1' }
You can also omit data and send the reference with its MIME type. libfx doesn't fetch the original. If an image can't be sent, the model receives the reference, so your tools can provide a suitable copy. An image with a sourceRef that exceeds the prompt's size limits is sent as its reference alone instead of failing the prompt. Your application authorizes access to the original. A reference-only block doesn't save the original file.
Saved images
A durable agent saves each image once with its session, whether a prompt, a steer, or one of your tools carries it. The saved conversation refers to each image by name, so images don't count toward the session's 4 MiB checkpoint limit. Sending the same image again doesn't store it twice.
A session keeps its newest 4,096 images, up to 512 MiB of them, counting prompt and tool images together. Older images leave the conversation, and the model sees a note in place of each.
These keep image data inline instead, so it counts toward their 4 MiB limit:
- A checkpoint you save yourself holds image data and references.
- An engine with a
persistencestore of its own keeps images inside its records and checkpoints. For a conversation with many images, use a durable agent.
Every model request carries the conversation's images again. When they would make a request too large to send, libfx leaves the oldest ones out of that request, each replaced by a note, and the conversation still keeps them.