PhoenixKitAgent.Attachments.Processor (PhoenixKitAgent v0.1.0)

Copy Markdown View Source

The work done on an attachment before the turn that sends it (spec §9), in a task under PhoenixKitAgent.Attachments.TaskSupervisor: the row goes from pending to ready with what the model will read, or to failed with a code in error; the chat's topic then hears {:agent_attachment_processed, conversation_uuid, attachment_uuid}.

  • text and sheet — the text, or the CSV, cut to the limits of PhoenixKitAgent.Attachments.Bounds; truncated says it was cut;
  • pdf — its text (pdftotext); a PDF with no text is a scan and is read like a picture, from its first pages;
  • image — checked to be a picture ImageMagick can read.

A picture (or a scan) is shown to the model in one of two ways, decided here by the user's model as it is now (PhoenixKitAgent.Models.image_input/1, picture_mode/1):

  • the model is known to take images: the row stays without text — the turn sends the pixels, in its own message only (PhoenixKitAgent.Attachments.pictures/1); later turns carry a line that names the file, and the assistant's own answer about it;
  • it is known not to, or its abilities are not known (a provider without a model listing, a model that declares nothing): it is taken for a blind one, and the vision model of the module (agent_vision_endpoint_uuid) describes the picture once, now; the description is the row's text, which the history keeps. With no vision model a picture is refused by PhoenixKitAgent.Attachments.add/4 before it is stored, and a scan (known to be one only here) failed as no_vision_model. The vision call is paid, so right before it is made the AI module must be on (ai_disabled otherwise), the session that added the file must still be one that may use the assistant (session_gone or forbidden), and the user must be within their limits (limit_reached) — the money limits as they apply to a description on the vision endpoint (PhoenixKitAgent.Limits.money_counted?/2 with :vision: priced like a chat call — the provider's cost, a catalogue, a manual or a list price): one that nothing prices is held by the turn limit only. One user's descriptions are made one at a time (a :global lock per user, on every connected node), so each is checked against a journal that holds what the one before cost. The call carries the user and source: "agent" — the spending limits count it — and its content is never written to the AI journal.

Failing safely

A task never lives for ever (process_deadline_ms, then timeout), is kept to a heap size (a file that needs more is too_large), and turns every exception into a code; the codes are unreadable, too_large, pdf_tools_missing, image_tools_missing, no_vision_model, ai_disabled, session_gone, forbidden, limit_reached, budget_exceeded, model_error, timeout, crashed, unavailable (no place within queue_wait_ms, see Attachments.Slots, or no scratch space), and the refusals of the AI guard for the agent.attachment task: task_blocked, task_limit_exceeded, model_unfit, model_unverified, no_provider_for_params.

The work is two processes: a job under TaskSupervisor, which waits for a place (at most concurrent_jobs jobs hold one, jobs_per_user of them one user's), makes the scratch directory (Attachments.Work), and the worker under WorkerSupervisor, which reads the file in it. The job removes the directory when the worker is done or killed. The file's content, what a parser said about it and what the model answered never reach the log — not as a message, nor in a crash report.

Summary

Types

What the task found: the fields of the row to store.

Functions

Starts the processing of the pending attachment uuid of the user user_uuid, added in the browser session session_ref (checked again before a paid call), and returns at once. The task waits for a place (PhoenixKitAgent.Attachments.Slots) before it does any work.

Types

result()

@type result() :: {:ok, map()} | {:error, atom()}

What the task found: the fields of the row to store.

Functions

start(uuid, user_uuid, session_ref)

Starts the processing of the pending attachment uuid of the user user_uuid, added in the browser session session_ref (checked again before a paid call), and returns at once. The task waits for a place (PhoenixKitAgent.Attachments.Slots) before it does any work.