do I need to use API and config file ? because I don't see any opportunity for these models on UI, like for gpt 4o or gpt 5, I want to test them for one promt and several test cases https://cdn.discordapp.com/attachments/1455361841365782568/1455361841764237362/2025-12-30_034535.png?ex=695472d6&is=69532156&hm=3adff818900d90bbf54ff1fe8237be8a0f09bd80e1d5c2c27fb8c5e3796ec297& https://cdn.discordapp.com/attachments/1455361841365782568/1455361842309234752/2025-12-30_034428.png?ex=695472d6&is=69532156&hm=56a7f05fd3e506cfbf0bd4558783228e2275f28c53285ceffd24fe7c6bdad6c1&

I'm having a fair number of cases where prompts generated via redteaming include {{purpose | trim}} which then yields a

Template render error: (unknown path) TypeError: Cannot read properties of undefined (reading 'replace')

from nunjucks. Is

trim

a field I'm meant to be providing alongside

purpose

in the redteaming config? I'm not seeing it mentioned anywhere in the docs but see it used in some of the prompts in the source code.

Hey team! 👋 I've been exploring promptfoo for LLM evaluation for the past few hours, but I'm hitting some friction in my workflow and wanted to get your thoughts. The friction: - The current workflow requires manually editing YAML to create/tweak tests - There's no "playground" mode to quickly iterate on prompts before formalizing them as tests - I find myself wanting to tinker with a prompt, see the output, then decide if it should become a test case - Basically: explore → validate → formalize (vs the current: formalize → validate) What I'm imagining: A UI layer where I can: - Edit prompts with live variable substitution preview - Run single prompts against a model on-the-fly - Mark good outputs as "save as test case" with auto-generated assertions - Export everything to promptfooconfig.yaml when I'm done tinkering My questions: 1. Does something like this already exist in promptfoo that I'm missing? (The web UI seems read-only for results) 2. Is this kind of interactive "prompt tinkering" layer consistent with promptfoo's design philosophy, or is the code-first approach intentional? 3. If I were to build this, would you recommend: - A standalone tool that exports to promptfoo format - A fork/PR to add this to the existing web UI Happy to contribute if this aligns with where the project is heading. Just don't want to duplicate effort or go against the grain. Happy New Year 💌

Hi, I have a generated test case that is using the promptfoo:redteam:iterative jailbreak strategy. About a month ago, I set this up and when I’d run the eval for it, I would use --grader vertex:gemini-2.5-flash on the command line and it worked fine. Now I’m noticing that promptfoo tries to use gpt-5-2025-08-07 even when I set the grader on the command line. However, if in my redteam block I put redteam.provider: vertex:gemini-2.5-flash then it uses gemini. Any ideas what I might be doing wrong here? Did the behaviour change recently? Thanks

When using GPT5 as a judge i get this very often. I don't get why. My intuition is that it's happening when the answer to be judged is pretty long and maybe there is a timeout in some level of the llm-call. Could this make sense? Should i set some timeout config in the provider to be sure? https://cdn.discordapp.com/attachments/1461732259353595916/1461732259525693703/image.png?ex=697cc340&is=697b71c0&hm=3f7ebcc13aa0c0e9d4491c7fede32c15e0fc04dc52c467256a369dad90aa1d10&

I am using the python provider and I am running some script before doing the proper query to the LLM. if would like the time of this script not to be in the Latency calculation. The strategy that i tried was to move it in the beforeEach/beforeAll hooks but it still looks that i have the same latencies in the output. do you have an idea if this make sense? should i use some other way to do it? thanks in advance peace, Marco

Right now, if you hide some table columns, the preference is lost the next time you run the evaluation, which doesn't make sense. After every run, I have to deselect all and click 5 times to select the columns I want to see in the dashboard THIS IS AN EXTREMELY FRUSTRATING EXPERIENCE.

Is there a reason why Gemini models don't have any pricing calculation available? All other major providers have it available, but nothing in the code is there to calculate costs for runs using Gemini models. And weirdest of all, there are no issues on GitHub with anyone requesting it. Looks like it's never been discussed at all.

When a provider throws rate limit error or token limit error,it will be handled by promotfoo internally and assertion process will be skipped for that.How we can catch that and apply assertion logic to that? Can anyone help regarding this. Thank you!

hello, created a pr for supporting concurrency between conversationIds. https://github.com/promptfoo, does anyone have time to look at it? i would like to know what needs to be done to merge this

Hey everyone! We’re thinking about contributing a new feature to Promptfoo - adding support for using Microsoft SQL Server as a database backend, in addition to the existing SQLite support. Before we dive into any design or implementation details, we wanted to check with the community first: Would you be open to a pull request that adds optional MS SQL Server support to the open‑source promptfoo?

I'm trying to run gemini version 3.1 through the foo prompt, but I'm getting this error:

Could not identify provider: gemini-3.1-pro-preview
Please check your configuration and ensure the provider is correctly specified.
For more information on supported providers, visit: https://promptfoo.dev/docs/providers/
file:///~/node_modules/promptfoo/dist/src/providers-BOvl9qv5.js:33135
    throw new Error(errorMessage);

Am I putting in the wrong name by any chance, or is there really no way to use this version yet?

Hey, I've been trying to use rag-poisoning plugin for documents generation, but upon running assessment these documents fails to be generated with fallowing error that I will upload to this post. https://cdn.discordapp.com/attachments/1475457665592332329/1475457720231399635/image.png?ex=699d8e95&is=699c3d15&hm=d8d40d925faeaf91071491b6e3b4dab48b550418c4525e76dfa1b8eca409f9c6&

I'm trying to run a test against a tool with an embedded form. I'm using css data selectors for the form input and elements, it works for the initial search; however, it timesout for future follow-ups despite the form field existing on the page. This is my browser config:

yaml

  - id: browser
    label: search-copilot
    config:
      persistSession: true
      connectOptions:
        debuggingPort: 9222 
      steps:
        # Wait before navigating to look more like a human
        - action: wait
          runOnce: true
          args:
            ms: 2000
        # Navigate to the search copilot page
        - action: navigate
          runOnce: true
          args:
            url: https://<REDACTED>/analytics/search
        # Wait for the page to load
        - action: wait
          runOnce: true
          args:
            ms: 5000
        # Type the prompt into the search copilot input
        - action: type
          args:
            selector: 'div[data-test-id="search-copilot-input"]'
            text: '{{prompt}}<enter>'
            delay: 100
        # Wait for the new chat entry to appear
        - action: wait
          args:
            ms: 1000 # Wait for the new chat entry to appear
        - action: waitForNewChildren
          args:
            parentSelector: 'div[data-test-id="search-copilot-latest-chat-entry"]'
            delay: 500 # Check every 500ms
            timeout: 30000 # Max wait time 30 seconds
        # Extract the text from the new chat entry and store it in the response variable
        - action: extract
          args:
            script: |
              const entry = document.querySelector('div[data-test-id="search-copilot-latest-chat-entry"]');

              if (!entry) {
                console.warn('Latest chat entry not found');
              } else {
                // Heuristic: paragraphs that appear in the main content stack (first block of p.MuiTypography-root after the title span)
                // 1) Find the title span (user prompt) within the entry
                const title = entry.querySelector('span.MuiTypography-root');

                // 2) From the title’s next siblings, find the first block that contains paragraph typography
                let contentRoot = null;
                for (let el = title?.parentElement?.nextElementSibling; el; el = el.nextElementSibling) {
                  const candidate = el.querySelector('p.MuiTypography-root');
                  if (candidate) {
                    contentRoot = el;
                    break;
                  }
                }

                const paras = contentRoot
                  ? Array.from(contentRoot.querySelectorAll('p.MuiTypography-root'))
                  : [];

                const paragraphs = paras.map(p => p.textContent.trim());
                const fullText = paragraphs.join('\n\n');

                return fullText || '';
              }
          name: aiResponse
        - action: wait
          args:
            ms: 1000
        - action: screenshot
          args:
            path: 'debug-{{sessionId}}-{{_attempt}}.png'
        - action: wait
          args:
            ms: 2000
    delay: 3000
    transformResponse: 'extracted.aiResponse'

The timeout error is:

text
Timeout waiting for selector: div[data-test-id="search-copilot-input"]

But the element is visible on the page when the timeout error presents. I'm not sure what I'm doing wrong or how to get it to recognize the input box repopulating on the page after the mui stream completes

Hii, love the vision for this product I'm highly enthusiastic to use this solution of promptfoo to have a project application wise analysis + evaluation dashboard along with a quick configurable playground, where everything right from the test iterations, model used, prompt set used, model configuration of the run, golden data run. Etc. all must be persisted and stored in a DB for generating a data point in the main application/project wise configurable dashboard, where the Cost, Accuracy, Precision, Recall, Latency, etc. values across various iterative runs are visualised simply and clearly, effectively allowing for a call for action so as to warrent a prompt update or a model change or something else. FYI: The usecsse is in the toxicity screening domain, i.e. moderation. Kindly share your thoughts and how we can go about this proactively, open to all lets keep this thread alive.

Hello there, I have just to start with AI cyber security. My question is pretty simple. Can I test Gandalf AI with prmptfoo if I can the next question is: how can I do that? Thanks in advance

I’m evaluating candidate LLMs for my app and need to compare models in a setup where context comes from local files, fetched by my own agent layer (OpenClaw), not by Promptfoo directly. Is it possible to evaluate models using this external context flow as part of Promptfoo evaluation? Promptfoo is currently running tests through a custom provider (defined in a local file), which calls my OpenClaw layer and returns the model output. https://cdn.discordapp.com/attachments/1486021607053856829/1486021607372754974/message.txt?ex=69c4a5b8&is=69c35438&hm=b348f0ae3d2b8691281ec3449d143202db44f1a6f1a40a23bc16be43fb86d55c&

I think there's a bug when prompts are loaded from python + doing

promptfoo:simulated-user

tests. Promptfoo is reading the file content as the prompt, not the function return. When the config and Python are the following,

# config.yaml
providers:
  - openai:gpt-5-mini

prompts:
  - file://generate_prompt.py:create_prompt

defaultTest:
  provider:
    id: "promptfoo:simulated-user"
    config:
      maxTurns: 2

tests:
  - vars:
      instructions: Learn how to use react.js

# -----------------------------------------
# generate_prompt.py

from system_prompt import my_prompt

def create_prompt(context):
    return my_prompt

# -----------------------------------------
# system_prompt.py (storing full prompt in another file)

my_prompt = "Always respond in 10 words or less"

Assistant doesn't follow

my_prompt

. Full response in image 1.png If I update

generate_prompt.py

to the following, assistant response will be <= 10 words

# generate_prompt.py
from system_prompt import my_prompt

def create_prompt(context):
    return "Always respond in 10 words or less"

Assistant will respond in 10 words. Full response in image 2.png. But I believe The prompt works fine when - loaded inline in yaml - loaded by json - there's no simulated provider for the tests. https://cdn.discordapp.com/attachments/1487832982466199666/1487832982839361836/Screenshot_2026-03-29_at_11.05.31_PM.png?ex=69ca93f1&is=69c94271&hm=3740559e5a811653ca0bf4403033ac9948a1d17ebcaf9da9dfc3e4d664e0f28b& https://cdn.discordapp.com/attachments/1487832982466199666/1487832983506260050/Screenshot_2026-03-29_at_11.06.59_PM.png?ex=69ca93f1&is=69c94271&hm=56ff17d097cf03825e0b138ae8845fb762af28215789fc190f69c53610f5798a&

How are you guys managing prompts in practice? I’m realizing prompt management is not one problem. Some prompts are easy to store/version in tools like PromptLayer or similar registries, and some can be evaluated with straightforward tests. But a lot of prompts seem harder to evaluate because their quality only shows up in a real workflow, with full context, tools, and downstream effects. In those cases, the judgment can be pretty subjective and closer to an end-to-end or integration test than a unit test. So I’m curious how people are actually handling this in real systems: • Where do you store/version different kinds of prompts? • How do you decide which prompts can be tested directly vs only evaluated in-context? • For subjective prompts like code review, planning, or UI generation, what does “good enough” evaluation look like for you? • Are you using prompt registries, Git, RAG, human review, E2E evals, or some combination?

from doc i see 2 version are 20.20, 22.22

My real environment is audio-to-audio with gemini 3.1 flash live model with multi turn conversation. I want to use promptfoo to simulate the conversation (

promptfoo:simulated-user

). 1. Is it possible to let

promptfoo:simulated-user

to do an audio-to-audio conversation with

gooogle:live:gemini-3.1-flash-live-preview

and run assert to validate the quality? 2. If not, do you know if using

response_modalities: ['text']

yields similar output? If output will be similar, i'm okay using text modality **Update**: I can't connect to

gooogle:live:gemini-3.1-flash-live-preview

at all even with

promptfoo init --example google-live-audio

. - In text modality, I get

WebSocket connection closed unexpectedly. Code: 1011, Reason: Internal error encountered.

. - In audio modality, I get

[ERROR] WebSocket request timed out

(

google:live:gemini-2.5-flash-native-audio-preview-12-2025

works fine)

Hi team, Is there a way to troubleshoot/debug errors from assert context-faithfulness ? I had one of these notification after running eval with verbose, At least one evaluation failure occurred. This might be caused by the underlying call to the provider, or a test failure. Context: [{"id":"8868fc514e5f7bd1a350480c5a825cd0f5b47b6c0c8da0ca62be71158f638c23","raw":"{{query}}","label":"{{query}}","provider":"http","metrics":{"score":0.6,"testPassCount":0,"testFailCount":1,"testErrorCount":0,"assertPassCount":0,"assertFailCount":1,"totalLatencyMs":8228,"tokenUsage":{"prompt":0,"completion":0,"cached":0,"total":0,"numRequests":1,"completionDetails":{"reasoning":0,"acceptedPrediction":0,"rejectedPrediction":0,"cacheReadInputTokens":0,"cacheCreationInputTokens":0},"assertions":{"total":4876,"prompt":3851,"completion":1025,"cached":0,"numRequests":0,"completionDetails":{"reasoning":0,"acceptedPrediction":0,"rejectedPrediction":0,"cacheReadInputTokens":0,"cacheCreationInputTokens":0}}},"namedScores":{"Faithfulness_Check":0.6},"namedScoresCount":{"Faithfulness_Check":1},"namedScoreWeights":{"Faithfulness_Check":1},"cost":0}}] Thanks

Hi Promptfoo team, I’m Zhi-Xin, a full-stack engineer from Taiwan, contacting you on behalf of our team at 緯大智控科技有限公司. We are currently evaluating Promptfoo and are interested in learning more about the Promptfoo Enterprise plan. I previously submitted our information through the official website contact form / email, but we have not received a response yet, so I’m following up here on Discord. We would like to know more about: 1. Enterprise features What additional features are included in Promptfoo Enterprise compared with the open-source/community version? 2. Data privacy and security Since we may use Promptfoo to evaluate internal or confidential data, we would like to understand how Enterprise handles sensitive information. Specifically, are prompts, model responses, evaluation results, uploaded data, or other confidential information sent to Promptfoo Cloud? Can Promptfoo guarantee that this data remains private, is not used for model training, and is not disclosed to third parties? 3. Pricing How is Enterprise pricing calculated? For example, is it based on users/seats, usage volume, number of evaluations, deployment type, or other factors? 4. Enterprise On-Prem deployment If available, we would like to learn whether Promptfoo Enterprise can be deployed fully inside our internal environment. Could someone from the Enterprise sales or support team contact me here, or let me know the best way to reach the right person? Thank you!

Hi, I understand that tool calls args are generated purely by the LLM, but most orchestrators 'know' the context/conversation under which the tool call is performed. At the moment, any function in [

functionToolCallbacks

](https://www.promptfoo.dev/docs/providers/openai/#using-functions:~:text=and%20parameters.-,functionToolCallbacks,-A%20map%20of) is invoked with just the param string. Since the callback has no information about the context, it is difficult to emulate the function as closely as one may like to the real implementation, during the eval. Adding context (

testCase.metadata

or

testCase.vars

) allows [function callbacks](https://www.promptfoo.dev/docs/providers/openai/#using-functions:~:text=and%20parameters.-,functionToolCallbacks,-A%20map%20of) to be somewhat stateful. Since there is little control over test case execution order in promptfoo, contextual callbacks cannot be used in cases where ordering is required. I'm not asking for that control. The examples in the doc led me to try this workaround

json
{
  type: "function",
  function: {
    name: "get_temperature",
    parameters: {
      type: "object",
        properties: {
          unit: {
            type: "string",
            enum: (vars) => vars.context_id,
          }
        },
    }
  }
}

It might work for the providers that fully support structured

json_schema

. However, I wasn't able to test this (my provider doesn't have this, it violated the schema and generated some nonsense string). I'm curious to hear if this is worthy of inclusion in promptfoo. I'd be happy to contribute if some one can share some pointers. PS: Congrats on the acquisition!

Hi, is there away set the config for azure openai reasoning models to see the reaosning the reasoning in promptfoo?

Examples: - JSON object -> quoted JSON string - scalar enum -> explanatory sentence - tool call / next step changes unexpectedly Do you usually catch this with promptfoo/schema assertions/logs, or is it still painful to debug? I’m sanity-checking whether this is a real problem outside my own workflows.

Hi guys. I have some skill I'd like to evaluate with Promptfoo. The structure looks like this

.
├── references
│   └── workflows
│       ├── sub-task-a.md
│       ├── sub-task-b.md
│       └── sub-task-c.md
└── SKILL.md

The skill has roughly 5 steps. First, it needs to decide which reference file to use based on the user prompt to execute a certain sub-task. The last step would call some

curl

command to validate the generated results is valid or not. My questions: 1. How should I configure promptfoo to verify that the right reference file is used for a given prompt? 2. How should I assert that

curl

command? Currently, I use custom provider with tracing enabled to verify the

curl

command and its payload. I'd like to learn from the community how would you guys do these evaluations. Thank you in advanced.

how to resume the existing scan as authentication token expires after 30 mins.

Does anyone use a prompt verification tool on a daily basis before executing a prompt? The benefits would be minimise cost, iterations, time prompting, error intend communication with AI. The goal is to get the best possible prompt in term of cost execution and quality of outputs. The idea would be to use different verification processes according to the complexity of the task asked so that for simple request a minimal verification is done but for more complex request, a deeper verification is made. The use cases can be regular prompts, custom gpts, skills, agent spec. I tried online tools for prompting check but it is hard to find a good balance between not enough verification and overkill prompt recommendations. I also tried to build a tool for that but the quality of outputs are not that good. Otherwise I asked also chat got to improve it but I end up with an heavy architecture for a prompt OS. I don't want the process to take longer or get more expensive than just executing the prompt and manually verifying the quality of the outputs. I don't want to increase complexity where it is not needed and at same time communicate as efficiently as possible my intention to the AI. What are your thoughts? How are your daily routines? What tools are you using?

How can I make questions that would fail chat gpt most of the time?