kimaai_eval_regression

When it fires

At the end of a scheduled run of the evaluation cases (Chatbot, Scheduled runs, Run on a schedule), when the share of passed cases is below “Alert threshold (%)”. The run’s batches run in the background through Action Scheduler, so the action fires in a background job, after the run was saved and after KimaAI sent its own alert to “Alert email” and “Alert webhook (optional)”, when they are set. Runs from WP-CLI do not fire it.

Use it to send the alert where your team works, in the format your tool expects, or to open a ticket.

Parameters

NameTypeDescription
summaryarrayThe run’s result, with total, passed, percent and categories (per category its total and pass).
failuresarrayThe failed cases, each with id, category, pass, failures (the reasons), answer and seconds.
thresholdintThe alert threshold in percent, 95 by default.

Example

add_action(
  'kimaai_eval_regression',
  function (array $summary, array $failures, int $threshold): void {
    // The team chat expects {"text": "..."}.
    $lines = [sprintf('Chatbot quality fell to %d%% (alert below %d%%).', $summary['percent'], $threshold)];
    foreach ($failures as $failure) {
      $lines[] = sprintf('- %s: %s', $failure['id'], implode('; ', $failure['failures']));
    }

    wp_remote_post('https://chat.example.com/hooks/quality', [
      'headers' => ['Content-Type' => 'application/json'],
      'body' => wp_json_encode(['text' => implode("\n", $lines)]),
    ]);
  },
  10,
  3
);
  • kimaai_chat_response_saved: fires for every saved answer, the evaluation’s included.

What are your feelings