Tasks#

class TasksClient(*, sdk_config: SDKConfiguration, generated_client: ApiClient)[source]#

Bases: object

Client for managing Arize tasks and task runs.

This class is primarily intended for internal use within the SDK. Users are highly encouraged to access resource-specific functionality via arize.ArizeClient.

The tasks client is a thin wrapper around the generated REST API client, using the shared generated API client owned by arize.config.SDKConfiguration.

Parameters:

  • sdk_config (SDKConfiguration) – Resolved SDK configuration.

  • generated_client (ApiClient) – Shared generated API client instance.

list(*, name: str | None = None, project: str | None = None, dataset: str | None = None, space: str | None = None, task_type: TaskType | None = None, limit: int = DEFAULT_LIST_LIMIT, cursor: str | None = None) → ListTasksResponse[source]#

List tasks the user has access to.

Results support cursor-based pagination. Optionally filter by space, project, dataset, or task type.

Parameters:

  • name (str | None) – Optional case-insensitive substring filter on the task name.

  • project (str | None) – Optional project name or identifier (base64) to filter results. If the value is a name, space must also be provided.

  • dataset (str | None) – Optional dataset name or identifier (base64) to filter results. If the value is a name, space must also be provided.

  • space (str | None) – Optional space name or ID used to disambiguate name-based resolution for project and dataset. If the value is a base64-encoded resource ID it is treated as a space ID; otherwise it is used as a case-insensitive substring filter on the space name.

  • task_type (TaskType | None) – Optional task type filter. One of "TEMPLATE_EVALUATION" or "CODE_EVALUATION".

  • limit (int) – Maximum number of tasks to return (1-100).

  • cursor (str | None) – Opaque pagination cursor from a previous response.

Returns:

A paginated task list response from the Arize REST API.

Raises:

ApiException – If the API request fails.

Return type:

ListTasksResponse

get(*, task: str, space: str | None = None) → Task[source]#

Get a task by name or ID.

Parameters:

  • task (str) – Task name or identifier (base64). If the value looks like an ID it is used directly; otherwise it is resolved by name.

  • space (str | None) – Optional space name or ID used to disambiguate the task lookup. Recommended when resolving by name.

Returns:

The task with its full configuration.

Raises:

ApiException – If the API request fails (for example, task not found).

Return type:

Task

create_evaluation_task(*, name: str, task_type: TaskType, evaluators: builtins.list[TaskEvaluatorInput], project: str | None = None, dataset: str | None = None, space: str | None = None, experiment_ids: builtins.list[str] | None = None, sampling_rate: float | None = None, is_continuous: bool | None = None, query_filter: str | None = None, query_filters: TaskQueryFilters | None = None) → Task[source]#

Create a new evaluation task.

A typed convenience wrapper around the internal task-creation logic for "TEMPLATE_EVALUATION" and "CODE_EVALUATION" task types. Prefer this method when creating evaluation tasks for a cleaner, narrowly-typed signature.

Parameters:

  • name (str) – Task name (must be unique within the space).

  • task_type (TaskType) – Task type: "TEMPLATE_EVALUATION" or "CODE_EVALUATION".

  • evaluators (builtins.list[TaskEvaluatorInput]) –

    List of evaluators to attach (at least one required). Each entry is a arize.tasks.types.TaskEvaluatorInput with the following fields:

    • evaluator_id — Evaluator identifier (base64). Required.

    • evaluator_version_id — Pins the evaluator to one version (base64). Optional; omit it to run the latest version.

    • query_filter — Per-evaluator filter. Optional (span shape).

    • column_mappings — Template variable name mappings. Optional (span shape).

    • query_mappings — Variable-to-query mappings. Optional (trace/session shape).

  • project (str | None) – Project name or identifier (base64). Required when dataset is not provided.

  • dataset (str | None) – Dataset name or identifier (base64). Required when project is not provided.

  • space (str | None) – Optional space name or ID used to disambiguate name-based resolution for project and dataset.

  • experiment_ids (builtins.list[str] | None) – Experiment identifiers (base64). Required (at least one) when dataset is provided.

  • sampling_rate (float | None) – Fraction of data to evaluate (0-1). Only valid for project-based tasks.

  • is_continuous (bool | None) – Whether to run the task continuously. Only valid for project-based tasks.

  • query_filter (str | None) – Task-level query filter for span-granularity evaluators. Mutually exclusive with query_filters.

  • query_filters (TaskQueryFilters | None) – Combined named query filters and boolean expression for trace/session evaluators. Pass a arize.tasks.types.TaskQueryFilters object. Mutually exclusive with query_filter.

Returns:

The newly created task.

Raises:

  • ValueError – If required fields are missing or mutually exclusive fields are combined.

  • ApiException – If the API request fails.

Return type:

Task

create_run_experiment_task(*, name: str, dataset: str, run_configuration: RunConfiguration | RunConfigurationRequest | AgentCallRunConfig | AgentCallRunConfigRequest | LlmGenerationRunConfig | LlmGenerationRunConfigRequest | TemplateEvaluationRunConfig | TemplateEvaluationRunConfigRequest | dict, space: str | None = None) → Task[source]#

Create a new run_experiment task.

A typed convenience wrapper around the internal task-creation logic for "RUN_EXPERIMENT" task types. The server drives all LLM calls using the AI integration specified in run_configuration — no local callable is required.

To create and immediately trigger a run in one call, use create_and_run_experiment_task (available separately).

Parameters:
Returns:

The newly created task.

Raises:

ApiException – If the API request fails.

Return type:

Task

update(*, task: str, space: str | None = None, name: str | UNSET = _UNSET, sampling_rate: float | UNSET = _UNSET, is_continuous: bool | UNSET = _UNSET, query_filter: str | None | UNSET = _UNSET, query_filters: TaskQueryFilters | None | UNSET = _UNSET, evaluators: builtins.list[TaskEvaluatorInput] | UNSET = _UNSET, run_configuration: RunConfiguration | RunConfigurationRequest | AgentCallRunConfig | AgentCallRunConfigRequest | LlmGenerationRunConfig | LlmGenerationRunConfigRequest | TemplateEvaluationRunConfig | TemplateEvaluationRunConfigRequest | dict | UNSET = _UNSET) → Task[source]#

Update mutable fields on an existing task.

Dispatches based on the task’s type — resolves the task by ID or name first, then GETs it to determine whether it is an evaluation task or a run_experiment task, and builds the appropriate PATCH body.

At least one mutable field must be provided. Pass None to query_filter to clear the existing filter; omit the argument to leave it unchanged.

For evaluation tasks (template_evaluation / code_evaluation):

  • Valid fields: name, sampling_rate, is_continuous, query_filter, query_filters, evaluators.

  • run_configuration must not be provided.

For run_experiment tasks:

  • Valid fields: name, run_configuration.

  • Evaluation-only fields (sampling_rate, is_continuous, query_filter, query_filters, evaluators) must not be provided.

Parameters:

  • task (str) – Task name or identifier (base64). Names are resolved within the space when space is provided.

  • space (str | None) – Optional space name or ID used to disambiguate task name resolution.

  • name (str | UNSET) – New display name for the task.

  • sampling_rate (float | UNSET) – Fraction of data to evaluate (0-1). Evaluation tasks only, project-based tasks only.

  • is_continuous (bool | UNSET) – Whether the task runs continuously. Evaluation tasks only.

  • query_filter (str | None | UNSET) – Task-level query filter for span-granularity evaluators, or None to clear. Mutually exclusive with query_filters. Evaluation tasks only.

  • query_filters (TaskQueryFilters | None | UNSET) – Combined named query filters and boolean expression for trace/session evaluators, or None to clear. Pass a arize.tasks.types.TaskQueryFilters object with an empty filters list to clear. Mutually exclusive with query_filter. Evaluation tasks only.

  • evaluators (builtins.list[TaskEvaluatorInput] | UNSET) – Full replacement list of evaluators (at least one when provided). Evaluation tasks only.

  • run_configuration (RunConfiguration | RunConfigurationRequest | AgentCallRunConfig | AgentCallRunConfigRequest | LlmGenerationRunConfig | LlmGenerationRunConfigRequest | TemplateEvaluationRunConfig | TemplateEvaluationRunConfigRequest | dict | UNSET) – Replacement run configuration. When provided the entire stored config is atomically replaced. run_experiment tasks only.

Returns:

The updated task.

Raises:

  • ValueError – If no update fields were provided, or if a field is not valid for the resolved task type.

  • ApiException – If the API request fails.

Return type:

Task

delete(*, task: str, space: str | None = None) → None[source]#

Delete a task and its associated configuration.

Parameters:

  • task (str) – Task name or identifier (base64).

  • space (str | None) – Optional space name or ID used when resolving by task name.

Raises:

ApiException – If the API request fails.

Return type:

None

trigger_run(*, task: str, space: str | None = None, data_start_time: datetime | None = None, data_end_time: datetime | None = None, max_spans: int | None = None, override_evaluations: bool | None = None, experiment_ids: builtins.list[str] | None = None, experiment_name: str | None = None, dataset_version_id: str | None = None, max_examples: int | None = None, example_ids: builtins.list[str] | None = None, evaluation_task_ids: builtins.list[str] | None = None, tracing_metadata: dict[str, Any] | None = None) → TaskRun[source]#

Trigger an on-demand run for a task.

Dispatches based on the task’s type — resolves the task by ID or name first, then GETs it to determine whether it is an evaluation task or a run_experiment task, and builds the appropriate trigger body.

For evaluation tasks (template_evaluation / code_evaluation):

  • Valid fields: data_start_time, data_end_time, max_spans, override_evaluations, experiment_ids.

  • All fields are optional; an empty trigger body uses server defaults.

For run_experiment tasks:

  • Valid fields: experiment_name (required), dataset_version_id, max_examples, example_ids, tracing_metadata, evaluation_task_ids.

  • experiment_name is the display name for the new experiment that will be created for this run.

  • example_ids and max_examples are mutually exclusive; at most one may be provided. When both are omitted, all examples are used.

Parameters:

  • task (str) – Task name or identifier (base64) to trigger a run for.

  • space (str | None) – Optional space name or ID used to disambiguate the task lookup. Recommended when resolving by name.

  • data_start_time (datetime | None) – Start of the data window to evaluate. Evaluation tasks only.

  • data_end_time (datetime | None) – End of the data window to evaluate. Defaults to now when omitted. Evaluation tasks only.

  • max_spans (int | None) – Maximum number of spans to process (default 10 000). Evaluation tasks only.

  • override_evaluations (bool | None) – Whether to re-evaluate data that already has evaluation labels. Defaults to False. Evaluation tasks only.

  • experiment_ids (builtins.list[str] | None) – Experiment identifiers (base64) to run against. Only applicable for dataset-based evaluation tasks.

  • experiment_name (str | None) – Display name for the experiment to be created. Must be unique within the dataset. Required for run_experiment tasks.

  • dataset_version_id (str | None) – Dataset version identifier (base64). Defaults to the latest version when omitted. run_experiment tasks only.

  • max_examples (int | None) – Maximum number of examples to run (dataset order). Mutually exclusive with example_ids. When both are omitted, all examples are used. run_experiment tasks only.

  • example_ids (builtins.list[str] | None) – Specific dataset example identifiers (base64) to run against. Mutually exclusive with max_examples. When both are omitted, all examples are used. run_experiment tasks only.

  • tracing_metadata (dict[str, Any] | None) – Arbitrary key-value metadata attached to the run’s traces. run_experiment tasks only.

  • evaluation_task_ids (builtins.list[str] | None) – Task identifiers (base64) of evaluation tasks to trigger after the experiment run completes. Supported for all run_experiment experiment types. run_experiment tasks only.

Returns:

The newly created task run (initially in "pending" status).

Raises:

  • ValueError – If a field is not valid for the resolved task type, if experiment_name is missing for a run_experiment task, or if both example_ids and max_examples are provided.

  • ApiException – If the API request fails.

Return type:

TaskRun

list_runs(*, task: str, space: str | None = None, status: RunStatus | None = None, limit: int = DEFAULT_LIST_LIMIT, cursor: str | None = None) → ListTaskRunsResponse[source]#

List runs for a task.

Results support cursor-based pagination. Optionally filter by run status.

Parameters:

  • task (str) – Task name or identifier (base64) to list runs for.

  • space (str | None) – Optional space name or ID used to disambiguate the task lookup. Recommended when resolving by name.

  • status (RunStatus | None) – Optional run status filter. One of "pending", "running", "completed", "failed", or "cancelled".

  • limit (int) – Maximum number of runs to return (1-100).

  • cursor (str | None) – Opaque pagination cursor from a previous response.

Returns:

A paginated task run list response from the Arize REST API.

Raises:

ApiException – If the API request fails.

Return type:

ListTaskRunsResponse

get_run(*, run_id: str) → TaskRun[source]#

Get a task run by its unique identifier.

Parameters:

run_id (str) – Task run identifier (base64) to retrieve.

Returns:

The task run with its current status and statistics.

Raises:

ApiException – If the API request fails (for example, run not found).

Return type:

TaskRun

cancel_run(*, run_id: str) → TaskRun[source]#

Cancel a task run.

Only valid when the run’s current status is "pending" or "running".

Parameters:

run_id (str) – Task run identifier (base64) to cancel.

Returns:

The updated task run with status "cancelled".

Raises:

ApiException – If the API request fails (for example, run not found or already in terminal state).

Return type:

TaskRun

wait_for_run(*, run_id: str, poll_interval: float = _DEFAULT_POLL_INTERVAL, timeout: float = _DEFAULT_TIMEOUT) → TaskRun[source]#

Poll a task run until it reaches a terminal state.

Repeatedly calls get_run at poll_interval-second intervals until the run’s status is one of "COMPLETED", "FAILED", or "CANCELLED", or until timeout seconds have elapsed.

Parameters:

  • run_id (str) – Task run identifier (base64) to wait for.

  • poll_interval (float) – Seconds between polling attempts. Defaults to 5.

  • timeout (float) – Maximum seconds to wait before raising TimeoutError. Defaults to 600.

Returns:

The task run in its terminal state.

Raises:

  • ValueError – If timeout or poll_interval is not positive.

  • TimeoutError – If the run does not reach a terminal state within timeout seconds.

  • ApiException – If any polling request fails.

Return type:

TaskRun

Usage Examples#

Span-granularity task#

Span tasks use a query_filter string at the task level and pass SpanEvaluatorInput entries to evaluators.

from arize.tasks import TasksClient, TaskType, SpanEvaluatorInput

client = TasksClient(...)

task = client.create_evaluation_task(
    name="Weekly Quality Check",
    task_type=TaskType.TEMPLATE_EVALUATION,
    space="my-space",
    project="my-project",
    query_filter="span_kind = 'LLM'",
    evaluators=[
        SpanEvaluatorInput(
            evaluator_id="your_evaluator_id",
            column_mappings={"input": "attributes.input.value", "output": "attributes.output.value"},
        )
    ],
)

Trace/session-granularity task (multi-span query)#

Trace/session tasks use query_filters (a TaskQueryFilters object with named filters and an optional boolean expression) at the task level and pass TraceOrSessionEvaluatorInput entries to evaluators.

from arize.tasks import (
    TasksClient,
    TaskType,
    TaskQueryFilter,
    TaskQueryFilters,
    TaskQueryMapping,
    TraceOrSessionEvaluatorInput,
)

client = TasksClient(...)

task = client.create_evaluation_task(
    name="Trace Quality Check",
    task_type=TaskType.TEMPLATE_EVALUATION,
    space="my-space",
    project="my-project",
    query_filters=TaskQueryFilters(
        filters=[
            TaskQueryFilter(id="A", filter="span_kind = 'LLM'"),
            TaskQueryFilter(id="B", filter="span_kind = 'RETRIEVER'"),
        ],
        expression="A AND B",
    ),
    evaluators=[
        TraceOrSessionEvaluatorInput(
            evaluator_id="your_evaluator_id",
            query_mappings=[
                TaskQueryMapping(variable_name="input", query_ids=["A"], attribute_path="attributes.input.value"),
                TaskQueryMapping(variable_name="output", query_ids=["B"], attribute_path="attributes.output.value"),
            ],
        )
    ],
)

Updating a task#

Both query_filter and query_filters can be updated; they are mutually exclusive — provide at most one.

# Switch to trace/session shape
task = client.update(
    task="your_task_id",
    space="my-space",
    query_filters=TaskQueryFilters(
        filters=[TaskQueryFilter(id="A", filter="span_kind = 'LLM'")],
        expression="A",
    ),
    evaluators=[
        TraceOrSessionEvaluatorInput(
            evaluator_id="your_evaluator_id",
            query_mappings=[
                TaskQueryMapping(variable_name="input", query_ids=["A"], attribute_path="attributes.input.value"),
            ],
        )
    ],
)

Response Types#

Public type re-exports for the tasks subdomain.

class AgentCallRunConfig(*, experiment_type: Annotated[str, Strict(strict=True)], integration_id: Annotated[str, Strict(strict=True)], input_template: Dict[str, Any], additional_properties: Dict[str, Any] = {})[source]#

Bases: BaseModel

Configuration for running an agent integration against each dataset example. The input_template is sent to the agent after Mustache substitution.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:
experiment_type: StrictStr#
integration_id: StrictStr#
input_template: Dict[str, Any]#
additional_properties: Dict[str, Any]#
classmethod experiment_type_validate_enum(value)[source]#

Validates the enum

model_config: ClassVar[ConfigDict] = {'populate_by_name': True, 'protected_namespaces': (), 'validate_assignment': True, 'validate_by_alias': True, 'validate_by_name': True}#

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

to_str() → str[source]#

Returns the string representation of the model using alias

Return type:

str

to_json() → str[source]#

Returns the JSON representation of the model using alias

Return type:

str

classmethod from_json(json_str: str) → Self | None[source]#

Create an instance of AgentCallRunConfig from a JSON string

Parameters:

json_str (str)

Return type:

Self | None

to_dict() → Dict[str, Any][source]#

Return the dictionary representation of the model using alias.

This has the following differences from calling pydantic’s self.model_dump(by_alias=True):

  • None is only added to the output dict for nullable fields that were set at model initialization. Other fields with value None are ignored.

  • Fields in self.additional_properties are added to the output dict.

Return type:

Dict[str, Any]

classmethod from_dict(obj: Dict[str, Any] | None) → Self | None[source]#

Create an instance of AgentCallRunConfig from a dict

Parameters:

obj (Dict[str, Any] | None)

Return type:

Self | None

class AgentCallRunConfigRequest(*, experiment_type: Annotated[str, Strict(strict=True)], integration_id: Annotated[str, Strict(strict=True)], input_template: Dict[str, Any])[source]#

Bases: BaseModel

Strict request configuration for running an agent integration.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:
experiment_type: StrictStr#
integration_id: StrictStr#
input_template: Dict[str, Any]#
classmethod experiment_type_validate_enum(value)[source]#

Validates the enum

model_config: ClassVar[ConfigDict] = {'populate_by_name': True, 'protected_namespaces': (), 'validate_assignment': True, 'validate_by_alias': True, 'validate_by_name': True}#

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

to_str() → str[source]#

Returns the string representation of the model using alias

Return type:

str

to_json() → str[source]#

Returns the JSON representation of the model using alias

Return type:

str

classmethod from_json(json_str: str) → Self | None[source]#

Create an instance of AgentCallRunConfigRequest from a JSON string

Parameters:

json_str (str)

Return type:

Self | None

to_dict() → Dict[str, Any][source]#

Return the dictionary representation of the model using alias.

This has the following differences from calling pydantic’s self.model_dump(by_alias=True):

  • None is only added to the output dict for nullable fields that were set at model initialization. Other fields with value None are ignored.

Return type:

Dict[str, Any]

classmethod from_dict(obj: Dict[str, Any] | None) → Self | None[source]#

Create an instance of AgentCallRunConfigRequest from a dict

Parameters:

obj (Dict[str, Any] | None)

Return type:

Self | None

class ListTaskRunsResponse(*, task_runs: List[TaskRun], pagination: PaginationMetadata)[source]#

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:
task_runs: List[TaskRun]#
pagination: PaginationMetadata#
model_config: ClassVar[ConfigDict] = {'populate_by_name': True, 'protected_namespaces': (), 'validate_assignment': True, 'validate_by_alias': True, 'validate_by_name': True}#

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

to_str() → str[source]#

Returns the string representation of the model using alias

Return type:

str

to_json() → str[source]#

Returns the JSON representation of the model using alias

Return type:

str

classmethod from_json(json_str: str) → Self | None[source]#

Create an instance of ListTaskRunsResponse from a JSON string

Parameters:

json_str (str)

Return type:

Self | None

to_dict() → Dict[str, Any][source]#

Return the dictionary representation of the model using alias.

This has the following differences from calling pydantic’s self.model_dump(by_alias=True):

  • None is only added to the output dict for nullable fields that were set at model initialization. Other fields with value None are ignored.

Return type:

Dict[str, Any]

classmethod from_dict(obj: Dict[str, Any] | None) → Self | None[source]#

Create an instance of ListTaskRunsResponse from a dict

Parameters:

obj (Dict[str, Any] | None)

Return type:

Self | None

to_df(by_alias: bool = False, exclude_none: str | bool = True, json_normalize: bool = False, convert_dtypes: bool = True, expand_field: str = 'additional_properties', expand_prefix: str = '') → pd.DataFrame#

Convert a list of objects to a pandas.DataFrame.

Behavior:

  • If an item is a Pydantic v2 model, use .model_dump(by_alias=…).

  • If an item is a mapping (dict-like), use it as-is.

  • Otherwise, raise a ValueError (unsupported row type).

Parameters:

  • self (object) – The object instance containing the field to convert.

  • by_alias (bool) – Use field aliases when dumping Pydantic models.

  • exclude_none (str | bool) – Control None/NaN column dropping. - False: keep Nones as-is - “all”: drop columns where all values are None/NaN - “any”: drop columns where any value is None/NaN - True: alias for “all”

  • json_normalize (bool) – If True, flatten nested dicts via pandas.json_normalize.

  • convert_dtypes (bool) – If True, call DataFrame.convert_dtypes() at the end.

  • expand_field (str) – If set, look for this field in each row and expand its keys into top-level columns.

  • expand_prefix (str) – If set, prefix expanded column names with this string.

Returns:

The converted DataFrame.

Return type:

pandas.DataFrame

class ListTasksResponse(*, tasks: list[Task], pagination: PaginationMetadata)[source]#

Bases: BaseModel

SDK view of the generated list response with each Task’s run_configuration unwrapped.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:
tasks: list[Task]#
pagination: PaginationMetadata#
model_config: ClassVar[ConfigDict] = {'from_attributes': True}#

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

to_df(by_alias: bool = False, exclude_none: str | bool = True, json_normalize: bool = False, convert_dtypes: bool = True, expand_field: str = 'additional_properties', expand_prefix: str = '') → pd.DataFrame#

Convert a list of objects to a pandas.DataFrame.

Behavior:

  • If an item is a Pydantic v2 model, use .model_dump(by_alias=…).

  • If an item is a mapping (dict-like), use it as-is.

  • Otherwise, raise a ValueError (unsupported row type).

Parameters:

  • self (object) – The object instance containing the field to convert.

  • by_alias (bool) – Use field aliases when dumping Pydantic models.

  • exclude_none (str | bool) – Control None/NaN column dropping. - False: keep Nones as-is - “all”: drop columns where all values are None/NaN - “any”: drop columns where any value is None/NaN - True: alias for “all”

  • json_normalize (bool) – If True, flatten nested dicts via pandas.json_normalize.

  • convert_dtypes (bool) – If True, call DataFrame.convert_dtypes() at the end.

  • expand_field (str) – If set, look for this field in each row and expand its keys into top-level columns.

  • expand_prefix (str) – If set, prefix expanded column names with this string.

Returns:

The converted DataFrame.

Return type:

pandas.DataFrame

class LlmGenerationRunConfig(*, experiment_type: Annotated[str, Strict(strict=True)], ai_integration_id: Annotated[str, Strict(strict=True)], model_name: Annotated[str, Strict(strict=True)] | None = None, messages: Annotated[List[LLMMessage], MinLen(min_length=1)], input_variable_format: InputVariableFormat, invocation_parameters: InvocationParams | None = None, provider_parameters: Dict[str, Any] | None = None, tool_config: ToolConfig | None = None, prompt_version_id: Annotated[str, Strict(strict=True)] | None = None, additional_properties: Dict[str, Any] = {})[source]#

Bases: BaseModel

Configuration for running an LLM prompt against each dataset example.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:

  • experiment_type (Annotated[str, Strict(strict=True)])

  • ai_integration_id (Annotated[str, Strict(strict=True)])

  • model_name (Annotated[str, Strict(strict=True)] | None)

  • messages (Annotated[List[LLMMessage], FieldInfo(annotation=NoneType, required=True, metadata=[MinLen(min_length=1)])])

  • input_variable_format (InputVariableFormat)

  • invocation_parameters (InvocationParams | None)

  • provider_parameters (Dict[str, Any] | None)

  • tool_config (ToolConfig | None)

  • prompt_version_id (Annotated[str, Strict(strict=True)] | None)

  • additional_properties (Dict[str, Any])

experiment_type: StrictStr#
ai_integration_id: StrictStr#
model_name: StrictStr | None#
messages: Annotated[List[LLMMessage], Field(min_length=1)]#
input_variable_format: InputVariableFormat#
invocation_parameters: InvocationParams | None#
provider_parameters: Dict[str, Any] | None#
tool_config: ToolConfig | None#
prompt_version_id: StrictStr | None#
additional_properties: Dict[str, Any]#
classmethod experiment_type_validate_enum(value)[source]#

Validates the enum

model_config: ClassVar[ConfigDict] = {'populate_by_name': True, 'protected_namespaces': (), 'validate_assignment': True, 'validate_by_alias': True, 'validate_by_name': True}#

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

to_str() → str[source]#

Returns the string representation of the model using alias

Return type:

str

to_json() → str[source]#

Returns the JSON representation of the model using alias

Return type:

str

classmethod from_json(json_str: str) → Self | None[source]#

Create an instance of LlmGenerationRunConfig from a JSON string

Parameters:

json_str (str)

Return type:

Self | None

to_dict() → Dict[str, Any][source]#

Return the dictionary representation of the model using alias.

This has the following differences from calling pydantic’s self.model_dump(by_alias=True):

  • None is only added to the output dict for nullable fields that were set at model initialization. Other fields with value None are ignored.

  • Fields in self.additional_properties are added to the output dict.

Return type:

Dict[str, Any]

classmethod from_dict(obj: Dict[str, Any] | None) → Self | None[source]#

Create an instance of LlmGenerationRunConfig from a dict

Parameters:

obj (Dict[str, Any] | None)

Return type:

Self | None

class LlmGenerationRunConfigRequest(*, experiment_type: Annotated[str, Strict(strict=True)], ai_integration_id: Annotated[str, Strict(strict=True)], model_name: Annotated[str, Strict(strict=True)] | None = None, messages: Annotated[List[LLMMessageRequest], MinLen(min_length=1)], input_variable_format: InputVariableFormat, invocation_parameters: InvocationParamsRequest | None = None, provider_parameters: Dict[str, Any] | None = None, tool_config: ToolConfigRequest | None = None, prompt_version_id: Annotated[str, Strict(strict=True)] | None = None)[source]#

Bases: BaseModel

Strict request configuration for running an LLM prompt against each dataset example.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:
experiment_type: StrictStr#
ai_integration_id: StrictStr#
model_name: StrictStr | None#
messages: Annotated[List[LLMMessageRequest], Field(min_length=1)]#
input_variable_format: InputVariableFormat#
invocation_parameters: InvocationParamsRequest | None#
provider_parameters: Dict[str, Any] | None#
tool_config: ToolConfigRequest | None#
prompt_version_id: StrictStr | None#
classmethod experiment_type_validate_enum(value)[source]#

Validates the enum

model_config: ClassVar[ConfigDict] = {'populate_by_name': True, 'protected_namespaces': (), 'validate_assignment': True, 'validate_by_alias': True, 'validate_by_name': True}#

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

to_str() → str[source]#

Returns the string representation of the model using alias

Return type:

str

to_json() → str[source]#

Returns the JSON representation of the model using alias

Return type:

str

classmethod from_json(json_str: str) → Self | None[source]#

Create an instance of LlmGenerationRunConfigRequest from a JSON string

Parameters:

json_str (str)

Return type:

Self | None

to_dict() → Dict[str, Any][source]#

Return the dictionary representation of the model using alias.

This has the following differences from calling pydantic’s self.model_dump(by_alias=True):

  • None is only added to the output dict for nullable fields that were set at model initialization. Other fields with value None are ignored.

Return type:

Dict[str, Any]

classmethod from_dict(obj: Dict[str, Any] | None) → Self | None[source]#

Create an instance of LlmGenerationRunConfigRequest from a dict

Parameters:

obj (Dict[str, Any] | None)

Return type:

Self | None

class RunConfiguration(*args, oneof_schema_1_validator: LlmGenerationRunConfig | None = None, oneof_schema_2_validator: TemplateEvaluationRunConfig | None = None, oneof_schema_3_validator: AgentCallRunConfig | None = None, actual_instance: AgentCallRunConfig | LlmGenerationRunConfig | TemplateEvaluationRunConfig | None = None, one_of_schemas: Set[str] = {'AgentCallRunConfig', 'LlmGenerationRunConfig', 'TemplateEvaluationRunConfig'}, discriminator_value_class_map: Dict[str, str] = {})[source]#

Bases: BaseModel

Experiment execution configuration for a RUN_EXPERIMENT task. Exactly one variant must be supplied, identified by experiment_type. All fields sit at the top level alongside experiment_type (flat — no wrapper sub-object).

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:
oneof_schema_1_validator: LlmGenerationRunConfig | None#
oneof_schema_2_validator: TemplateEvaluationRunConfig | None#
oneof_schema_3_validator: AgentCallRunConfig | None#
actual_instance: AgentCallRunConfig | LlmGenerationRunConfig | TemplateEvaluationRunConfig | None#
one_of_schemas: Set[str]#
model_config: ClassVar[ConfigDict] = {'protected_namespaces': (), 'validate_assignment': True}#

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

discriminator_value_class_map: Dict[str, str]#
classmethod actual_instance_must_validate_oneof(v)[source]#
classmethod from_dict(obj: str | Dict[str, Any]) → Self[source]#
Parameters:

obj (str | Dict[str, Any])

Return type:

Self

classmethod from_json(json_str: str) → Self[source]#

Returns the object represented by the json string

Parameters:

json_str (str)

Return type:

Self

to_json() → str[source]#

Returns the JSON representation of the actual instance

Return type:

str

to_dict() → Dict[str, Any] | AgentCallRunConfig | LlmGenerationRunConfig | TemplateEvaluationRunConfig | None[source]#

Returns the dict representation of the actual instance

Return type:

Dict[str, Any] | AgentCallRunConfig | LlmGenerationRunConfig | TemplateEvaluationRunConfig | None

to_str() → str[source]#

Returns the string representation of the actual instance

Return type:

str

class RunConfigurationRequest(*args, oneof_schema_1_validator: LlmGenerationRunConfigRequest | None = None, oneof_schema_2_validator: TemplateEvaluationRunConfigRequest | None = None, oneof_schema_3_validator: AgentCallRunConfigRequest | None = None, actual_instance: AgentCallRunConfigRequest | LlmGenerationRunConfigRequest | TemplateEvaluationRunConfigRequest | None = None, one_of_schemas: Set[str] = {'AgentCallRunConfigRequest', 'LlmGenerationRunConfigRequest', 'TemplateEvaluationRunConfigRequest'}, discriminator_value_class_map: Dict[str, str] = {})[source]#

Bases: BaseModel

Strict request form of an experiment execution configuration. Exactly one variant must be supplied, identified by experiment_type.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:
oneof_schema_1_validator: LlmGenerationRunConfigRequest | None#
oneof_schema_2_validator: TemplateEvaluationRunConfigRequest | None#
oneof_schema_3_validator: AgentCallRunConfigRequest | None#
actual_instance: AgentCallRunConfigRequest | LlmGenerationRunConfigRequest | TemplateEvaluationRunConfigRequest | None#
one_of_schemas: Set[str]#
model_config: ClassVar[ConfigDict] = {'protected_namespaces': (), 'validate_assignment': True}#

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

discriminator_value_class_map: Dict[str, str]#
classmethod actual_instance_must_validate_oneof(v)[source]#
classmethod from_dict(obj: str | Dict[str, Any]) → Self[source]#
Parameters:

obj (str | Dict[str, Any])

Return type:

Self

classmethod from_json(json_str: str) → Self[source]#

Returns the object represented by the json string

Parameters:

json_str (str)

Return type:

Self

to_json() → str[source]#

Returns the JSON representation of the actual instance

Return type:

str

to_dict() → Dict[str, Any] | AgentCallRunConfigRequest | LlmGenerationRunConfigRequest | TemplateEvaluationRunConfigRequest | None[source]#

Returns the dict representation of the actual instance

Return type:

Dict[str, Any] | AgentCallRunConfigRequest | LlmGenerationRunConfigRequest | TemplateEvaluationRunConfigRequest | None

to_str() → str[source]#

Returns the string representation of the actual instance

Return type:

str

RunStatus#

alias of TaskRunStatus

class Task(*, id: str, name: str, type: str, project_id: str | None = None, dataset_id: str | None = None, sampling_rate: float | None = None, is_continuous: bool, query_filter: str | None = None, query_filters: TaskQueryFilters | None = None, evaluators: list[TaskEvaluator], experiment_ids: list[str], run_configuration: AgentCallRunConfig | LlmGenerationRunConfig | TemplateEvaluationRunConfig | None = None, last_run_at: datetime | None = None, created_at: datetime, updated_at: datetime, created_by_user_id: str | None = None)[source]#

Bases: BaseModel

SDK view of the generated Task with run_configuration unwrapped.

The run_configuration field holds the concrete inner type (AgentCallRunConfig, LlmGenerationRunConfig, or TemplateEvaluationRunConfig) instead of the oneOf wrapper RunConfiguration.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:
id: str#
name: str#
type: str#
project_id: str | None#
dataset_id: str | None#
sampling_rate: float | None#
is_continuous: bool#
query_filter: str | None#
query_filters: TaskQueryFilters | None#
evaluators: list[TaskEvaluator]#
experiment_ids: list[str]#
run_configuration: AgentCallRunConfig | LlmGenerationRunConfig | TemplateEvaluationRunConfig | None#
last_run_at: datetime | None#
created_at: datetime#
updated_at: datetime#
created_by_user_id: str | None#
model_config: ClassVar[ConfigDict] = {'from_attributes': True}#

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class TaskEvaluator(*, evaluator_id: Annotated[str, Strict(strict=True)], evaluator_name: Annotated[str, Strict(strict=True)], evaluator_version_id: Annotated[str, Strict(strict=True)] | None, query_filter: Annotated[str, Strict(strict=True)] | None, column_mappings: Dict[str, Annotated[str, Strict(strict=True)]] | None, query_mappings: List[TaskQueryMapping] | None = None)[source]#

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:
evaluator_id: StrictStr#
evaluator_name: StrictStr#
evaluator_version_id: StrictStr | None#
query_filter: StrictStr | None#
column_mappings: Dict[str, StrictStr] | None#
query_mappings: List[TaskQueryMapping] | None#
model_config: ClassVar[ConfigDict] = {'populate_by_name': True, 'protected_namespaces': (), 'validate_assignment': True, 'validate_by_alias': True, 'validate_by_name': True}#

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

to_str() → str[source]#

Returns the string representation of the model using alias

Return type:

str

to_json() → str[source]#

Returns the JSON representation of the model using alias

Return type:

str

classmethod from_json(json_str: str) → Self | None[source]#

Create an instance of TaskEvaluator from a JSON string

Parameters:

json_str (str)

Return type:

Self | None

to_dict() → Dict[str, Any][source]#

Return the dictionary representation of the model using alias.

This has the following differences from calling pydantic’s self.model_dump(by_alias=True):

  • None is only added to the output dict for nullable fields that were set at model initialization. Other fields with value None are ignored.

Return type:

Dict[str, Any]

classmethod from_dict(obj: Dict[str, Any] | None) → Self | None[source]#

Create an instance of TaskEvaluator from a dict

Parameters:

obj (Dict[str, Any] | None)

Return type:

Self | None

class TaskEvaluatorInput(*args, oneof_schema_1_validator: SpanEvaluatorInput | None = None, oneof_schema_2_validator: TraceOrSessionEvaluatorInput | None = None, actual_instance: SpanEvaluatorInput | TraceOrSessionEvaluatorInput | None = None, one_of_schemas: Set[str] = {'SpanEvaluatorInput', 'TraceOrSessionEvaluatorInput'})[source]#

Bases: BaseModel

An evaluator attachment supplied when creating or updating a task. At least one entry is required on evaluation-task requests. Evaluators carry one of two mutually exclusive shapes: span evaluators use query_filter + column_mappings; trace/session evaluators use query_mappings.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:

  • oneof_schema_1_validator (SpanEvaluatorInput | None)

  • oneof_schema_2_validator (TraceOrSessionEvaluatorInput | None)

  • actual_instance (SpanEvaluatorInput | TraceOrSessionEvaluatorInput | None)

  • one_of_schemas (Set[str])

oneof_schema_1_validator: SpanEvaluatorInput | None#
oneof_schema_2_validator: TraceOrSessionEvaluatorInput | None#
actual_instance: SpanEvaluatorInput | TraceOrSessionEvaluatorInput | None#
one_of_schemas: Set[str]#
model_config: ClassVar[ConfigDict] = {'protected_namespaces': (), 'validate_assignment': True}#

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

classmethod actual_instance_must_validate_oneof(v)[source]#
classmethod from_dict(obj: str | Dict[str, Any]) → Self[source]#
Parameters:

obj (str | Dict[str, Any])

Return type:

Self

classmethod from_json(json_str: str) → Self[source]#

Returns the object represented by the json string

Parameters:

json_str (str)

Return type:

Self

to_json() → str[source]#

Returns the JSON representation of the actual instance

Return type:

str

to_dict() → Dict[str, Any] | SpanEvaluatorInput | TraceOrSessionEvaluatorInput | None[source]#

Returns the dict representation of the actual instance

Return type:

Dict[str, Any] | SpanEvaluatorInput | TraceOrSessionEvaluatorInput | None

to_str() → str[source]#

Returns the string representation of the actual instance

Return type:

str

class TaskQueryFilter(*, id: Annotated[str, Strict(strict=True)], filter: Annotated[str, Strict(strict=True)])[source]#

Bases: BaseModel

A single named task-level query filter (trace/session shape). The id is a logical label used in query_filters.expression and per-evaluator query_mappings.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:

  • id (Annotated[str, FieldInfo(annotation=NoneType, required=True, metadata=[Strict(strict=True)])])

  • filter (Annotated[str, Strict(strict=True)])

id: Annotated[str, Field(strict=True)]#
filter: StrictStr#
classmethod id_validate_regular_expression(value)[source]#

Validates the regular expression

model_config: ClassVar[ConfigDict] = {'populate_by_name': True, 'protected_namespaces': (), 'validate_assignment': True, 'validate_by_alias': True, 'validate_by_name': True}#

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

to_str() → str[source]#

Returns the string representation of the model using alias

Return type:

str

to_json() → str[source]#

Returns the JSON representation of the model using alias

Return type:

str

classmethod from_json(json_str: str) → Self | None[source]#

Create an instance of TaskQueryFilter from a JSON string

Parameters:

json_str (str)

Return type:

Self | None

to_dict() → Dict[str, Any][source]#

Return the dictionary representation of the model using alias.

This has the following differences from calling pydantic’s self.model_dump(by_alias=True):

  • None is only added to the output dict for nullable fields that were set at model initialization. Other fields with value None are ignored.

Return type:

Dict[str, Any]

classmethod from_dict(obj: Dict[str, Any] | None) → Self | None[source]#

Create an instance of TaskQueryFilter from a dict

Parameters:

obj (Dict[str, Any] | None)

Return type:

Self | None

class TaskQueryFilters(*, filters: Annotated[List[TaskQueryFilter], MinLen(min_length=1), MaxLen(max_length=5)], expression: Annotated[str, Strict(strict=True)] | None = None)[source]#

Bases: BaseModel

Combined named-query filters and boolean expression for the trace/session shape. Supply this object OR query_filter (span shape) — not both.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:

  • filters (Annotated[List[TaskQueryFilter], FieldInfo(annotation=NoneType, required=True, metadata=[MinLen(min_length=1), MaxLen(max_length=5)])])

  • expression (Annotated[str, Strict(strict=True)] | None)

filters: Annotated[List[TaskQueryFilter], Field(min_length=1, max_length=5)]#
expression: StrictStr | None#
model_config: ClassVar[ConfigDict] = {'populate_by_name': True, 'protected_namespaces': (), 'validate_assignment': True, 'validate_by_alias': True, 'validate_by_name': True}#

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

to_str() → str[source]#

Returns the string representation of the model using alias

Return type:

str

to_json() → str[source]#

Returns the JSON representation of the model using alias

Return type:

str

classmethod from_json(json_str: str) → Self | None[source]#

Create an instance of TaskQueryFilters from a JSON string

Parameters:

json_str (str)

Return type:

Self | None

to_dict() → Dict[str, Any][source]#

Return the dictionary representation of the model using alias.

This has the following differences from calling pydantic’s self.model_dump(by_alias=True):

  • None is only added to the output dict for nullable fields that were set at model initialization. Other fields with value None are ignored.

Return type:

Dict[str, Any]

classmethod from_dict(obj: Dict[str, Any] | None) → Self | None[source]#

Create an instance of TaskQueryFilters from a dict

Parameters:

obj (Dict[str, Any] | None)

Return type:

Self | None

class TaskQueryMapping(*, variable_name: Annotated[str, Strict(strict=True)], query_ids: Annotated[List[Annotated[str, FieldInfo(annotation=NoneType, required=True, metadata=[Strict(strict=True)])]], MinLen(min_length=0)], attribute_path: Annotated[str, Strict(strict=True)])[source]#

Bases: BaseModel

Maps one evaluator variable to one or more query ids and an attribute path (trace/session shape).

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:

  • variable_name (Annotated[str, Strict(strict=True)])

  • query_ids (Annotated[List[Annotated[str, FieldInfo(annotation=NoneType, required=True, metadata=[Strict(strict=True)])]], FieldInfo(annotation=NoneType, required=True, metadata=[MinLen(min_length=0)])])

  • attribute_path (Annotated[str, Strict(strict=True)])

variable_name: StrictStr#
query_ids: Annotated[List[Annotated[str, Field(strict=True)]], Field(min_length=0)]#
attribute_path: StrictStr#
model_config: ClassVar[ConfigDict] = {'populate_by_name': True, 'protected_namespaces': (), 'validate_assignment': True, 'validate_by_alias': True, 'validate_by_name': True}#

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

to_str() → str[source]#

Returns the string representation of the model using alias

Return type:

str

to_json() → str[source]#

Returns the JSON representation of the model using alias

Return type:

str

classmethod from_json(json_str: str) → Self | None[source]#

Create an instance of TaskQueryMapping from a JSON string

Parameters:

json_str (str)

Return type:

Self | None

to_dict() → Dict[str, Any][source]#

Return the dictionary representation of the model using alias.

This has the following differences from calling pydantic’s self.model_dump(by_alias=True):

  • None is only added to the output dict for nullable fields that were set at model initialization. Other fields with value None are ignored.

Return type:

Dict[str, Any]

classmethod from_dict(obj: Dict[str, Any] | None) → Self | None[source]#

Create an instance of TaskQueryMapping from a dict

Parameters:

obj (Dict[str, Any] | None)

Return type:

Self | None

class TaskRun(*, id: Annotated[str, Strict(strict=True)], task_id: Annotated[str, Strict(strict=True)], experiment_id: Annotated[str, Strict(strict=True)] | None = None, status: TaskRunStatus, run_started_at: datetime | None, run_finished_at: datetime | None, data_start_time: datetime | None, data_end_time: datetime | None, num_successes: Annotated[int, Strict(strict=True)], num_errors: Annotated[int, Strict(strict=True)], num_skipped: Annotated[int, Strict(strict=True)], created_at: datetime, created_by_user_id: Annotated[str, Strict(strict=True)] | None, failure_reason: Annotated[str, Strict(strict=True)] | None = None, additional_properties: Dict[str, Any] = {})[source]#

Bases: BaseModel

A task run is an async job that executes the work defined on a task. Runs are created by triggering an existing task (POST /v2/tasks/{task_id}/trigger). For RUN_EXPERIMENT tasks, experiment_id is populated after the experiment is provisioned; poll GET /v2/task-runs/{run_id} until status reaches a terminal state.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:
id: StrictStr#
task_id: StrictStr#
experiment_id: StrictStr | None#
status: TaskRunStatus#
run_started_at: datetime | None#
run_finished_at: datetime | None#
data_start_time: datetime | None#
data_end_time: datetime | None#
num_successes: StrictInt#
num_errors: StrictInt#
num_skipped: StrictInt#
created_at: datetime#
created_by_user_id: StrictStr | None#
failure_reason: StrictStr | None#
additional_properties: Dict[str, Any]#
model_config: ClassVar[ConfigDict] = {'populate_by_name': True, 'protected_namespaces': (), 'validate_assignment': True, 'validate_by_alias': True, 'validate_by_name': True}#

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

to_str() → str[source]#

Returns the string representation of the model using alias

Return type:

str

to_json() → str[source]#

Returns the JSON representation of the model using alias

Return type:

str

classmethod from_json(json_str: str) → Self | None[source]#

Create an instance of TaskRun from a JSON string

Parameters:

json_str (str)

Return type:

Self | None

to_dict() → Dict[str, Any][source]#

Return the dictionary representation of the model using alias.

This has the following differences from calling pydantic’s self.model_dump(by_alias=True):

  • None is only added to the output dict for nullable fields that were set at model initialization. Other fields with value None are ignored.

  • Fields in self.additional_properties are added to the output dict.

Return type:

Dict[str, Any]

classmethod from_dict(obj: Dict[str, Any] | None) → Self | None[source]#

Create an instance of TaskRun from a dict

Parameters:

obj (Dict[str, Any] | None)

Return type:

Self | None

class TaskType(value, names=_not_given, *values, module=None, qualname=None, type=None, start=1, boundary=None)[source]#

Bases: str, Enum

The task type. - TEMPLATE_EVALUATION - An LLM template-based evaluation task. - CODE_EVALUATION - A code-based evaluation task. - RUN_EXPERIMENT - A task that runs experiments.

TEMPLATE_EVALUATION = 'TEMPLATE_EVALUATION'#
CODE_EVALUATION = 'CODE_EVALUATION'#
RUN_EXPERIMENT = 'RUN_EXPERIMENT'#
classmethod from_json(json_str: str) → Self[source]#

Create an instance of TaskType from a JSON string

Parameters:

json_str (str)

Return type:

Self

class TemplateEvaluationRunConfig(*, experiment_type: Annotated[str, Strict(strict=True)], ai_integration_id: Annotated[str, Strict(strict=True)], model_name: Annotated[str, Strict(strict=True)] | None = None, template: Annotated[str, Strict(strict=True), MinLen(min_length=1)], provide_explanation: Annotated[bool, Strict(strict=True)], classification_choices: Dict[str, Annotated[float, Strict(strict=True)] | Annotated[int, Strict(strict=True)]] | None = None, column_mapping: Dict[str, Annotated[str, Strict(strict=True)]] | None = None, evaluator_version_id: Annotated[str, Strict(strict=True)] | None = None, invocation_parameters: InvocationParams | None = None, provider_parameters: Dict[str, Any] | None = None, additional_properties: Dict[str, Any] = {})[source]#

Bases: BaseModel

Configuration for running a template-based LLM evaluator against each dataset example.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:

  • experiment_type (Annotated[str, Strict(strict=True)])

  • ai_integration_id (Annotated[str, Strict(strict=True)])

  • model_name (Annotated[str, Strict(strict=True)] | None)

  • template (Annotated[str, FieldInfo(annotation=NoneType, required=True, metadata=[Strict(strict=True), MinLen(min_length=1)])])

  • provide_explanation (Annotated[bool, Strict(strict=True)])

  • classification_choices (Dict[str, Annotated[float, Strict(strict=True)] | Annotated[int, Strict(strict=True)]] | None)

  • column_mapping (Dict[str, Annotated[str, Strict(strict=True)]] | None)

  • evaluator_version_id (Annotated[str, Strict(strict=True)] | None)

  • invocation_parameters (InvocationParams | None)

  • provider_parameters (Dict[str, Any] | None)

  • additional_properties (Dict[str, Any])

experiment_type: StrictStr#
ai_integration_id: StrictStr#
model_name: StrictStr | None#
template: Annotated[str, Field(min_length=1, strict=True)]#
provide_explanation: StrictBool#
classification_choices: Dict[str, StrictFloat | StrictInt] | None#
column_mapping: Dict[str, StrictStr] | None#
evaluator_version_id: StrictStr | None#
invocation_parameters: InvocationParams | None#
provider_parameters: Dict[str, Any] | None#
additional_properties: Dict[str, Any]#
classmethod experiment_type_validate_enum(value)[source]#

Validates the enum

model_config: ClassVar[ConfigDict] = {'populate_by_name': True, 'protected_namespaces': (), 'validate_assignment': True, 'validate_by_alias': True, 'validate_by_name': True}#

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

to_str() → str[source]#

Returns the string representation of the model using alias

Return type:

str

to_json() → str[source]#

Returns the JSON representation of the model using alias

Return type:

str

classmethod from_json(json_str: str) → Self | None[source]#

Create an instance of TemplateEvaluationRunConfig from a JSON string

Parameters:

json_str (str)

Return type:

Self | None

to_dict() → Dict[str, Any][source]#

Return the dictionary representation of the model using alias.

This has the following differences from calling pydantic’s self.model_dump(by_alias=True):

  • None is only added to the output dict for nullable fields that were set at model initialization. Other fields with value None are ignored.

  • Fields in self.additional_properties are added to the output dict.

Return type:

Dict[str, Any]

classmethod from_dict(obj: Dict[str, Any] | None) → Self | None[source]#

Create an instance of TemplateEvaluationRunConfig from a dict

Parameters:

obj (Dict[str, Any] | None)

Return type:

Self | None

class TemplateEvaluationRunConfigRequest(*, experiment_type: Annotated[str, Strict(strict=True)], ai_integration_id: Annotated[str, Strict(strict=True)], model_name: Annotated[str, Strict(strict=True)] | None = None, template: Annotated[str, Strict(strict=True), MinLen(min_length=1)], provide_explanation: Annotated[bool, Strict(strict=True)], classification_choices: Dict[str, Annotated[float, Strict(strict=True)] | Annotated[int, Strict(strict=True)]] | None = None, column_mapping: Dict[str, Annotated[str, Strict(strict=True)]] | None = None, evaluator_version_id: Annotated[str, Strict(strict=True)] | None = None, invocation_parameters: InvocationParamsRequest | None = None, provider_parameters: Dict[str, Any] | None = None)[source]#

Bases: BaseModel

Strict request configuration for running a template-based LLM evaluator.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:
experiment_type: StrictStr#
ai_integration_id: StrictStr#
model_name: StrictStr | None#
template: Annotated[str, Field(min_length=1, strict=True)]#
provide_explanation: StrictBool#
classification_choices: Dict[str, StrictFloat | StrictInt] | None#
column_mapping: Dict[str, StrictStr] | None#
evaluator_version_id: StrictStr | None#
invocation_parameters: InvocationParamsRequest | None#
provider_parameters: Dict[str, Any] | None#
classmethod experiment_type_validate_enum(value)[source]#

Validates the enum

model_config: ClassVar[ConfigDict] = {'populate_by_name': True, 'protected_namespaces': (), 'validate_assignment': True, 'validate_by_alias': True, 'validate_by_name': True}#

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

to_str() → str[source]#

Returns the string representation of the model using alias

Return type:

str

to_json() → str[source]#

Returns the JSON representation of the model using alias

Return type:

str

classmethod from_json(json_str: str) → Self | None[source]#

Create an instance of TemplateEvaluationRunConfigRequest from a JSON string

Parameters:

json_str (str)

Return type:

Self | None

to_dict() → Dict[str, Any][source]#

Return the dictionary representation of the model using alias.

This has the following differences from calling pydantic’s self.model_dump(by_alias=True):

  • None is only added to the output dict for nullable fields that were set at model initialization. Other fields with value None are ignored.

Return type:

Dict[str, Any]

classmethod from_dict(obj: Dict[str, Any] | None) → Self | None[source]#

Create an instance of TemplateEvaluationRunConfigRequest from a dict

Parameters:

obj (Dict[str, Any] | None)

Return type:

Self | None

On this page