Evaluators#

class EvaluatorsClient(*, sdk_config: SDKConfiguration, generated_client: ApiClient)[source]#

Bases: object

Client for managing Arize evaluators and evaluator versions.

This class is primarily intended for internal use within the SDK. Users are highly encouraged to access resource-specific functionality via arize.ArizeClient.

The evaluators client is a thin wrapper around the generated REST API client, using the shared generated API client owned by arize.config.SDKConfiguration.

Parameters:

  • sdk_config (SDKConfiguration) – Resolved SDK configuration.

  • generated_client (ApiClient) – Shared generated API client instance.

list(*, name: str | None = None, space: str | None = None, limit: int = DEFAULT_LIST_LIMIT, cursor: str | None = None) → ListEvaluatorsResponse[source]#

List evaluators the user has access to.

Results are sorted by update date (most recent first). This endpoint supports cursor-based pagination. When space is provided, results are limited to that space; otherwise evaluators from all permitted spaces are returned.

Parameters:

  • name (str | None) – Optional case-insensitive substring filter on the evaluator name.

  • space (str | None) – Optional space filter. If the value is a base64-encoded resource ID it is treated as a space ID; otherwise it is used as a case-insensitive substring filter on the space name.

  • limit (int) – Maximum number of evaluators to return (1-100).

  • cursor (str | None) – Opaque pagination cursor from a previous response.

Returns:

A paginated evaluator list response from the Arize REST API.

Raises:

ApiException – If the API request fails.

Return type:

ListEvaluatorsResponse

get(*, evaluator: str, space: str | None = None, version_id: str | None = None) → EvaluatorWithVersion[source]#

Get an evaluator by name or ID, with its resolved version.

By default, the latest version is returned. Pass version_id to resolve a specific version instead.

Parameters:

  • evaluator (str) – Evaluator name or identifier (base64) to retrieve.

  • space (str | None) – Optional space name or ID. Required when evaluator is a name rather than an ID.

  • version_id (str | None) – Optional version identifier (base64). If omitted, the latest version is returned.

Returns:

The evaluator with its resolved version.

Raises:

ApiException – If the API request fails (for example, evaluator not found).

Return type:

EvaluatorWithVersion

create_template_evaluator(*, name: str, space: str, commit_message: str, template_config: TemplateConfigInput, description: str | None = None) → EvaluatorWithVersion[source]#

Create a new template evaluator with an initial version.

The evaluator name must be unique within the given space.

Parameters:

  • name (str) – Evaluator name (must be unique within the space).

  • space (str) – Space name or ID to create the evaluator in.

  • commit_message (str) – Commit message for the initial version.

  • template_config (TemplateConfigInput) –

    Template configuration for the evaluator. Build with arize.evaluators.types.TemplateConfigInput. Required fields:

    • name — eval column name; must match ^[a-zA-Z0-9_\\s\\-&()]+$.

    • template — prompt template string with {variable} placeholders referencing span/trace attributes.

    • include_explanations — whether the LLM should include a reasoning explanation alongside the score.

    • use_function_calling — prefer structured function-call output over free-text parsing when the model supports it.

    • classification_choices — required map of choice label to numeric score, e.g. {"relevant": 1, "irrelevant": 0}.

    • llm_config — arize.evaluators.types.EvaluatorLlmConfig specifying the model provider, model name, and API key.

    Optional fields: use_structured_output, direction, data_granularity.

  • description (str | None) – Optional human-readable description of the evaluator.

Returns:

The created evaluator with its initial version.

Raises:

ApiException – If the API request fails (for example, name conflict or invalid payload).

Return type:

EvaluatorWithVersion

create_code_evaluator(*, name: str, space: str, commit_message: str, code_config: CodeConfig | CustomCodeConfig | ManagedCodeConfig | CodeConfigRequest | CustomCodeConfigRequest | ManagedCodeConfigRequest | dict, description: str | None = None) → EvaluatorWithVersion[source]#

Create a new code evaluator with an initial version.

The evaluator name must be unique within the given space.

Parameters:
Returns:

The created evaluator with its initial version.

Raises:

ApiException – If the API request fails (for example, name conflict or invalid payload).

Return type:

EvaluatorWithVersion

create_remote_evaluator(*, name: str, space: str, integration_id: str, commit_message: str, description: str | None = None) → EvaluatorWithVersion[source]#

Create a new remote evaluator with an initial version.

Remote evaluators call a customer-hosted HTTP endpoint on each evaluation run. The endpoint must already exist as an EVALUATOR integration (see arize.ai_integrations.AiIntegrationsClient). The feature flag enableRemoteEvalTasks must be enabled on the account.

Parameters:

  • name (str) – Evaluator name (must be unique within the space).

  • space (str) – Space name or ID to create the evaluator in.

  • integration_id (str) – Global ID of an EVALUATOR integration that defines the remote endpoint, headers, and input schema.

  • commit_message (str) – Commit message for the initial version.

  • description (str | None) – Optional human-readable description of the evaluator.

Returns:

The created evaluator with its initial version.

Raises:

ApiException – If the API request fails (for example, name conflict, invalid payload, or feature flag disabled).

Return type:

EvaluatorWithVersion

update(*, evaluator: str, space: str | None = None, name: str | UNSET | None = _UNSET, description: str | UNSET | None = _UNSET) → Evaluator[source]#

Update an evaluator’s metadata.

Parameters:

  • evaluator (str) – Evaluator name or identifier (base64) to update.

  • space (str | None) – Optional space name or ID. Required when evaluator is a name rather than an ID.

  • name (str | UNSET | None) – New evaluator name (must be unique within its space). Omit it or pass None to leave the existing name unchanged.

  • description (str | UNSET | None) – New description for the evaluator. Omit to leave unchanged; pass None to clear it.

Returns:

The updated evaluator.

Raises:

ApiException – If the API request fails.

Return type:

Evaluator

delete(*, evaluator: str, space: str | None = None) → None[source]#

Delete an evaluator and all its versions.

This operation is irreversible.

Parameters:

  • evaluator (str) – Evaluator name or identifier (base64) to delete.

  • space (str | None) – Optional space name or ID. Required when evaluator is a name rather than an ID.

Returns:

None.

Raises:

ApiException – If the API request fails (for example, evaluator not found).

Return type:

None

list_versions(*, evaluator: str, space: str | None = None, limit: int = DEFAULT_LIST_LIMIT, cursor: str | None = None) → ListEvaluatorVersionsResponse[source]#

List all versions of an evaluator.

Results are returned with cursor-based pagination.

Parameters:

  • evaluator (str) – Evaluator name or identifier (base64) to list versions for.

  • space (str | None) – Optional space name or ID. Required when evaluator is a name rather than an ID.

  • limit (int) – Maximum number of versions to return (1-100).

  • cursor (str | None) – Opaque pagination cursor from a previous response.

Returns:

A paginated evaluator version list response.

Raises:

ApiException – If the API request fails.

Return type:

ListEvaluatorVersionsResponse

delete_versions(*, evaluator: str, space: str | None = None, version_ids: builtins.list[str]) → DeleteEvaluatorVersionsResponse[source]#

Delete a batch of evaluator versions.

This operation is irreversible. The delete is partial-tolerant: versions that exist and belong to the evaluator are deleted; requested IDs that were not deleted appear in not_deleted_version_ids. The API still returns HTTP 200.

version_ids must contain 1-100 IDs. Duplicates are accepted and silently collapsed server-side.

On success, completed is always true; it does NOT mean every ID was deleted. Each requested ID appears in exactly one of deleted_version_ids or not_deleted_version_ids.

Deleting a version pinned to a running online task un-pins that task (falls back to latest).

Parameters:

  • evaluator (str) – Evaluator name or identifier (base64) to delete versions from.

  • space (str | None) – Optional space name or ID. Required when evaluator is a name rather than an ID.

  • version_ids (builtins.list[str]) – Version identifiers (base64) to delete (1-100).

Returns:

A DeleteEvaluatorVersionsResponse with completed, deleted_version_ids, and not_deleted_version_ids.

Raises:

ApiException – If the API request fails.

Return type:

DeleteEvaluatorVersionsResponse

get_version(*, version_id: str) → EvaluatorVersionCode | EvaluatorVersionTemplate | EvaluatorVersionHarness | EvaluatorVersionRemote[source]#

Get a specific evaluator version by its global ID.

Parameters:

version_id (str) – Evaluator version identifier (base64).

Returns:

The evaluator version — a EvaluatorVersionCode for code evaluators (with code_config already unwrapped), an EvaluatorVersionTemplate for template evaluators, or an EvaluatorVersionHarness / EvaluatorVersionRemote for harness and remote evaluators. Remote evaluators include remote_config with the referenced integration ID.

Raises:

ApiException – If the API request fails (for example, version not found).

Return type:

EvaluatorVersionCode | EvaluatorVersionTemplate | EvaluatorVersionHarness | EvaluatorVersionRemote

create_template_version(*, evaluator: str, space: str | None = None, commit_message: str, template_config: TemplateConfigInput) → EvaluatorVersionTemplate[source]#

Create a new template version of an existing evaluator.

The new version becomes the latest version immediately (versioning is append-only). Versions are immutable once created; to change the configuration, create a new version.

Parameters:

  • evaluator (str) – Evaluator name or identifier (base64) to add a version to.

  • space (str | None) – Optional space name or ID. Required when evaluator is a name rather than an ID.

  • commit_message (str) – Commit message describing the changes in this version.

  • template_config (TemplateConfigInput) – Updated template configuration for this version. Build with arize.evaluators.types.TemplateConfigInput. classification_choices is required.

Returns:

The newly created evaluator version.

Raises:

ApiException – If the API request fails.

Return type:

EvaluatorVersionTemplate

create_code_version(*, evaluator: str, space: str | None = None, commit_message: str, code_config: CodeConfig | CustomCodeConfig | ManagedCodeConfig | CodeConfigRequest | CustomCodeConfigRequest | ManagedCodeConfigRequest | dict) → EvaluatorVersionCode[source]#

Create a new code version of an existing evaluator.

The new version becomes the latest version immediately (versioning is append-only). Versions are immutable once created; to change the configuration, create a new version.

Parameters:
Returns:

The newly created evaluator version.

Raises:

ApiException – If the API request fails.

Return type:

EvaluatorVersionCode

create_remote_version(*, evaluator: str, space: str | None = None, integration_id: str, commit_message: str) → EvaluatorVersionRemote[source]#

Create a new remote version of an existing evaluator.

The new version becomes the latest version immediately (versioning is append-only). Versions are immutable once created; to change the configuration, create a new version.

The feature flag enableRemoteEvalTasks must be enabled on the account, and the referenced integration must be an EVALUATOR integration.

Parameters:

  • evaluator (str) – Evaluator name or identifier (base64) to add a version to.

  • space (str | None) – Optional space name or ID. Required when evaluator is a name rather than an ID.

  • integration_id (str) – Global ID of an EVALUATOR integration that defines the remote endpoint, headers, and input schema.

  • commit_message (str) – Commit message describing the changes in this version.

Returns:

The newly created remote evaluator version.

Raises:

ApiException – If the API request fails (for example, feature flag disabled, integration not found, or wrong integration type).

Return type:

EvaluatorVersionRemote

Response Types#

Public type re-exports for the evaluators subdomain.

class CodeConfig(*args, oneof_schema_1_validator: ManagedCodeConfig | None = None, oneof_schema_2_validator: CustomCodeConfig | None = None, actual_instance: CustomCodeConfig | ManagedCodeConfig | None = None, one_of_schemas: Set[str] = {'CustomCodeConfig', 'ManagedCodeConfig'}, discriminator_value_class_map: Dict[str, str] = {})[source]#

Bases: BaseModel

Discriminated union representing either a managed (built-in) or custom (user-supplied Python) code evaluator configuration, resolved by the nested type field (MANAGED -> ManagedCodeConfig, CUSTOM -> CustomCodeConfig). This inner type is independent of the parent evaluator version’s type (which is always CODE here).

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:
oneof_schema_1_validator: ManagedCodeConfig | None#
oneof_schema_2_validator: CustomCodeConfig | None#
actual_instance: CustomCodeConfig | ManagedCodeConfig | None#
one_of_schemas: Set[str]#
model_config: ClassVar[ConfigDict] = {'protected_namespaces': (), 'validate_assignment': True}#

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

discriminator_value_class_map: Dict[str, str]#
classmethod actual_instance_must_validate_oneof(v)[source]#
classmethod from_dict(obj: str | Dict[str, Any]) → Self[source]#
Parameters:

obj (str | Dict[str, Any])

Return type:

Self

classmethod from_json(json_str: str) → Self[source]#

Returns the object represented by the json string

Parameters:

json_str (str)

Return type:

Self

to_json() → str[source]#

Returns the JSON representation of the actual instance

Return type:

str

to_dict() → Dict[str, Any] | CustomCodeConfig | ManagedCodeConfig | None[source]#

Returns the dict representation of the actual instance

Return type:

Dict[str, Any] | CustomCodeConfig | ManagedCodeConfig | None

to_str() → str[source]#

Returns the string representation of the actual instance

Return type:

str

class CodeConfigRequest(*args, oneof_schema_1_validator: ManagedCodeConfigRequest | None = None, oneof_schema_2_validator: CustomCodeConfigRequest | None = None, actual_instance: CustomCodeConfigRequest | ManagedCodeConfigRequest | None = None, one_of_schemas: Set[str] = {'CustomCodeConfigRequest', 'ManagedCodeConfigRequest'}, discriminator_value_class_map: Dict[str, str] = {})[source]#

Bases: BaseModel

Strict request form of CodeConfig. Discriminated union of ManagedCodeConfigRequest and CustomCodeConfigRequest. Use in write request bodies.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:
oneof_schema_1_validator: ManagedCodeConfigRequest | None#
oneof_schema_2_validator: CustomCodeConfigRequest | None#
actual_instance: CustomCodeConfigRequest | ManagedCodeConfigRequest | None#
one_of_schemas: Set[str]#
model_config: ClassVar[ConfigDict] = {'protected_namespaces': (), 'validate_assignment': True}#

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

discriminator_value_class_map: Dict[str, str]#
classmethod actual_instance_must_validate_oneof(v)[source]#
classmethod from_dict(obj: str | Dict[str, Any]) → Self[source]#
Parameters:

obj (str | Dict[str, Any])

Return type:

Self

classmethod from_json(json_str: str) → Self[source]#

Returns the object represented by the json string

Parameters:

json_str (str)

Return type:

Self

to_json() → str[source]#

Returns the JSON representation of the actual instance

Return type:

str

to_dict() → Dict[str, Any] | CustomCodeConfigRequest | ManagedCodeConfigRequest | None[source]#

Returns the dict representation of the actual instance

Return type:

Dict[str, Any] | CustomCodeConfigRequest | ManagedCodeConfigRequest | None

to_str() → str[source]#

Returns the string representation of the actual instance

Return type:

str

class CreateCodeEvaluatorVersionRequest(*, commit_message: Annotated[str, Strict(strict=True)], code_config: CodeConfigRequest)[source]#

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:
commit_message: StrictStr#
code_config: CodeConfigRequest#
model_config: ClassVar[ConfigDict] = {'populate_by_name': True, 'protected_namespaces': (), 'validate_assignment': True, 'validate_by_alias': True, 'validate_by_name': True}#

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

to_str() → str[source]#

Returns the string representation of the model using alias

Return type:

str

to_json() → str[source]#

Returns the JSON representation of the model using alias

Return type:

str

classmethod from_json(json_str: str) → Self | None[source]#

Create an instance of CreateCodeEvaluatorVersionRequest from a JSON string

Parameters:

json_str (str)

Return type:

Self | None

to_dict() → Dict[str, Any][source]#

Return the dictionary representation of the model using alias.

This has the following differences from calling pydantic’s self.model_dump(by_alias=True):

  • None is only added to the output dict for nullable fields that were set at model initialization. Other fields with value None are ignored.

Return type:

Dict[str, Any]

classmethod from_dict(obj: Dict[str, Any] | None) → Self | None[source]#

Create an instance of CreateCodeEvaluatorVersionRequest from a dict

Parameters:

obj (Dict[str, Any] | None)

Return type:

Self | None

class CreateEvaluatorVersionRequest(*args, oneof_schema_1_validator: CreateTemplateEvaluatorVersionRequest | None = None, oneof_schema_2_validator: CreateCodeEvaluatorVersionRequest | None = None, oneof_schema_3_validator: CreateRemoteEvaluatorVersionRequest | None = None, actual_instance: CreateCodeEvaluatorVersionRequest | CreateRemoteEvaluatorVersionRequest | CreateTemplateEvaluatorVersionRequest | None = None, one_of_schemas: Set[str] = {'CreateCodeEvaluatorVersionRequest', 'CreateRemoteEvaluatorVersionRequest', 'CreateTemplateEvaluatorVersionRequest'})[source]#

Bases: BaseModel

Payload for an evaluator version: exactly one of template_config, code_config, or remote_config. Used both when creating an evaluator (initial version) and when appending a version.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:
oneof_schema_1_validator: CreateTemplateEvaluatorVersionRequest | None#
oneof_schema_2_validator: CreateCodeEvaluatorVersionRequest | None#
oneof_schema_3_validator: CreateRemoteEvaluatorVersionRequest | None#
actual_instance: CreateCodeEvaluatorVersionRequest | CreateRemoteEvaluatorVersionRequest | CreateTemplateEvaluatorVersionRequest | None#
one_of_schemas: Set[str]#
model_config: ClassVar[ConfigDict] = {'protected_namespaces': (), 'validate_assignment': True}#

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

classmethod actual_instance_must_validate_oneof(v)[source]#
classmethod from_dict(obj: str | Dict[str, Any]) → Self[source]#
Parameters:

obj (str | Dict[str, Any])

Return type:

Self

classmethod from_json(json_str: str) → Self[source]#

Returns the object represented by the json string

Parameters:

json_str (str)

Return type:

Self

to_json() → str[source]#

Returns the JSON representation of the actual instance

Return type:

str

to_dict() → Dict[str, Any] | CreateCodeEvaluatorVersionRequest | CreateRemoteEvaluatorVersionRequest | CreateTemplateEvaluatorVersionRequest | None[source]#

Returns the dict representation of the actual instance

Return type:

Dict[str, Any] | CreateCodeEvaluatorVersionRequest | CreateRemoteEvaluatorVersionRequest | CreateTemplateEvaluatorVersionRequest | None

to_str() → str[source]#

Returns the string representation of the actual instance

Return type:

str

class CreateRemoteEvaluatorVersionRequest(*, commit_message: Annotated[str, Strict(strict=True)], remote_config: RemoteConfigInput)[source]#

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:
commit_message: StrictStr#
remote_config: RemoteConfigInput#
model_config: ClassVar[ConfigDict] = {'populate_by_name': True, 'protected_namespaces': (), 'validate_assignment': True, 'validate_by_alias': True, 'validate_by_name': True}#

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

to_str() → str[source]#

Returns the string representation of the model using alias

Return type:

str

to_json() → str[source]#

Returns the JSON representation of the model using alias

Return type:

str

classmethod from_json(json_str: str) → Self | None[source]#

Create an instance of CreateRemoteEvaluatorVersionRequest from a JSON string

Parameters:

json_str (str)

Return type:

Self | None

to_dict() → Dict[str, Any][source]#

Return the dictionary representation of the model using alias.

This has the following differences from calling pydantic’s self.model_dump(by_alias=True):

  • None is only added to the output dict for nullable fields that were set at model initialization. Other fields with value None are ignored.

Return type:

Dict[str, Any]

classmethod from_dict(obj: Dict[str, Any] | None) → Self | None[source]#

Create an instance of CreateRemoteEvaluatorVersionRequest from a dict

Parameters:

obj (Dict[str, Any] | None)

Return type:

Self | None

class CreateTemplateEvaluatorVersionRequest(*, commit_message: Annotated[str, Strict(strict=True)], template_config: TemplateConfigInput)[source]#

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:
commit_message: StrictStr#
template_config: TemplateConfigInput#
model_config: ClassVar[ConfigDict] = {'populate_by_name': True, 'protected_namespaces': (), 'validate_assignment': True, 'validate_by_alias': True, 'validate_by_name': True}#

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

to_str() → str[source]#

Returns the string representation of the model using alias

Return type:

str

to_json() → str[source]#

Returns the JSON representation of the model using alias

Return type:

str

classmethod from_json(json_str: str) → Self | None[source]#

Create an instance of CreateTemplateEvaluatorVersionRequest from a JSON string

Parameters:

json_str (str)

Return type:

Self | None

to_dict() → Dict[str, Any][source]#

Return the dictionary representation of the model using alias.

This has the following differences from calling pydantic’s self.model_dump(by_alias=True):

  • None is only added to the output dict for nullable fields that were set at model initialization. Other fields with value None are ignored.

Return type:

Dict[str, Any]

classmethod from_dict(obj: Dict[str, Any] | None) → Self | None[source]#

Create an instance of CreateTemplateEvaluatorVersionRequest from a dict

Parameters:

obj (Dict[str, Any] | None)

Return type:

Self | None

class CustomCodeConfig(*, data_granularity: DataGranularity | None = None, query_filter: Annotated[str, Strict(strict=True)] | None = None, type: Annotated[str, Strict(strict=True)], name: Annotated[str, Strict(strict=True)], code: Annotated[str, Strict(strict=True)], imports: Annotated[str, Strict(strict=True)] | None = None, variables: List[Annotated[str, Strict(strict=True)]], static_params: List[StaticParam] | None = None)[source]#

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:
data_granularity: DataGranularity | None#
query_filter: StrictStr | None#
type: StrictStr#
name: Annotated[str, Field(strict=True)]#
code: StrictStr#
imports: StrictStr | None#
variables: List[StrictStr]#
static_params: List[StaticParam] | None#
classmethod type_validate_enum(value)[source]#

Validates the enum

classmethod name_validate_regular_expression(value)[source]#

Validates the regular expression

model_config: ClassVar[ConfigDict] = {'populate_by_name': True, 'protected_namespaces': (), 'validate_assignment': True, 'validate_by_alias': True, 'validate_by_name': True}#

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

to_str() → str[source]#

Returns the string representation of the model using alias

Return type:

str

to_json() → str[source]#

Returns the JSON representation of the model using alias

Return type:

str

classmethod from_json(json_str: str) → Self | None[source]#

Create an instance of CustomCodeConfig from a JSON string

Parameters:

json_str (str)

Return type:

Self | None

to_dict() → Dict[str, Any][source]#

Return the dictionary representation of the model using alias.

This has the following differences from calling pydantic’s self.model_dump(by_alias=True):

  • None is only added to the output dict for nullable fields that were set at model initialization. Other fields with value None are ignored.

Return type:

Dict[str, Any]

classmethod from_dict(obj: Dict[str, Any] | None) → Self | None[source]#

Create an instance of CustomCodeConfig from a dict

Parameters:

obj (Dict[str, Any] | None)

Return type:

Self | None

class CustomCodeConfigRequest(*, data_granularity: DataGranularity | None = None, query_filter: Annotated[str, Strict(strict=True)] | None = None, type: Annotated[str, Strict(strict=True)], name: Annotated[str, Strict(strict=True)], code: Annotated[str, Strict(strict=True)], imports: Annotated[str, Strict(strict=True)] | None = None, variables: List[Annotated[str, Strict(strict=True)]], static_params: List[StaticParamRequest] | None = None)[source]#

Bases: BaseModel

Custom (user-supplied Python) code evaluator configuration in a write request (strict form of CustomCodeConfig)

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:
data_granularity: DataGranularity | None#
query_filter: StrictStr | None#
type: StrictStr#
name: Annotated[str, Field(strict=True)]#
code: StrictStr#
imports: StrictStr | None#
variables: List[StrictStr]#
static_params: List[StaticParamRequest] | None#
classmethod type_validate_enum(value)[source]#

Validates the enum

classmethod name_validate_regular_expression(value)[source]#

Validates the regular expression

model_config: ClassVar[ConfigDict] = {'populate_by_name': True, 'protected_namespaces': (), 'validate_assignment': True, 'validate_by_alias': True, 'validate_by_name': True}#

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

to_str() → str[source]#

Returns the string representation of the model using alias

Return type:

str

to_json() → str[source]#

Returns the JSON representation of the model using alias

Return type:

str

classmethod from_json(json_str: str) → Self | None[source]#

Create an instance of CustomCodeConfigRequest from a JSON string

Parameters:

json_str (str)

Return type:

Self | None

to_dict() → Dict[str, Any][source]#

Return the dictionary representation of the model using alias.

This has the following differences from calling pydantic’s self.model_dump(by_alias=True):

  • None is only added to the output dict for nullable fields that were set at model initialization. Other fields with value None are ignored.

Return type:

Dict[str, Any]

classmethod from_dict(obj: Dict[str, Any] | None) → Self | None[source]#

Create an instance of CustomCodeConfigRequest from a dict

Parameters:

obj (Dict[str, Any] | None)

Return type:

Self | None

class DataGranularity(value, names=_not_given, *values, module=None, qualname=None, type=None, start=1, boundary=None)[source]#

Bases: str, Enum

Data granularity level for evaluation. - SPAN - Evaluate at the individual span level. - TRACE - Evaluate at the full trace level. - SESSION - Evaluate at the session level.

SPAN = 'SPAN'#
TRACE = 'TRACE'#
SESSION = 'SESSION'#
classmethod from_json(json_str: str) → Self[source]#

Create an instance of DataGranularity from a JSON string

Parameters:

json_str (str)

Return type:

Self

class DeleteEvaluatorVersionsResponse(*, completed: Annotated[bool, Strict(strict=True)], deleted_version_ids: List[Annotated[str, Strict(strict=True)]], not_deleted_version_ids: List[Annotated[str, Strict(strict=True)]])[source]#

Bases: BaseModel

Result of a DELETE /v2/evaluators/{evaluator_id}/versions request. The delete is partial-tolerant: requested versions that exist and belong to evaluator_id are deleted; every requested ID that was not deleted is reported in not_deleted_version_ids. An ID may be not-deleted because it does not exist or belongs to a different evaluator. completed is true when this response is returned because the synchronous delete has fully processed the request. It does not mean every requested version was found and deleted: each requested ID appears in exactly one of deleted_version_ids or not_deleted_version_ids. The delete operation is idempotent — re-submitting already-deleted IDs is safe and simply reports them as not deleted. Deleting a version that is currently pinned to a running online task un-pins that task, which then falls back to resolving the evaluator’s latest version.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:
completed: StrictBool#
deleted_version_ids: List[StrictStr]#
not_deleted_version_ids: List[StrictStr]#
model_config: ClassVar[ConfigDict] = {'populate_by_name': True, 'protected_namespaces': (), 'validate_assignment': True, 'validate_by_alias': True, 'validate_by_name': True}#

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

to_str() → str[source]#

Returns the string representation of the model using alias

Return type:

str

to_json() → str[source]#

Returns the JSON representation of the model using alias

Return type:

str

classmethod from_json(json_str: str) → Self | None[source]#

Create an instance of DeleteEvaluatorVersionsResponse from a JSON string

Parameters:

json_str (str)

Return type:

Self | None

to_dict() → Dict[str, Any][source]#

Return the dictionary representation of the model using alias.

This has the following differences from calling pydantic’s self.model_dump(by_alias=True):

  • None is only added to the output dict for nullable fields that were set at model initialization. Other fields with value None are ignored.

Return type:

Dict[str, Any]

classmethod from_dict(obj: Dict[str, Any] | None) → Self | None[source]#

Create an instance of DeleteEvaluatorVersionsResponse from a dict

Parameters:

obj (Dict[str, Any] | None)

Return type:

Self | None

class Evaluator(*, id: Annotated[str, Strict(strict=True)], name: Annotated[str, Strict(strict=True)], description: Annotated[str, Strict(strict=True)] | None = None, type: EvaluatorType, space_id: Annotated[str, Strict(strict=True)], created_at: datetime, updated_at: datetime, created_by_user_id: Annotated[str, Strict(strict=True)] | None)[source]#

Bases: BaseModel

An evaluator defines reusable evaluation logic that can be attached to evaluation tasks. The type field determines the kind of evaluation: TEMPLATE (LLM-based template evaluation), CODE (custom code evaluation), or REMOTE (externally hosted evaluation).

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:
id: StrictStr#
name: StrictStr#
description: StrictStr | None#
type: EvaluatorType#
space_id: StrictStr#
created_at: datetime#
updated_at: datetime#
created_by_user_id: StrictStr | None#
model_config: ClassVar[ConfigDict] = {'populate_by_name': True, 'protected_namespaces': (), 'validate_assignment': True, 'validate_by_alias': True, 'validate_by_name': True}#

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

to_str() → str[source]#

Returns the string representation of the model using alias

Return type:

str

to_json() → str[source]#

Returns the JSON representation of the model using alias

Return type:

str

classmethod from_json(json_str: str) → Self | None[source]#

Create an instance of Evaluator from a JSON string

Parameters:

json_str (str)

Return type:

Self | None

to_dict() → Dict[str, Any][source]#

Return the dictionary representation of the model using alias.

This has the following differences from calling pydantic’s self.model_dump(by_alias=True):

  • None is only added to the output dict for nullable fields that were set at model initialization. Other fields with value None are ignored.

Return type:

Dict[str, Any]

classmethod from_dict(obj: Dict[str, Any] | None) → Self | None[source]#

Create an instance of Evaluator from a dict

Parameters:

obj (Dict[str, Any] | None)

Return type:

Self | None

class EvaluatorLlmConfig(*, ai_integration_id: Annotated[str, Strict(strict=True)], model_name: Annotated[str, Strict(strict=True)], invocation_parameters: InvocationParams, provider_parameters: ProviderParams)[source]#

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:
ai_integration_id: StrictStr#
model_name: StrictStr#
invocation_parameters: InvocationParams#
provider_parameters: ProviderParams#
model_config: ClassVar[ConfigDict] = {'populate_by_name': True, 'protected_namespaces': (), 'validate_assignment': True, 'validate_by_alias': True, 'validate_by_name': True}#

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

to_str() → str[source]#

Returns the string representation of the model using alias

Return type:

str

to_json() → str[source]#

Returns the JSON representation of the model using alias

Return type:

str

classmethod from_json(json_str: str) → Self | None[source]#

Create an instance of EvaluatorLlmConfig from a JSON string

Parameters:

json_str (str)

Return type:

Self | None

to_dict() → Dict[str, Any][source]#

Return the dictionary representation of the model using alias.

This has the following differences from calling pydantic’s self.model_dump(by_alias=True):

  • None is only added to the output dict for nullable fields that were set at model initialization. Other fields with value None are ignored.

Return type:

Dict[str, Any]

classmethod from_dict(obj: Dict[str, Any] | None) → Self | None[source]#

Create an instance of EvaluatorLlmConfig from a dict

Parameters:

obj (Dict[str, Any] | None)

Return type:

Self | None

class EvaluatorLlmConfigRequest(*, ai_integration_id: Annotated[str, Strict(strict=True)], model_name: Annotated[str, Strict(strict=True)], invocation_parameters: InvocationParamsRequest, provider_parameters: ProviderParamsRequest)[source]#

Bases: BaseModel

LLM configuration for an evaluator in a write request (strict form of EvaluatorLlmConfig)

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:
ai_integration_id: StrictStr#
model_name: StrictStr#
invocation_parameters: InvocationParamsRequest#
provider_parameters: ProviderParamsRequest#
model_config: ClassVar[ConfigDict] = {'populate_by_name': True, 'protected_namespaces': (), 'validate_assignment': True, 'validate_by_alias': True, 'validate_by_name': True}#

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

to_str() → str[source]#

Returns the string representation of the model using alias

Return type:

str

to_json() → str[source]#

Returns the JSON representation of the model using alias

Return type:

str

classmethod from_json(json_str: str) → Self | None[source]#

Create an instance of EvaluatorLlmConfigRequest from a JSON string

Parameters:

json_str (str)

Return type:

Self | None

to_dict() → Dict[str, Any][source]#

Return the dictionary representation of the model using alias.

This has the following differences from calling pydantic’s self.model_dump(by_alias=True):

  • None is only added to the output dict for nullable fields that were set at model initialization. Other fields with value None are ignored.

Return type:

Dict[str, Any]

classmethod from_dict(obj: Dict[str, Any] | None) → Self | None[source]#

Create an instance of EvaluatorLlmConfigRequest from a dict

Parameters:

obj (Dict[str, Any] | None)

Return type:

Self | None

class EvaluatorType(value, names=_not_given, *values, module=None, qualname=None, type=None, start=1, boundary=None)[source]#

Bases: str, Enum

The evaluator type: - TEMPLATE — LLM-based evaluator. - CODE — managed built-in evaluators or custom Python code (both are subtypes of CODE, discriminated by the nested CodeConfig.type = MANAGED | CUSTOM). - HARNESS — test harness evaluator. - REMOTE — remote evaluator. Applies to both the parent Evaluator.type field and every version’s type discriminator — a version’s type must always match its parent evaluator’s type.

TEMPLATE = 'TEMPLATE'#
CODE = 'CODE'#
HARNESS = 'HARNESS'#
REMOTE = 'REMOTE'#
classmethod from_json(json_str: str) → Self[source]#

Create an instance of EvaluatorType from a JSON string

Parameters:

json_str (str)

Return type:

Self

class EvaluatorVersionCode(*, id: str, evaluator_id: str, commit_hash: str, commit_message: str | None = None, created_at: datetime, created_by_user_id: str | None = None, type: str, code_config: CustomCodeConfig | ManagedCodeConfig)[source]#

Bases: BaseModel

SDK view of the generated EvaluatorVersionCode with code_config unwrapped.

The code_config field holds the concrete inner type (CustomCodeConfig or ManagedCodeConfig) instead of the oneOf wrapper CodeConfig, so model_dump() and attribute access work as expected.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:
id: str#
evaluator_id: str#
commit_hash: str#
commit_message: str | None#
created_at: datetime#
created_by_user_id: str | None#
type: str#
code_config: CustomCodeConfig | ManagedCodeConfig#
model_config: ClassVar[ConfigDict] = {'from_attributes': True}#

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class EvaluatorVersionHarness(*, id: Annotated[str, Strict(strict=True)], evaluator_id: Annotated[str, Strict(strict=True)], commit_hash: Annotated[str, Strict(strict=True)], commit_message: Annotated[str, Strict(strict=True)] | None, created_at: datetime, created_by_user_id: Annotated[str, Strict(strict=True)] | None, type: Annotated[str, Strict(strict=True)])[source]#

Bases: BaseModel

Evaluator version backed by a harness evaluation config. Only common version metadata (id, commit info, timestamps) is returned — the harness configuration is not yet accessible and will be a future addition.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:
id: StrictStr#
evaluator_id: StrictStr#
commit_hash: StrictStr#
commit_message: StrictStr | None#
created_at: datetime#
created_by_user_id: StrictStr | None#
type: StrictStr#
classmethod type_validate_enum(value)[source]#

Validates the enum

model_config: ClassVar[ConfigDict] = {'populate_by_name': True, 'protected_namespaces': (), 'validate_assignment': True, 'validate_by_alias': True, 'validate_by_name': True}#

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

to_str() → str[source]#

Returns the string representation of the model using alias

Return type:

str

to_json() → str[source]#

Returns the JSON representation of the model using alias

Return type:

str

classmethod from_json(json_str: str) → Self | None[source]#

Create an instance of EvaluatorVersionHarness from a JSON string

Parameters:

json_str (str)

Return type:

Self | None

to_dict() → Dict[str, Any][source]#

Return the dictionary representation of the model using alias.

This has the following differences from calling pydantic’s self.model_dump(by_alias=True):

  • None is only added to the output dict for nullable fields that were set at model initialization. Other fields with value None are ignored.

Return type:

Dict[str, Any]

classmethod from_dict(obj: Dict[str, Any] | None) → Self | None[source]#

Create an instance of EvaluatorVersionHarness from a dict

Parameters:

obj (Dict[str, Any] | None)

Return type:

Self | None

class EvaluatorVersionRemote(*, id: Annotated[str, Strict(strict=True)], evaluator_id: Annotated[str, Strict(strict=True)], commit_hash: Annotated[str, Strict(strict=True)], commit_message: Annotated[str, Strict(strict=True)] | None, created_at: datetime, created_by_user_id: Annotated[str, Strict(strict=True)] | None, type: Annotated[str, Strict(strict=True)], remote_config: RemoteConfig)[source]#

Bases: BaseModel

Evaluator version backed by a remote evaluator integration.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:
id: StrictStr#
evaluator_id: StrictStr#
commit_hash: StrictStr#
commit_message: StrictStr | None#
created_at: datetime#
created_by_user_id: StrictStr | None#
type: StrictStr#
remote_config: RemoteConfig#
classmethod type_validate_enum(value)[source]#

Validates the enum

model_config: ClassVar[ConfigDict] = {'populate_by_name': True, 'protected_namespaces': (), 'validate_assignment': True, 'validate_by_alias': True, 'validate_by_name': True}#

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

to_str() → str[source]#

Returns the string representation of the model using alias

Return type:

str

to_json() → str[source]#

Returns the JSON representation of the model using alias

Return type:

str

classmethod from_json(json_str: str) → Self | None[source]#

Create an instance of EvaluatorVersionRemote from a JSON string

Parameters:

json_str (str)

Return type:

Self | None

to_dict() → Dict[str, Any][source]#

Return the dictionary representation of the model using alias.

This has the following differences from calling pydantic’s self.model_dump(by_alias=True):

  • None is only added to the output dict for nullable fields that were set at model initialization. Other fields with value None are ignored.

Return type:

Dict[str, Any]

classmethod from_dict(obj: Dict[str, Any] | None) → Self | None[source]#

Create an instance of EvaluatorVersionRemote from a dict

Parameters:

obj (Dict[str, Any] | None)

Return type:

Self | None

class EvaluatorVersionTemplate(*, id: Annotated[str, Strict(strict=True)], evaluator_id: Annotated[str, Strict(strict=True)], commit_hash: Annotated[str, Strict(strict=True)], commit_message: Annotated[str, Strict(strict=True)] | None, created_at: datetime, created_by_user_id: Annotated[str, Strict(strict=True)] | None, type: Annotated[str, Strict(strict=True)], template_config: TemplateConfig)[source]#

Bases: BaseModel

Evaluator version carrying a template (LLM) configuration.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:
id: StrictStr#
evaluator_id: StrictStr#
commit_hash: StrictStr#
commit_message: StrictStr | None#
created_at: datetime#
created_by_user_id: StrictStr | None#
type: StrictStr#
template_config: TemplateConfig#
classmethod type_validate_enum(value)[source]#

Validates the enum

model_config: ClassVar[ConfigDict] = {'populate_by_name': True, 'protected_namespaces': (), 'validate_assignment': True, 'validate_by_alias': True, 'validate_by_name': True}#

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

to_str() → str[source]#

Returns the string representation of the model using alias

Return type:

str

to_json() → str[source]#

Returns the JSON representation of the model using alias

Return type:

str

classmethod from_json(json_str: str) → Self | None[source]#

Create an instance of EvaluatorVersionTemplate from a JSON string

Parameters:

json_str (str)

Return type:

Self | None

to_dict() → Dict[str, Any][source]#

Return the dictionary representation of the model using alias.

This has the following differences from calling pydantic’s self.model_dump(by_alias=True):

  • None is only added to the output dict for nullable fields that were set at model initialization. Other fields with value None are ignored.

Return type:

Dict[str, Any]

classmethod from_dict(obj: Dict[str, Any] | None) → Self | None[source]#

Create an instance of EvaluatorVersionTemplate from a dict

Parameters:

obj (Dict[str, Any] | None)

Return type:

Self | None

class EvaluatorWithVersion(*, id: str, name: str, description: str | None = None, type: EvaluatorType, space_id: str, created_at: datetime, updated_at: datetime, created_by_user_id: str | None = None, version: EvaluatorVersionCode | EvaluatorVersionTemplate | EvaluatorVersionHarness | EvaluatorVersionRemote)[source]#

Bases: BaseModel

SDK view of the generated EvaluatorWithVersion with version unwrapped.

The version field holds the concrete inner type (EvaluatorVersionCode for code evaluators, EvaluatorVersionTemplate for template evaluators, EvaluatorVersionHarness for harness evaluators, or EvaluatorVersionRemote for remote evaluators) instead of the oneOf wrapper.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:
id: str#
name: str#
description: str | None#
type: EvaluatorType#
space_id: str#
created_at: datetime#
updated_at: datetime#
created_by_user_id: str | None#
version: EvaluatorVersionCode | EvaluatorVersionTemplate | EvaluatorVersionHarness | EvaluatorVersionRemote#
model_config: ClassVar[ConfigDict] = {'from_attributes': True}#

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class ListEvaluatorVersionsResponse(*, evaluator_versions: list[EvaluatorVersionCode | EvaluatorVersionTemplate | EvaluatorVersionHarness | EvaluatorVersionRemote], pagination: PaginationMetadata)[source]#

Bases: BaseModel

SDK view of the generated ListEvaluatorVersionsResponse with each version unwrapped.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:
evaluator_versions: list[EvaluatorVersionCode | EvaluatorVersionTemplate | EvaluatorVersionHarness | EvaluatorVersionRemote]#
pagination: PaginationMetadata#
model_config: ClassVar[ConfigDict] = {'from_attributes': True}#

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

to_df(by_alias: bool = False, exclude_none: str | bool = True, json_normalize: bool = False, convert_dtypes: bool = True, expand_field: str = 'additional_properties', expand_prefix: str = '') → pd.DataFrame#

Convert a list of objects to a pandas.DataFrame.

Behavior:

  • If an item is a Pydantic v2 model, use .model_dump(by_alias=…).

  • If an item is a mapping (dict-like), use it as-is.

  • Otherwise, raise a ValueError (unsupported row type).

Parameters:

  • self (object) – The object instance containing the field to convert.

  • by_alias (bool) – Use field aliases when dumping Pydantic models.

  • exclude_none (str | bool) – Control None/NaN column dropping. - False: keep Nones as-is - “all”: drop columns where all values are None/NaN - “any”: drop columns where any value is None/NaN - True: alias for “all”

  • json_normalize (bool) – If True, flatten nested dicts via pandas.json_normalize.

  • convert_dtypes (bool) – If True, call DataFrame.convert_dtypes() at the end.

  • expand_field (str) – If set, look for this field in each row and expand its keys into top-level columns.

  • expand_prefix (str) – If set, prefix expanded column names with this string.

Returns:

The converted DataFrame.

Return type:

pandas.DataFrame

class ListEvaluatorsResponse(*, evaluators: List[Evaluator], pagination: PaginationMetadata)[source]#

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:
evaluators: List[Evaluator]#
pagination: PaginationMetadata#
model_config: ClassVar[ConfigDict] = {'populate_by_name': True, 'protected_namespaces': (), 'validate_assignment': True, 'validate_by_alias': True, 'validate_by_name': True}#

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

to_str() → str[source]#

Returns the string representation of the model using alias

Return type:

str

to_json() → str[source]#

Returns the JSON representation of the model using alias

Return type:

str

classmethod from_json(json_str: str) → Self | None[source]#

Create an instance of ListEvaluatorsResponse from a JSON string

Parameters:

json_str (str)

Return type:

Self | None

to_dict() → Dict[str, Any][source]#

Return the dictionary representation of the model using alias.

This has the following differences from calling pydantic’s self.model_dump(by_alias=True):

  • None is only added to the output dict for nullable fields that were set at model initialization. Other fields with value None are ignored.

Return type:

Dict[str, Any]

classmethod from_dict(obj: Dict[str, Any] | None) → Self | None[source]#

Create an instance of ListEvaluatorsResponse from a dict

Parameters:

obj (Dict[str, Any] | None)

Return type:

Self | None

to_df(by_alias: bool = False, exclude_none: str | bool = True, json_normalize: bool = False, convert_dtypes: bool = True, expand_field: str = 'additional_properties', expand_prefix: str = '') → pd.DataFrame#

Convert a list of objects to a pandas.DataFrame.

Behavior:

  • If an item is a Pydantic v2 model, use .model_dump(by_alias=…).

  • If an item is a mapping (dict-like), use it as-is.

  • Otherwise, raise a ValueError (unsupported row type).

Parameters:

  • self (object) – The object instance containing the field to convert.

  • by_alias (bool) – Use field aliases when dumping Pydantic models.

  • exclude_none (str | bool) – Control None/NaN column dropping. - False: keep Nones as-is - “all”: drop columns where all values are None/NaN - “any”: drop columns where any value is None/NaN - True: alias for “all”

  • json_normalize (bool) – If True, flatten nested dicts via pandas.json_normalize.

  • convert_dtypes (bool) – If True, call DataFrame.convert_dtypes() at the end.

  • expand_field (str) – If set, look for this field in each row and expand its keys into top-level columns.

  • expand_prefix (str) – If set, prefix expanded column names with this string.

Returns:

The converted DataFrame.

Return type:

pandas.DataFrame

class ManagedCodeConfig(*, data_granularity: DataGranularity | None = None, query_filter: Annotated[str, Strict(strict=True)] | None = None, type: Annotated[str, Strict(strict=True)], name: Annotated[str, Strict(strict=True)], managed_evaluator: ManagedCodeEvaluator, variables: List[Annotated[str, Strict(strict=True)]], static_params: List[StaticParam] | None = None)[source]#

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:
data_granularity: DataGranularity | None#
query_filter: StrictStr | None#
type: StrictStr#
name: Annotated[str, Field(strict=True)]#
managed_evaluator: ManagedCodeEvaluator#
variables: List[StrictStr]#
static_params: List[StaticParam] | None#
classmethod type_validate_enum(value)[source]#

Validates the enum

classmethod name_validate_regular_expression(value)[source]#

Validates the regular expression

model_config: ClassVar[ConfigDict] = {'populate_by_name': True, 'protected_namespaces': (), 'validate_assignment': True, 'validate_by_alias': True, 'validate_by_name': True}#

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

to_str() → str[source]#

Returns the string representation of the model using alias

Return type:

str

to_json() → str[source]#

Returns the JSON representation of the model using alias

Return type:

str

classmethod from_json(json_str: str) → Self | None[source]#

Create an instance of ManagedCodeConfig from a JSON string

Parameters:

json_str (str)

Return type:

Self | None

to_dict() → Dict[str, Any][source]#

Return the dictionary representation of the model using alias.

This has the following differences from calling pydantic’s self.model_dump(by_alias=True):

  • None is only added to the output dict for nullable fields that were set at model initialization. Other fields with value None are ignored.

Return type:

Dict[str, Any]

classmethod from_dict(obj: Dict[str, Any] | None) → Self | None[source]#

Create an instance of ManagedCodeConfig from a dict

Parameters:

obj (Dict[str, Any] | None)

Return type:

Self | None

class ManagedCodeConfigRequest(*, data_granularity: DataGranularity | None = None, query_filter: Annotated[str, Strict(strict=True)] | None = None, type: Annotated[str, Strict(strict=True)], name: Annotated[str, Strict(strict=True)], managed_evaluator: ManagedCodeEvaluator, variables: List[Annotated[str, Strict(strict=True)]], static_params: List[StaticParamRequest] | None = None)[source]#

Bases: BaseModel

Managed (built-in) code evaluator configuration in a write request (strict form of ManagedCodeConfig)

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:
data_granularity: DataGranularity | None#
query_filter: StrictStr | None#
type: StrictStr#
name: Annotated[str, Field(strict=True)]#
managed_evaluator: ManagedCodeEvaluator#
variables: List[StrictStr]#
static_params: List[StaticParamRequest] | None#
classmethod type_validate_enum(value)[source]#

Validates the enum

classmethod name_validate_regular_expression(value)[source]#

Validates the regular expression

model_config: ClassVar[ConfigDict] = {'populate_by_name': True, 'protected_namespaces': (), 'validate_assignment': True, 'validate_by_alias': True, 'validate_by_name': True}#

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

to_str() → str[source]#

Returns the string representation of the model using alias

Return type:

str

to_json() → str[source]#

Returns the JSON representation of the model using alias

Return type:

str

classmethod from_json(json_str: str) → Self | None[source]#

Create an instance of ManagedCodeConfigRequest from a JSON string

Parameters:

json_str (str)

Return type:

Self | None

to_dict() → Dict[str, Any][source]#

Return the dictionary representation of the model using alias.

This has the following differences from calling pydantic’s self.model_dump(by_alias=True):

  • None is only added to the output dict for nullable fields that were set at model initialization. Other fields with value None are ignored.

Return type:

Dict[str, Any]

classmethod from_dict(obj: Dict[str, Any] | None) → Self | None[source]#

Create an instance of ManagedCodeConfigRequest from a dict

Parameters:

obj (Dict[str, Any] | None)

Return type:

Self | None

class ManagedCodeEvaluator(value, names=_not_given, *values, module=None, qualname=None, type=None, start=1, boundary=None)[source]#

Bases: str, Enum

Built-in managed code evaluator name

MATCHES_REGEX = 'MATCHES_REGEX'#
JSON_PARSEABLE = 'JSON_PARSEABLE'#
CONTAINS_ANY_KEYWORD = 'CONTAINS_ANY_KEYWORD'#
CONTAINS_ALL_KEYWORDS = 'CONTAINS_ALL_KEYWORDS'#
EXACT_MATCH = 'EXACT_MATCH'#
classmethod from_json(json_str: str) → Self[source]#

Create an instance of ManagedCodeEvaluator from a JSON string

Parameters:

json_str (str)

Return type:

Self

class OptimizationDirection(value, names=_not_given, *values, module=None, qualname=None, type=None, start=1, boundary=None)[source]#

Bases: str, Enum

The direction for optimization. Defaults to NONE when omitted. - MAXIMIZE: higher scores are better - MINIMIZE: lower scores are better - NONE: higher or lower scores are neither better nor worse

MAXIMIZE = 'MAXIMIZE'#
MINIMIZE = 'MINIMIZE'#
NONE = 'NONE'#
classmethod from_json(json_str: str) → Self[source]#

Create an instance of OptimizationDirection from a JSON string

Parameters:

json_str (str)

Return type:

Self

class RemoteConfig(*, integration_id: Annotated[str, Strict(strict=True)] | None = None)[source]#

Bases: BaseModel

Remote configuration for a REMOTE evaluator version. The backing EVALUATOR integration is referenced by this version and may also be referenced by other versions. Updating it affects every version that references it. integration_id is included only when the caller has permission to read the backing integration. Callers without that permission still receive the remote version metadata, but the integration reference is omitted.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:

integration_id (Annotated[str, Strict(strict=True)] | None)

integration_id: StrictStr | None#
model_config: ClassVar[ConfigDict] = {'populate_by_name': True, 'protected_namespaces': (), 'validate_assignment': True, 'validate_by_alias': True, 'validate_by_name': True}#

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

to_str() → str[source]#

Returns the string representation of the model using alias

Return type:

str

to_json() → str[source]#

Returns the JSON representation of the model using alias

Return type:

str

classmethod from_json(json_str: str) → Self | None[source]#

Create an instance of RemoteConfig from a JSON string

Parameters:

json_str (str)

Return type:

Self | None

to_dict() → Dict[str, Any][source]#

Return the dictionary representation of the model using alias.

This has the following differences from calling pydantic’s self.model_dump(by_alias=True):

  • None is only added to the output dict for nullable fields that were set at model initialization. Other fields with value None are ignored.

Return type:

Dict[str, Any]

classmethod from_dict(obj: Dict[str, Any] | None) → Self | None[source]#

Create an instance of RemoteConfig from a dict

Parameters:

obj (Dict[str, Any] | None)

Return type:

Self | None

class RemoteConfigInput(*, integration_id: Annotated[str, Strict(strict=True)])[source]#

Bases: BaseModel

Remote configuration for a REMOTE evaluator version in write requests. The referenced integration may also be used by other versions; editing it affects every version that references it.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:

integration_id (Annotated[str, Strict(strict=True)])

integration_id: StrictStr#
model_config: ClassVar[ConfigDict] = {'populate_by_name': True, 'protected_namespaces': (), 'validate_assignment': True, 'validate_by_alias': True, 'validate_by_name': True}#

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

to_str() → str[source]#

Returns the string representation of the model using alias

Return type:

str

to_json() → str[source]#

Returns the JSON representation of the model using alias

Return type:

str

classmethod from_json(json_str: str) → Self | None[source]#

Create an instance of RemoteConfigInput from a JSON string

Parameters:

json_str (str)

Return type:

Self | None

to_dict() → Dict[str, Any][source]#

Return the dictionary representation of the model using alias.

This has the following differences from calling pydantic’s self.model_dump(by_alias=True):

  • None is only added to the output dict for nullable fields that were set at model initialization. Other fields with value None are ignored.

Return type:

Dict[str, Any]

classmethod from_dict(obj: Dict[str, Any] | None) → Self | None[source]#

Create an instance of RemoteConfigInput from a dict

Parameters:

obj (Dict[str, Any] | None)

Return type:

Self | None

class StaticParam(*, name: Annotated[str, Strict(strict=True)], type: StaticParamType, default_value: StaticParamDefaultValue)[source]#

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:
name: StrictStr#
type: StaticParamType#
default_value: StaticParamDefaultValue#
model_config: ClassVar[ConfigDict] = {'populate_by_name': True, 'protected_namespaces': (), 'validate_assignment': True, 'validate_by_alias': True, 'validate_by_name': True}#

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

to_str() → str[source]#

Returns the string representation of the model using alias

Return type:

str

to_json() → str[source]#

Returns the JSON representation of the model using alias

Return type:

str

classmethod from_json(json_str: str) → Self | None[source]#

Create an instance of StaticParam from a JSON string

Parameters:

json_str (str)

Return type:

Self | None

to_dict() → Dict[str, Any][source]#

Return the dictionary representation of the model using alias.

This has the following differences from calling pydantic’s self.model_dump(by_alias=True):

  • None is only added to the output dict for nullable fields that were set at model initialization. Other fields with value None are ignored.

Return type:

Dict[str, Any]

classmethod from_dict(obj: Dict[str, Any] | None) → Self | None[source]#

Create an instance of StaticParam from a dict

Parameters:

obj (Dict[str, Any] | None)

Return type:

Self | None

class StaticParamDefaultValue(*args, oneof_schema_1_validator: Annotated[str, Strict(strict=True)] | None = None, oneof_schema_2_validator: List[Annotated[str, Strict(strict=True)]] | None = None, actual_instance: List[str] | str | None = None, one_of_schemas: Set[str] = {'List[str]', 'str'})[source]#

Bases: BaseModel

Default value. Must be a string when type is STRING or REGEX, and a string array when type is STRING_ARRAY. Mismatches are rejected with 400 by the server.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:
oneof_schema_1_validator: StrictStr | None#
oneof_schema_2_validator: List[StrictStr] | None#
actual_instance: List[str] | str | None#
one_of_schemas: Set[str]#
model_config: ClassVar[ConfigDict] = {'protected_namespaces': (), 'validate_assignment': True}#

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

classmethod actual_instance_must_validate_oneof(v)[source]#
classmethod from_dict(obj: str | Dict[str, Any]) → Self[source]#
Parameters:

obj (str | Dict[str, Any])

Return type:

Self

classmethod from_json(json_str: str) → Self[source]#

Returns the object represented by the json string

Parameters:

json_str (str)

Return type:

Self

to_json() → str[source]#

Returns the JSON representation of the actual instance

Return type:

str

to_dict() → Dict[str, Any] | List[str] | str | None[source]#

Returns the dict representation of the actual instance

Return type:

Dict[str, Any] | List[str] | str | None

to_str() → str[source]#

Returns the string representation of the actual instance

Return type:

str

class StaticParamRequest(*, name: Annotated[str, Strict(strict=True)], type: StaticParamType, default_value: StaticParamDefaultValue)[source]#

Bases: BaseModel

Static evaluator parameter in a write request (strict form of StaticParam)

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:
name: StrictStr#
type: StaticParamType#
default_value: StaticParamDefaultValue#
model_config: ClassVar[ConfigDict] = {'populate_by_name': True, 'protected_namespaces': (), 'validate_assignment': True, 'validate_by_alias': True, 'validate_by_name': True}#

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

to_str() → str[source]#

Returns the string representation of the model using alias

Return type:

str

to_json() → str[source]#

Returns the JSON representation of the model using alias

Return type:

str

classmethod from_json(json_str: str) → Self | None[source]#

Create an instance of StaticParamRequest from a JSON string

Parameters:

json_str (str)

Return type:

Self | None

to_dict() → Dict[str, Any][source]#

Return the dictionary representation of the model using alias.

This has the following differences from calling pydantic’s self.model_dump(by_alias=True):

  • None is only added to the output dict for nullable fields that were set at model initialization. Other fields with value None are ignored.

Return type:

Dict[str, Any]

classmethod from_dict(obj: Dict[str, Any] | None) → Self | None[source]#

Create an instance of StaticParamRequest from a dict

Parameters:

obj (Dict[str, Any] | None)

Return type:

Self | None

class StaticParamType(value, names=_not_given, *values, module=None, qualname=None, type=None, start=1, boundary=None)[source]#

Bases: str, Enum

Argument type for static evaluator parameters. - STRING - A single string value. - STRING_ARRAY - An array of string values. - REGEX - A regular expression string.

STRING = 'STRING'#
STRING_ARRAY = 'STRING_ARRAY'#
REGEX = 'REGEX'#
classmethod from_json(json_str: str) → Self[source]#

Create an instance of StaticParamType from a JSON string

Parameters:

json_str (str)

Return type:

Self

class TemplateConfig(*, name: Annotated[str, Strict(strict=True)], template: Annotated[str, Strict(strict=True)], include_explanations: Annotated[bool, Strict(strict=True)], use_function_calling: Annotated[bool, Strict(strict=True)] | None = None, use_structured_output: Annotated[bool, Strict(strict=True)] | None = True, classification_choices: Dict[str, Annotated[float, Strict(strict=True)] | Annotated[int, Strict(strict=True)]] | None = None, direction: OptimizationDirection | None = None, data_granularity: DataGranularity | None = None, llm_config: EvaluatorLlmConfig)[source]#

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:
name: StrictStr#
template: StrictStr#
include_explanations: StrictBool#
use_function_calling: StrictBool | None#
use_structured_output: StrictBool | None#
classification_choices: Dict[str, StrictFloat | StrictInt] | None#
direction: OptimizationDirection | None#
data_granularity: DataGranularity | None#
llm_config: EvaluatorLlmConfig#
model_config: ClassVar[ConfigDict] = {'populate_by_name': True, 'protected_namespaces': (), 'validate_assignment': True, 'validate_by_alias': True, 'validate_by_name': True}#

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

to_str() → str[source]#

Returns the string representation of the model using alias

Return type:

str

to_json() → str[source]#

Returns the JSON representation of the model using alias

Return type:

str

classmethod from_json(json_str: str) → Self | None[source]#

Create an instance of TemplateConfig from a JSON string

Parameters:

json_str (str)

Return type:

Self | None

to_dict() → Dict[str, Any][source]#

Return the dictionary representation of the model using alias.

This has the following differences from calling pydantic’s self.model_dump(by_alias=True):

  • None is only added to the output dict for nullable fields that were set at model initialization. Other fields with value None are ignored.

Return type:

Dict[str, Any]

classmethod from_dict(obj: Dict[str, Any] | None) → Self | None[source]#

Create an instance of TemplateConfig from a dict

Parameters:

obj (Dict[str, Any] | None)

Return type:

Self | None

class TemplateConfigInput(*, name: Annotated[str, Strict(strict=True)], template: Annotated[str, Strict(strict=True)], include_explanations: Annotated[bool, Strict(strict=True)], use_function_calling: Annotated[bool, Strict(strict=True)], use_structured_output: Annotated[bool, Strict(strict=True)] | None = True, classification_choices: Dict[str, Annotated[float, Strict(strict=True)] | Annotated[int, Strict(strict=True)]] | None = None, direction: OptimizationDirection | None = None, data_granularity: DataGranularity | None = None, llm_config: EvaluatorLlmConfigRequest)[source]#

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:
name: StrictStr#
template: StrictStr#
include_explanations: StrictBool#
use_function_calling: StrictBool#
use_structured_output: StrictBool | None#
classification_choices: Dict[str, StrictFloat | StrictInt] | None#
direction: OptimizationDirection | None#
data_granularity: DataGranularity | None#
llm_config: EvaluatorLlmConfigRequest#
model_config: ClassVar[ConfigDict] = {'populate_by_name': True, 'protected_namespaces': (), 'validate_assignment': True, 'validate_by_alias': True, 'validate_by_name': True}#

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

to_str() → str[source]#

Returns the string representation of the model using alias

Return type:

str

to_json() → str[source]#

Returns the JSON representation of the model using alias

Return type:

str

classmethod from_json(json_str: str) → Self | None[source]#

Create an instance of TemplateConfigInput from a JSON string

Parameters:

json_str (str)

Return type:

Self | None

to_dict() → Dict[str, Any][source]#

Return the dictionary representation of the model using alias.

This has the following differences from calling pydantic’s self.model_dump(by_alias=True):

  • None is only added to the output dict for nullable fields that were set at model initialization. Other fields with value None are ignored.

Return type:

Dict[str, Any]

classmethod from_dict(obj: Dict[str, Any] | None) → Self | None[source]#

Create an instance of TemplateConfigInput from a dict

Parameters:

obj (Dict[str, Any] | None)

Return type:

Self | None

On this page