Evaluators#

class EvaluatorsClient(*, sdk_config: SDKConfiguration, generated_client: ApiClient)[source]#

Bases: object

Client for managing Arize evaluators and evaluator versions.

This class is primarily intended for internal use within the SDK. Users are highly encouraged to access resource-specific functionality via arize.ArizeClient.

The evaluators client is a thin wrapper around the generated REST API client, using the shared generated API client owned by arize.config.SDKConfiguration.

Parameters:

sdk_config (SDKConfiguration) – Resolved SDK configuration.
generated_client (ApiClient) – Shared generated API client instance.

list(*, name: str | None = None, space: str | None = None, limit: int = DEFAULT_LIST_LIMIT, cursor: str | None = None) → EvaluatorListResponse[source]#

List evaluators the user has access to.

Results are sorted by update date (most recent first). This endpoint supports cursor-based pagination. When space is provided, results are limited to that space; otherwise evaluators from all permitted spaces are returned.

Parameters:

name (str | None) – Optional case-insensitive substring filter on the evaluator name.
space (str | None) – Optional space filter. If the value is a base64-encoded resource ID it is treated as a space ID; otherwise it is used as a case-insensitive substring filter on the space name.
limit (int) – Maximum number of evaluators to return (1-100).
cursor (str | None) – Opaque pagination cursor from a previous response.

Returns:

A paginated evaluator list response from the Arize REST API.

Raises:

ApiException – If the API request fails.

Return type:

EvaluatorListResponse

get(*, evaluator: str, space: str | None = None, version_id: str | None = None) → EvaluatorWithVersion[source]#

Get an evaluator by name or ID, with its resolved version.

By default, the latest version is returned. Pass version_id to resolve a specific version instead.

Parameters:

evaluator (str) – Evaluator name or identifier (base64) to retrieve.
space (str | None) – Optional space name or ID. Required when evaluator is a name rather than an ID.
version_id (str | None) – Optional version identifier (base64). If omitted, the latest version is returned.

Returns:

The evaluator with its resolved version.

Raises:

ApiException – If the API request fails (for example, evaluator not found).

Return type:

EvaluatorWithVersion

create_template_evaluator(*, name: str, space: str, commit_message: str, template_config: TemplateConfig, description: str | None = None) → EvaluatorWithVersion[source]#

Create a new template evaluator with an initial version.

The evaluator name must be unique within the given space.

Parameters:

name (str) – Evaluator name (must be unique within the space).
space (str) – Space name or ID to create the evaluator in.
commit_message (str) – Commit message for the initial version.
template_config (TemplateConfig) –
Template configuration for the evaluator. Build with arize.evaluators.types.TemplateConfig. Required fields:
- name — eval column name; must match ^[a-zA-Z0-9_\\s\\-&()]+$.
- template — prompt template string with {variable} placeholders referencing span/trace attributes.
- include_explanations — whether the LLM should include a reasoning explanation alongside the score.
- use_function_calling_if_available — prefer structured function-call output over free-text parsing when the model supports it.
- llm_config — arize.evaluators.types.EvaluatorLlmConfig specifying the model provider, model name, and API key.
Optional fields: classification_choices, direction, data_granularity.
description (str | None) – Optional human-readable description of the evaluator.

Returns:

The created evaluator with its initial version.

Raises:

ApiException – If the API request fails (for example, name conflict or invalid payload).

Return type:

EvaluatorWithVersion

create_code_evaluator(*, name: str, space: str, commit_message: str, code_config: CodeConfig | CustomCodeConfig | ManagedCodeConfig | dict, description: str | None = None) → EvaluatorWithVersion[source]#

Create a new code evaluator with an initial version.

The evaluator name must be unique within the given space.

Parameters:

name (str) – Evaluator name (must be unique within the space).
space (str) – Space name or ID to create the evaluator in.
commit_message (str) – Commit message for the initial version.
code_config (CodeConfig | CustomCodeConfig | ManagedCodeConfig | dict) – Code configuration for the evaluator. Accepts a arize.evaluators.types.CodeConfig wrapper, an unwrapped arize.evaluators.types.ManagedCodeConfig or arize.evaluators.types.CustomCodeConfig, or a plain dict matching one of those schemas.
description (str | None) – Optional human-readable description of the evaluator.

Returns:

The created evaluator with its initial version.

Raises:

ApiException – If the API request fails (for example, name conflict or invalid payload).

Return type:

EvaluatorWithVersion

update(*, evaluator: str, space: str | None = None, name: str | None = None, description: str | None = None) → Evaluator[source]#

Update an evaluator’s metadata.

Parameters:

evaluator (str) – Evaluator name or identifier (base64) to update.
space (str | None) – Optional space name or ID. Required when evaluator is a name rather than an ID.
name (str | None) – New evaluator name (must be unique within its space).
description (str | None) – New description for the evaluator.

Returns:

The updated evaluator.

Raises:

ApiException – If the API request fails.

Return type:

Evaluator

delete(*, evaluator: str, space: str | None = None) → None[source]#

Delete an evaluator and all its versions.

This operation is irreversible.

Parameters:

evaluator (str) – Evaluator name or identifier (base64) to delete.
space (str | None) – Optional space name or ID. Required when evaluator is a name rather than an ID.

Returns:

None.

Raises:

ApiException – If the API request fails (for example, evaluator not found).

Return type:

None

list_versions(*, evaluator: str, space: str | None = None, limit: int = DEFAULT_LIST_LIMIT, cursor: str | None = None) → EvaluatorVersionListResponse[source]#

List all versions of an evaluator.

Results are returned with cursor-based pagination.

Parameters:

evaluator (str) – Evaluator name or identifier (base64) to list versions for.
space (str | None) – Optional space name or ID. Required when evaluator is a name rather than an ID.
limit (int) – Maximum number of versions to return (1-100).
cursor (str | None) – Opaque pagination cursor from a previous response.

Returns:

A paginated evaluator version list response.

Raises:

ApiException – If the API request fails.

Return type:

EvaluatorVersionListResponse

get_version(*, version_id: str) → EvaluatorVersionCode | EvaluatorVersionTemplate | EvaluatorVersionHarness | EvaluatorVersionRemote[source]#

Get a specific evaluator version by its global ID.

Parameters:: version_id (str) – Evaluator version identifier (base64).
Returns:: The evaluator version — a EvaluatorVersionCode for code evaluators (with code_config already unwrapped), an EvaluatorVersionTemplate for template evaluators, or an EvaluatorVersionHarness / EvaluatorVersionRemote for harness and remote evaluators (common version metadata only — their configurations are not yet accessible via the REST API).
Raises:: ApiException – If the API request fails (for example, version not found).
Return type:: EvaluatorVersionCode | EvaluatorVersionTemplate | EvaluatorVersionHarness | EvaluatorVersionRemote

create_template_version(*, evaluator: str, space: str | None = None, commit_message: str, template_config: TemplateConfig) → EvaluatorVersionTemplate[source]#

Create a new template version of an existing evaluator.

The new version becomes the latest version immediately (versioning is append-only). Versions are immutable once created; to change the configuration, create a new version.

Parameters:

evaluator (str) – Evaluator name or identifier (base64) to add a version to.
space (str | None) – Optional space name or ID. Required when evaluator is a name rather than an ID.
commit_message (str) – Commit message describing the changes in this version.
template_config (TemplateConfig) – Updated template configuration for this version. Build with arize.evaluators.types.TemplateConfig.

Returns:

The newly created evaluator version.

Raises:

ApiException – If the API request fails.

Return type:

EvaluatorVersionTemplate

create_code_version(*, evaluator: str, space: str | None = None, commit_message: str, code_config: CodeConfig | CustomCodeConfig | ManagedCodeConfig | dict) → EvaluatorVersionCode[source]#

Create a new code version of an existing evaluator.

The new version becomes the latest version immediately (versioning is append-only). Versions are immutable once created; to change the configuration, create a new version.

Parameters:

evaluator (str) – Evaluator name or identifier (base64) to add a version to.
space (str | None) – Optional space name or ID. Required when evaluator is a name rather than an ID.
commit_message (str) – Commit message describing the changes in this version.
code_config (CodeConfig | CustomCodeConfig | ManagedCodeConfig | dict) – Updated code configuration for this version. Accepts a arize.evaluators.types.CodeConfig wrapper, an unwrapped arize.evaluators.types.ManagedCodeConfig or arize.evaluators.types.CustomCodeConfig, or a plain dict matching one of those schemas.

Returns:

The newly created evaluator version.

Raises:

ApiException – If the API request fails.

Return type:

EvaluatorVersionCode

Response Types#

class CodeConfig(*args, oneof_schema_1_validator: ManagedCodeConfig | None = None, oneof_schema_2_validator: CustomCodeConfig | None = None, actual_instance: CustomCodeConfig | ManagedCodeConfig | None = None, one_of_schemas: Set[str] = {'CustomCodeConfig', 'ManagedCodeConfig'}, discriminator_value_class_map: Dict[str, str] = {})[source]#

Bases: BaseModel

Discriminated union representing either a managed (built-in) or custom (user-supplied Python) code evaluator configuration, resolved by the nested type field (managed -> ManagedCodeConfig, custom -> CustomCodeConfig). This inner type is independent of the parent evaluator version’s type (which is always code here).

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:

oneof_schema_1_validator (ManagedCodeConfig | None)
oneof_schema_2_validator (CustomCodeConfig | None)
actual_instance (CustomCodeConfig | ManagedCodeConfig | None)
one_of_schemas (Set[str])
discriminator_value_class_map (Dict[str, str])

oneof_schema_1_validator: ManagedCodeConfig | None#

oneof_schema_2_validator: CustomCodeConfig | None#

actual_instance: CustomCodeConfig | ManagedCodeConfig | None#

one_of_schemas: Set[str]#

model_config: ClassVar[ConfigDict] = {'protected_namespaces': (), 'validate_assignment': True}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

discriminator_value_class_map: Dict[str, str]#

classmethod actual_instance_must_validate_oneof(v)[source]#

classmethod from_dict(obj: str | Dict[str, Any]) → Self[source]#

Parameters:: obj (str | Dict[str, Any])
Return type:: Self

classmethod from_json(json_str: str) → Self[source]#

Returns the object represented by the json string

Parameters:: json_str (str)
Return type:: Self

to_json() → str[source]#

Returns the JSON representation of the actual instance

Return type:: str

to_dict() → Dict[str, Any] | CustomCodeConfig | ManagedCodeConfig | None[source]#

Returns the dict representation of the actual instance

Return type:: Dict[str, Any] | CustomCodeConfig | ManagedCodeConfig | None

to_str() → str[source]#

Returns the string representation of the actual instance

Return type:: str

class CustomCodeConfig(*, data_granularity: DataGranularity | None = None, query_filter: Annotated[str, Strict(strict=True)] | None = None, type: Annotated[str, Strict(strict=True)], name: Annotated[str, Strict(strict=True)], code: Annotated[str, Strict(strict=True)], imports: Annotated[str, Strict(strict=True)] | None = None, variables: List[Annotated[str, Strict(strict=True)]], static_params: List[StaticParam] | None = None)[source]#

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:

data_granularity (DataGranularity | None)
query_filter (Annotated[str, Strict(strict=True)] | None)
type (Annotated[str, Strict(strict=True)])
name (Annotated[str, FieldInfo(annotation=NoneType, required=True, metadata=[Strict(strict=True)])])
code (Annotated[str, Strict(strict=True)])
imports (Annotated[str, Strict(strict=True)] | None)
variables (List[Annotated[str, Strict(strict=True)]])
static_params (List[StaticParam] | None)

data_granularity: DataGranularity | None#

query_filter: StrictStr | None#

type: StrictStr#

name: Annotated[str, Field(strict=True)]#

code: StrictStr#

imports: StrictStr | None#

variables: List[StrictStr]#

static_params: List[StaticParam] | None#

classmethod type_validate_enum(value)[source]#: Validates the enum

classmethod name_validate_regular_expression(value)[source]#: Validates the regular expression

model_config: ClassVar[ConfigDict] = {'populate_by_name': True, 'protected_namespaces': (), 'validate_assignment': True, 'validate_by_alias': True, 'validate_by_name': True}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

to_str() → str[source]#

Returns the string representation of the model using alias

Return type:: str

to_json() → str[source]#

Returns the JSON representation of the model using alias

Return type:: str

classmethod from_json(json_str: str) → Self | None[source]#

Create an instance of CustomCodeConfig from a JSON string

Parameters:: json_str (str)
Return type:: Self | None

to_dict() → Dict[str, Any][source]#

Return the dictionary representation of the model using alias.

This has the following differences from calling pydantic’s self.model_dump(by_alias=True):

None is only added to the output dict for nullable fields that were set at model initialization. Other fields with value None are ignored.

Return type:: Dict[str, Any]

classmethod from_dict(obj: Dict[str, Any] | None) → Self | None[source]#

Create an instance of CustomCodeConfig from a dict

Parameters:: obj (Dict[str, Any] | None)
Return type:: Self | None

class DataGranularity(value, names=_not_given, *values, module=None, qualname=None, type=None, start=1, boundary=None)[source]#

Bases: str, Enum

Data granularity level for evaluation. - span - Evaluate at the individual span level. - trace - Evaluate at the full trace level. - session - Evaluate at the session level.

SPAN = 'span'#

TRACE = 'trace'#

SESSION = 'session'#

classmethod from_json(json_str: str) → Self[source]#

Create an instance of DataGranularity from a JSON string

Parameters:: json_str (str)
Return type:: Self

class Evaluator(*, id: Annotated[str, Strict(strict=True)], name: Annotated[str, Strict(strict=True)], description: Annotated[str, Strict(strict=True)] | None = None, type: EvaluatorType, space_id: Annotated[str, Strict(strict=True)], created_at: datetime, updated_at: datetime, created_by_user_id: Annotated[str, Strict(strict=True)] | None)[source]#

Bases: BaseModel

An evaluator defines reusable evaluation logic that can be attached to evaluation tasks. The type field determines the kind of evaluation: template (LLM-based template evaluation) or code (custom code evaluation).

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:

id (Annotated[str, Strict(strict=True)])
name (Annotated[str, Strict(strict=True)])
description (Annotated[str, Strict(strict=True)] | None)
type (EvaluatorType)
space_id (Annotated[str, Strict(strict=True)])
created_at (datetime)
updated_at (datetime)
created_by_user_id (Annotated[str, Strict(strict=True)] | None)

id: StrictStr#

name: StrictStr#

description: StrictStr | None#

type: EvaluatorType#

space_id: StrictStr#

created_at: datetime#

updated_at: datetime#

created_by_user_id: StrictStr | None#

model_config: ClassVar[ConfigDict] = {'populate_by_name': True, 'protected_namespaces': (), 'validate_assignment': True, 'validate_by_alias': True, 'validate_by_name': True}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

to_str() → str[source]#

Returns the string representation of the model using alias

Return type:: str

to_json() → str[source]#

Returns the JSON representation of the model using alias

Return type:: str

classmethod from_json(json_str: str) → Self | None[source]#

Create an instance of Evaluator from a JSON string

Parameters:: json_str (str)
Return type:: Self | None

to_dict() → Dict[str, Any][source]#

Return the dictionary representation of the model using alias.

This has the following differences from calling pydantic’s self.model_dump(by_alias=True):

None is only added to the output dict for nullable fields that were set at model initialization. Other fields with value None are ignored.

Return type:: Dict[str, Any]

classmethod from_dict(obj: Dict[str, Any] | None) → Self | None[source]#

Create an instance of Evaluator from a dict

Parameters:: obj (Dict[str, Any] | None)
Return type:: Self | None

class EvaluatorListResponse(*, evaluators: List[Evaluator], pagination: PaginationMetadata)[source]#

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:

evaluators (List[Evaluator])
pagination (PaginationMetadata)

evaluators: List[Evaluator]#

pagination: PaginationMetadata#

model_config: ClassVar[ConfigDict] = {'populate_by_name': True, 'protected_namespaces': (), 'validate_assignment': True, 'validate_by_alias': True, 'validate_by_name': True}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

to_str() → str[source]#

Returns the string representation of the model using alias

Return type:: str

to_json() → str[source]#

Returns the JSON representation of the model using alias

Return type:: str

classmethod from_json(json_str: str) → Self | None[source]#

Create an instance of EvaluatorListResponse from a JSON string

Parameters:: json_str (str)
Return type:: Self | None

to_dict() → Dict[str, Any][source]#

Return the dictionary representation of the model using alias.

This has the following differences from calling pydantic’s self.model_dump(by_alias=True):

None is only added to the output dict for nullable fields that were set at model initialization. Other fields with value None are ignored.

Return type:: Dict[str, Any]

classmethod from_dict(obj: Dict[str, Any] | None) → Self | None[source]#

Create an instance of EvaluatorListResponse from a dict

Parameters:: obj (Dict[str, Any] | None)
Return type:: Self | None

to_df(by_alias: bool = False, exclude_none: str | bool = True, json_normalize: bool = False, convert_dtypes: bool = True, expand_field: str = 'additional_properties', expand_prefix: str = '') → pd.DataFrame#

Convert a list of objects to a pandas.DataFrame.

Behavior:

If an item is a Pydantic v2 model, use .model_dump(by_alias=…).
If an item is a mapping (dict-like), use it as-is.
Otherwise, raise a ValueError (unsupported row type).

Parameters:

self (object) – The object instance containing the field to convert.
by_alias (bool) – Use field aliases when dumping Pydantic models.
exclude_none (str | bool) – Control None/NaN column dropping. - False: keep Nones as-is - “all”: drop columns where all values are None/NaN - “any”: drop columns where any value is None/NaN - True: alias for “all”
json_normalize (bool) – If True, flatten nested dicts via pandas.json_normalize.
convert_dtypes (bool) – If True, call DataFrame.convert_dtypes() at the end.
expand_field (str) – If set, look for this field in each row and expand its keys into top-level columns.
expand_prefix (str) – If set, prefix expanded column names with this string.

Returns:

The converted DataFrame.

Return type:

pandas.DataFrame

class EvaluatorLlmConfig(*, ai_integration_id: Annotated[str, Strict(strict=True)], model_name: Annotated[str, Strict(strict=True)], invocation_parameters: InvocationParams, provider_parameters: ProviderParams)[source]#

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:

ai_integration_id (Annotated[str, Strict(strict=True)])
model_name (Annotated[str, Strict(strict=True)])
invocation_parameters (InvocationParams)
provider_parameters (ProviderParams)

ai_integration_id: StrictStr#

model_name: StrictStr#

invocation_parameters: InvocationParams#

provider_parameters: ProviderParams#

model_config: ClassVar[ConfigDict] = {'populate_by_name': True, 'protected_namespaces': (), 'validate_assignment': True, 'validate_by_alias': True, 'validate_by_name': True}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

to_str() → str[source]#

Returns the string representation of the model using alias

Return type:: str

to_json() → str[source]#

Returns the JSON representation of the model using alias

Return type:: str

classmethod from_json(json_str: str) → Self | None[source]#

Create an instance of EvaluatorLlmConfig from a JSON string

Parameters:: json_str (str)
Return type:: Self | None

to_dict() → Dict[str, Any][source]#

Return the dictionary representation of the model using alias.

This has the following differences from calling pydantic’s self.model_dump(by_alias=True):

None is only added to the output dict for nullable fields that were set at model initialization. Other fields with value None are ignored.

Return type:: Dict[str, Any]

classmethod from_dict(obj: Dict[str, Any] | None) → Self | None[source]#

Create an instance of EvaluatorLlmConfig from a dict

Parameters:: obj (Dict[str, Any] | None)
Return type:: Self | None

class EvaluatorType(value, names=_not_given, *values, module=None, qualname=None, type=None, start=1, boundary=None)[source]#

Bases: str, Enum

The evaluator type: - template — LLM-based evaluator. - code — managed built-in evaluators or custom Python code (both are subtypes of code, discriminated by the nested CodeConfig.type = managed | custom). - harness — test harness evaluator. - remote — remote evaluator. Applies to both the parent Evaluator.type field and every version’s type discriminator — a version’s type must always match its parent evaluator’s type.

TEMPLATE = 'template'#

CODE = 'code'#

HARNESS = 'harness'#

REMOTE = 'remote'#

classmethod from_json(json_str: str) → Self[source]#

Create an instance of EvaluatorType from a JSON string

Parameters:: json_str (str)
Return type:: Self

class EvaluatorVersionCode(*, id: str, evaluator_id: str, commit_hash: str, commit_message: str | None = None, created_at: datetime, created_by_user_id: str | None = None, type: str, code_config: CustomCodeConfig | ManagedCodeConfig)[source]#

Bases: BaseModel

SDK view of the generated EvaluatorVersionCode with code_config unwrapped.

The code_config field holds the concrete inner type (CustomCodeConfig or ManagedCodeConfig) instead of the oneOf wrapper CodeConfig, so model_dump() and attribute access work as expected.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:

id (str)
evaluator_id (str)
commit_hash (str)
commit_message (str | None)
created_at (datetime)
created_by_user_id (str | None)
type (str)
code_config (CustomCodeConfig | ManagedCodeConfig)

id: str#

evaluator_id: str#

commit_hash: str#

commit_message: str | None#

created_at: datetime#

created_by_user_id: str | None#

type: str#

code_config: CustomCodeConfig | ManagedCodeConfig#

model_config: ClassVar[ConfigDict] = {'from_attributes': True}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class EvaluatorVersionCodeCreate(*, commit_message: Annotated[str, Strict(strict=True)], code_config: CodeConfig)[source]#

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:

commit_message (Annotated[str, Strict(strict=True)])
code_config (CodeConfig)

commit_message: StrictStr#

code_config: CodeConfig#

model_config: ClassVar[ConfigDict] = {'populate_by_name': True, 'protected_namespaces': (), 'validate_assignment': True, 'validate_by_alias': True, 'validate_by_name': True}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

to_str() → str[source]#

Returns the string representation of the model using alias

Return type:: str

to_json() → str[source]#

Returns the JSON representation of the model using alias

Return type:: str

classmethod from_json(json_str: str) → Self | None[source]#

Create an instance of EvaluatorVersionCodeCreate from a JSON string

Parameters:: json_str (str)
Return type:: Self | None

to_dict() → Dict[str, Any][source]#

Return the dictionary representation of the model using alias.

This has the following differences from calling pydantic’s self.model_dump(by_alias=True):

None is only added to the output dict for nullable fields that were set at model initialization. Other fields with value None are ignored.

Return type:: Dict[str, Any]

classmethod from_dict(obj: Dict[str, Any] | None) → Self | None[source]#

Create an instance of EvaluatorVersionCodeCreate from a dict

Parameters:: obj (Dict[str, Any] | None)
Return type:: Self | None

class EvaluatorVersionCreate(*args, oneof_schema_1_validator: EvaluatorVersionTemplateCreate | None = None, oneof_schema_2_validator: EvaluatorVersionCodeCreate | None = None, actual_instance: EvaluatorVersionCodeCreate | EvaluatorVersionTemplateCreate | None = None, one_of_schemas: Set[str] = {'EvaluatorVersionCodeCreate', 'EvaluatorVersionTemplateCreate'})[source]#

Bases: BaseModel

Payload for an evaluator version: exactly one of template_config or code_config. Used both when creating an evaluator (initial version) and when appending a version.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:

oneof_schema_1_validator (EvaluatorVersionTemplateCreate | None)
oneof_schema_2_validator (EvaluatorVersionCodeCreate | None)
actual_instance (EvaluatorVersionCodeCreate | EvaluatorVersionTemplateCreate | None)
one_of_schemas (Set[str])

oneof_schema_1_validator: EvaluatorVersionTemplateCreate | None#

oneof_schema_2_validator: EvaluatorVersionCodeCreate | None#

actual_instance: EvaluatorVersionCodeCreate | EvaluatorVersionTemplateCreate | None#

one_of_schemas: Set[str]#

model_config: ClassVar[ConfigDict] = {'protected_namespaces': (), 'validate_assignment': True}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

classmethod actual_instance_must_validate_oneof(v)[source]#

classmethod from_dict(obj: str | Dict[str, Any]) → Self[source]#

Parameters:: obj (str | Dict[str, Any])
Return type:: Self

classmethod from_json(json_str: str) → Self[source]#

Returns the object represented by the json string

Parameters:: json_str (str)
Return type:: Self

to_json() → str[source]#

Returns the JSON representation of the actual instance

Return type:: str

to_dict() → Dict[str, Any] | EvaluatorVersionCodeCreate | EvaluatorVersionTemplateCreate | None[source]#

Returns the dict representation of the actual instance

Return type:: Dict[str, Any] | EvaluatorVersionCodeCreate | EvaluatorVersionTemplateCreate | None

to_str() → str[source]#

Returns the string representation of the actual instance

Return type:: str

class EvaluatorVersionListResponse(*, evaluator_versions: list[EvaluatorVersionCode | EvaluatorVersionTemplate | EvaluatorVersionHarness | EvaluatorVersionRemote], pagination: PaginationMetadata)[source]#

Bases: BaseModel

SDK view of the generated EvaluatorVersionListResponse with each version unwrapped.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:

evaluator_versions (list[EvaluatorVersionCode | EvaluatorVersionTemplate | EvaluatorVersionHarness | EvaluatorVersionRemote])
pagination (PaginationMetadata)

evaluator_versions: list[EvaluatorVersionCode | EvaluatorVersionTemplate | EvaluatorVersionHarness | EvaluatorVersionRemote]#

pagination: PaginationMetadata#

model_config: ClassVar[ConfigDict] = {'from_attributes': True}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

to_df(by_alias: bool = False, exclude_none: str | bool = True, json_normalize: bool = False, convert_dtypes: bool = True, expand_field: str = 'additional_properties', expand_prefix: str = '') → pd.DataFrame#

Convert a list of objects to a pandas.DataFrame.

Behavior:

If an item is a Pydantic v2 model, use .model_dump(by_alias=…).
If an item is a mapping (dict-like), use it as-is.
Otherwise, raise a ValueError (unsupported row type).

Parameters:

self (object) – The object instance containing the field to convert.
by_alias (bool) – Use field aliases when dumping Pydantic models.
exclude_none (str | bool) – Control None/NaN column dropping. - False: keep Nones as-is - “all”: drop columns where all values are None/NaN - “any”: drop columns where any value is None/NaN - True: alias for “all”
json_normalize (bool) – If True, flatten nested dicts via pandas.json_normalize.
convert_dtypes (bool) – If True, call DataFrame.convert_dtypes() at the end.
expand_field (str) – If set, look for this field in each row and expand its keys into top-level columns.
expand_prefix (str) – If set, prefix expanded column names with this string.

Returns:

The converted DataFrame.

Return type:

pandas.DataFrame

class EvaluatorVersionTemplate(*, id: Annotated[str, Strict(strict=True)], evaluator_id: Annotated[str, Strict(strict=True)], commit_hash: Annotated[str, Strict(strict=True)], commit_message: Annotated[str, Strict(strict=True)] | None, created_at: datetime, created_by_user_id: Annotated[str, Strict(strict=True)] | None, type: Annotated[str, Strict(strict=True)], template_config: TemplateConfig)[source]#

Bases: BaseModel

Evaluator version carrying a template (LLM) configuration.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:

id (Annotated[str, Strict(strict=True)])
evaluator_id (Annotated[str, Strict(strict=True)])
commit_hash (Annotated[str, Strict(strict=True)])
commit_message (Annotated[str, Strict(strict=True)] | None)
created_at (datetime)
created_by_user_id (Annotated[str, Strict(strict=True)] | None)
type (Annotated[str, Strict(strict=True)])
template_config (TemplateConfig)

id: StrictStr#

evaluator_id: StrictStr#

commit_hash: StrictStr#

commit_message: StrictStr | None#

created_at: datetime#

created_by_user_id: StrictStr | None#

type: StrictStr#

template_config: TemplateConfig#

classmethod type_validate_enum(value)[source]#: Validates the enum

model_config: ClassVar[ConfigDict] = {'populate_by_name': True, 'protected_namespaces': (), 'validate_assignment': True, 'validate_by_alias': True, 'validate_by_name': True}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

to_str() → str[source]#

Returns the string representation of the model using alias

Return type:: str

to_json() → str[source]#

Returns the JSON representation of the model using alias

Return type:: str

classmethod from_json(json_str: str) → Self | None[source]#

Create an instance of EvaluatorVersionTemplate from a JSON string

Parameters:: json_str (str)
Return type:: Self | None

to_dict() → Dict[str, Any][source]#

Return the dictionary representation of the model using alias.

This has the following differences from calling pydantic’s self.model_dump(by_alias=True):

None is only added to the output dict for nullable fields that were set at model initialization. Other fields with value None are ignored.

Return type:: Dict[str, Any]

classmethod from_dict(obj: Dict[str, Any] | None) → Self | None[source]#

Create an instance of EvaluatorVersionTemplate from a dict

Parameters:: obj (Dict[str, Any] | None)
Return type:: Self | None

class EvaluatorVersionTemplateCreate(*, commit_message: Annotated[str, Strict(strict=True)], template_config: TemplateConfig)[source]#

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:

commit_message (Annotated[str, Strict(strict=True)])
template_config (TemplateConfig)

commit_message: StrictStr#

template_config: TemplateConfig#

model_config: ClassVar[ConfigDict] = {'populate_by_name': True, 'protected_namespaces': (), 'validate_assignment': True, 'validate_by_alias': True, 'validate_by_name': True}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

to_str() → str[source]#

Returns the string representation of the model using alias

Return type:: str

to_json() → str[source]#

Returns the JSON representation of the model using alias

Return type:: str

classmethod from_json(json_str: str) → Self | None[source]#

Create an instance of EvaluatorVersionTemplateCreate from a JSON string

Parameters:: json_str (str)
Return type:: Self | None

to_dict() → Dict[str, Any][source]#

Return the dictionary representation of the model using alias.

This has the following differences from calling pydantic’s self.model_dump(by_alias=True):

None is only added to the output dict for nullable fields that were set at model initialization. Other fields with value None are ignored.

Return type:: Dict[str, Any]

classmethod from_dict(obj: Dict[str, Any] | None) → Self | None[source]#

Create an instance of EvaluatorVersionTemplateCreate from a dict

Parameters:: obj (Dict[str, Any] | None)
Return type:: Self | None

class EvaluatorWithVersion(*, id: str, name: str, description: str | None = None, type: EvaluatorType, space_id: str, created_at: datetime, updated_at: datetime, created_by_user_id: str | None = None, version: EvaluatorVersionCode | EvaluatorVersionTemplate | EvaluatorVersionHarness | EvaluatorVersionRemote)[source]#

Bases: BaseModel

SDK view of the generated EvaluatorWithVersion with version unwrapped.

The version field holds the concrete inner type (EvaluatorVersionCode for code evaluators, EvaluatorVersionTemplate for template evaluators, EvaluatorVersionHarness for harness evaluators, or EvaluatorVersionRemote for remote evaluators) instead of the oneOf wrapper.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:

id (str)
name (str)
description (str | None)
type (EvaluatorType)
space_id (str)
created_at (datetime)
updated_at (datetime)
created_by_user_id (str | None)
version (EvaluatorVersionCode | EvaluatorVersionTemplate | EvaluatorVersionHarness | EvaluatorVersionRemote)

id: str#

name: str#

description: str | None#

type: EvaluatorType#

space_id: str#

created_at: datetime#

updated_at: datetime#

created_by_user_id: str | None#

version: EvaluatorVersionCode | EvaluatorVersionTemplate | EvaluatorVersionHarness | EvaluatorVersionRemote#

model_config: ClassVar[ConfigDict] = {'from_attributes': True}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class ManagedCodeConfig(*, data_granularity: DataGranularity | None = None, query_filter: Annotated[str, Strict(strict=True)] | None = None, type: Annotated[str, Strict(strict=True)], name: Annotated[str, Strict(strict=True)], managed_evaluator: ManagedCodeEvaluator, variables: List[Annotated[str, Strict(strict=True)]], static_params: List[StaticParam] | None = None)[source]#

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:

data_granularity (DataGranularity | None)
query_filter (Annotated[str, Strict(strict=True)] | None)
type (Annotated[str, Strict(strict=True)])
name (Annotated[str, FieldInfo(annotation=NoneType, required=True, metadata=[Strict(strict=True)])])
managed_evaluator (ManagedCodeEvaluator)
variables (List[Annotated[str, Strict(strict=True)]])
static_params (List[StaticParam] | None)

data_granularity: DataGranularity | None#

query_filter: StrictStr | None#

type: StrictStr#

name: Annotated[str, Field(strict=True)]#

managed_evaluator: ManagedCodeEvaluator#

variables: List[StrictStr]#

static_params: List[StaticParam] | None#

classmethod type_validate_enum(value)[source]#: Validates the enum

classmethod name_validate_regular_expression(value)[source]#: Validates the regular expression

model_config: ClassVar[ConfigDict] = {'populate_by_name': True, 'protected_namespaces': (), 'validate_assignment': True, 'validate_by_alias': True, 'validate_by_name': True}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

to_str() → str[source]#

Returns the string representation of the model using alias

Return type:: str

to_json() → str[source]#

Returns the JSON representation of the model using alias

Return type:: str

classmethod from_json(json_str: str) → Self | None[source]#

Create an instance of ManagedCodeConfig from a JSON string

Parameters:: json_str (str)
Return type:: Self | None

to_dict() → Dict[str, Any][source]#

Return the dictionary representation of the model using alias.

This has the following differences from calling pydantic’s self.model_dump(by_alias=True):

None is only added to the output dict for nullable fields that were set at model initialization. Other fields with value None are ignored.

Return type:: Dict[str, Any]

classmethod from_dict(obj: Dict[str, Any] | None) → Self | None[source]#

Create an instance of ManagedCodeConfig from a dict

Parameters:: obj (Dict[str, Any] | None)
Return type:: Self | None

class ManagedCodeEvaluator(value, names=_not_given, *values, module=None, qualname=None, type=None, start=1, boundary=None)[source]#

Bases: str, Enum

Built-in managed code evaluator name

MATCHESREGEX = 'MatchesRegex'#

JSONPARSEABLE = 'JSONParseable'#

CONTAINSANYKEYWORD = 'ContainsAnyKeyword'#

CONTAINSALLKEYWORDS = 'ContainsAllKeywords'#

EXACTMATCH = 'ExactMatch'#

classmethod from_json(json_str: str) → Self[source]#

Create an instance of ManagedCodeEvaluator from a JSON string

Parameters:: json_str (str)
Return type:: Self

class OptimizationDirection(value, names=_not_given, *values, module=None, qualname=None, type=None, start=1, boundary=None)[source]#

Bases: str, Enum

The direction for optimization. Defaults to none when omitted. - maximize: higher scores are better - minimize: lower scores are better - none: higher or lower scores are neither better nor worse

MAXIMIZE = 'maximize'#

MINIMIZE = 'minimize'#

NONE = 'none'#

classmethod from_json(json_str: str) → Self[source]#

Create an instance of OptimizationDirection from a JSON string

Parameters:: json_str (str)
Return type:: Self

class StaticParam(*, name: Annotated[str, Strict(strict=True)], type: StaticParamType, default_value: StaticParamDefaultValue)[source]#

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:

name (Annotated[str, Strict(strict=True)])
type (StaticParamType)
default_value (StaticParamDefaultValue)

name: StrictStr#

type: StaticParamType#

default_value: StaticParamDefaultValue#

model_config: ClassVar[ConfigDict] = {'populate_by_name': True, 'protected_namespaces': (), 'validate_assignment': True, 'validate_by_alias': True, 'validate_by_name': True}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

to_str() → str[source]#

Returns the string representation of the model using alias

Return type:: str

to_json() → str[source]#

Returns the JSON representation of the model using alias

Return type:: str

classmethod from_json(json_str: str) → Self | None[source]#

Create an instance of StaticParam from a JSON string

Parameters:: json_str (str)
Return type:: Self | None

to_dict() → Dict[str, Any][source]#

Return the dictionary representation of the model using alias.

This has the following differences from calling pydantic’s self.model_dump(by_alias=True):

None is only added to the output dict for nullable fields that were set at model initialization. Other fields with value None are ignored.

Return type:: Dict[str, Any]

classmethod from_dict(obj: Dict[str, Any] | None) → Self | None[source]#

Create an instance of StaticParam from a dict

Parameters:: obj (Dict[str, Any] | None)
Return type:: Self | None

class StaticParamDefaultValue(*args, oneof_schema_1_validator: Annotated[str, Strict(strict=True)] | None = None, oneof_schema_2_validator: List[Annotated[str, Strict(strict=True)]] | None = None, actual_instance: List[str] | str | None = None, one_of_schemas: Set[str] = {'List[str]', 'str'})[source]#

Bases: BaseModel

Default value. Must be a string when type is STRING or REGEX, and a string array when type is STRING_ARRAY. Mismatches are rejected with 400 by the server.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:

oneof_schema_1_validator (Annotated[str, Strict(strict=True)] | None)
oneof_schema_2_validator (List[Annotated[str, Strict(strict=True)]] | None)
actual_instance (List[str] | str | None)
one_of_schemas (Set[str])

oneof_schema_1_validator: StrictStr | None#

oneof_schema_2_validator: List[StrictStr] | None#

actual_instance: List[str] | str | None#

one_of_schemas: Set[str]#

model_config: ClassVar[ConfigDict] = {'protected_namespaces': (), 'validate_assignment': True}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

classmethod actual_instance_must_validate_oneof(v)[source]#

classmethod from_dict(obj: str | Dict[str, Any]) → Self[source]#

Parameters:: obj (str | Dict[str, Any])
Return type:: Self

classmethod from_json(json_str: str) → Self[source]#

Returns the object represented by the json string

Parameters:: json_str (str)
Return type:: Self

to_json() → str[source]#

Returns the JSON representation of the actual instance

Return type:: str

to_dict() → Dict[str, Any] | List[str] | str | None[source]#

Returns the dict representation of the actual instance

Return type:: Dict[str, Any] | List[str] | str | None

to_str() → str[source]#

Returns the string representation of the actual instance

Return type:: str

class StaticParamType(value, names=_not_given, *values, module=None, qualname=None, type=None, start=1, boundary=None)[source]#

Bases: str, Enum

Argument type for static evaluator parameters. - STRING - A single string value. - STRING_ARRAY - An array of string values. - REGEX - A regular expression string.

STRING = 'STRING'#

STRING_ARRAY = 'STRING_ARRAY'#

REGEX = 'REGEX'#

classmethod from_json(json_str: str) → Self[source]#

Create an instance of StaticParamType from a JSON string

Parameters:: json_str (str)
Return type:: Self

class TemplateConfig(*, name: Annotated[str, Strict(strict=True)], template: Annotated[str, Strict(strict=True)], include_explanations: Annotated[bool, Strict(strict=True)], use_function_calling_if_available: Annotated[bool, Strict(strict=True)], classification_choices: Dict[str, Annotated[float, Strict(strict=True)] | Annotated[int, Strict(strict=True)]] | None = None, direction: OptimizationDirection | None = None, data_granularity: DataGranularity | None = None, llm_config: EvaluatorLlmConfig)[source]#

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:

name (Annotated[str, Strict(strict=True)])
template (Annotated[str, Strict(strict=True)])
include_explanations (Annotated[bool, Strict(strict=True)])
use_function_calling_if_available (Annotated[bool, Strict(strict=True)])
classification_choices (Dict[str, Annotated[float, Strict(strict=True)] | Annotated[int, Strict(strict=True)]] | None)
direction (OptimizationDirection | None)
data_granularity (DataGranularity | None)
llm_config (EvaluatorLlmConfig)

name: StrictStr#

template: StrictStr#

include_explanations: StrictBool#

use_function_calling_if_available: StrictBool#

classification_choices: Dict[str, StrictFloat | StrictInt] | None#

direction: OptimizationDirection | None#

data_granularity: DataGranularity | None#

llm_config: EvaluatorLlmConfig#

model_config: ClassVar[ConfigDict] = {'populate_by_name': True, 'protected_namespaces': (), 'validate_assignment': True, 'validate_by_alias': True, 'validate_by_name': True}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

to_str() → str[source]#

Returns the string representation of the model using alias

Return type:: str

to_json() → str[source]#

Returns the JSON representation of the model using alias

Return type:: str

classmethod from_json(json_str: str) → Self | None[source]#

Create an instance of TemplateConfig from a JSON string

Parameters:: json_str (str)
Return type:: Self | None

to_dict() → Dict[str, Any][source]#

Return the dictionary representation of the model using alias.

This has the following differences from calling pydantic’s self.model_dump(by_alias=True):

None is only added to the output dict for nullable fields that were set at model initialization. Other fields with value None are ignored.

Return type:: Dict[str, Any]

classmethod from_dict(obj: Dict[str, Any] | None) → Self | None[source]#

Create an instance of TemplateConfig from a dict

Parameters:: obj (Dict[str, Any] | None)
Return type:: Self | None