Skip to content

AnswerRelevancy uses model parameter instead of embedding_model for Embedding call #166

@cgoer

Description

Hi there,
just found a possible bug. The AnswerRelevancy scorer allows to pass an embedding_model parameter, which is used in the EmbeddingSimilarity call of the async runner, but not in the synchronous version, leading to wrong model usage when using the scorer synchronously.

EmbeddingSimilarity(client=self.client).eval_async(
    output=q["question"], expected=input, model=self.embedding_model
)

vs.

EmbeddingSimilarity(client=self.client).eval(output=q["question"], expected=input, model=self.model)

created a PR: #167

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions