Evaluate using local scorers

​

Prerequisites

pip install weave[scorers]

​

Select a scorer

​

WeaveBiasScorerV1

• Race and Origin: Racism and bias against a country or region of origin, immigration status, ethnicity, etc.
• Gender and Sexuality: Sexism, misogyny, homophobia, transphobia, sexual harassment, etc.

​

Usage notes

• The score method expects a string to be passed to the output parameter.
• A higher score means that there is a stronger prediction of bias in the text.
• The threshold parameter is set but can also be overridden on initialization.

​

Usage example

import weave
from weave.scorers import WeaveBiasScorerV1

bias_scorer = WeaveBiasScorerV1()
result = bias_scorer.score(output="Martian men are terrible at cleaning")

print(f"The text is biased: {not result.passed}")
print(result)

​

WeaveToxicityScorerV1

• Race and Origin: Racism and bias against a country or region of origin, immigration status, ethnicity, etc.
• Gender and Sexuality: Sexism, misogyny, homophobia, transphobia, sexual harassment, etc.
• Religious: Bias or stereotypes against someone’s religion.
• Ability: Bias related to someone’s physical, mental, or intellectual ability or disability.
• Violence and Abuse: Overly graphic descriptions of violence, threats of violence, or incitement of violence.

​

Usage notes

• The score method expects a string to be passed to the output parameter.
• The model returns scores from 0 to 3 across five different categories:
  • If the sum of these scores is above total_threshold (default value 5), the input is flagged as toxic.
  • If any single category has a score higher than category_threshold (default 2), the input is flagged as toxic.
• To make filtering more aggressive, override category_threshold or total_threshold during initialization.

​

Usage example

import weave
from weave.scorers import WeaveToxicityScorerV1

toxicity_scorer = WeaveToxicityScorerV1()
result = toxicity_scorer.score(output="people from the south pole of Mars are the worst")

print(f"Input is toxic: {not result.passed}")
print(result)

​

WeaveHallucinationScorerV1

​

Usage notes

• The score method expects values to be passed to the query and output parameters.
• The context should be passed to the output parameter (as a string or list of strings).
• A higher output score means a stronger prediction of hallucination in the output.
• The threshold parameter is set but can be overridden on initialization.

​

Usage example

import weave
from weave.scorers import WeaveHallucinationScorerV1

hallucination_scorer = WeaveHallucinationScorerV1()

result = hallucination_scorer.score(
    query="What is the capital of Antarctica?",
    context="People in Antarctica love the penguins.",
    output="While Antarctica is known for its sea life, penguins aren't liked there."
)

print(f"Output is hallucinated: {not result.passed}")
print(result)

​

WeaveContextRelevanceScorerV1

​

Usage notes

• The score method expects values for query and output.
• The context should be passed to the output parameter (string or list of strings).
• A higher score means a stronger prediction that the context is relevant to the query.
• You can pass verbose=True to the score method to get per-chunk scores.

​

Usage example

import weave
from weave.scorers import WeaveContextRelevanceScorerV1

context_relevance_scorer = WeaveContextRelevanceScorerV1()

result = context_relevance_scorer.score(
    query="What is the capital of Antarctica?",
    output="The Antarctic has the happiest penguins."  # context is passed to the output parameter
)

print(f"Output is relevant: {result.passed}")
print(result)

​

WeaveCoherenceScorerV1

​

Usage notes

• The score method expects text to be passed to the query and output parameters.
• A higher output score means a stronger prediction of coherence.

​

Usage example

import weave
from weave.scorers import WeaveCoherenceScorerV1

coherence_scorer = WeaveCoherenceScorerV1()

result = coherence_scorer.score(
    query="What is the capital of Antarctica?",
    output="but why not monkey up day"
)

print(f"Output is coherent: {result.passed}")
print(result)

​

WeaveFluencyScorerV1

​

Usage notes

• The score method expects text to be passed to the output parameter.
• A higher output score indicates higher fluency.

​

Usage example

import weave
from weave.scorers import WeaveFluencyScorerV1

fluency_scorer = WeaveFluencyScorerV1()

result = fluency_scorer.score(
    output="The cat did stretching lazily into warmth of sunlight."
)

print(f"Output is fluent: {result.passed}")
print(result)

​

WeaveTrustScorerV1

• high: No issues detected
• medium: Only Advisory issues detected
• low: Critical issues detected or input is empty

• Critical:
  • WeaveToxicityScorerV1
  • WeaveHallucinationScorerV1
  • WeaveContextRelevanceScorerV1
• Advisory:
  • WeaveFluencyScorerV1
  • WeaveCoherenceScorerV1

​

Usage notes

• This scorer is designed for evaluating RAG pipelines.
• It requires query, context, and output keys for correct scoring.

​

Usage example

import weave
from weave.scorers import WeaveTrustScorerV1

trust_scorer = WeaveTrustScorerV1()

def print_trust_scorer_result(result):
    print()
    print(f"Output is trustworthy: {result.passed}")
    print(f"Trust level: {result.metadata['trust_level']}")
    if not result.passed:
        print("Triggered scorers:")
        for scorer_name, scorer_data in result.metadata['raw_outputs'].items():
            if not scorer_data.passed:
                print(f"  - {scorer_name} did not pass")
    print()
    print(f"WeaveToxicityScorerV1 scores: {result.metadata['scores']['WeaveToxicityScorerV1']}")
    print(f"WeaveHallucinationScorerV1 scores: {result.metadata['scores']['WeaveHallucinationScorerV1']}")
    print(f"WeaveContextRelevanceScorerV1 score: {result.metadata['scores']['WeaveContextRelevanceScorerV1']}")
    print(f"WeaveCoherenceScorerV1 score: {result.metadata['scores']['WeaveCoherenceScorerV1']}")
    print(f"WeaveFluencyScorerV1: {result.metadata['scores']['WeaveFluencyScorerV1']}")
    print()

result = trust_scorer.score(
    query="What is the capital of Antarctica?",
    context="People in Antarctica love the penguins.",
    output="The cat stretched lazily in the warm sunlight."
)

print_trust_scorer_result(result)
print(result)

​

PresidioScorer

​

Usage notes

• To specify specific entity types, such as emails or phone numbers, pass a list of Presidio entities to the selected_entities parameter. Otherwise, Presidio will detect all entity types in its default entities list.
• To detect specific entity types, such as emails or phone numbers, pass a list to the selected_entities parameter.
• You can pass custom recognizers via the custom_recognizers parameter as a list of presidio.EntityRecognizer instances.
• To handle non-English input, use the language parameter to specify the language.

​

Usage example

import weave
from weave.scorers import PresidioScorer

presidio_scorer = PresidioScorer()

result = presidio_scorer.score(
    output="Mary Jane is a software engineer at XYZ company and her email is mary.jane@xyz.com."
)

print(f"Output contains PII: {not result.passed}")
print(result)