Factuality hits json parse errors caused by exceeding token limit

[CodingCanuck]

I'm experimenting with the Factuality scorer, and with some inputs I see json parsing failures caused by unterminated strings, e.g.:

FactualityAccuracyScorer: |-
    SyntaxError: Unterminated string in JSON at position 2196
        at JSON.parse (<anonymous>)
        at parseResponse 

This failure seems deterministic in my limited testing. But when I increase the hardcoded max token limit of 512 here, the failure goes away, suggesting the JSON parsing error is caused by a truncated json string when we hit a max token limit.

Could we make max token limit something that callers can override, the same way they can override the model used for inference here?

Alternatively, I see this pending PR from march to simply remove the max token count, authored by Ankur Goyal (@ankrgyl) and approved by Olmo Maldonado (@ibolmo) : #120 would that be a preferred approach? (I'm not sure why that PR was approved and not merged: did it turn out to cause more problems than it fixed?)

Let me know if you'd like me to send a PR to add a maxTokens argument to LLMClassifierFromTemplate and modelGradedSpecSchema to allow callers to override this max token setting: I can happily send you a change.

[ankrgyl]

#120 is the right approach but indeed caused a bunch of test failures on our end that we did not have bandwidth to work through at the time.

[CodingCanuck]

#120 is the right approach but indeed caused a bunch of test failures on our end that we did not have bandwidth to work through at the time.

For my interest, do you remember if the test failures were those listed in the report on the PR (so mostly duration regressions), or in another test suite that isn't visible there?