Use the vitals package with ellmer to evaluate and compare the accuracy of LLMs, including writing evals to test local models.

Large language models (LLMs) like ChatGPT show reasoning errors across many domains. Identifying vulnerabilities is good for public safety, industry, and the scientists making these models. The human ...