pairwiseLLM

1.1.0

Pairwise Comparison Tools for Large Language Model-Based Writing Evaluation

0packages depend

3.4Kdownloads / year

84.5%test coverage

13/13checks pass

Overview

About

Maintained by Sterett H. MercerFirst published 2025-12-221 releasesCRAN page ↗GitHub ↗

Provides a unified framework for generating, submitting, and analyzing pairwise comparisons of writing quality using large language models (LLMs). The package supports live and/or batch evaluation workflows across multiple providers ('OpenAI', 'Anthropic', 'Google Gemini', 'Together AI', and locally-hosted 'Ollama' models), includes bias-tested prompt templates and a flexible template registry, and offers tools for constructing forward and reversed comparison sets to analyze consistency and positional bias. Results can be modeled using Bradley–Terry (1952) doi:10.2307/2334029 or Elo rating methods to derive writing quality scores. For information on the method of pairwise comparisons, see Thurstone (1927) doi:10.1037/h0070288 and Heldsinger & Humphry (2010) doi:10.1007/BF03216919. For information on Elo ratings, see Clark et al. (2018) doi:10.1371/journal.pone.0190393.

Install

R (CRAN)pak (dev)Ubuntu (r2u)openSUSE (OBS)

install.packages("pairwiseLLM", repos = "https://cloud.r-project.org/")

Installs the current CRAN release. Prebuilt binaries on Windows and macOS.

if (!requireNamespace("pak", quietly = TRUE))
  install.packages("pak", repos = "https://cloud.r-project.org/")
pak::pak("shmercer/pairwiseLLM")

Development version from shmercer/pairwiseLLM. Installs pak first if it is not already available.

Release

# enable cranapt (r2u) if it is not already set up, then install
grep -rqs r2u.stat.illinois.edu /etc/apt/sources.list /etc/apt/sources.list.d/ \
  || curl -fsSL https://raw.githubusercontent.com/eddelbuettel/r2u/master/inst/scripts/add_cranapt_noble.sh | sudo bash
sudo apt install -y r-cran-pairwisellm

Fast Ubuntu binaries via r2u (no compilation). The cranapt setup runs once and is skipped when it is already enabled.

bspm (used by r2u) also needs make: sudo apt install make

# add the autoCRAN OBS repo (Tumbleweed) if it is missing, then install
zypper -q lr | grep -qi autoCRAN \
  || sudo zypper addrepo https://download.opensuse.org/repositories/devel:/languages:/R:/autoCRAN/openSUSE_Tumbleweed/devel:languages:R:autoCRAN.repo
sudo zypper install -y R-pairwiseLLM

RPMs auto-built from CRAN on openSUSE OBS (Tumbleweed).

Health

CRAN checks

13OK

Slowest check: 4.6 min · r-oldrel-macos-x86_64

Check history

OK2026-06-08
13 OK · 0 NOTE · 0 WARNING · 0 ERROR · 0 FAILURE
ERROR2026-06-07
12 OK · 0 NOTE · 0 WARNING · 1 ERROR · 0 FAILURE
OK2026-03-30
14 OK · 0 NOTE · 0 WARNING · 0 ERROR · 0 FAILURE
ERROR2026-03-28
13 OK · 0 NOTE · 0 WARNING · 1 ERROR · 0 FAILURE
OK2026-03-10
14 OK · 0 NOTE · 0 WARNING · 0 ERROR · 0 FAILURE

Documentation

Documentation signals are not tracked yet for this package.

Downloads

3.4K

CRAN downloads in the past year

Rank #8,340 · ~9/day · ~284/mo

pairwiseLLM Releases R versions

Daily download trend is not available in this view yet.

28730 days

1.3K90 days

3.4K1 year

Compare downloads with other packages →

Also on39 r2u37 autocran

Repository

GitHub ↗ shmercer/pairwisellm

3Stars

0Forks

0Open issues

0Open PRs

2Releases

324Commits

1Contributors

assessment-toolelo-ratinglarge-language-modelsollamapairwise-comparisonrr-packagerstats

324 commits · Last activity 2026-05-03 · 0% stars, 30d

Repository practices

Upstream repositoryBeta

8 development-tooling and community-health practices detected across 6 families in the upstream repository

Checks run against github.com/shmercer/pairwisellm on 2026-07-19.

Continuous integration (1)

GitHub Actions

Reproducibility and dev environment (1)

data-raw/

CRAN release process (2)

cran-comments.mdCRAN-SUBMISSION

Docs source (1)

README.Rmd

Show all practices

Research and citation (1)

codemeta.json

Lint, format, editor (2)

lintrRStudio project

How this is detected·Detection ruleset v1 (2026-07-18)

Dependencies

Declared dependencies

19 external dependencies (excludes base and recommended)

Depends (1)

R >= 4.1

Imports (10)

curl dplyr httr2 jsonlite rlangstatstibble tidyselecttoolsutils

LinkingTo (0)

none

Suggests (12)

BradleyTerry2 EloChoice knitr mockery purrr readr rmarkdown sirt stringr testthat tidyr withr

Enhances (0)

none

Dependency license breakdown →

Reverse dependencies

0direct

0indirect

Nothing depends on this yet.

Code & Tests

Datasets

People & History

People (1)

Maintainer (1)

Sterett H. Mercer

Author, Maintainer

Authors (1)

Sterett H. Mercer

Author, Maintainer

Package Timeline

1 release. R releases are shown for context.

R
R 4.6.0 released · 2026-04-24
1.1.0Latest
2026-03-10 · current release
R
R 4.5.0 released · 2025-04-11

Package metadata

First published: 2025-12-22
Total releases: 1 / 1 yrs
License: MIT + file LICENSE OSI
Download size: 263 KB
Installed size: not tracked yet
With dependencies: not tracked yet

Report a problem with this page →