What is the Plottery benchmark?

A benchmark for local LLM roleplay. Instead of ranking models on general impressions, it tests the small, single-purpose prompts a story engine runs under the hood and scores how each local model handles each one.

Which model is best for local AI roleplay?

It depends on the task. The benchmark scores each local model on each story task and each world setting, so you can compare models on the specific jobs a roleplay engine actually runs rather than on a single overall number.

How does the benchmark grade a model?

In two phases. First each model under test answers every case; then a separate judge pass grades those answers as pass rates. Because the Plottery app itself runs the prompts, a score reflects the exact engine you play rather than an approximation of it.

How good is your local model at telling a story?

Plottery runs a stack of small AI prompts under the hood to keep your story consistent. The Plottery benchmark measures how well each local model handles them.

A story engine is more than one prompt

When the narrator writes your next story beat, that isn't a single request to a model. Behind it, Plottery runs a set of small, single-purpose prompts: one decides when a scene has ended, one keeps a character's facts consistent, one tracks the in-story time, one folds old turns into a summary, etc. Each of these is a task.

The benchmark scores each task on its own, instead of judging a model's storytelling from a short chat or a general impression.

Two ways to read the grid

The benchmark is a grid: tasks down the side, models across the top. Each run adds results to the cells, and a cell shows the average across its runs. The same numbers answer two different questions depending on which way you read them.

failing basic 12/12

Model A

example placeholder

82%N=240

Model B

example placeholder

72%N=240

Model C

example placeholder

58%N=252

▸Ending a scene(2)

100%N=40

90%N=40

65%N=40

▸Tracking time(3)

95%N=40

80%N=40

60%N=80

▸Generating arcs(2)

90%N=20

20%N=20

10%N=40

Each column header sums up one model. The large percentage is its mean pass rate over judged runs, and N is how many runs were graded (test cases times iterations). Each task row repeats the same summary for that task alone, with a bar filled to its pass rate.

Read a column

How one model does across every task. This is the model comparison: which local model to load for the best stories.

Read a row

How one prompt holds up across models. A weak row means the prompt itself needs work, not the model.

An example run

The image below is a heatmap. The app can export the same run a few other ways too: a summary, a detailed table, grouped bars, a radar chart, or a scatter plot.

Plottery benchmark results matrix: story tasks scored as pass rates across eight local models in a classic-fantasy world, graded by an LLM judge.

The app is the benchmark engine

The benchmark prompts are the same prompts the game uses. There's no separate test setup to fall out of sync. If a prompt changes in the game, the benchmark uses the new one too.

A run has two passes: a generation pass where every model answers every case, then a judge pass where a single judge model grades those answers. Using one judge for all of them keeps the scores comparable from one model to the next.

“When a measure becomes a target, it ceases to be a good measure”

Goodhart's law is the trap every LLM benchmark runs into. If the grader never changes, you can tune a prompt until it scores well on that one judge without actually getting better. So nothing here stays fixed: the prompts keep getting better at scoring, the judge keeps getting better at catching mistakes, and a good score gets harder to fake over time.

Built into the app

The benchmark isn't a separate download. It's part of Plottery, so its scores come straight from the game's own engine.

Read the guide Download Plottery