Why Star Ratings Lie: How We Read Recipe Comments Instead

The first time we trusted star ratings to rank recipes, the top result for "easy weeknight pasta" was a recipe with 4.9 stars and 1,400 comments. Half the comments were variations of "this was way harder than the recipe suggested" and "sauce broke, had to start over". The 4.9 was a marketing artifact, not a quality signal.

We threw out star-weighted ranking the next day. Here's what replaced it and why.

Star ratings have a mean problem

Across the major recipe sites, the median rating on any recipe with more than 50 reviews is between 4.6 and 4.9. That's not because the recipes are good. It's because:

• People who didn't like the recipe usually don't rate at all.
• People who succeeded feel obligated to leave 5 stars.
• The site nudges raters toward 5 with copy like "loved it?"
• Recipe authors moderate their own comments and quietly remove low-rated reviews.

The result is a ratings distribution that looks more like a vote of confidence in the chef than a measurement of recipe quality.

Comments are messy. Comments are honest.

We pull every comment on a recipe (when public) and run it through a sentiment pipeline tuned for cooking-specific language. We're not asking "did they like it" — we're asking five much harder questions:

1. Did the recipe work as written?Comments like "had to add 30 more minutes" or "sauce never thickened" are signal that the timing or technique is off, regardless of the rating.
2. Did people have to substitute?"Used X instead of Y" tells us the original ingredient is unavailable or impractical, which dings the recipe's ingredient-availability score.
3. Did people add steps the recipe didn't mention?"Make sure to deglaze first" means the author skipped a step the dish actually requires.
4. Did experienced cooks succeed and beginners fail?A cluster of "easy for me but my friend struggled" is a technique-difficulty signal we weight heavily.
5. Did anyone say the recipe lied?"Says easy but it wasn't" is the canary. We surface those terms with a separate weight.

The numbers

We built two ranking versions side-by-side: one weighted on stars, one weighted on comment sentiment. We had a small panel of cooks rate the top 20 results from each on five queries ("easy weeknight pasta," "quick chicken," "30-min salmon," "sheet pan dinner," "weeknight stir-fry") and asked the same question: how confident are you that this recipe will work the first time?

• Star-weighted ranking: 51% of top results rated "confident" or higher.
• Sentiment-weighted ranking: 78% rated "confident" or higher.

Sample size was small (n=12) but the gap was consistent across all five queries. We shipped the sentiment version that week and never looked back.

What this looks like in the ForkScore

Every recipe gets a 1–100 ForkScore. About 35% of that score comes from comment sentiment — specifically:

• Positive sentiment density (positive language per comment): +15
• "Worked as written" markers: +10
• "Lied about difficulty" markers: −15
• Substitution density (high = ingredients hard to source): −5

Star ratings still contribute a small amount (around 8% of the score) because at the extremes — a recipe with a 3.2 average is unusual enough that the rating is signal again — but the dominant input is what people actually wrote.

Why this matters past recipes

Anywhere humans rate a thing, the rating regresses to 4.something. Restaurant reviews, product reviews, app store ratings — the same dynamics flatten the signal. The comments contain the truth. Most products just don't bother reading them.

We do, because we're a recipe app for people who want to cook, not for people who want to feel like they cook. The ranking has to work.

The takeaway

If you're building a discovery product, the rating system is probably the worst signal you'll ever consume. Read the comments. Build a parser. Trust the words more than the numbers.