When Does Relevance Become Manipulation?

Relevance becomes manipulation when a system uses what it knows about a person to steer their behavior while making that steering difficult to recognize, question, or refuse.

A relevant recommendation helps someone pursue a purpose they can identify as their own. A manipulative one takes advantage of a weakness, conceals a competing objective, or narrows the available choices so that one response becomes unusually likely.

The boundary is not whether a recommendation has an effect. Every useful recommendation does. The boundary is whether the person remains able to understand and direct the choice being influenced.

A Recommendation Can Respect the Person

Imagine looking for a reliable explanation of a local housing proposal. You read two reports about zoning and save a map of the affected neighborhood. A feed recommends a public meeting notice and a careful account of how the proposal might change rents.

The recommendation is relevant. It builds on an interest you demonstrated and gives you information you can use. You may agree, disagree, investigate further, or ignore it. The system has influenced what you encounter, but that alone does not make the influence manipulative.

Now imagine a different sequence. After you read the reports, the feed repeatedly presents frightening claims that the proposal will destroy the neighborhood. Each post invites immediate anger. The original documents and measured explanations become harder to find. You spend longer on the app, but come away less able to judge the proposal.

That pattern raises a serious question. Is the system helping you understand something you chose to investigate, or using your concern to keep you reacting?

We cannot infer a platform’s intention from one person’s feed. We can, however, examine the design and its effects.

Relevance Is Not a Neutral Label

A post may be relevant because it answers your question. It may also be relevant in the narrower sense that you are likely to click it. Those meanings overlap, but they are not identical.

Someone worried about the housing proposal may find a frightening rumor almost impossible to ignore. A system that predicts attention accurately could place that rumor near the top. Its prediction might be correct even if the post makes the person less informed.

This is why “the algorithm showed it because you were interested” is an incomplete explanation. Interested in what way? Curious enough to learn? Anxious enough to check? Angry enough to reply? The answer matters when the system is designed to act on the difference.

The Person’s Larger Intentions Matter

People have preferences about their preferences. You may enjoy watching a heated exchange while also wishing to spend less time in arguments. You may want to understand local politics without becoming suspicious of every neighbor. You may be tempted by a dramatic story but prefer accurate information when deciding how to vote.

Researchers examining preference manipulation in recommender systems have argued that respecting users requires attention to these higher-order wishes: not only what someone responds to now, but what kinds of preferences they want a system to support.

That idea does not give a platform a perfect way to read a person’s mind. It points to a missing question. If someone repeatedly watches confrontational posts but says, “I want a calmer, more informative feed,” which signal should have authority?

When a system treats the immediate reaction as the only real preference, it can override an intention the person has stated clearly.

Look for Concealment and Unequal Friction

Manipulation becomes easier when people cannot see why they are receiving a post, distinguish an advertisement from independent advice, or find a practical way to change the pattern.

The OECD’s work on dark commercial patterns examines digital designs that can subvert consumer decision-making through the way choices are presented. Its focus extends beyond social feeds, so it should not be read as a verdict on every recommendation. It offers a useful test: does the design help people make an independent, informed choice, or does it make that choice harder?

Consider two controls. “Show me less of this” is easy to find and works consistently. “Show me less” is hidden behind several menus while the next provocative post appears with a single swipe. Both systems technically offer a choice. They do not give that choice the same practical weight.

The effort required to refuse can matter as much as the option to refuse.

Persuasion Is Not Automatically Manipulation

A persuasive writer presents evidence and tries to change a reader’s mind. A friend recommends a candidate or a book. A public-health organization asks people to consider a risk. These acts influence others, but influence is part of ordinary social life.

The distinction is not simply whether a message changes someone. It concerns the method: whether claims and motives can be examined, whether important information is hidden, and whether the person can disagree without the surrounding system continually exploiting the very reaction that disagreement produces.

A feed complicates this distinction because it can personalize the order, repetition, and timing of messages for each individual. The visible argument may be ordinary persuasion. The invisible distribution strategy around it may deserve separate scrutiny.

A Practical Boundary

Ask four questions about a recommendation. Does it connect to a purpose the person can recognize? Can they understand why it appeared? Can they easily change or reject the pattern? Does the system respond to their considered feedback, including a wish for less of something they sometimes click?

No single answer settles every case. A feed cannot disclose every technical detail, and people will sometimes disagree with a recommendation without being manipulated. But repeated failures across these questions suggest that relevance is being defined mainly by what the system can induce rather than what the person wants to accomplish.

A relevant feed says, “This may help with something you care about.” A manipulative feed acts as though the most predictable response is permission to keep producing it.

The difference becomes especially consequential when the response being rewarded is not understanding or satisfaction, but measurable engagement. Why does that measure so often become the default definition of human interest?