Orcish Genocide and the Reaction Roll

No mechanic can prevent people who are committed to playing orcish genocide, but I do think that one of the reasons it has remained a constant problematic possibility within D&D is the abandonment of the reaction roll

The reaction roll is a useful tool that pushes many potentially violent encounters to at least start off nonviolently. Without it, experience shows that many referees, especially newer ones, will default to encounters that are automatically hostile. 

This automatic hostility then has to be rationalised, and the intellectual prop that is leaned on to explain it is "racial alignment", one of the stupider notions ever to occur in the game. "Racial alignment" as a concept, in turn, is shaped to serve this need and becomes ever more rigid and universal.

Eventually, you end up with nonsense like "all orcs are innately evil" with some shady reasons why, mostly either racist 19th-century biological nonsense or the same thing but with "magic" in place of the actual "race science". In-game, this translate to the orcs show up, automatically attack, and get killed by the PCs without remorse over and over again. 

Throwing out the reheated "race science" is a good start - you can simply have some orcish polities that encourage selfish, cruel and violent behaviour and focus in on these as the source of antagonists without needing every orc everywhere to sign off on this behaviour (even within the polity itself!). This opens up some interesting and fun strategic options beyond orcish genocide. 

But, this change won't make much difference without some mechanical supplement. Saying "Not all orcs are bad" but still having every orc who appears in-game automatically charge in to slay the PCs just means that the PCs will nod their heads at how enlightened they are while still committing orcish genocide. This still represents an imaginative failure, but one the PCs can't really be blamed for.

One mechanical supplement that I think can help people break out of this rut is consistent use of the 2d6 reaction roll, or a similar kind of check of attitudes at the start of the encounter adapted to whatever system. This system should be set up (and is, in most old school versions) so that a simple failure doesn't lead to automatic hostilities (that is, there should be at least one unfriendly-but-not-trying-to-kill-you state). 

One of the functions of rules is to define the incidence of various possibilities. A rule or mechanic where the rest is that the vast majority of the time the enemy will not immediately charge to attack is far more useful for shaping PC behaviour and opening up possibilities beyond mass murder than simply verbally rejecting the bioessentialist fluff is. 

In my old Necrocarcerus campaign, the PCs at one point encountered some Inhumanoids, which are basically vat-grown cannibal soldiers who are brainwashed into serving their evil creators. Necrocarcerus parodies regular D&D tropes, so Inhumanoids basically dial-up all of the bioessentialist / evil magic nonsense about orcs to 11. 

But, in the sessions where the PCs were dealing with them, I just consistently rolled for reaction rolls every time the PCs encountered a group of Inhumanoids. This resulted in far more positive encounters with the Inhumanoids (thanks to some good rolls) than I would have ever planned, and more importantly, the possibility of positive encounters incentivised the PCs to adopt a strategy that didn't require them to kill more than a handful of Inhumanoids at the very start. 

One of the PCs gave a performance to an indifferent group of Inhumanoids, who shifted to being friendly since they'd never heard music before. They kidnapped him, he gave the performance of a lifetime to distract the entire Inhumanoid guard force, and the rest of the PCs used the distraction to steal the nuclear reactor fuel they were there for.

All of this was emergent, rather than planned, of course, but I think that without the reaction roll system working its magic, this adventure would have turned into a fairly typical "orcs in a hole" murder march.

So in brief, while changing fluff to avoid regurgitating inane 19th-century nonsense is good, and worth doing, using mechanics like the reaction roll or similar mechanics that interrupt the automatic leap to hostility are actually just as important for getting to a kind of play that offers more options than just murder simulation.