In his wonderful and sadly lapsed Consequentialism FAQ, Scott Alexander offered the following parable on 'why morality must live in the world', an argument that he ultimately uses for a form of preference utilitarianism:

In the deep jungles of Clamzoria across the Freptane Sea is a tall and snow-capped mountain. Within this mountain is a cave which is the lair of the dreaded Hrogmorph, Slayer of Men. Encased within the chest of Hrogmorph is a massive ruby called the Heartstone, a ruby with legendary magic powers. The stories say that whoever wears the Heartstone is immune from the moral law, and may commit any actions he desires without them being even the mildest of venial sins.

Lured by the legend of the stone, you sail the Freptane Sea and trek through the Clamzorian jungle. You defeat the dreaded Hrogmorph, Slayer of Men, in single combat, take the Heartstone from his body, and place it around your neck as an amulet. Upon returning home, you decide to test its powers, so you adopt a kitten from the local shelter, then kill it.

You feel absolutely awful. You just want to curl up in a ball and never show your face again. “Well, what did you expect?" asks the ghost of Hrogmorph, who has decided to haunt you. “The power of the Heartstone isn't to prevent you from feeling guilty. Guilt comes from chemicals in the brain, chemicals that live in the world like everything else - not from the metaphysical essence of morality. Look, if it makes you feel better, you didn't actually do anything wrong, since you do have the amulet. You just feel like you did."

Then Animal Control Services knocks on your door. They've gotten an anonymous tip - probably that darned ghost of Hrogmorph again - that you've drowned a kitten. They bring you to court for animal cruelty. The judge admits, since you're wearing the Heartstone, that you technically didn't commit an immoral act - but you did break the law, so he's going to have to fine you and sentence you to a few months of community service.

While you're on your community service, you meet a young girl who is looking for her lost kitten. She describes the cat to you, and it sounds exactly like the one you adopted from the shelter. You tell her she should stop looking, because the cat was taken to the animal shelter and then you killed it. She starts crying, telling you that she loved that cat and it was the only bright spot in her otherwise sad life and now she doesn't know how she can go on. Despite still having the Heartstone on, you feel really bad for her and wish you could make her stop crying.

If morality is just some kind of metaphysical rule, the magic powers of the Heartstone should be sufficient to cancel that rule and make morality irrelevant. But the Heartstone, for all its legendary powers, is utterly worthless and in fact totally indistinguishable, by any possible or conceivable experiment, from a fake. Whatever metaphysical effects it produces have nothing to do with the sort of things that make us consider morality important.

But the parable as told above is only half-complete. What Alexander hasn’t told us is how the Heartstone came to have its wondrous power - you didn’t think something that powerful just worked by magic, did you? Let’s find out in the (unlicensed) prequel:

Underneath Clamzoria, several miles below the base of the mountain, live the malevolent Hantaka civilisation. Once, they terrorised Clamzoria, stealing, raping and pillaging from their neighbours, talking in the cinema, and loudly tutting every time someone obstructed them during rush hour.

Eventually the surrounding Glourshin and Kinewurthy kingdoms found a champion to unite them against the Hantakans. Hrogmorph, saviour of men, led his people to a gruelling victory that nearly wrought the ruin of the three kingdoms. At the foot of Mount Freptoil, with their backs to a sheer cliff, the Hantakans startled their foes by choosing surrender over a glorious conflagration.

Their terms were brief, and hard to refuse – that they be allowed to live on unmolested in the ancient Freptoilan lava tunnels, sealed away from the outside world; that a symbol of their existence should remain on the surface; and that they could select one of their own to remain on the surface to guard it eternally.

With little dissent, the allied kingdoms accepted their terms. The Hantakans duly chose the exile Hrogmorph, and bequeathed to his vigil the same Heartstone pendant he'd stolen from King Hanta all those years ago.

Just before triggering the avalanche that would seal them in the dark for all time, they revealed their final doom to the onlookers: from this day forth, all Hantakans, woman, man and child, would passionately desire any bearer of the Heartstone to commit every single heinous act they could imagine. All future generations would be raised to desire the same; Hantakan art and science would dedicate itself to thinking of further heinous acts, so that there would be no crime the Heartstone bearer could commit that its craftsmen hadn't en masse desired him to do so.

Thus, far beyond merely freeing the bearer from moral law, as the harrowed craven scribes would tell the story, the Heartstone would actively link his every harmful act to the satisfaction of countless preferences. To this day, the weight of their de re desires still taints the stone. The Hantakans, meanwhile, are content to thrive in the Earth's mantle, never learning of the Heartstone's fate, but nourishing their civilisation on lichen and things that scuttle in the dark, always preferring. Preferring...

And as their numbers have swelled, so the power of the Heartstone has grown. Should anyone ever claim it, and, say, kill a kitten, that simple act of malice will be the most preference-satisfyingly benevolent act the world has ever known...

So now we know. The Heartstone is even worse than we imagined, and powered by the actual preferences of living people. Does that change any of Alexander's conclusions, if we reread his story? The stone, separated from its 'power' by an insuperable epistemological-cum-physical barrier that prevents the Hantakans from ever learning whether their preferences have been satisfied, is as worthless as ever - and just as immune to experimental verification.