The most common piece of advice people have to resolve out-of-character problems is "Have a mature and reasonable discussion with the person about the problems". This is not bad advice, so much as vacuous and too abstract. Structured, effective conversations that end in consensus are actually reasonably difficult to pull off, especially if there's more than two people involved. There's a lot of transferable material from conflict resolution, negotiation and business management that can help you deal with tricky situations.
Wikipedia on Conflict Resolution, on Conflict Resolution Research and on Active Listening
On Conflict and Consensus (pdf), a free ebook that lays out how Formal Consensus works.
Handling Disagreements, from Frank Hecker's experience working on Mozilla.
You may want to read Getting to Yes, which is as useful for resolving interpersonal disputes in games as it is for handling business negotiations.
This is a short piece on hiring people who claim to have good "interpersonal skills". While I don't agree with everything it says, I think it gives some good tips to keep in mind when you're recruiting people for games to avoid bringing on board toxic personalities in the first place.
People are frequently advised to stop gaming with troublesome individuals who resist all attempts to change, and while I don't disagree with this advice, my experience has been that people are extremely reluctant to go through the actual process of kicking someone out, and instead basically sit around hoping they're not going to show up this week. Kicking someone out of a game group is similar, though not identical, to firing employees. While not everything is identical, the core practices of the article are sound. The basic principles from it and from other articles about firing employees.
1) Do it with other people around to witness what goes on, but not in public.
2) It's not a negotiation.
3) Cite specific examples of problems. Hold them accountable for what they've done.
4) Keep the process succinct and on topic.
5) Don't gossip about the person who was kicked out.
6) Don't say you're sorry. Don't apologise.
7) Set aside a specific time and place to kick them out - don't do it right before the game starts.
8) Resolve any entanglements as quickly as possible (e.g. they have other people's books)