Hostile creatures that are asleep or fail a mental resistance check against Spellpower are randomly divided into 5 groups. Members within each group are friendly to each other. All 5 groups are hostile to each other, hostile to the party and hostile to enemies not affected by the spell.
The distribution of affected enemies into groups works like this: A random number between 1 and 5 is generated for each affected enemy. This number determines the group the enemy is put in. Thus, groups can be empty and in rare cases all affected enemies may be in the same group.
Using AoE abilities / spells that draw threat from affected enemies makes the spell ineffective (affected enemies will stop attacking each other and target the AoE ability user).
On Hard / Nightmare difficulties, Miasma draws 20 extra threat from all enemies in the AoE, which will often be sufficient to make the caster the highest-threat target (and keep enemies affected by Waking Nightmare from attacking each other).
Running away from affected enemies who attack party members can cause them to choose a different target, since threat decreases with more distance.
Using this spell effectively is very challenging, since even a single enemy chasing after the party can trigger a chain effect of most enemies attacking or chasing after the party. Using Sleep first can be extremely helpful in setting up hostile groups that are closer to each other than the party and will thus attack each other due to high distance-based Threat. This will often result in a very tight cluster of enemies that is an excellent setup for Walking Bomb and Virulent Walking Bomb. The bomb must be immobilized to prevent the enemy cluster from going after the party.