I use a d6-based group initiative roll resolved using some simple principles from Courtney Campbell. The basic principles are that one wants to roll high, with odd ties going to enemies, and even ties going to PCs (So a roll of 6 always favours the PCs), and initiative is rerolled each round. I also use a phase system within a round, so a high roll lets your side act first within each phase, though both sides act within each phase before moving on to the next phase.
As of Necrocarcerus version 1.2, I will also be using a skill system that includes the Tactics skill. Increasing the tactics skill allows one to roll a larger die type for one's initiative roll (d6 to d8 to d10 to a d12 maximum).
I've been debating a slight variation in the initiative rules. I haven't playtested this before, but I am proposing it to solicit feedback. The rule is that the roll on your die not only determines whether you go first or last, but how many actions your side can take during their turn. So a roll of 1 means no matter the number of PCs, your side must choose only one PC to act. A 6 means up to six actions may be taken, split amongst the PCs as they please (or amongst the monsters as you please). If there are more actions than PCs, the PCs may choose who acts a second time. The actions would not be split per phase, but could be distributed across each phase as each side pleased. Alternately, one might drop the phase system entirely and use simply this action distribution to push the PCs to make tactical choices.
Here are some outcomes I can foresee from adopting this system:
Single enemies become much tougher, especially single "boss" type monsters, since they will almost always be acting multiple times in a round.
Mobs become somewhat less overwhelming, though also easier to run, since only a portion of the mob will be acting in any given round.
This makes the Tactics skill much more valuable. Rolling a d12 to determine how many times you act per round is a huge advantage over opponents only rolling d6s (Monsters can have the Tactics skill).
Combats may take longer to resolve, since the median die result of a d6 (3.5) is less than the usual # of characters on each side who could act if they each got one action (I'm usually running games with 4+ PCs). On the other hand, the fewer number of actions per round might make rounds progress more quickly.
It would be important to track and differentiate abilities that can be used "once per round" from abilities that could be used "any number of times per round".
PCs would spend time each round resolving who gets to act in a given round. This might favour preferring certain characters acting, especially if they're somewhat mechanically superior to others, doubly rewarding players with powerful characters. As a positive side of this, it imposes scarcity limitations on PCs to help shape their decisions within a round.
If phases are retained, then PCs must make a separate decision each phase about how many actions they want to pursue in that phase. I would allow unused actions to cascade through the phases instead of requiring PCs to determine at the start of the round, after the initiative roll, which phase they wanted to dedicate the actions to. I think the order of phases will assist in this (movement is the second last phase, coming only before resolving spells), but I can foresee that there will probably be some weird edge case somewhere down the line where the PCs are like "We dump the remaining actions".
It could be difficult to plan between rounds, since one doesn't know how many actions will be coming up.
One side might be entirely ganked simply through a series of singular bad rolls each round - repeatedly rolling 1s while the other side rolls higher.
I'm not sure what my ruling would be if the PCs split into two independent groups fighting a single monster, or group of monsters. My inclination is to require them to still make a single initiative roll, but I can see a legitimate argument that the two groups, because they are acting independently, should make separate rolls (but then, of course, the lack of a boundary condition here allows one to drill the scope of the roll down to a single PC, which I would prefer not to do).
If anyone has tried this previously, I'd be interested in hearing your experiences.