A common argument against proportional representation is that it “gives too much power to small parties”. For the sake of argument, imagine an election result like this: a party wins 48% of the votes against its main opponent of 47%, but it cannot form a government on its own; it needs the support of the third party with 5%.
The intuition that this is a problem can also be supported mathematically. The Banzhaf power index tells us how much pure mathematical “power” a party has in terms of possible coalitions, based on how necessary its presence is in all possible (majority) coalitions. For example, if there is a 51% party, then its strength is 100%, because no one else can form a majority against it, and the only possible ‘coalition’ always requires it. If there are parties of 48%, 47%, 4% and 1%, then the strength of the 1% party is 0, since it is not needed for any coalition. However, the strength of the 4% party is relatively 33%, the same as for the two big parties. Why? Because the 3 parties are key players in 2 of the most possible coalitions. Thus, the 4-5% party in the above examples is ‘punching above its weight’.
Of course, this math does not take into account the political dimension, i.e., that not every party would be willing to enter into a coalition with every other party. In practice, it may seem that a small party is really in a coercive position, because only one coalition is possible on ideological grounds. But let's not forget that politics can limit how much a small party can realistically demand. Another interesting issue is political responsibility: do we take it for granted that it’s the responsibility of the small party to behave appropriately, not the other large one? In such a case, the two big parties could always join forces against the small party.
Returning to proportional representation: what should we think about this counterargument? It has a valid core, but if we think about it for longer, it is not that convincing. We could even say that the problem is not so much a problem of proportional systems, but of any system that is more inclined to produce a situation where a large party can be coerced by 1-2 small parties. However, the more proportional the representation, the more likely it is that there will be more than just 3-4 parties in parliament, so the number of possible coalition configurations also increases rapidly. At the same time, the mathematical power of each party begins to converge to its vote share, and in fact, it is generally more typical that the larger parties have more outsized power: they are the ones, without whom it is not really possible to form a coalition. If we take political realities into account, then possibly even more so.
Therefore, if we have an electoral system that tends to produce one or two large parties (but occasionally lets small parties break through and become kingmakers), the solution might not be to try to push out the small parties. Rather, to try to encourage the development of a healthier (possibly more natural) multi-party system, where a few larger, several medium-sized, and several small parties are represented (proportionally). This way, the large ones can choose between coalition partners, and the influence of each small party will be smaller. If we consider the issue of government formation and survival to be of paramount importance (in parliamentary and semi-presidential systems), then we can also introduce negative parliamentarism and a constructive motion of no confidence.
Another option, based on a similar principle, is to try to loosen party cohesion. If all representatives of each party always vote together, then it is easier to get into gridlock, while if there are representatives going their own way, then it is easier to build a majority along specific issues. (Of course, this also has its disadvantages…)
Overall, the concern seems legitimate, but the topic is complex. It is worth trying not to think about this in binary terms. Also, perhaps this problem becomes most apparent not in a pure PR system, but in semi-proportional and district-based winner-take-all ones, where there are likely two major parties, but one or two small parties also have a chance to enter, not more. In any case, this problem must be considered in the light of the alternatives. Maybe pondering the questions could be more fruitful:
How much of a problem is it if small parties can have power in a legislature that is not justified by their support?
Does it matter whether the pressure to compromise with the small party in question pulls the government coalition in a centrist or extremist direction?
Does it matter whether the large and small parties in the coalition agree on, say, less than 50% or more than 90% of the issues?
Does the second-largest party have any responsibility in forming a government, can a grand coalition be expected or is it to be avoided?
How should we view this problem when designing electoral systems?
That's interesting.