Surely there are strong unions in the private sector - what were they doing?
I cannot fully answer, but I can speculate.
Pensions are expensive. The shift from Defined Benefit schemes to Defined Contribution ones was driven because companies noticed that the risk of those pensions vested with them, not the employee. As people lived longer that risk (from the employer's point of view) was too much, the schemes were too expensive. This was for a variety of reasons, including shocking investment decisions, insufficient contributions from members/employers, too many pensions “holidays” and aggressive actuarial valuations etc. The change was sold to the workforce in a positive way (see negative below) by saying “it’s more flexible for you, and LOOK, LOOK at the amazing stock market growth! If that continues, you can retire earlier on a bigger income!” That way it was presented as a win-win. Cheaper for the company, less risk for the company, fliexible and better for the individual. Of course, none of that worked out. Companies won, employees lost. The first fly in the ointment was the over estimation of stockmarket growth, and the underestimation of how much pension firms would charge for the privilege of taking your money off you. Cue many mis-selling scandals.
That’s the first part – Unions would be no help there, if they were “duped” (and I don’t think there was duping, I just believe everyone got carried away).
The second part is all about having an employer, or not, there are “market forces” at play. In the private sector if you are more expensive than your competitor for the same product people are quite likely to go to your competitor. If you don’t sell stuff, you are in trouble. Being bust is good for nobody, all the employees are now without a wage or indeed a pension, so companies cut costs. A massive cost is the pension (and if you wish this would be a good point to insert a rant about boardroom pay compared to shop floor pay). So you can present the argument as “you could have a nice pension but no job to fund it, so no pension, or a job and a poor pension. Your move”. Then there is a threat of “strike, and we have nothing to sell. Our customers will go elsewhere, and we will not get them back. You hurt me (the employer) but you hurt yourself too.” More recently it would be “you know, I’ll just outsource your job to India. Save some money.”
So – unions were hamstrung both ways.
There is no such “market” in the public sector, or at least much of it. This is not a good place to argue the merits or otherwise of going that way. To illustrate, I don't have much choice about a fire service or a refuse collection service. So there is no particular incentive to “do it cheaper” because another person would do it instead, and take your business away, leaving you out of a job. Instead costs and revenues are all “regulated” by specific Governmental edicts, such as capping council tax, or overall pay scales etc.
For years and years pensions has been a brewing issue (a quick look at how your council tax is spent is quite illuminating) but at the same time it’s a political nightmare to deal with, and many Governments of different colours have chosen to place it in the “too difficult” pile. As a result the public sector pensions have not been messed around with as much as private sector ones. Until now…
I agree completely with your assertion that we should not be in a race to the bottom, to find the cheapest for all. We should aspire to having the best provision for all. I, for one, cannot see how we fund that however.