I’ve installed a lot of passenger side mirrors back when I was at dealership (as well as cruise installs, etc.). I will say that those knock-out holes are not all the same. Most are easy to knock out. Some . . . not so much, and would be easier to get a hole-saw and make your own hole. So, maybe 1 troublesome one for every 9 easy ones.
I know the dies used to stamp steel with, wears out and need to be replaced every so often, I suspect when dies are close to wore-out, are the troublesome ones appear (me shrugs).
That said, I found it faster to jack up R/F corner of car and place on jack stand. Remove R/F tire and the plastic inner fender. I had a 2-foot-long piece of 1/2 inch thick solid steel round bar stock that I used to knock the door plugs out with. That tool was overkill (but worked great) and could be done with a long punch, long screwdriver or something similar.
The plug on body side, I generally had more fits with – but it is also more accessible. The important part is to take round file and smooth out any sharp edges, as well as block of wood and hammer to make sure the “hole” is (mostly) flat, before mirror install.
Same thing with power window wiring holes – but those holes are a lot larger. I only installed power windows on four cars. Maybe closer to 100+ on Right mirror installs.
The location of mirror on door is not critical – so if mirror is ¼” up/down or ½” front/rear – NO ONE will notice (except for you). The plastic mirror gasket (if you want to call it that item) works for a wonderful guide as to where to drill holes at.
I have drilled a ton of holes in doors - but found later what works great is to mark center of holes, drill a pilot hole slightly smaller than hole-saw center bit or drill hole to use a hole punch set. Later, I found the hole punches are faster, make a lot cleaner hole and works wonderful on non-flat surfaces (like concave or convex door skins). The hole punches also work great for those door edge holes for the cable (on the ones that refuse to easily pop out).
The trick on hole punches is the more expensive ones give very little problems. The cheaper ones give, well, more problems. It comes down to you get what you pay for (but . . . if you might only need to use it once?).
BudW