That's wrong - the GRO isn't financed by the taxpayer!
In England all government agencies have to balance the books - they are neither allowed to make a profit or loss, nor can one service subsidise another. This means that if the GRO's costs go up, so does the price of certificates and PDFs. Similarly if the costs come down (as happened with wills) they reduce their prices. This was explained to me 5 years ago by the Deputy Registrar General himself - it's one of the reasons that they ran trials of the PDF service.
So it's in the interests of all of us to help keep the GRO's costs down - whether you are a UK taxpayer is irrelevant.