The interest in the history of gauge invariance or gauge symmetry has been predominantly focused on two disconnected moments: the early development of gauge invariance in the 1920s (H. Weyl, F. London) and the generalization of gauge invariance to non-abelian groups after 1954 (Yang-Mills). However, it is difficult to explain the initial excitement generated by Yang-Mills gauge theories (which from the beginning were plagued by the non-trivial zero-mass problem) without a closer investigation of the role played by gauge invariance in the formulations of quantum electrodynamics as a quantum field theory. This presentation will explore the role of gauge invariance in the Pauli-Heisenberg canonical formalism for the quantization of fields, and in the Schwinger covariant formalism for the evaluation of vacuum polarization.