Gauge symmetries provide one of the most puzzling examples of the applicability of mathematics in modern physics. The presented work focuses on the role of analogical reasoning in the gauge argument, and is motivated by Mark Steiner's claim that the application of the gauge principle relies on a Pythagorean analogy whose success undermines naturalist philosophy. We present two different accounts of the analogy between gravity and electromagnetism, which first motivated the gauge principle, and discuss two ways in which it can be understood. Each account provides a different philosophical response to Steiner and the problem of the applicability of mathematics in the natural sciences. The first is based on an account of Weyl's original work and the idealism underwriting his appeal to mathematical analogy. The second examines possible naturalist alternatives, arguing that the most promising direction is a material understanding of the analogy based on Rovelli's relational account of gauge.