The entanglement entropy (EE) of gauge theories in three spacetime dimensions is analyzed using manifestly gauge-invariant variables defined directly in the continuum. Specifically, we focus on the Maxwell, Maxwell-Chern-Simons (MCS), and non-Abelian Yang-Mills theories. Special attention is paid to the analysis of edge modes and their contribution to EE. The contact term is derived without invoking the replica method and its physical origin is traced to the phase space volume measure for the edge modes. The topological contribution to the EE for the MCS case is calculated. For all the Abelian cases, the EE presented in this paper agrees with known results in the literature. The EE for the non-Abelian theory is computed in a gauge-invariant Gaussian approximation, which incorporates the dynamically generated mass gap. A formulation of the contact term for the non-Abelian case is also presented.