The large sieve inequality is of fundamental importance in analytic number theory. Its theory started with Linnik's investigation of the least quadratic non-residue modulo primes on average. These days, there is a whole zoo of large sieve inequalities in all kind of contexts (for number fields, automorphic forms, etc.). L. Zhao started a systematic study of the large sieve with sparse sets of moduli, in particular, square moduli, in his thesis in 2003. A result in this direction was previously established by Dieter Wolke in 1971 for prime moduli, where a saving over the classical large sieve with the full set of moduli was obtained. In joint work of Zhao with the speaker, these results have been extended and improved using a combination of elementary arguments and Fourier analytic tools. Meanwhile, the large sieve with square moduli has found many applications, in particular, in questions regarding elliptic curves. In my talk, I will give a summary of this work and present new extensions to number and function fields which have recently been established in joint work with Arpit Bansal and Rajneesh Kumar Singh. It turns out that we get substantially better results in the function field case, which allows us to make progress on certain combinatorial problems considered by Erdös, Rivat, Sárközy and Schoen in the function field setting.