We investigate the structure of singlet states (i.e., states supported on the trivial-representation subspace of the angular momentum algebra) in an ensemble of N qudits. Such states can be created in experiments with cold atoms, and they play an important role in quantum metrology, especially in measuring gradients of magnetic fields. It follows form the Schur-Weyl duality theorem that in the case of qubit ensembles (with even number of particles) there exists a unique permutation invariant (PI) singlet state, however, for higher spin ensembles the structure of PI singlet states is much richer. Using recent plethysm results, we classify all PI singlet states. Moreover, by applying tools from asymptotic representation theory, all reduced density matrices are calculated in the large N limit, which allows us to extract the Quantum Fisher Information corresponding to gradient metrology.