Nonlinear optical spectroscopies are powerful tools for investigating both static material properties and light-induced dynamics. Terahertz (THz) emission spectroscopy has emerged in the past several decades as a versatile method for directly tracking the ultrafast evolution of physical properties, quasiparticle distributions, and order parameters within bulk materials and nanoscale interfaces. Ultrafast optically-induced THz radiation is often analyzed mechanistically in terms of relative contributions from nonlinear polarization, magnetization, and various transient free charge currents. While this offers material-specific insights, more fundamental symmetry considerations enable the generalization of measured nonlinear tensors to much broader classes of systems. We thus frame the present discussion in terms of underlying broken symmetries, which enable THz emission by defining a system directionality in space and/or time, as well as more detailed point group symmetries that determine the nonlinear response tensors. Within this framework, we survey a selection of recent studies that utilize THz emission spectroscopy to uncover basic properties and complex behaviors of emerging materials, including strongly correlated, magnetic, multiferroic, and topological systems. We then turn to low-dimensional systems to explore the role of designer nanoscale structuring and corresponding symmetries that enable or enhance THz emission. This serves as a promising route for probing nanoscale physics and ultrafast light-matter interactions, as well as facilitating advances in integrated THz systems. Furthermore, the interplay between intrinsic and extrinsic material symmetries, in addition to hybrid structuring, may stimulate the discovery of exotic properties and phenomena beyond existing material paradigms.