Thermoresponsive microgels undergo a volume phase transition from a swollen state under good solvent conditions to a collapsed state under poor solvent conditions. The most prominent examples of such responsive systems are based on poly-(N-isopropylacrylamide). When cross-linked with N,N'-methylenebisacrylamide, such microgels typically possess a fuzzy-spherelike morphology with a higher cross-linked core and a loosely cross-linked fuzzy shell. Despite the efforts devoted to understanding the internal structure of microgels and their kinetics during collapse/swelling, the origins of the accompanying changes in light scattering intensity have barely been addressed. In this work, we study core-shell microgels that contain small gold nanoparticle cores with microgel shells of different thicknesses and cross-linker densities. All microgels are small enough to fulfill the Rayleigh-Debye-Gans criterion at all stages of swelling. Due to the high X-ray contrast of the gold cores, we can use absolute intensity small-angle X-ray scattering to determine the number density in the dilute dispersions. This allows us to extract polymer volume fractions of the microgels at different stages of swelling from form factor analysis of small-angle neutron scattering data. We match our findings to results from temperature-dependent absorbance measurements. The increase in absorbance during the shrinkage of the microgels is related to the transition from fuzzy spheres to hard sphere-like scattering objects with a rather homogeneous density profile. We provide a first attempt to model experimental spectra using finite difference time domain simulations that take into account the structural changes during the volume phase transition. Our findings significantly contribute to the understanding of the optical properties of thermoresponsive microgels. Further, we provide polymer volume fractions and microgel refractive indices as a function of the swelling state.