Evidence has shown that both sleep loss and daily caffeine intake can induce changes in grey matter (GM). Caffeine is frequently used to combat sleepiness and impaired performance caused by insufficient sleep. It is unclear (1) whether daily use of caffeine could prevent or exacerbate the GM alterations induced by 5-day sleep restriction (i.e. chronic sleep restriction, CSR), and (2) whether the potential impact on GM plasticity depends on individual differences in the availability of adenosine receptors, which are involved in mediating effects of caffeine on sleep and waking function. Thirty-six healthy adults participated in this double-blind, randomized, controlled study (age = 28.9 ± 5.2 y/; FM = 1521; habitual level of caffeine intake < 450 mg; 29 homozygous C/C allele carriers of rs5751876 of ADORA2A, an A2A adenosine receptor gene variant). Each participant underwent a 9-day laboratory visit consisting of one adaptation day, 2 baseline days (BL), 5-day sleep restriction (5 h time-in-bed), and a recovery day (REC) after an 8-h sleep opportunity. Nineteen participants received 300 mg caffeine in coffee through the 5 days of CSR (CAFF group), while 17 matched participants received decaffeinated coffee (DECAF group). We examined GM changes on the 2nd BL Day, 5th CSR Day, and REC Day using magnetic resonance imaging and voxel-based morphometry. Moreover, we used positron emission tomography with [18F]-CPFPX to quantify the baseline availability of A1 adenosine receptors (A1R) and its relation to the GM plasticity. The results from the voxel-wise multimodal whole-brain analysis on the Jacobian-modulated T1-weighted images controlled for variances of cerebral blood flow indicated a significant interaction effect between caffeine and CSR in four brain regions (a) right temporal-occipital region, (b) right dorsomedial prefrontal cortex (DmPFC), (c) left dorsolateral prefrontal cortex (DLPFC), and (d) right thalamus. The post-hoc analyses on the signal intensity of these GM clusters indicated that, compared to BL, GM on the CSR day was increased in the DECAF group in all clusters  but decreased in the thalamus, DmPFC, and DLPFC in the CAFF group. Furthermore, lower baseline subcortical A1R availability predicted a larger GM reduction in the CAFF group after CSR of all brain regions except for the thalamus. In conclusion, our data suggest an adaptive GM upregulation after 5-day CSR, while concomitant use of caffeine instead leads to a GM reduction. The lack of consistent association with individual A1R availability may suggest that CSR and caffeine affect thalamic GM plasticity predominantly by a different mechanism. Future studies on the role of adenosine A2A receptors in CSR-induced GM plasticity are warranted.