In plant and animal breeding, genomic prediction models are established to select new lines based on genomic data, without the need for laborious phenotyping. Prediction models can be trained on recent or historic phenotypic data and increasingly available genotypic data. This enables the adoption of genomic selection also in under-used legume crops such as common bean. Beans are an important staple food in the tropics and mainly grown by smallholders under limiting environmental conditions such as drought or low soil fertility. Therefore, genotype-by-environment interactions (G × E) are an important consideration when developing new bean varieties. However, G × E are often not considered in genomic prediction models nor are these models implemented in current bean breeding programs. Here we show the prediction abilities of four agronomic traits in common bean under various environmental stresses based on twelve field trials. The dataset includes 481 elite breeding lines characterized by 5,820 SNP markers. Prediction abilities over all twelve trials ranged between 0.6 and 0.8 for yield and days to maturity, respectively, predicting new lines into new seasons. In all four evaluated traits, the prediction abilities reached about 50-80% of the maximum accuracies given by phenotypic correlations and heritability. Predictions under drought and low phosphorus stress were up to 10 and 20% improved when G × E were included in the model, respectively. Our results demonstrate the potential of genomic selection to increase the genetic gain in common bean breeding. Prediction abilities improved when more phenotypic data was available and G × E could be accounted for. Furthermore, the developed models allowed us to predict genotypic performance under different environmental stresses. This will be a key factor in the development of common bean varieties adapted to future challenging conditions.