Sex-specific transcription characterizes hundreds of genes in mouse liver, many implicated in sex-differential drug and lipid metabolism and disease susceptibility. While the regulation of liver sex differences by growth hormone-activated STAT5 is well established, little is known about autosomal genetic factors regulating the sex-specific liver transcriptome. Here we show, using genotyping and expression data from a large population of Diversity Outbred mice, that genetic factors work in tandem with growth hormone to control the individual variability of hundreds of sex-biased genes, including many long non-coding RNA genes. Significant associations between single nucleotide polymorphisms and sex-specific gene expression were identified as expression quantitative trait loci (eQTLs), many of which showed strong sex-dependent associations. Remarkably, autosomal genetic modifiers of sex-specific genes were found to account for more than 200 instances of gain or loss of sex-specificity across eight Diversity Outbred mouse founder strains. Sex-biased STAT5 binding sites and open chromatin regions with strain-specific variants were significantly enriched at eQTL regions regulating correspondingly sex-specific genes, supporting the proposed functional regulatory nature of the eQTL regions identified. Binding of the male-biased, growth hormone-regulated repressor BCL6 was most highly enriched at trans-eQTL regions controlling female-specific genes. Co-regulated gene clusters defined by overlapping eQTLs included sets of highly correlated genes from different chromosomes, further supporting trans-eQTL action. These findings elucidate how an unexpectedly large number of autosomal factors work in tandem with growth hormone signaling pathways to regulate the individual variability associated with sex differences in liver metabolism and disease.