Although genome-wide association studies are widely used in mining genes for quantitative traits, effects to be estimated are confounded and methodologies of detecting interactions are imperfect. To address these issues, first, the mixed model proposed here estimates the genotypic effects for AA, Aa, and aa, while genotypic polygenic background replaces additive and dominance polygenic backgrounds. Then, the estimated genotypic effects are partitioned into additive and dominance effects using one-way analysis-of-variance model. This strategy was further expanded to cover QTN-by-environment interaction (QEI) and QTN-by-QTN interaction (QQI) using the same mixed model framework. Thus, a three variance components mixed model was integrated with our mrMLM method to establish a new methodological framework, 3VmrMLM, that detects all types of loci and estimates their effects. In Monte Carlo studies, 3VmrMLM correctly detected all types of loci and almost unbiasedly estimated their effects, with high powers and accuracies and low false positive rate. In the re-analyses of ten traits in 1439 rice hybrids, 269 known genes, 45 known gene-by-environment interactions and 20 known gene-by-gene interactions strongly validated 3VmrMLM. Further analyses of known genes showed more small (67.49%), minor allele frequency (35.52%), and pleiotropic (30.54%) genes, with higher repeatability across datasets (54.36%), and more dominance loci. Heteroscedasticity mixed model in multiple environments and dimension reduction methods in quite a number of environments were developed to detect QEI, while variable selection under polygenic background was proposed to detect QQI. In addition, the compressed mixed model was incorporated into our GCIM to propose GCIM-QEI-random and GCIM-QEI-fixed, respectively, under random and fixed models of genetic effects in immortalized F2 and F2:3 design. GCIM-QEI was validated by a series of simulation studies and real data analyses owing to higher powers in small and linked QTL and QEI detection compared to the ICIM method. GCIM-QEI-random was slightly better than GCIM-QEI-fixed. The softwares are available at https://github.com/YuanmingZhang65/IIIVmrMLM and R website. This study provides a new approach to reveal the genetic architecture of quantitative traits in association and linkage analyses.

genome-wide association study;QTN-by-environment interaction;QTN-by-QTN interaction;3VmrMLM;rice