Detection of Mid-parent Heterosis Genes in Large-Scale Unreplicated RNA-Seq Experiments

Mid-parent heterosis (MPH), characterized by hybrid trait values deviating from the mid-parent average, is a well-documented phenomenon whose genetic basis remains poorly understood. Identifying genes associated with MPH is crucial for uncovering the molecular mechanisms underlying heterosis. Recent large-scale RNA-sequencing (RNA-seq) experiments enable the evaluation of heterosis genes across numerous families; however, replication is often infeasible due to cost and labor constraints, resulting in unreplicated large-scale datasets and posing statistical challenges for dispersion estimation and reliable inference. To address this issue, we propose a novel two-stage likelihood ratio test (2sLRT) for detecting MPH genes in unreplicated RNA-seq experiments. In the first stage, genes and families without evidence of differential expression across varieties are identified, and the corresponding varieties are used as pseudo-replicates to estimate dispersion. In the second stage, a likelihood ratio test based on the negative binomial distribution is employed to test for MPH. Simulation studies demonstrate that 2sLRT achieves higher power and better false discovery rate control compared to existing approaches. Application of 2sLRT to a maize RNA-seq dataset with 599 families further highlights the effectiveness of the method in revealing meaningful patterns of MPH gene expression.