The establishment of social hierarchies is a naturally occurring, evolutionarily conserved phenomenon with a well-established impact on fitness and health. Investigations of complex social group dynamics may offer novel opportunities for translational studies of autism spectrum disorder. Here we describe a robust behavioral paradigm using an automated version of the tube test. Isogenic groups of male and female mice establish linear social hierarchies that remain highly stable for at least 14 days, the longest interval tested. Remarkably, however, their social strategy is sex-specific: females primarily utilize intrinsic attributes, whereas males are strongly influenced by prior social experience. Using both genetic and pharmacological manipulations, we identify testosterone as a critical sex-specific factor for determining which social strategy is used. Males inheriting a null mutation of the sex-determining region Y (Sry) gene used a similar social cognitive strategy as females. In contrast, females with transgenic expression of Sry utilized a typically male social strategy. Analogously, castration of males and testosterone supplementation of females yielded similar outcomes, with a reversal of their social cognitive strategy. Together, our results demonstrate a sex-specific mechanism underlying social hierarchy, in which both males and females retain the functional capacity to adapt their social strategy. More generally, we expect the automated tube test to provide an important complementary approach for both fundamental and translational studies of social behavior.