Objectives: Short sleep is becoming more common in modern society. This study aimed to explore the relationship between accelerometer-measured sleep duration and cognitive performance among young adults as well as the underlying hemodynamic mechanisms. Methods: A total of 58 participants were included in this study. Participants were asked to wear an ActiGraph GT3X+ accelerometer to identify their sleep duration for 7 consecutive days. Cognitive function was assessed by the Stroop test. Two conditions, including the congruent and incongruent Stroop, were set. In addition, stratified analyses were used to examine sensitivity. 24-channel functional near-infrared spectroscopy (fNIRS) equipment was applied to measure hemodynamic changes of the prefrontal cortex (PFC) during cognitive tasks. Results: Results showed that sleep duration was positively associated with accuracy of the incongruent Stroop test (0.001 (0.000, 0.002), p = 0.042). Compared with the regular sleep (≥7 h) group, lower accuracy of the incongruent Stroop test (−0.012 (−0.023, −0.002), p = 0.024) was observed in the severe short sleep (textless6 h). Moreover, a stratified analysis was conducted to examining gender, age, BMI, birthplace, and education’s impact on sleep duration and the incongruent Stroop test accuracy, confirming a consistent correlation across all demographics. In the severe short sleep group, the activation of left middle frontal gyri and right dorsolateral superior frontal gyri were negatively associated with the cognitive performance. Conclusions: This study emphasized the importance of maintaining enough sleep schedules in young college students from a fNIRS perspective. The findings of this study could potentially be used to guide sleep time in young adults and help them make sleep schemes.