Heart rate variability (HRV) has been considered as a potential physiological marker for driver fatigue. However, consensus has not been reached for how HRV changes during the development of fatigue, due to inconsistent results in the literature. One potential cause for inconsistent results is that different causal factors were used to introduce fatigue. The aim of this study is to investigate how HRV parameters change during driving in relation to fatigue caused by sleep related and task related factors. Data from a real road experiment, with 89 participants who drove four times over a 180 km route, were used for the analysis. We investigated how time of day and time-on-task factors influence HRV parameters. The result shows that different HRV parameters react differently in relation to time of day and time-on-task factors. The result emphasizes the importance of considering the causal factors when interpreting results from driver fatigue studies and when developing fatigue detectors based on physiological measures.