It is well known that mineral fillers have an important influence on the performance of asphalt mixes. For example, usually micaceous fillers are disregarded in manufacturing asphalt mixes. Substituting the fine fraction in an asphalt mixture with a fine fraction of a noticeably different material with respect to physical-mineralogical properties will change the asphalt mixture properties and its performance in the field, especially under severe winter condition. The effect of the surface area of the fine materials is already known, however, the present paper studies mineralogically related factors of the filler fraction that influences the asphalt concrete resistance against water impact and freeze-thaw cycles. Five fillers derived from common crushed rock aggregates for construction purposes were analysed. The samples have been mineralogically characterised, the specific surface area (SSA) has been determined together with geometrical parameters and surface texture to differentiate between samples. The analysed data confirms a correlation between the SSA and mineralogical composition of fine material and the deterioration of the stiffness modulus of asphalt concrete under severe winter condition. This study indicates the vitality of the mineralogical composition of the fine material for the performance of bituminous layers. This work should give rise to better quality assurance and economise on natural resources in the production of asphalt mixes.