That statement is true.
If we use a same standard to measure the physical fitness between men and women, most of the women would definitely have lower score since Men naturally have more muscle mass compared to women.
Because of this, The median score from the fitness test taken from people in your similar age and gender is used as the standard measurement.
