Answer
Aerodynamic drag is likely a major part of the penalty, because drag generally increases with speed while the payload mass stays constant.
Explanation
The way performance changes with speed can help separate aerodynamic losses from pure weight effects.