Why Does HPC Need InfiniBand?

When people first learn about HPC clusters, they often notice something unusual. The compute nodes don't just have Ethernet. Many HPC clusters also have InfiniBand. So a natural question is: Why? If Ethernet already connects computers together, why do HPC clusters need another network?