Everyone says that, it's because they don't know properly how to use the speed-bag. Here's a good idea of what it SHOULD look like to work hands, shoulder endurance, rythm, breathing, timing, interchanging of punches, foot-movement, etc.:
http://www.youtube.com/watch?v=f5WyDGUIoQA
This issue comes up a lot. And out of many people even in the gym, most people look like this when they hit the speed-bag:
http://www.youtube.com/watch?v=d6Fz2r9zjh4
The difference should be obvious. One way will make you faster and more fluid, the other will not, and only work your shoulders, and hand-eye coordination.