You should stick with drums but buy an electric set so your neighbors/roommates wont have to hear you practice (you can use headphones). Those sets are getting affordable and they don't take up too much room (the electric kind). Just don't get flustered-- try playing along to songs on the radio/your computer/ipod and you'll get the hang of it.
I think your problem is that you're putting too much conscious effort into learning, which is like putting too much conscious effort into every word when speaking a foreign language (gets choppy, bad results).
Just have fun and groove along without caring or worrying too much about mistakes- that's how people who get good mostly do it (look at them, they're not all geniuses). Then, once they've attained decent basic proficiency and natural command, then they keep it up but supplement with a little more study (the studying doesn't bother them because they [are able to] still just have fun and improvise here and there for most of their practice, instead of being frustrated and then having to learn theory on top of that which they can't even apply-- compounded frustrating)
