Yeah, you're not as small as you think. There should be plenty of guys your weight--if not your height ...
I'd second what OuchThatHurt said about technique. That's what will eventually make the difference (a big difference, I might add) if you stick with it and steadily improve.
You'll spend some time trying to go strength-on-strength with bigger guys--it's a natural impulse and it will work some of the time (it did for me). But try to resist it and focus on escaping from their "pins" (i.e., side control, mount, rear mount) using the techniques your instructor shows you.
Just remember that there are more experienced grapplers your size who can completely school the bigger guys you'll have trouble with. The difference is all in the technique. And the technique can be learned.