Manny matured into a WW and had already fought Oscar at 147 and was fighting Cotto at 145. Mosley is the same size as both of them and they wouldn't fight him at anything above 142lbs, so size clearly wasn't the only factor. Ability and risk definitely featured in the decision and Roach said so himself. Weight wasn't an issue once Mosley was shot either, he was fine with 147 once that risk was taken away. The size thing is overstated, Roach preferred Pac against bigger guys, he speed and mobility terrorised them. Margarito had a huge size advantage that they couldn't have cared less about. Roach has never worried about match Pac up against bigger guys.
Floyd isn't a much bigger person either, they both way the same on fight night. Just because Floyd is taller and has longer arms doesn't mean he is much bigger. All body types have their advantages, go compare Pacquiao's legs to Floyds, they are much thicker. Floyd has advantages that come as a result of his length, and I'm sure Pacquiao's naturally thick legs are at the route of his punching power and resistance. Ultimately they both weigh around the same, with Floyd's walking around weight being slightly higher, but he isn't much bigger that in the frame of general welterweights, they are both small welters.