There must be a cultural component to being good at math. My 4-year-old can add and subtract with her fingers, and she has a bunch of small number addition memorized. She has it because I play learning / dice games with her, and praise her for knowing / remembering math facts. If a kid gets to school and doesn't have that background, it has got to be way harder to learn math and not be forever behind.
Same with reading. If you talk to and read to your kid a lot, they will have a big vocabulary. Then, when they learn to read, all they have to do is associate a word with the letters. If a kid doesn't even know the fucking word, they aren't going to have brain power for remembering the letters because they have to focus on the new word.
Having the leisure, money, education, and interest to educate your kids yourself makes a big difference on what they will get out of school.
Most of these kids don't have anyone fucking teaching them anything at home.