A lot of poor people do spend time reading to underprivileged kids — except they just call it parenting.
I think the part you’re missing is that people who have financial freedom have a freedom of mind that bleeds into their time off.
People who are two paychecks away from broke are usually juggling so much that if they have “time” they’re more likely to spend it figuring out how to feed everyone on their budget, or just spending time with their family in the hours they aren’t commuting or working.
13 million US workers have more than one job. It’s a real stretch to think they can spend their “leisure time” the same way the wealthy can.