I suspect it's because our rights don't include the right to put others at risk.

Someone who has a great immune system can carry the virus, never get sick and never show symptoms and unknowingly infect people who do have a compromised immune system.

Masks are how we protect the people who are most vulnerable. Even if we're not worried about ourselves.