It took me ages to learn that privilege is very hard to spot in part because it makes things *easier*, not easy. Being poor instead of dirt poor, having to hide your ethnicity, queerness or disability rather than being unable to avoid any flak you might take for it, waiting for an elevator being an annoyance rather than a necessity.
This stacks badly with a human tendency towards dualisms, and baggage from seeing suffering as virtuous (things which were bad but not miserable get discounted).