I'll use myself as an example. My grandparents came from Italy, struggled through the Depression, and, through hard work, scraped together enough money to buy a house. Although there was some prejudice towards Italians in those days, they made it into the white club pretty early on, and my grandparents were allowed access to a middle-class, white neighborhood. (This is important, because as my grandparents were buying their house, people of color were being systematically kept out of white neighborhoods through the government-approved practice of redlining.)
Enter, my parents. My father never graduated from high school, and worked at a printing company his whole life, earning very little. My mother stayed home with the kids (there were five of us) until we were all in school, and then worked as a caregiver for children and the elderly. Although they worked hard for their money, they would never have been able to afford our middle-class neighborhood if not for the fact that we lived upstairs from my grandparents in a two-family house, paying very little rent. (The same scenario holds true for my husband's parents.) When my grandparents died, my parents received a small inheritance that they used to pay off my father's gambling debt (interesting to wonder what might have happened to him, and to us, had that money not been there) and as a down payment on their own home, which they bought for $21,000 and sold twenty-five years later for $220,000. So, because my grandparents benefited from their whiteness long before I was born, I grew up in a safe neighborhood, received a good education at the good schools in that neighborhood, and had a financial safety net in my parents if needed.
So I've worked hard for my money, yes. But without my realizing it, my whiteness was working, too.