That's the point: privilege is the flipside of discrimination. If people of color face discrimination, in housing, employment and elsewhere, then the rest of us are receiving a de facto subsidy, a privilege, an advantage in those realms of daily life. There can be no down without an up, in other words.
None of this means that white folks don't face challenges. Of course we do, and some of them (based on class, gender, sexual orientation, disability status, or other factors) are systemic and institutionalized. But on balance, we can take for granted that we will receive a leg-up on those persons of color with whom we share a nation.
And no, affirmative action doesn't change any of this.
Despite white fears to the contrary, even with affirmative action in place (which, contrary to popular belief does not allow quotas or formal set-asides except in those rare cases where blatant discrimination has been proven) whites hold about ninety percent of all the management level jobs in this country, receive about ninety-four percent of government contract dollars, and hold ninety percent of tenured faculty positions on college campuses. And in spite of affirmative action programs, whites are more likely than members of any other racial group to be admitted to their college of first choice.* And according to a study released last year, for every student of color who received even the slightest consideration from an affirmative action program in college, there are two whites who failed to meet normal qualification requirements at the same school, but who got in anyway because of parental influence, alumni status or because other favors were done.
Furthermore, although white students often think that so-called minority scholarships are a substantial drain on financial aid resources that would otherwise be available to them, nothing could be further from the truth. According to a national study by the General Accounting Office, less than four percent of scholarship money in the U.S. is represented by awards that consider race as a factor at all, while only 0.25 percent (that's one quarter of one percent for the math challenged) of all undergrad scholarship dollars come from awards that are restricted to persons of color alone. What's more, the idea that large numbers of students of color receive the benefits of race-based scholarships is lunacy of the highest order. In truth, only 3.5 percent of college students of color receive any scholarship even partly based on race, suggesting that such programs remain a pathetically small piece of the financial aid picture in this country, irrespective of what a gaggle of reactionary white folks might believe.**
In other words, despite the notion that somehow we have attained an equal opportunity, or color-blind society, the fact is, we are far from an equitable nation. People of color continue to face obstacles based solely on color, and whites continue to reap benefits from the same. None of this makes whites bad people, and none of it means we should feel guilty or beat ourselves up. But it does mean we need to figure out how we're going to be accountable for our unearned advantages. One way is by fighting for a society in which those privileges will no longer exist, and in which we will be able to stand on our own two feet, without the artificial crutch of racial advantage to prop us up. We need to commit to fighting for racial equity and challenging injustice at every turn, not only because it harms others, but because it diminishes us as well (even as it pays dividends), and because it squanders the promise of fairness and equity to which we claim to adhere as Americans.
It's about responsibility, not guilt.