Since Rep. Peter DeFazio (D-OR) called for Timothy Geithner and Larry Summers to step down this Wednesday, the question has become a part of the political conversation. Republicans predictably took the opportunity to attack the Obama economic team, but progressive Democrats have started to get the question as well, a sign of things to come.

Sherrod Brown has had to answer it twice in recent days, telling ABC that Geithner had his confidence, while adding that there needed to be a greater focus on manufacturing and job creation; then telling CNN that some of the President’s advisers were too focused on Wall Street.

While ultimately Brown tried to remain positive and say that the Administration has turned the corner, the perception that the White House has renewed GDP growth and financial industry profits without giving the same attention to unemployment is a nagging problem that progressives will continue to need to address. As DeFazio told the Huffington Post this week, it’s “embarrassing” that the party traditionally attempting to rein in Wall Street has now become so closely identified with them:

“It is pretty embarrassing for a Democratic administration and a Democratic Congress to be identified with total attention to Wall Street and nothing for Main Street and jobs,” he said. “There are a lot of Democrats who… want to see something more effective done to create employment.”

DeFazio insisted that President Obama and, by extension, the Democratic Party were hampered by Geithner’s policies for economic recovery. He pointed to the inability of the administration to spur small business lending and the lack of effective TARP oversight as particularly egregious examples of mismanagement. More than anything else, the Oregon Democrat deemed it untenable for the president to continue employing his current economic team given the taint of Wall Street that clings to many of those advisers.

“I have had a number of people say to me, ‘I feel the same way you do but I’m not going to say it.’ People are worried it will rub off on the president who still enjoys popularity,” he said. “I tell them I still support the president. I just think he is being poorly served by his economic team.”

“The truth of the matter,” DeFazio added, “is that we have not changed the way the money is being used. It is not being used for the purpose it was supposed to be used for. We are not creating jobs and we have not aggressively taken on the culture of Wall Street.”

This growing undercurrent of dissatisfaction is starting to play itself out within the House of Representatives. It’s why you’re likely to see some kind of jobs bill before the end of the year. Darcy Burner, the Executive Director of the Progressive Caucus, says that this is a more serious challenge to the President’s ways of dealing with the economic crisis to this point, and that you’ll see more of a more widespread effort on this in the near future.

I think John Conyers’ release of frustrations with Obama this week was another signal, although the issues were not the same. Unemployment in the African-American and Hispanic communities is unsustainably high – 15.7% for the former, 13.2% for the latter. Ben Jealous of the NAACP said this week that it’s harder for a black man with no criminal record to get a job than a white man with a criminal record. Members representing these districts are in dangerous territory if they cannot get aid and support from a Reagan-like destruction of their communities, even with a biracial man in the White House. They’ll find someone to blame, and it’ll be their immediate member of Congress if there’s nobody removed at the top.

This is a dynamic that bears watching in the next several weeks, and it’s certainly on my radar screen.