Part of this is due to the old 18th/19th century "one drop rule", which basically says that if a white person and a black person have a child, that child is seen as black. If that child and a white person then have a child, their child is also seen as black, and so on and so on. This is still a somewhat common view, even though many people don't know where it originates from.

However, in practice it is hard to gauge race. This lead to the "brown paper bag test" within the African-American middle class; if you were lighter than a brown paper bag, there is a chance you could "pass" as non-black. The reason the African-American middle class wanted to do this is because race and social class are hopelessly jumbled together in the U.S., partially because the U.S. officially is supposed to be a classless society. (For instance when someone says that a person is "ghetto", it's not clear if they mean the person is black or working-class, as it is used interchangeably for both). Passing (both culturally and racially) allows the black middle-class a shot at joining the upper class. There is an equally strong reaction against this from black nationalists, who accuse blacks who do this of "acting white". By acting white, they don't mean acting like rednecks (the white rural working class), but acting like WASPs (the white upper class).

Race relations in the U.S. are very complicated. :)