The main reason that poor Southerners chose to fought was because of Southern politics. Basically, Abraham Lincoln didn't want slavery to expand any further. If there was no bartering over new states to become slave or free, then the South was afraid that they would lose power in Congress. It's also worth noting that slavery is what was keeping the South afloat in terms of economy. All of those plantation homes were so lavish because the North funded the institution of slavery by purchasing goods produced by slaves from Southern slave keepers.
