Here is something to think about from James Oakes’s Freedom National: The Destruction of Slavery in the United States, 1861-1865.
It is not hard to understand the flurry of support for colonization during the Civil War. Notwithstanding the opposition of radical abolitionists, colonization presupposed emancipation, and whenever talk of emancipation arose, so too did talk of colonization. The more difficult question to answer is why it came to so little. In the modern world, wars of unification, especially civil wars inflamed by ethnic nationalism, commonly lead to forced population transfers and sometimes genocide. The Civil War in the United States was certainly a war of national unification, and the Republicans exhibited more than their fare share of ethnic nationalism. Nor was the idea of forced expulsion unheard of in the United States. Most Republican policymakers were old enough to remember the brutal “removal” of the southeastern Indians during Andrew Jackson’s administration. And during the Civil War itself the Union army forcibly expelled some ten thousand whites from their homes in Missouri. The same army systematically uprooted tens of thousands of slaves from their plantations to relocate them in areas safe from the reach of their former masters. And yet not a single emancipated slave was involuntarily “removed” from the United States in the wake of emancipation. (p. 281)
Oakes goes on to suggest an explanation, but for now I am going to leave you with just the excerpt.