Several studies have found that there is in general a negative effect of job displacement on a large set of various labor market outcomes. However, many of these studies only compare the workers who remain in a firm upon closure (the stayers), and do not consider the workers displacing in the process leading up to this final displacement event (the early leavers). Neglecting these early leavers may lead to biased estimates of the costs of displacement. Using Norwegian administrative register data, I identify the early leavers and the stayers by determining when the selective labor turnover directly related to the downsizing process starts by comparing the post-separation outcomes of workers separating from closing and non-closing firms using a difference-in-difference approach. Having established that selective labor turnover starts two years prior to firm closure, I estimate the costs of displacement using a double difference-in-difference approach. I find that the cost of displacement in terms of annual pensionable income is greater for the stayers compared to the early leavers. Specifically, the annual income loss for the stayers is from 2.2 to 4.5 percent higher in the five years after the displacement year. Using a three-way fixed effects log-wage model estimated prior to the start of the downsizing process, I construct empirical human capital distributions for the early leavers and the stayers. The resulting distributions lend support to the hypothesis that the stayers are negatively selected in terms of general human capital and positively selected in terms of firm-specific human capital. From a search-theoretical point of view, this may explain why the negative effect of displacement on income is greater for the stayers than for the early leavers.