This is exactly my knee-jerk opposition to 'refactoring'. In many cases the cause for it isn't that the factor (seams along which the problem was separated) has changed, but rather the factor wasn't known at the time and was more like primitives composed without context or deduplicating patterns/machinery. Going top-down you'd be conceptualizing/picking/naming the factors as you go. Sometimes they change too but not at arbitrary points in layers of implementation, and this all happens sooner than later.