Do you mean time and energy for optimising things that make no sense, so you have no time and energy to optimise things that make sense?

Not optimising things that "make no sense" is how we ended up with operating systems needing gigabytes of RAM just to boot, and AAA video games taking over 100GB each. A reasonably efficient system is the sum of reasonably efficient parts, or else you end up in UniformlySlowCode hell.