Optimizing memory layout and reducing allocations are pretty ISA-independent. But yeah, at some point you will have to start looking at the assembly to optimize further (even if only to know when to tell the compiler to stop inlining a function that causes tons of register spilling inside a tight loop).

In most cases there’s a lot you can do by just peeking at the source code. Almost all programs ship without having been run through a profiler at all.