When interleaved DOM queries and modifications in your JS cause the DOM to be successively invalidated and then laid out again, multiple times within a single script.
Modern browsers use a dirty-bit system for DOM modifications: they don't perform a layout every time it's changed, they just note the change and then perform the layout when the script returns. However, if the DOM is queried in the meantime, they have to perform layout, because otherwise they won't have up-to-date dimensions and positions. So if you interleave queries with modifications, the dirty bit gets set, the DOM gets laid out, the dirty bit gets set, the DOM gets laid out, etc, defeating all the optimizations browser vendors have put in place.
Layout is an expensive operation, too - on a moderately complex website like Google Search it takes about 17ms on desktop, and that can increase by 10x on mobile devices. 170ms is well past the point of visual perception.
It's not a formal term; the author was just drawing the parallel between memory + disk thrashing and the DOM + layout thrashing. Basically slowdown due to excessive reads and writes/redundantly moving data back and forth.
Anyone want to offer the net's very first ever explicit definition of this term?