You'd be surprised how many people use non-GTK X11 Emacs. The reasons they give include being able to connect and disconnect from multiple X servers (GTK has longstanding unfixed bugs here) and being generally lighter on the system than GTK is.
"Pulling [GTK] out into its own frontend" means taking GTK support out of where it is now, Emacs' general-purpose X11 code (where GTK is one of many supported X11 toolkits) and putting it in a new top-level window-system that's a peer of NS (for OS X) and Windows support. The new GTK window system wouldn't be allowed to use X11 functions directly. Under this architecture, Emacs would be much closer to a well-behaved GTK program and could take advantage of cool GTK tricks like Broadway support.
I am one of those X11-only, no-toolkit Emacs users. The only times I need gtk support is when I run a variant build with gtk3 support in order to have a native Wayland window which doesn't require Xwayland. But I primarily use X11 so the no-toolkit, X11-only build.
Also, are the Xt and motif versions still maintained? Surely the user base of those must be pretty close to 0 these days?