Have you guys started isolating any patterns to the bugs you missed pre-ASAN? What were your fuzzers (for instance) failing to do? Are there specific kinds of code paths (things buried in specific state transitions, things that were timing specific) that you couldn't catch without this level of instrumentation?
If you knew then what you knew now, short of implementing ASAN, how would you change your fuzzing cluster?
Hey Tom. We haven't really tracked the difference in what ASAN finds. If I had to ballpark it I'd say that ASAN wins significantly more than half of the time--meaning a given repro faults under ASAN but not under an uninstrumented build. And for a bit more context, the vast majority of the bugs we deal with are use-after-free issues, which is something that ASAN really excels at detecting and clearly reporting.
In terms of process, our scaled out fuzzing is rapidly evolving anyway. So, I think it was early enough that our approach wouldn't have changed too much. It's more that ASAN has enabled us to move faster, with fewer resources. And the improved turnaround time has significantly reduced the number of bugs that get past trunk.
If you want, I can put you in touch with the guys on my team doing the real work on the fuzzing cluster (I've been assisting in only an advisory capacity). They're also planning on doing a Chromium blog post on the project soon, so you could wait for that if you prefer.
Have you guys started isolating any patterns to the bugs you missed pre-ASAN? What were your fuzzers (for instance) failing to do? Are there specific kinds of code paths (things buried in specific state transitions, things that were timing specific) that you couldn't catch without this level of instrumentation?
If you knew then what you knew now, short of implementing ASAN, how would you change your fuzzing cluster?