Hi Adam! If I may ask, why did you go with BuildKit (instead of, say, just calling runc on your own) and what has your experience been like?

We use a lot of what buildkit does.

For instance, building up the DAG of all the build steps and then scheduling things so that various parts can be built in parallel. Buildkit does a lot that we use beyond just being a way do build things inside a runc container.

That said, we have a fork of buildkit for the various things we add that don't fit well in the upstream.

Already our auto-skip feature and our branching are implemented on top of Buildkit rather than using it. Probably as we grow and add more build centric features we will continue to diverge.

I'm just a DevRel person though, so that's just my 2 cents. The core team may disagree with me.

> building up the DAG of all the build steps and then scheduling things so that various parts can be built in parallel

This is one of my hobbies as a programmer and I'm extremely interested in this area. Can you share more or is it a trade secret?

Furthermore, do you have links to tools or papers that deal with this?

Just saw this now.

I don't have any great references to be honest. It's not that its a trade-secret, its just that its an area I don't work on.

The AWK book describes how to build you own Make using AWK. You might like that. And an explanation of it and python implementation is found here:

https://benhoyt.com/writings/awk-make/

Hopefully that helps.

Should be great, thank you.

I am considering writing a small library / framework to optimize scheduled tasks by making a DAG and inferring which ones are safe to be parallel and which aren't. So this can be useful.