If you’ve tried, has it been clunky to run non-python based workflows? I.e if you want to run bedtools or diamond without having to run a bunch of subprocess.run commands?
Right, for most of our workflows, we stay in python land, which is great and seamless with Metaflow being in python. But yes, there are occasions that we have to make a system call to run an old R script or even a compiled C++ executable :shrug: (Metaflow does have some native R support tho) I have not had to use the specific tools you called out, bedtools or diamond.
Most of the time this not a blocking problem since each step in a flow is mapped to a Docker image and/or your choice of EC2 instance (e.g. one step on a GPU, another on a memory optimized instance). You can have one step use an image with all of your python-based ML stuff, and another step have a different image with compiled exectuables that are triggered by a system call. If needed, outputs from such a system call would then need to be persisted in a database/S3 or read back into the python flow for persistence. So, it is not as seamless as a flow in all python, but it can work "good enough".
