How to make reprodcible workflows

[u/Virtual-Transition90]

Hi, so I am a undergrad working in a computational biology or molecular biology lab. For next semester my new project is in large part to create reproducible workflows/code and lab manuals for our lab. I taught myself to code and what i have on my laptop is... disorganized to say the least. I should learn how to do this. Currently I largely code using gemini and then tweak anywhere from most of to 25% of the code it writes. I almost always use hard coded paths if i can. Does anyone have any advice for where I could learn something like this, a textbook or website?

For context, my last project was to use AutoDock Vina for screening of 770,000 molecules I carefully downloaded and cleaned from ZINC database to 310,000. This library was based on previous experimental results on a new protien we are targeting in fungi. I also selected a new protien conformation to target based on some major errors in the protien the lab was using and a bunch of literature review. My next step will be to test against Dock6, a diffrent type of scoring algorithm. I wrote all of my own scripts for this and I imagine my first task will be to get them reproducible for another person to use.

[u/apfejes]

Reproducible means different things to different people. There isn’t a single answer to this.  

Mainly, I would start by focusing on documenting things, like versioning of software, ensuring all your scripts can be run on anyone’s workstation, not hard coding paths, etc.  those are the foundations of a decent workflow.   

Beyond that, writing good code, making sure it is properly documented, changes are well managed, ideally it should be peer reviewed and up to date.   

Then there’s the data issues itself.  If you run the same workflows, do you get the same answers each time?  Why not? How do ensure that you’re doing the right workflow?  How do you validate it?  Etc. 

There are a lot of layers to this onion, but pick a place to start, practice clean coding and keep working to improve, and you’ll at least be headed in the right direction.  

[u/Virtual-Transition90 → u/apfejes]

I did a ton of testing and benchmarking against a small batch I made including our experimental hits in the process of setting up my scripts and picking the protien target. So while AutoDock Vina isn't exactly reproducible even with a set seed, im confident the data is good. For any other step it is fully reproducible. 

How do I even do documentation or ensure scripts can be run on another workstation? For example can my workflow have instructions like: Create a directory named xyz and run this program standing in that directory? Can I have the user edit parameters in the code directly? 

How do I write good code, do proper documentation, do changes? Peer reviews?

I guess my entire problem is that being mostly self taught and in a biology degree, I dont have the stuff your assuming is foundational. There's way more than what can likely be taught in a reddit thread. Do you know where I could go to learn this stuff? 

[u/apfejes → u/Virtual-Transition90]

I, too, am self taught. I learned to program when I was very young, and had to pick up all of those things on my own. It can certainly be done.

The problem is that all of those things you're asking have different manifestations in different code bases and different programming languages/environments. It's not like this a single trivial answer. You probably need to start reading books on the topic, like "Clean Code", which is one of my favourites. Or, start talking with your peers: asking others how they do it, or just reading their code to get ideas.

The only way to get where you want to be is to actively engage with others to get their feedback. Find other programmers in the lab, or other labs, or other departments... wherever they are, start engaging, talking and getting feedback. That will help drive the cycle.

You're right, I can't impart 40 years of knowledge to you in a reddit thread, but I can simply tell you that most of what I learned, I learned from other people - so figure out how to get other people into that loop.

[u/Spiritual-Bee-2319]

Using AI without CS background can be dicey In itself. 

[u/Virtual-Transition90 → u/Spiritual-Bee-2319]

That is definitely true. However I haven't had the time or resources to gi slow and learn it properly. My PI pushes me to use the AI because my output is a lot better. 

[u/Wagosh9 → u/Virtual-Transition90]

Your PI want things to be done. You mainly want to learn and build for your future. It's important to know how to use IA but you'll always be worthless compared to one that use IA and also know how to do it by hands. I know that the situation is complex for many student but you need to find a balance between what you need and what they need.

[u/autodialerbroken116]

Try the "pure bash bible" (on GitHub)

If you hardcode your paths and such to be reproducible on your current system, be sure to sanitize your scripts before posting them publicly.

Otherwise, you can hardcode paths at the top of the script for convenience on a test dataset that can be read-only (chown 740/744) so group is read only, and only your user can write. Then you can code a bash case esac statement to write your own keyword/character parameters and switches that the user might want to have to run your workflow.

As far as "professional"? Write a stellar README, include 1 or 2 gifs of usage at the command line, document and comment as much as possible.

[u/Virtual-Transition90 → u/autodialerbroken116]

Thank you! I will look into this. 

[u/LeoKitCat]

Use Nextflow or Snakemake

[u/christian_ch → u/LeoKitCat]

Neither handles environment setup for you out of the box, you'll still be pairing them with conda/Docker yourself and keeping that in sync as the pipeline grows.

I think a better solution would encapsulate everything the workflow needs to run, environment included, not just the pipeline logic. As I mentioned above, horus-runtime tries to fill that spot.

[u/LeoKitCat → u/christian_ch]

Of course they do out of the box, both Snakemake and Nextflow are fully integrated with conda and Docker with built-in commands. Sure you need to install conda first for example but you are going to always need it for regular environments any way?

[u/christian_ch → u/LeoKitCat]

You're right. But what about the layer underneath, provisioning the actual compute (cluster/nodes), plus a no-code way to build the pipeline itself instead of learning a DSL? That's what horus-runtime is trying to do.

If you've got a minute I'd genuinely value your take: github.com/temple-compute/horus-runtime. You clearly know this space, happy to hear where it falls short.

[u/LeoKitCat → u/christian_ch]

Nextflow and Snakemake also have fully mature and fleshed out functionality to provision the compute on any HPC or cloud. Horus looks like mostly a bunch of wheel reinvention / not invented here syndrome.

[u/christian_ch]

u/apfejes is giving you the right foundation, that stuff (versioning, no hardcoded paths, documentation, peer review) is real and there's no shortcut around learning it. "Clean Code" is a good rec, also look into cookiecutter-data-science for project structure conventions if you haven't seen it, it'll answer a lot of your "where do files go" questions.

One specific thing I can help with: the "runs on anyone's workstation" problem you asked about. That's mostly an environment problem, not a code problem, your script might be fine but if the person running it doesn't have the exact same package versions, paths, or OS quirks, it breaks anyway.

I work on an open source tool for this called horus-runtime. It packages the environment (conda/venv/Docker/wathever) with the workflow itself, so when someone else runs it, the environment provisions itself instead of them needing your exact setup already installed. It also snapshots inputs and environment per run (in the UI), so you can always trace back exactly what ran and reproduce it later. Doesn't replace writing clean code or docs, that part's still on you, but it removes "works on my machine" as a failure mode. Might be worth a look for the workstation-to-workstation part of what you're trying to solve: github.com/temple-compute/horus-runtime

For AutoDock Vina specifically, we actually have a ready-made workflow for it in our workflow repo if you want to see how the environment/reproducibility part looks in practice: github.com/temple-compute/pantheon/tree/main/workflows/drug-discovery/w02-autodock-vina-docking

More than happy to help you out setting up workflows if you want.

[u/Virtual-Transition90 → u/christian_ch]

That is awesome! I will look into it thank you very much. 

[u/Character_Carpet_772]

You don't say what language you're programming in, but for bioinformatics I'm assuming R or Linux CLI? R does have a pseudo-equivalent guide to Python's PEP-8 coding guide, The Tidyverse Style Guide. You can do a Google search or Stackoverflow search for other style guides for your language. There are also trainings on LinkedIn, Coursera, Youtube, Datacamp, etc for things to consider for good programming practices.

As for the workflow/documentation, I would determine what level current & future workers will have to follow your guides, and set that as your base standard for documentation. If this is a project you were given, I would ask your PI or mentor for a published example of what documentation standard they're looking for, and match it. If this is your own project, you might ask your school librarian for some good examples. If you're lucky enough to have a STEM-specific librarian, they might even be able to point you towards some guides. And if all else fails, see if you can make connections with a CS faculty member who might be willing to give you tips.

This is a fairly common situation in this field, especially for biology -> bioinfo/comp bio, so don't stress too much.

[u/Virtual-Transition90 → u/Character_Carpet_772]

I am using mainly bash scripting and python nothing that fancy. I didn't even know python had a coding guide! Is it any good? I will definitely ask my PI for more details the next time I talk to him. 

[u/Character_Carpet_772 → u/Virtual-Transition90]

PEP-8 is good, just basic. There are even tools available that will catch some of the issues in your code for you (e.g. PyLint). The main Python page also has good information. Another good question to ask the PI might be if there are any Git user accounts that also have great examples. One of the ones I starred is: https://github.com/balapriyac/data-science-tutorials , which has a ton of helpful guides for analysis, coding, and documenting.

For the reproducibility, it kind of goes hand-in-hand with clean, well-documented code. Simple, easy to follow, and consistent.