Last updated: 2023-09-22
Checks: 2 0
Knit directory: snakemake_tutorial/
This reproducible R Markdown analysis was created with workflowr (version 1.7.0). The Checks tab describes the reproducibility checks that were applied when the results were created. The Past versions tab lists the development history.
Great! Since the R Markdown file has been committed to the Git repository, you know the exact version of the code that produced these results.
Great! You are using Git for version control. Tracking code development and connecting the code version to the results is critical for reproducibility.
The results in this page were generated with repository version bc62309. See the Past versions tab to see a history of the changes made to the R Markdown and HTML files.
Note that you need to be careful to ensure that all relevant files for
the analysis have been committed to Git prior to generating the results
(you can use wflow_publish or
wflow_git_commit). workflowr only checks the R Markdown
file, but you know if there are other scripts or data files that it
depends on. Below is the status of the Git repository when the results
were generated:
Ignored files:
Ignored: .Rhistory
Ignored: .Rproj.user/
Note that any generated files, e.g. HTML, png, CSS, etc., are not included in this status report because it is ok for generated content to have uncommitted changes.
These are the previous versions of the repository in which changes were
made to the R Markdown (analysis/introduction.Rmd) and HTML
(docs/introduction.html) files. If you’ve configured a
remote Git repository (see ?wflow_git_remote), click on the
hyperlinks in the table below to view the files as they were in that
past version.
| File | Version | Author | Date | Message |
|---|---|---|---|---|
| Rmd | bc62309 | Jean Morrison | 2023-09-22 | updates |
| html | bc62309 | Jean Morrison | 2023-09-22 | updates |
| Rmd | 1939b86 | Jean Morrison | 2023-09-21 | add code and update site |
| html | 1939b86 | Jean Morrison | 2023-09-21 | add code and update site |
| html | 5757821 | Jean Morrison | 2023-09-21 | build site |
| Rmd | 7957b05 | Jean Morrison | 2023-09-21 | add installation instructions |
| html | 7957b05 | Jean Morrison | 2023-09-21 | add installation instructions |
A workflow is a series of analytically steps to go from raw data to an analysis result. Often in biostatistics, statistical genetics, or bioinformatics, our workflows include several steps with intermediate outputs. We might need to combine multiple different programs in sequence. If we are unlucky, these programs might even have conflicting dependencies.
For example, to run a genome-wide association study, we need to format phenotype and genotype data, perform quality control checks and filters, compute a genetic relatedness matrix, and then use one of several available packages to perform association tests for each variant.
A natural way to run a workflow is to execute each step one at a time “by hand”. Every analysis process will involve this type of process to some degree, and this is not a bad way to start working through a procedure.
However! This is not a great way to produce repeatable results that you have high confidence in.
The “by hand” strategy is very vulnerable to common errors. For example, you may modify an early data normalization or processing step but forget to replace the file name in all of the downstream steps.
This is also not a scalable strategy. For example, suppose we are running GWAS of protein levels measured by a high-throughput technology. Rather than running association tests for one phenotype, we need to run association tests for thousands of phenotypes. One option would be to write a bash script loops through the genes and submits the job to the cluster. However, writing this code and then monitoring job submission and completion takes time and can be a bit fiddly.
Finally, the “by hand” strategy is hard to repeat and hard to accurately document
Snakemake is a tool for creating and executing workflows.
A Snakemake workflow is repeatable. This is especially true if you use Conda inside of Snakemake to control the software environment of each step.
A Snakemake workflow is scalable. It is easy to modify a workflow to be performed many times or with multiple options. Snakemake can automate the cluster submission process so you don’t have to manage it.
A Snakemake workflow is human readable. You will have a single document that describes exactly what you did at each step, rather than having scrpts spread over many directories.
To write a Snakemake workflow, we need to describe each step in the procedure using three attributes: input files, output files, and the action (shell command). Let’s practice by writing the inputs, outputs, and action for the process of making tea. The steps we want to perform are:
What are inputs/outputs for each step? What is the final target result?
We could write this procedure in the style of a Snakemake workflow by
writing each step as a rule with inputs and outputs. The
rule all at the top indicates the final target. We will
talk more about this in the next section.
rule all:
input: "tasty cup of tea"
rule boil:
input: "cold water"
output: "boiling water"
shell: "Put the cold water in the kettle and boil it."
rule steep:
input: "boiling water", "mug", "tea bag"
output: "black tea"
shell: "Put the mug in the tea bag. Pour the boiling water over the tea bag."
rule flavor:
input: "black tea", "milk", "sugar"
output: "tasty cup of tea"
shell: "Add milk and sugar to black tea"You are guaranteed to get some errors when you are developing a Snakemake pipeline. Generally, Snakemake error messages are fairly informative and it is helpful to get used to reading them. If you get an error during this tutorial, don’t worry! Share your error with the group and we will probably all learn something.