Organizing large-scale biological data around standardized projects

# Organizing large-scale biological data around standardized projects

Nathan Sheffield

<span style="font-size:0.6em"><a href="http://www.databio.org/slides">www.databio.org/slides</a></span>

</section>
---

<div class="small">
<a href="{{project_url}}">{{project_url}}</a><br>
</div>
<span class="small bullet"><img src="/_modules/project-cover/icons/paper.svg" height="25" class="bullet" style="vertical-align: text-bottom; margin-right: 5px;">{{project_citations}}</span>
---

## Data is becoming more...

<div style="display: flex; justify-content: space-between;">
<div style="width: 30%; text-align: center;">
abundant
<img src="/_modules/pep/sequencing_costs_2015.jpg" width="100%">
</div>
<div style="width: 30%; text-align: center;">
available
<img src="/_modules/pep/cos_black.jpg" width="100%">
</div>
<div style="width: 30%; text-align: center;">
powerful
<img src="/_modules/pep/gwasw.jpg" width="100%">
</div>
</div>

---

## So why are the world's problems not solved?

---

---

## First step in bioinformatics analysis

<div style="display: flex; justify-content: space-between;">
<div style="width: 45%; text-align: center;">
<img src="/_modules/pep/icons/pipeline.svg" width="200">
pipeline
</div>
<div class="fragment" style="width: 45%; text-align: center;">
<img src="/_modules/pep/search_trends.svg" width="100%">
Papers with "bioinformatics pipeline" in title
</div>
</div>

---

## Problem solved?

---

## Problem solved?

---

## Problem solved?

---

## Data munging

---

## Then, downstream tools need a different organization

---

## What if?

---

## The solution

---

## The solution

---

## PEP: A standard format for project metadata

---

## PEP Ecosystem

---

## <img src="/_modules/pep/pep_logo.svg" width="70" style="vertical-align: middle;"> PEP format

<span class="bullet"><img src="/_modules/pep/icons/file.svg" width="30" class="bullet"> project_config.yaml</span>

```yaml
metadata:
  sample_annotation: /path/to/samples.csv
  output_dir: /path/to/output/folder
```

---

<span class="bullet"><img src="/_modules/pep/icons/file.svg" width="30" class="bullet"> samples.csv</span>

```csv
sample_name, protocol, organism, data_source
frog_0h, RNA-seq, frog, /path/to/frog0.gz
frog_1h, RNA-seq, frog, /path/to/frog1.gz
frog_2h, RNA-seq, frog, /path/to/frog2.gz
frog_3h, RNA-seq, frog, /path/to/frog3.gz
```
---

#### Microwave syndrome

<div class="well fragment">In user interface design, prioritizing easy access to integrated functions over their individual components. </div>

---

---

---

---

### The UNIX philosophy

<div class="col2">
<img src="/_modules/microwave-syndrome/unix_book.jpg" height="450">
</div>
<div class="col2">
<div class="well"><span style="color:#ffb; font-weight:bold">[T]he power of a system comes more from the relationships among programs than from the programs themselves.</span><br><br>
<span style="font-size: 0.8em">Many UNIX programs do quite trivial tasks in isolation, but, combined with other programs, become general and useful tools.</span><br/><br/>
<span class="small">- Kernighan and Pike, The UNIX Programming Environment (1983, p. viii)</span>
</div>
</div>

---

---

---

---

---

---

---

---

### Problem

### Solution

<img src="/_modules/microwave-syndrome/pep/data_input_plug.svg" width="375">
</div>
---

## PEP Ecosystem

---

## peppy package <img src="/_modules/pep-details/logo/logo_python.svg" height="50" style="vertical-align: middle;">

```python
import peppy

prj = Project("pep_config.yaml")
samples = prj.get_samples()

for sample in samples:
    print(sample.name)
    # do further analysis to each sample
```

<span class="bullet"><img src="/_modules/pep-details/icons/ftnetwork-connected.svg" width="50" class="bullet">Project API</span>

---

## pepr package <img src="/_modules/pep-details/logo/logo_R.svg" height="50" style="vertical-align: middle;">

```r
library("pepr")

prj = pepr::Project("pep_config.yaml")
samples = pepr::pepSamples(prj)

for (sample in samples) {
    message(pepr::sampleName(sample))
    # do further analysis to each sample
}
```

---

## Conclusion

- PEP format is a novel approach to standardize *projects*.
- Initial tools like `geofetch` and `looper` build PEP projects and connect them to pipelines
- Python and R packages provide a universal interface to PEP metadata for tools and analysis

More information at [pepkit.github.io](http://pepkit.github.io/).
---

# Thank You

<span class="small bullet"><img src="/images/external/github_bug_black.svg" height="20" class="bullet"><a href="http://github.com/nsheff">nsheff</a></span> &middot;
<span class="small bullet"><img src="/images/icons/web.svg" height="25" class="bullet"><a href="http://databio.org">databio.org</a></span> &middot;
<span class="small bullet"><img src="/images/icons/letter.svg" height="25" class="bullet"><a href="mailto:nsheffield@virginia.edu">nsheffield@virginia.edu</a></span>

</section>