Packages submission and reviews; how does it work?

class: center, middle, inverse, title-slide

# Packages submission and reviews; how does it work?
### <a href="https://llrs.dev">Lluís Revilla</a> <br> <a href="https://user2021.llrs.dev"><svg viewBox="0 0 448 512" style="height:1em;position:relative;display:inline-block;top:.1em;" xmlns="http://www.w3.org/2000/svg"> <path d="M400 32H48C21.5 32 0 53.5 0 80v352c0 26.5 21.5 48 48 48h352c26.5 0 48-21.5 48-48V80c0-26.5-21.5-48-48-48zm-6 400H54c-3.3 0-6-2.7-6-6V86c0-3.3 2.7-6 6-6h340c3.3 0 6 2.7 6 6v340c0 3.3-2.7 6-6 6z"></path></svg></a> <a href="https://github.com/llrs/user2021/"><svg viewBox="0 0 496 512" style="height:1em;position:relative;display:inline-block;top:.1em;" xmlns="http://www.w3.org/2000/svg"> <path d="M165.9 397.4c0 2-2.3 3.6-5.2 3.6-3.3.3-5.6-1.3-5.6-3.6 0-2 2.3-3.6 5.2-3.6 3-.3 5.6 1.3 5.6 3.6zm-31.1-4.5c-.7 2 1.3 4.3 4.3 4.9 2.6 1 5.6 0 6.2-2s-1.3-4.3-4.3-5.2c-2.6-.7-5.5.3-6.2 2.3zm44.2-1.7c-2.9.7-4.9 2.6-4.6 4.9.3 2 2.9 3.3 5.9 2.6 2.9-.7 4.9-2.6 4.6-4.6-.3-1.9-3-3.2-5.9-2.9zM244.8 8C106.1 8 0 113.3 0 252c0 110.9 69.8 205.8 169.5 239.2 12.8 2.3 17.3-5.6 17.3-12.1 0-6.2-.3-40.4-.3-61.4 0 0-70 15-84.7-29.8 0 0-11.4-29.1-27.8-36.6 0 0-22.9-15.7 1.6-15.4 0 0 24.9 2 38.6 25.8 21.9 38.6 58.6 27.5 72.9 20.9 2.3-16 8.8-27.1 16-33.7-55.9-6.2-112.3-14.3-112.3-110.5 0-27.5 7.6-41.3 23.6-58.9-2.6-6.5-11.1-33.3 2.6-67.9 20.9-6.5 69 27 69 27 20-5.6 41.5-8.5 62.8-8.5s42.8 2.9 62.8 8.5c0 0 48.1-33.6 69-27 13.7 34.7 5.2 61.4 2.6 67.9 16 17.7 25.8 31.5 25.8 58.9 0 96.5-58.9 104.2-114.8 110.5 9.2 7.9 17 22.9 17 46.4 0 33.7-.3 75.4-.3 83.6 0 6.5 4.6 14.4 17.3 12.1C428.2 457.8 496 362.9 496 252 496 113.3 383.5 8 244.8 8zM97.2 352.9c-1.3 1-1 3.3.7 5.2 1.6 1.6 3.9 2.3 5.2 1 1.3-1 1-3.3-.7-5.2-1.6-1.6-3.9-2.3-5.2-1zm-10.8-8.1c-.7 1.3.3 2.9 2.3 3.9 1.6 1 3.6.7 4.3-.7.7-1.3-.3-2.9-2.3-3.9-2-.6-3.6-.3-4.3.7zm32.4 35.6c-1.6 1.3-1 4.3 1.3 6.2 2.3 2.3 5.2 2.6 6.5 1 1.3-1.3.7-4.3-1.3-6.2-2.2-2.3-5.2-2.6-6.5-1zm-11.4-14.7c-1.6 1-1.6 3.6 0 5.9 1.6 2.3 4.3 3.3 5.6 2.3 1.6-1.3 1.6-3.9 0-6.2-1.4-2.3-4-3.3-5.6-2z"></path></svg></a> <a href="https://twitter.com/Lluis_Revilla"><svg viewBox="0 0 512 512" style="height:1em;position:relative;display:inline-block;top:.1em;" xmlns="http://www.w3.org/2000/svg"> <path d="M459.37 151.716c.325 4.548.325 9.097.325 13.645 0 138.72-105.583 298.558-298.558 298.558-59.452 0-114.68-17.219-161.137-47.106 8.447.974 16.568 1.299 25.34 1.299 49.055 0 94.213-16.568 130.274-44.832-46.132-.975-84.792-31.188-98.112-72.772 6.498.974 12.995 1.624 19.818 1.624 9.421 0 18.843-1.3 27.614-3.573-48.081-9.747-84.143-51.98-84.143-102.985v-1.299c13.969 7.797 30.214 12.67 47.431 13.319-28.264-18.843-46.781-51.005-46.781-87.391 0-19.492 5.197-37.36 14.294-52.954 51.655 63.675 129.3 105.258 216.365 109.807-1.624-7.797-2.599-15.918-2.599-24.04 0-57.828 46.782-104.934 104.934-104.934 30.213 0 57.502 12.67 76.67 33.137 23.715-4.548 46.456-13.32 66.599-25.34-7.798 24.366-24.366 44.833-46.132 57.827 21.117-2.273 41.584-8.122 60.426-16.243-14.292 20.791-32.161 39.308-52.628 54.253z"></path></svg>Lluis_Revilla</a>

---

name:intro
# Brief introduction

Goals of a *submission*

- Sharing something of quality that can be useful to others.
- Make it easier for others to build upon your package.
- Other: work, grant, prestige ...

???

Submissions are though specially if coming from places with poor training
Lack of confidence/experience with reviews.

|            Archives reviewing packages             |                                Objectives of the reviews?                                |
|:--------------------------------------------------:|:----------------------------------------------------------------------------------------:|
|    <a href=https://cran.r-project.org/>CRAN</a>    |                        Non-trivial publication quality packages.                         |
| <a href=https://bioconductor.org/>Bioconductor</a> |                 Promote high-quality, well documented and interoperable.                 |
|     <a href=https://ropensci.org/>rOpenSci</a>     | Drive the adoption of best practices with useful, transparent and constructive feedback. |

???

Differences in objectives but all looking for quality
CRAN: Point errors, comments
Bioconductor: In detail comment of style, classes, dependencies, structure…
rOpenSci: guideline for reviewers (about style, tests, functions, description, documentation, …)

CRAN ~16000 packages, Bioconductor ~2000,  rOpenSci ~300
To work with this slides use xaringan::infinite_moon_reader()

---
name:projects
class: center

# Project differences

|          |                                      CRAN                                       |                                  Bioconductor                                  |                                rOpenSci                                |
|:---------|:-------------------------------------------------------------------------------:|:------------------------------------------------------------------------------:|:----------------------------------------------------------------------:|
|Guides    | <a href=https://cran.r-project.org/doc/manuals/r-release/R-exts.html>R-exts</a> | <a href=https://www.bioconductor.org/developers/package-guidelines>Website</a> |       <a href=https://devguide.ropensci.org/index.html>Book</a>        |
|Submit    |         <a href=https://cran.r-project.org/submit.html>tar.gz file</a>          |    <a href=https://github.com/Bioconductor/Contributions/>fill an issue</a>    | <a href=https://github.com/ropensci/software-review/>fill an issue</a> |
|Review    |                                   email & ftp                                   |                                     Github                                     |                                 Github                                 |
|Setup     |                                      None                                       |                           ssh key, subscribe mailing                           |                                CI tests                                |
|Checks    |                                 check --as-cran                                 |                                check; BiocCheck                                |                            check --as-cran                             |
|OS        |                               Windows, Unix, iOS                                |                               Windows, Unix, iOS                               |                           Windows, Unix, iOS                           |
|Versions  |                         oldrel, release, patched, devel                         |                                 release, devel                                 |                         oldrel, release, devel                         |
|Cycle     |                                   Always open                                   |                               2 annual releases                                |                              Always open                               |
|Editors   |                                        0                                        |                                       0                                        |                                  ~10                                   |
|Reviewers |                                    <b>~5</b>                                    |                                      ~10                                       |                               Volunteers                               |

.middle[Different setup, different review.]

???

The different projects/archives have different setups.
*Read the table*
All of them first you need to pass the automatic checks in place before a human looks into it.
Will use data from the three projects but mostly refer to CRAN.

---
name:submissions
# Submissions

.center[
CRAN data thanks to the [incoming dashboard](https://lockedata.github.io/cransays/articles/dashboard.html).

]

???

One order of magnitude of difference between each other CRAN > Bioconductor > rOpenSci
Many variability on month
Also very few data collected from CRAN so far (Also there are some hiccups on CRAN collection, near the end of May the CRON job stopped working for a week. )

---
name:organization
# Organization

.center[
Packages are moved by reviewers [between folders](https://llrs.dev/2021/01/cran-review/#cran-load).
]

???

Many folders but these two are the most important.
There isn't an explanation from CRAN about how do they work.
Pretest is resubmission (newer versions of packages) and also for newbies

---

# Workload after holidays

.center[Big volume of work! Patience!]

???
 
2 months to get back to normal for new packages.
First served are the resubmissions of packages.
 
---
name:submission-patterns
# Submissions patterns

.center[
Check [dashboard](https://lockedata.github.io/cransays/articles/dashboard.html) before submitting?
]

???

Submit when you are ready, better on the queue than outside.

---
name:review-time
# Review time

.center[Reviews are short, brief and to the point.]

???

Median time on submissions ~10 hours, mean time ~37.9882859 hours.
1, 2, 10, 37.9882859, 43, 1941

---

# Review speed

.center[Expect 3-7 days till your new package is on CRAN.]

???

Different time, can be shorter or longer.
Most longer need resubmission.
Resubmit with different version (makes it easier to track how many are).

CRAN: 80h
Bioconductor: most of them in 1 month
rOpenSci: in 2 months (seeking 2 reviewers and posting them).

```
## # A tibble: 2 x 2
##   new     time
##   <chr>  <dbl>
## 1 New     76.5
## 2 Update   5
```

---
name: users
# Users role

.center[Some users are very involved.]

???

Bioconductor reviewers do a lot
rOpenSci editors too
Both organizations have a group of users involved on the package review system.
Even if Bioconductor doesn't explicitly ask for reviewers from the community.
Bioconductor are considering now how to improve the review system.
Omitted bots bioc-issue-bot and ropensci-review-bot (new March 2021).

---

# Comments

.center[
A dialog between authors and reviewers & editors. 
]

???

Non reviewers users on bioconductor still chime in to help.

---
name:bot

# Bot role

.center[Bot helps on the process and changes with the process]

???

Bot provides feedback of many issues and actions performed. 
It can be changed/adapted to change in requirements or errors.
rOpenSci is going to have a bot too [ropensci-review-bot](https://github.com/ropensci-review-bot/).

---
exclude: true
name: labels

# Labels

.bottom[ .center[ Labels are used to indicate progress on the submission. ] ]

???

On bioconductor most problems with the submissions are not the package itself but not replying or chosing another venue.
rOpenSci provides more detailed questioning for scope of a package.

---
name:success-submissions
# Success submissions

.bottom[.center[High approval rates!!]]

???

Bioconductor & rOpenSci 50%, some submissions are abandoned or do not fit the project.
Different problems faced by new packages and older ones. 
More indeepth review requires 1 month for each reviewer.

```
## # A tibble: 35 x 7
##    submission_n new    Accepted     n  perc suspended perc_suspended
##    <fct>        <chr>  <lgl>    <int> <dbl>     <dbl>          <dbl>
##  1 1            New    FALSE      451 19.4        451          32.4 
##  2 1            New    TRUE      1868 80.6          0           0   
##  3 1            Update FALSE      494 12.2        494          35.5 
##  4 1            Update TRUE      3551 87.8          0           0   
##  5 2            New    FALSE      109 27.1        109           7.84
##  6 2            New    TRUE       293 72.9          0           0   
##  7 2            Update FALSE      162  8.15       162          11.7 
##  8 2            Update TRUE      1826 91.9          0           0   
##  9 3            New    FALSE       26 22.4         26           1.87
## 10 3            New    TRUE        90 77.6          0           0   
## # … with 25 more rows
```

```
## # A tibble: 2 x 3
##   Approved     n  perc
##   <chr>    <int> <dbl>
## 1 No         983 0.474
## 2 Yes       1092 0.526
```

```
## # A tibble: 1 x 4
##   `1. awaiting moderation` `2. review in progress` `3a. accepted` `3b. declined`
##                      <dbl>                   <dbl>          <dbl>          <dbl>
## 1                0.0000231                   0.222           35.9           19.1
```

|name                            | Median days| Total days|
|:-------------------------------|-----------:|----------:|
|1/editor-checks                 |         2.7|        2.7|
|2/seeking-reviewer(s)           |         2.1|        4.8|
|3/reviewer(s)-assigned          |         6.9|       11.7|
|4/review(s)-in-awaiting-changes |        26.3|       38.0|
|5/awaiting-reviewer(s)-response |        17.1|       55.1|
|6/approved                      |        12.7|       67.8|

---
name:submit
# Submit!

.pull-left[

.tip-submission[
Prepare
.center[
 
  Manual to [create R packages](https://cran.r-project.org/doc/manuals/r-release/R-exts.html#Creating-R-packages), [R Packages](https://r-pkgs.org/)  
  Follow policies ([CRAN](https://cran.r-project.org/web/packages/policies.html)) and **guidelines** ([Bioconductor](https://www.bioconductor.org/developers/package-submission/), [rOpenSci](https://devguide.ropensci.org/)).
]
]

.tip-submission[
.center[
**Check**

[Use Rhub](https://builder.r-hub.io/), [Github Actions](https://github.com/r-lib/actions)]
]

.tip-submission[ 
.right[(Re)***Submit***]
.center[[Fix and explain](https://cran.r-project.org/web/packages/policies.html#Re_002dsubmission) on re-submission.]
]

]
???

Follow the detailed guidelines from Bioconductor and rOpenSci.
Fix any problem that you haven't detected previously (double check the policy on CRAN). 
Resubmit

.pull-right[

.center[
# Thanks

R core and CRAN team,
Bioconductor core, 
rOpenSci editors and reviewers
]

.bottom[

.center[***Q&A ?***

Some answers on [Lluís's blog](https://llrs.dev/post/) posts: [Bioconductor](https://llrs.dev/2020/07/bioconductor-submissions-reviews/), [rOpenSci](https://llrs.dev/2020/09/ropensci-submissions/), [CRAN](https://llrs.dev/2021/01/cran-review/).

]
]
]

???

Thank also to the package authors (mainly tidyverse, ggplot2 and rhub, and gh).
Maëlle Salmon and Stephanie Locke for the dashboard.
rOpenSci review: [Video](https://www.youtube.com/watch?v=iJnn_9xKkqk)