Lecture 3

class: center, middle, inverse, title-slide

# Lecture 3
### Jung-Jin Lee
### Jan 30, 2019

---

## Exploring data

```r
d <- iris # iris data set comes with base R
dim(d)
```

```
## [1] 150   5
```

So `d` has 150 observations and 5 variables.

```r
head(d)
```

```
##   Sepal.Length Sepal.Width Petal.Length Petal.Width Species
## 1          5.1         3.5          1.4         0.2  setosa
## 2          4.9         3.0          1.4         0.2  setosa
## 3          4.7         3.2          1.3         0.2  setosa
## 4          4.6         3.1          1.5         0.2  setosa
## 5          5.0         3.6          1.4         0.2  setosa
## 6          5.4         3.9          1.7         0.4  setosa
```

---

## Exploring data, continued

```r
table(d$Species)
```

```
## 
##     setosa versicolor  virginica 
##         50         50         50
```

.pull-left[
<p align = "center">
  <img src = iris_species.jpeg>
</p>
]
.pull-right[
<p align = "center">
  <img src = iris_petal_sepal.png width = "180">
</p>  
]

---

## Exploring data, continued

```r
g <- ggplot(d, aes(Sepal.Length)) + geom_histogram()
print(g)
```

---

## Data manipulation

Create a data set consists of `setosa` only.

```r
library(dplyr) # dplyr is a part of tidyverse
# you could run library(tidyverse) instead
```

`dplyr` provides the following _verbs_ that are useful in data manipulation:
- `select()`: subset using columns (variables)
- `filter()`: subset using rows (observations)
- `mutate()`: transform variables
- `arrange()`: reorder observations

These verbs, when combined with the pipe operator `%>%`, become even more powerful!

---

## select()

**Example**: create a data set consisting only of `Sepal.Length` and `Sepal.Width`.

```r
s1 <- select(d, Sepal.Length, Sepal.Width)
dim(s1)
```

```
## [1] 150   2
```

```r
head(s1)
```

```
##   Sepal.Length Sepal.Width
## 1          5.1         3.5
## 2          4.9         3.0
## 3          4.7         3.2
## 4          4.6         3.1
## 5          5.0         3.6
## 6          5.4         3.9
```

---
 
## select(), continued

**Exercise**: create a data set consisting of `Sepal.Length`, `Sepal.Width`, `Petal.Length` and `Petal.Width`.

```r
s2 <- select(d, Sepal.Length, Sepal.Width, Petal.Length, Petal.Width)
dim(s2)
```

```
## [1] 150   4
```

```r
head(s2)
```

```
##   Sepal.Length Sepal.Width Petal.Length Petal.Width
## 1          5.1         3.5          1.4         0.2
## 2          4.9         3.0          1.4         0.2
## 3          4.7         3.2          1.3         0.2
## 4          4.6         3.1          1.5         0.2
## 5          5.0         3.6          1.4         0.2
## 6          5.4         3.9          1.7         0.4
```

---

## select(), continued

```r
s3 <- select(d, -Species)
head(s3)
```

---

## filter()

**Example**: create a data set consisting only of `setosa` species.

```r
f1 <- filter(d, Species == "setosa") # note use of ==, not =
dim(f1)
```

```
## [1] 50  5
```

```r
head(f1)
```

---

## filter(), continued

**Exercise**: create a data set consisting only of `setosa` and `virginica` species.

```r
f2 <- filter(d, Species %in% c("setosa", "virginica")) # %in%
dim(f2)
```

```
## [1] 100   5
```

```r
tail(f2)
```

```
##     Sepal.Length Sepal.Width Petal.Length Petal.Width   Species
## 95           6.7         3.3          5.7         2.5 virginica
## 96           6.7         3.0          5.2         2.3 virginica
## 97           6.3         2.5          5.0         1.9 virginica
## 98           6.5         3.0          5.2         2.0 virginica
## 99           6.2         3.4          5.4         2.3 virginica
## 100          5.9         3.0          5.1         1.8 virginica
```

```r
f3 <- filter(d, Species != "versicolor") # != means "not equal"
dim(f3)
```

```
## [1] 100   5
```

---

## filter(), continued

**Exercise**: create a data set with `Sepal.Length` < 6.

```r
f4 <- filter(d, Sepal.Length < 6) 
dim(f4)
```

```
## [1] 83  5
```

```r
head(f4)
```

---

## filter(), continued

**Exercise**: how many `virginica` species have `Sepal.Length` at least 5.7?

```r
f5 <- filter(d, Sepal.Length >= 5.7) # >= "greater than or equal to"
```

```r
table(f5$Species)
```

```
## 
##     setosa versicolor  virginica 
##          3         34         48
```

```r
Sep.Len.Cut <- 5.7
f6 <- filter(d, Sepal.Length >= Sep.Len.Cut) 
table(f6$Species)
```

```
## 
##     setosa versicolor  virginica 
##          3         34         48
```

---

## mutate()

**Example**: create a new variable `Sepal.Sum` which is defined as `Sepal.Length` + `Sepal.Width`.

```r
m1 <- mutate(d, Sepal.Sum = Sepal.Length + Sepal.Width)
dim(m1)
```

```
## [1] 150   6
```

```r
head(m1)
```

```
##   Sepal.Length Sepal.Width Petal.Length Petal.Width Species Sepal.Sum
## 1          5.1         3.5          1.4         0.2  setosa       8.6
## 2          4.9         3.0          1.4         0.2  setosa       7.9
## 3          4.7         3.2          1.3         0.2  setosa       7.9
## 4          4.6         3.1          1.5         0.2  setosa       7.7
## 5          5.0         3.6          1.4         0.2  setosa       8.6
## 6          5.4         3.9          1.7         0.4  setosa       9.3
```

---

## mutate(), continued

**Exercise**: create a new variable `size` which is defined as "big" if `Sepal.Length` `$\geq$` 5 and "small" if `Sepal.Length` `$<$` 5 (_Hint_: you might want to look up the function `ifelse()`).

```r
m2 <- mutate(d, size = ifelse(Sepal.Length >= 5, "big", "small"))
dim(m2)
```

```
## [1] 150   6
```

```r
head(m2)
```

```
##   Sepal.Length Sepal.Width Petal.Length Petal.Width Species  size
## 1          5.1         3.5          1.4         0.2  setosa   big
## 2          4.9         3.0          1.4         0.2  setosa small
## 3          4.7         3.2          1.3         0.2  setosa small
## 4          4.6         3.1          1.5         0.2  setosa small
## 5          5.0         3.6          1.4         0.2  setosa   big
## 6          5.4         3.9          1.7         0.4  setosa   big
```

---

## arrange()

**Example**: sort `d` by `Sepal.Length` (ascending order)

```r
a1 <- arrange(d, Sepal.Length)
dim(a1)
```

```
## [1] 150   5
```

```r
head(a1)
```

```
##   Sepal.Length Sepal.Width Petal.Length Petal.Width Species
## 1          4.3         3.0          1.1         0.1  setosa
## 2          4.4         2.9          1.4         0.2  setosa
## 3          4.4         3.0          1.3         0.2  setosa
## 4          4.4         3.2          1.3         0.2  setosa
## 5          4.5         2.3          1.3         0.3  setosa
## 6          4.6         3.1          1.5         0.2  setosa
```

---

## arrange(), continued

**Example**: sort `d` by `Sepal.Length` (descending order)

```r
a2 <- arrange(d, desc(Sepal.Length))
dim(a2)
```

```
## [1] 150   5
```

```r
head(a2)
```

```
##   Sepal.Length Sepal.Width Petal.Length Petal.Width   Species
## 1          7.9         3.8          6.4         2.0 virginica
## 2          7.7         3.8          6.7         2.2 virginica
## 3          7.7         2.6          6.9         2.3 virginica
## 4          7.7         2.8          6.7         2.0 virginica
## 5          7.7         3.0          6.1         2.3 virginica
## 6          7.6         3.0          6.6         2.1 virginica
```

---

## Piping

**Exercise**: report all `setosa` elements with the difference `Diff` defined by `Diff` =  `Sepal.Length` - `Sepal.Width` > 1.5. Show only `Sepal.Length`, `Sepal.Width` and `Diff`, and sort by descending order of `Diff`.

```r
r1 <- mutate(d, Diff = Sepal.Length - Sepal.Width)
```

```r
r2 <- filter(r1, Diff > 1.5, Species == "setosa")
```

```r
r3 <- select(r2, Sepal.Length, Sepal.Width, Diff)
```

```r
r4 <- arrange(r3, desc(Diff))
head(r4)
```

```
##   Sepal.Length Sepal.Width Diff
## 1          4.5         2.3  2.2
## 2          5.4         3.4  2.0
## 3          5.4         3.4  2.0
## 4          5.0         3.0  2.0
## 5          5.5         3.5  2.0
## 6          4.9         3.0  1.9
```

---

## Piping, continued

Piping operator `%>%` takes the LHS of `%>%` as the first argument of the RHS of `%>%`.

```r
r <- d %>% 
  mutate(Diff = Sepal.Length - Sepal.Width) %>%
  filter(Diff > 1.5, Species == "setosa") %>%
  select(Sepal.Length, Sepal.Width, Diff) %>%
  arrange(desc(Diff))
head(r)
```

---

## group_by() / summarize()

**Exercise**: find the mean `Sepal.Length` for each `Species`.

```r
setosa_df <- iris %>% 
  filter(Species == "setosa")
mean(setosa_df$Sepal.Length)
```

```
## [1] 5.006
```

```r
versi_df <- iris %>% 
  filter(Species == "versicolor")
mean(versi_df$Sepal.Length)
```

```
## [1] 5.936
```

```r
virgi_df <- iris %>% 
  filter(Species == "virginica")
mean(virgi_df$Sepal.Length)
```

```
## [1] 6.588
```

---

## group_by() / summarize()

`group_by()` and `summarize()` combination: summary of data (e.g. mean, max, min, median) by each category of a categorical variable.

```r
iris %>% group_by(Species) %>%
  summarize(SL.mean = mean(Sepal.Length))
```

```
## # A tibble: 3 x 2
##   Species    SL.mean
##   <fct>        <dbl>
## 1 setosa        5.01
## 2 versicolor    5.94
## 3 virginica     6.59
```

---

## group_by() / summarize()

**Exercise**: What is the maximum (and minimum) of `Sepal.Length` in each `Species`?

```r
iris %>% group_by(Species) %>%
  summarize(SL.max = max(Sepal.Length), SL.min= min(Sepal.Length))
```

```
## # A tibble: 3 x 3
##   Species    SL.max SL.min
##   <fct>       <dbl>  <dbl>
## 1 setosa        5.8    4.3
## 2 versicolor    7      4.9
## 3 virginica     7.9    4.9
```

---

## Wide and long form of data

Calories taken by four patients on a certain day:

```r
cal_wide <- data.frame(subject = paste0("Patient", 1:4),
                      breakfast = c(300, 350, 300, 250),
                      lunch = c(500, 500, 500, 450),
                      dinner = c(600, 650, 450, 500))
print(cal_wide)
```

```
##    subject breakfast lunch dinner
## 1 Patient1       300   500    600
## 2 Patient2       350   500    650
## 3 Patient3       300   500    450
## 4 Patient4       250   450    500
```

---

## Wide and long form of data

At times, _long_ format of data is more convenient to deal with:

`$$\begin{array}{c|c|c} \text{subject} & \text{meal} & \text{calorie} \\ \hline   
\text{Patient1} & \text{breakfast} & 300   \\  
\text{Patient1} & \text{lunch} & 500  \\
\text{Patient1} & \text{dinner} & 600  \\
\text{Patient2} & \text{breakfast} & 350   \\  
\text{Patient2} & \text{lunch} & 500  \\
\text{Patient2} & \text{dinner} & 650  \\
\vdots & \vdots & \vdots
\end{array}$$`

---

## Wide and long form of data

`gather()` transforms wide format into long format:

```r
cal_long <- cal_wide %>%
  gather(key = meal, value = calorie, breakfast, lunch, dinner) %>%
  arrange(subject)

head(cal_long)
```

```
##    subject      meal calorie
## 1 Patient1 breakfast     300
## 2 Patient1     lunch     500
## 3 Patient1    dinner     600
## 4 Patient2 breakfast     350
## 5 Patient2     lunch     500
## 6 Patient2    dinner     650
```

---

## Wide and long form of data

`gather()` can be used in a way excluding column(s) not to be transformed into `key`:

```r
cal_long_2 <- cal_wide %>%
  gather(key = meal, value = calorie, -subject) %>%
  arrange(subject)

head(cal_long_2)
```

---

## Wide and long form of data

`spread()` transforms long format to wide format:

```r
cal_wide_2 <- cal_long %>%
  spread(key = meal, value = calorie)

head(cal_wide_2)
```

```
##    subject breakfast dinner lunch
## 1 Patient1       300    600   500
## 2 Patient2       350    650   500
## 3 Patient3       300    450   500
## 4 Patient4       250    500   450
```

---

## Box plot

**Example**: describe the distribution of `Sepal.Length` using box plot.

```r
g <- ggplot(d, aes(x = Species, y = Sepal.Length)) +
  geom_boxplot()
print(g)
```

---

## Box plot, continued

- The thick line indicates the median.
- The lower and upper hinges correspond to the 25% and 75% quantiles. 
- The upper whisker extends from the hinge to the largest value no further than 1.5 `$\times$` IQR from the hinge, where IQR is the inter-quartile range (distance between the 25% and 75% quantiles). Similarly for the lower whisker.

**Exercise**: compute the maximum, minimum, 25% quantile, 75% quantile and median for `versicola` species and compare with the previous plot.

```r
ds <- d %>%
  filter(Species == "versicolor")
quantile(ds$Sepal.Length)
```

```
##   0%  25%  50%  75% 100% 
##  4.9  5.6  5.9  6.3  7.0
```

---

## Box plot, continued

**Exercise**: create the following plots.

---

## Scatter plots

How are `Sepal.Length` and `Sepal.Width` associated?

```r
g <- ggplot(d, aes(x = Sepal.Length, y = Sepal.Width)) +
  geom_point()
print(g)
```

---

## Scatter plots, continued

```r
g <- ggplot(d, aes(Sepal.Length, Sepal.Width, color = Species)) +
  geom_point()
print(g)
```

---

## Scatter plots, continued

```r
g2 <- g + theme(aspect.ratio = 1)
print(g2)
```