Lecture 3

class: center, middle, inverse, title-slide

# Lecture 3
## Data manipulation
### Jung-Jin Lee
### Jan 26, 2021

---

## Exploring data

```r
d <- iris # iris data set comes with base R
dim(d)
```

```
## [1] 150   5
```

So `d` has 150 observations and 5 variables.

```r
head(d)
```

```
##   Sepal.Length Sepal.Width Petal.Length Petal.Width Species
## 1          5.1         3.5          1.4         0.2  setosa
## 2          4.9         3.0          1.4         0.2  setosa
## 3          4.7         3.2          1.3         0.2  setosa
## 4          4.6         3.1          1.5         0.2  setosa
## 5          5.0         3.6          1.4         0.2  setosa
## 6          5.4         3.9          1.7         0.4  setosa
```

---

## Exploring data, continued

```r
table(d$Species) 
```

```
## 
##     setosa versicolor  virginica 
##         50         50         50
```

.pull-left[
<p align = "center">
  <img src = iris_species.jpeg>
</p>
]
.pull-right[
<p align = "center">
  <img src = iris_petal_sepal.png width = "180">
</p>  
]

---

## Exploring data, continued

```r
g <- ggplot(d, aes(Sepal.Length)) + geom_histogram()
print(g)
```

---

## Data manipulation

Reshaping data is an essenstial part of data analysis. We will use a package called `dplyr`, which is a part of `tidyverse`.

```r
library(dplyr) 
# you could run library(tidyverse) instead
```

`dplyr` provides the following _verbs_ that are useful in data manipulation:
- `select()`: subset using columns (variables)
- `filter()`: subset using rows (observations)
- `mutate()`: transform variables
- `arrange()`: reorder observations

These verbs, when combined with the pipe operator `%>%`, become even more powerful!

---

## Piping

For a function `f`, the expression `x %>% f(y, z)` is the same as `f(x, y, z)`.

```r
x.in <- c(1, 3, 6, 2, 5, 4) 
*x.in %>% sum() # this is same as sum(x.in)
```

```
## [1] 21
```

```r
x.in %>% sort()
```

```
## [1] 1 2 3 4 5 6
```

```r
x.in %>% sort(decreasing = T)
```

```
## [1] 6 5 4 3 2 1
```

Piping is even more useful when it is used as a chain.

---

## Piping, continued

**Example**: compute `$\sqrt{\log 1} + \sqrt{\log 2} + \sqrt{\log 3} + \cdots + \sqrt{\log 100}$`.

```r
num <- 1:100
*log.num <- log(num) # log() computes the natural logarithm
*sqrt.log.num <- sqrt(log.num) # sqrt() computes the square root
sum(sqrt.log.num)
```

```
## [1] 188.104
```

This can be done in the following ways too:

```r
sum(sqrt(log(num))) # repeated application of functions
```

```
## [1] 188.104
```

```r
num %>% log() %>% sqrt() %>% sum() # chain reaction!
```

```
## [1] 188.104
```

---

## select()

**Example**: create a data set consisting only of `Sepal.Length` and `Sepal.Width`.

```r
*s1 <- select(d, Sepal.Length, Sepal.Width)
dim(s1)
```

```
## [1] 150   2
```

```r
head(s1)
```

```
##   Sepal.Length Sepal.Width
## 1          5.1         3.5
## 2          4.9         3.0
## 3          4.7         3.2
## 4          4.6         3.1
## 5          5.0         3.6
## 6          5.4         3.9
```

---
 
## select(), continued

**Exercise**: create a data set consisting of `Sepal.Length`, `Sepal.Width`, `Petal.Length` and `Petal.Width`.

```r
s2 <- select(d, Sepal.Length, Sepal.Width, Petal.Length, Petal.Width)
dim(s2)
```

```
## [1] 150   4
```

```r
head(s2)
```

```
##   Sepal.Length Sepal.Width Petal.Length Petal.Width
## 1          5.1         3.5          1.4         0.2
## 2          4.9         3.0          1.4         0.2
## 3          4.7         3.2          1.3         0.2
## 4          4.6         3.1          1.5         0.2
## 5          5.0         3.6          1.4         0.2
## 6          5.4         3.9          1.7         0.4
```

---

## select(), continued

One can specify columns to be excluded:

```r
*s3 <- select(d, -Species)
head(s3)
```

---

## filter()

**Example**: create a data set consisting only of `setosa` species.

```r
*f1 <- filter(d, Species == "setosa") # note use of ==, not =
dim(f1)
```

```
## [1] 50  5
```

```r
head(f1)
```

---

## filter(), continued

**Exercise**: create a data set consisting only of `setosa` and `virginica` species.

```r
f2 <- filter(d, Species %in% c("setosa", "virginica")); dim(f2)
```

```
## [1] 100   5
```

```r
tail(f2)
```

```
##     Sepal.Length Sepal.Width Petal.Length Petal.Width   Species
## 95           6.7         3.3          5.7         2.5 virginica
## 96           6.7         3.0          5.2         2.3 virginica
## 97           6.3         2.5          5.0         1.9 virginica
## 98           6.5         3.0          5.2         2.0 virginica
## 99           6.2         3.4          5.4         2.3 virginica
## 100          5.9         3.0          5.1         1.8 virginica
```

```r
f3 <- filter(d, Species != "versicolor"); dim(f3)
```

```
## [1] 100   5
```

---

## filter(), continued

**Exercise**: create a data set with `Sepal.Length` < 6.

```r
f4 <- filter(d, Sepal.Length < 6)
dim(f4)
```

```
## [1] 83  5
```

```r
head(f4)
```

---

## filter(), continued

Multiple conditions can be supplied to `filter()`:

**Exercise**: how many `virginica` species have `Sepal.Length` at least 5.7?

```r
# multiple conditions (comma separated) can be used in filter()
f5 <- filter(d, Sepal.Length >= 5.7, Species == "virginica")
dim(f5)
```

```
## [1] 48  5
```

---

## mutate()

**Example**: create a new variable `Sepal.Sum` which is defined as `Sepal.Length` + `Sepal.Width`.

```r
*m1 <- mutate(d, Sepal.Sum = Sepal.Length + Sepal.Width)
dim(m1)
```

```
## [1] 150   6
```

```r
head(m1)
```

```
##   Sepal.Length Sepal.Width Petal.Length Petal.Width Species Sepal.Sum
## 1          5.1         3.5          1.4         0.2  setosa       8.6
## 2          4.9         3.0          1.4         0.2  setosa       7.9
## 3          4.7         3.2          1.3         0.2  setosa       7.9
## 4          4.6         3.1          1.5         0.2  setosa       7.7
## 5          5.0         3.6          1.4         0.2  setosa       8.6
## 6          5.4         3.9          1.7         0.4  setosa       9.3
```

---

## mutate(), continued

**Exercise**: create a new variable `size` which is defined as "big" if `Sepal.Length` `$\geq$` 5 and "small" if `Sepal.Length` `$<$` 5 (_Hint_: you might want to look up the function `ifelse()`).

```r
*m2 <- mutate(d, size = ifelse(Sepal.Length >= 5, "big", "small"))
dim(m2)
```

```
## [1] 150   6
```

```r
head(m2)
```

```
##   Sepal.Length Sepal.Width Petal.Length Petal.Width Species  size
## 1          5.1         3.5          1.4         0.2  setosa   big
## 2          4.9         3.0          1.4         0.2  setosa small
## 3          4.7         3.2          1.3         0.2  setosa small
## 4          4.6         3.1          1.5         0.2  setosa small
## 5          5.0         3.6          1.4         0.2  setosa   big
## 6          5.4         3.9          1.7         0.4  setosa   big
```

---

## mutate(), continued

**Exercise**: create a new variable `ID` which gives the identifier for each observation such as `obs1`, `obs2`, `$\ldots$`, `obs150`.

```r
m3 <- mutate(d, ID = paste0("obs", 1:nrow(d))); head(m3)
```

```
##   Sepal.Length Sepal.Width Petal.Length Petal.Width Species   ID
## 1          5.1         3.5          1.4         0.2  setosa obs1
## 2          4.9         3.0          1.4         0.2  setosa obs2
## 3          4.7         3.2          1.3         0.2  setosa obs3
## 4          4.6         3.1          1.5         0.2  setosa obs4
## 5          5.0         3.6          1.4         0.2  setosa obs5
## 6          5.4         3.9          1.7         0.4  setosa obs6
```

---

## arrange()

**Example**: sort `d` by `Sepal.Length` (ascending order)

```r
*a1 <- arrange(d, Sepal.Length)
dim(a1)
```

```
## [1] 150   5
```

```r
head(a1)
```

```
##   Sepal.Length Sepal.Width Petal.Length Petal.Width Species
## 1          4.3         3.0          1.1         0.1  setosa
## 2          4.4         2.9          1.4         0.2  setosa
## 3          4.4         3.0          1.3         0.2  setosa
## 4          4.4         3.2          1.3         0.2  setosa
## 5          4.5         2.3          1.3         0.3  setosa
## 6          4.6         3.1          1.5         0.2  setosa
```

---

## arrange(), continued

**Example**: sort `d` by `Sepal.Length` (descending order)

```r
*a2 <- arrange(d, desc(Sepal.Length))
dim(a2)
```

```
## [1] 150   5
```

```r
head(a2)
```

```
##   Sepal.Length Sepal.Width Petal.Length Petal.Width   Species
## 1          7.9         3.8          6.4         2.0 virginica
## 2          7.7         3.8          6.7         2.2 virginica
## 3          7.7         2.6          6.9         2.3 virginica
## 4          7.7         2.8          6.7         2.0 virginica
## 5          7.7         3.0          6.1         2.3 virginica
## 6          7.6         3.0          6.6         2.1 virginica
```

---

## Wrap-up

**Exercise**: report all `setosa` elements with the difference `Diff` defined by `Diff` =  `Sepal.Length` - `Sepal.Width` > 1.5. Show only `Sepal.Length`, `Sepal.Width` and `Diff`, and sort by descending order of `Diff`.

```r
r1 <- mutate(d, Diff = Sepal.Length - Sepal.Width)
```

```r
r2 <- filter(r1, Diff > 1.5, Species == "setosa")
```

```r
r3 <- select(r2, Sepal.Length, Sepal.Width, Diff)
```

```r
r4 <- arrange(r3, desc(Diff)); head(r4)
```

```
##   Sepal.Length Sepal.Width Diff
## 1          4.5         2.3  2.2
## 2          5.4         3.4  2.0
## 3          5.4         3.4  2.0
## 4          5.0         3.0  2.0
## 5          5.5         3.5  2.0
## 6          4.9         3.0  1.9
```

---

## Wrap-up, continued

Piping operator `%>%` enhances readability.

```r
r5 <- d %>% 
  mutate(Diff = Sepal.Length - Sepal.Width) %>%
  filter(Diff > 1.5, Species == "setosa") %>%
  select(Sepal.Length, Sepal.Width, Diff) %>%
  arrange(desc(Diff))
head(r5)
```

---

## Dot(.) notation

In a chain of piping, dot(.) represents the input for the current pipe operator:

```r
r6 <- d %>%
  select(Species, Petal.Width) %>%
* mutate(ID = paste0("obs", 1:nrow(.)))
head(r6)
```

```
##   Species Petal.Width   ID
## 1  setosa         0.2 obs1
## 2  setosa         0.2 obs2
## 3  setosa         0.2 obs3
## 4  setosa         0.2 obs4
## 5  setosa         0.2 obs5
## 6  setosa         0.4 obs6
```

---

## Other useful functions

- `count()`: gives a frequency table:

```r
*d %>% count(Species)
```

```
##      Species  n
## 1     setosa 50
## 2 versicolor 50
## 3  virginica 50
```

- `pull()`: extract a column (acts like `$`):

```r
*d %>% pull(Sepal.Length) %>% head()
```

```
## [1] 5.1 4.9 4.7 4.6 5.0 5.4
```

---

## Other useful functions, continued

- `slice()`: exctract selected rows:

```r
d %>% select(Species, Petal.Width) %>%
  mutate(ID = paste0("obs", 1:nrow(.))) %>%
* slice(5:8) # rows 5 through 8
```

```
##   Species Petal.Width   ID
## 1  setosa         0.2 obs5
## 2  setosa         0.4 obs6
## 3  setosa         0.3 obs7
## 4  setosa         0.2 obs8
```

- `top_n()`: top `n` elements for a given `n` and a variable:

```r
*d %>% top_n(3, Sepal.Width)
```

```
##   Sepal.Length Sepal.Width Petal.Length Petal.Width Species
## 1          5.7         4.4          1.5         0.4  setosa
## 2          5.2         4.1          1.5         0.1  setosa
## 3          5.5         4.2          1.4         0.2  setosa
```

---

## Wide and long form of data

**Example**: calories taken by four patients on a certain day are given below:

```r
cal_wide <- data.frame(subject = paste0("Patient", 1:4),
                      breakfast = c(300, 350, 300, 250),
                      lunch = c(500, 500, 500, 450),
                      dinner = c(600, 650, 450, 500))
print(cal_wide)
```

```
##    subject breakfast lunch dinner
## 1 Patient1       300   500    600
## 2 Patient2       350   500    650
## 3 Patient3       300   500    450
## 4 Patient4       250   450    500
```

Make a box plot to compare breakfast/lunch/dinner in terms of calories taken by patients.

---