3_DataStructure.qmd

---
title: "3. Data Structure"
author: "Yi-Ju Tseng"
format:
  revealjs:
    slide-number: c/t
    show-slide-number: all
editor: visual
---

## Common Data Structure 資料結構

A particular way of organizing data in a computer

::::: columns
::: {.column width="50%"}
-   序列 (sequence)
    -   表 (list)
    -   定值表 (tuple)
    -   範圍 (range)
-   映射 (mapping)
    -   字典 (dist)
:::

::: {.column width="50%"}
-   矩陣 (matrix, array)
    -   一般矩陣
    -   numpy array
-   資料框 (data frame)
    -   pandas 資料框
:::
:::::

## Common Data Structure 資料結構

::::: columns
::: {.column width="50%"}
-   **序列 (sequence)**
    -   表 (list)
    -   定值表 (tuple)
    -   範圍 (range)
-   映射 (mapping)
    -   字典 (dist)
:::

::: {.column width="50%"}
-   矩陣 (matrix, array)
    -   一般矩陣
    -   numpy array
-   資料框 (data frame)
    -   pandas 資料框
:::
:::::

# 序列 (sequence)

## 表 (list)

A collection allows us to put many values in a single “variable”

-   surrounded by square brackets`[ ]`
-   separated by commas`,`
-   element can be *any* Python object
-   can be empty
-   可作運算

```{python}
#| echo: true
list1 = [3.5, 4.6, 5.7]
print(list1)
```

## list and for loop

依序將list結構的內容取出，設定為x，並印出來

```{python}
#| echo: true
friends = ['Joseph', 'Glenn', 'Sally']
for x in friends:
    print('Happy New Year:',  x)
print('Done!')
```

## list 取值

-   sequence包含：list, tuple, range
-   使用中括號加上index取值`變數名稱[index]`
-   `index`從0開始

| 位置  |  1  |  2  |  3  |  4  | ... |
|:-----:|:---:|:---:|:---:|:---:|:---:|
| index |  0  |  1  |  2  |  3  | ... |

```{python}
#| echo: true
print(friends)
```

```{python}
#| echo: true
print(friends[0])
```

```{python}
#| echo: true
print(friends[1])
```

```{python}
#| echo: true
print(friends[2])
```

## list 取值 - slice

`start`:`up to but not including`

```{python}
#| echo: true
print(friends)
```

```{python}
#| echo: true
print(friends[1:2])
```

```{python}
#| echo: true
print(friends[:2])
```

```{python}
#| echo: true
print(friends[:1])
```

## 範圍 (range)

-   使用`range(數列起點,數列終點[不包含],間隔)`宣告
-   若只有輸入一個參數，視為數列終點，起點設為預設值0
-   若只有輸入兩個參數，則間隔設為預設值1

```{python}
#| echo: true
r1 = range(10)
print(r1)
```

```{python}
#| echo: true
r2 = range(10,20)
print(r2)
```

```{python}
#| echo: true
r3 = range(10,20,2)
print(r3)
```

## 範圍 (range)

::::: columns
::: column
```{python}
print(r1)
```

```{python}
#| echo: true
for i in r1:
  print(i)
```
:::

::: column
```{python}
print(r3)
```

```{python}
#| echo: true
for i in r3:
  print(i)
```
:::
:::::

## list and for loop - range

```{python}
#| echo: true
friends = ['Joseph', 'Glenn', 'Sally']
print(len(friends))
```

```{python}
#| echo: true
print(range(len(friends)))
```

::::: columns
::: column
```{python}
#| echo: true
for x in friends:
    print('Happy New Year:',  x)
print('Done!')
```
:::

::: column
```{python}
#| echo: true
for i in range(len(friends)) :
    friend = friends[i]
    print(friend)
print('Done!')
```
:::
:::::

## list 新增修改

-   Lists are “mutable”
    -   we can change an element of a list using the index operator

```{python}
#| echo: true
print(friends)
```

```{python}
#| echo: true
friends[0]='new friend'
print(friends)
```

## list 新增修改 +

We can create a new list by adding `+` two existing lists together

```{python}
#| echo: true
friend_cs = ['Joseph', 'Glenn', 'Sally']
friend_md = ['Alex', 'John', 'Ray']
print(friend_cs+friend_md)
```

## list 新增修改 append

We can create an empty list and then add elements using the `append` method

```{python}
#| echo: true
print(friend_cs)
```

```{python}
#| echo: true
friend_cs.append('Megan')
print(friend_cs)
```

## list的運算功能

```{python}
#| echo: true
list1 = [3.5, 4.6, 5.7]
print(list1)
```

`sum(`list物件名稱`)` 物件加總

```{python}
#| echo: true
sum(list1)
```

`max(`list物件名稱`)` 物件最大值

```{python}
#| echo: true
list1 = [3.5, 4.6, 5.7]
max(list1)
```

可參考[文件](https://docs.python.org/zh-cn/3/library/functions.html)

## 定值表 (tuple)

-   Like `list`...
-   surrounded by square brackets`( )`
-   separated by commas`,`
-   Tuples are “immutable” --\> *More Efficient*
    -   No `.append()`, `.sort()`, etc

```{python}
#| echo: true
tuple1 = (3.5, 4.6, 5.7, "111", "222")
print(tuple1)
```

## 定值表 (tuple) assignment

We can also put a tuple on the left-hand side of an assignment statement

```{python}
#| echo: true
(x, y) = (4, 'fred')
print(y)
```

## Hands-on

-   新增一sequence `a`，包含數字1到10
-   新增一sequence `b`，包含數字1到20中的所有偶數
-   取出`a` sequence 的第*4*個值

## Common Data Structure 資料結構

::::: columns
::: {.column width="50%"}
-   序列 (sequence)
    -   表 (list)
    -   定值表 (tuple)
    -   範圍 (range)
-   **映射 (mapping)**
    -   字典 (dist)
:::

::: {.column width="50%"}
-   矩陣 (matrix, array)
    -   一般矩陣
    -   numpy array
-   資料框 (data frame)
    -   pandas 資料框
:::
:::::

# 映射 (mapping)

## 字典 Dictionaries (dist)

-   Dictionaries are Python’s most powerful data collection
-   使用大括號 `{ }`宣告
-   內容為`key : value`的組合，並以`,`分隔
-   key不能重複，通常為字串

```{python}
#| echo: true
dist1 = {"id":1, "name":"Ryan","age":20, "School":"NYCU"}
print(dist1)
```

## mapping 取值 \[ \]

-   mapping包含：dist
-   使用中括號加上key取值`變數名稱[key]`

```{python}
#| echo: true
print(dist1)
```

```{python}
#| echo: true
print(dist1['age'])
```

## mapping 取值 .keys()

\[key1, key2, ...\]

```{python}
#| echo: true
print(dist1)
```

```{python}
#| echo: true
print(dist1.keys())
```

## mapping 取值 .values()

\[value1, value2, ...\]

```{python}
#| echo: true
print(dist1)
```

```{python}
#| echo: true
print(dist1.values())
```

## mapping 取值 .items()

\[(key1, value1), (key2, value2),...\]

```{python}
#| echo: true
print(dist1)
```

```{python}
#| echo: true
print(dist1.items())
```

## mapping 新增修改 .update()

-   mapping包含：dist
-   `mapping物件.update()`可新增或修改內容(key:value pairs)至mapping中

```{python}
#| echo: true
print(dist1)
```

```{python}
#| echo: true
dist1.update({"age":25, "dept":"CS"})
print(dist1)
```

## mapping 新增修改 \[ \]

`[ ]`取值後寫回

```{python}
#| echo: true
print(dist1)
```

```{python}
#| echo: true
dist1['age']=dist1['age']+10
print(dist1)
```

## mapping 新增修改 .get()

-   `.get(key, 0)` 取值
    -   key: key
    -   0: any default value

```{python}
#| echo: true
print(dist1)
```

```{python}
#| echo: true
dist1['age']=dist1.get('age', 0)+10
print(dist1)
```

## Hands-on

-   新增一mapping `dist1`，內容為

```{python}
#| echo: true
dist1 = {"id":[1,2,3,4], 
        "name":["Ryan","Tom","Emma","Amy"], 
        "School":"NYCU"}
```

-   取出`id`中的第*3*個值
-   取出`name`中的第*2*個值的第1個字元（字母）

## mapping + for

-   mapping結構（dist）預設取得部份為key
-   使用`.values()`即可取value (值)

```{python}
#| echo: true
print(dist1)
```

::::: columns
::: column
Key

```{python}
#| echo: true
for i in dist1:
  print(i)
```
:::

::: column
Value

```{python}
#| echo: true
for i in dist1.values():
  print(i)
```
:::
:::::

## mapping + for - key and value

```{python}
#| echo: true
print(dist1)
```

先得到key，再取得value

```{python}
#| echo: true
for key in dist1:
  print(key, dist1[key])
```

或是直接使用`.items()`

```{python}
#| echo: true
for key,value in dist1.items():
  print(key, value)
```

## Hands-on

Most Common Name? Edit the ???? parts

```{python}
#| echo: true
names = ['csev', 'cwen', 'csev', 'zqian', 'cwen']
```

Use dist and for to get the most common name and its count

::::: columns
::: column
```{python}
#| echo: true
#| eval: false
names = ['csev', 'cwen', 'csev', 'zqian', 'cwen']
counts = dict()
for name in ??? : # edit here
    if name not in counts: 
      counts[name] = ??? # edit here
    else :
      ??? # edit here
print(counts)
```
:::

::: column
```{python}
#| echo: true
#| eval: false
names = ['csev', 'cwen', 'csev', 'zqian', 'cwen']
counts = dict()
for name in ??? : # edit here
    counts[name] = ??? + 1 # edit here
print(counts)
```
:::
:::::

## Common Data Structure 資料結構

::::: columns
::: {.column width="50%"}
-   序列 (sequence)
    -   表 (list)
    -   定值表 (tuple)
    -   範圍 (range)
-   映射 (mapping)
    -   字典 (dist)
:::

::: {.column width="50%"}
-   **矩陣 (matrix, array)**
    -   一般矩陣
    -   numpy array
-   資料框 (data frame)
    -   pandas 資料框
:::
:::::

# 矩陣 (matrix, array)

## 一般矩陣 (matrix, array)

-   可想成sequence (1維矩陣)的堆疊
-   sequence為row的組成
-   直接以中括號分隔matrix的row和column

| Col1 | Col2 | Col3 |
|:----:|:----:|:----:|
|  1   |  2   |  3   |
|  4   |  5   |  6   |

::::: columns
::: column
```{python}
#| echo: true
matrix1 = [[1, 2, 3], 
          [4, 5, 6]]
print(matrix1)
```
:::

::: column
```{python}
#| echo: true
row1 = [1, 2, 3]
row2 = [4, 5, 6]
matrix2 = [row1, row2]
print(matrix2)
```
:::
:::::

## numpy array - 1維

-   from `numpy` library (`Numerical Python`)
-   like `list` type, but arrays provide much more efficient storage and data operations (fixed-type)
-   Use `.array(list)` to create numpy array from list

```{python}
#| echo: true
import numpy as np
a = np.array([0, 0.5, 1.0, 1.5, 2.0]) 
list_string = ['a', 'b', 'c']
b = np.array(list_string) 
print([a,b])
```

## numpy array - 1維

或是用`.arrange(起點,終點[不包含],間隔)`來生成numpy array，如同`range()`的用法

```{python}
#| echo: true
c = np.arange(0, 10, 2) 
print(c)
```

## numpy array - 2維

-   可用兩個sequence組合(一般2維矩陣)新增numpy array
-   預設以row為維度堆疊

```{python}
#| echo: true
two_d = np.array([row1,row2]) 
print(two_d)
```

## numpy array - Creating Arrays from Scratch

::::: columns
::: column
-   `np.full(dim, value)`: repeated `value`s
-   2d dimension: (row, column)

```{python}
#| echo: true
np.full((2, 4), 3.14)
```
:::

::: column
-   `np.random.random(dim)`
-   `np.random.normal(mean,sd,dim)`

```{python}
#| echo: true
np.random.random((2,3))
```

```{python}
#| echo: true
np.random.normal(0,1,(2,3))
```
:::
:::::

## numpy array - 取值

Will be covered in Ch 5

## numpy array - 新增修改

Will be covered in Ch 5

## numpy array - 計算功能

`sum()`、`mean()`、`std()`、`cumsum()`、`max()`、`min()`、`count()`

::::: columns
::: column
```{python}
#| echo: true
print(a)
```

```{python}
#| echo: true
print(a.sum())
```

```{python}
#| echo: true
print(a.mean())
```

```{python}
#| echo: true
print(a.std())
```

```{python}
#| echo: true
print(a.cumsum())
```
:::

::: column
```{python}
#| echo: true
print(two_d)
```

```{python}
#| echo: true
print(two_d.sum()) 
```

```{python}
#| echo: true
print(two_d.mean()) 
```

```{python}
#| echo: true
print(two_d.std()) 
```

```{python}
#| echo: true
print(two_d.cumsum()) 
```
:::
:::::

## numpy array - 計算功能

`sum()`、`mean()`、`std()`、`cumsum()`、`max()`、`min()`、`count()`

`axis`= 0 BY COLUMN, 1 BY ROW

::::: columns
::: column
```{python}
#| echo: true
print(two_d)
```

```{python}
#| echo: true
print(two_d.sum(axis=0))
```

```{python}
#| echo: true
print(two_d.sum(axis=1))
```
:::

::: column
```{python}
#| echo: true
print(two_d.mean(axis=0))
```

```{python}
#| echo: true
print(two_d.mean(axis=1))
```
:::
:::::

## Hands-on

-   The first student (index) has wrong score, the correct score is 73. Please correct the score
-   Please compute the average score and determine which students passed (scores above a certain threshold)

```{python}
#| echo: true
import numpy as np
scores = np.array([75, 82, 90, 65, 88, 55, 66, 77, 44, 100])
```

## Common Data Structure 資料結構

::::: columns
::: {.column width="50%"}
-   序列 (sequence)
    -   表 (list)
    -   定值表 (tuple)
    -   範圍 (range)
-   映射 (mapping)
    -   字典 (dist)
:::

::: {.column width="50%"}
-   矩陣 (matrix, array)
    -   一般矩陣
    -   numpy array
-   **資料框 (data frame)**
    -   pandas 資料框
:::
:::::

# pandas

Series, DataFrame, and Index

## pandas Series

-   使用前須載入`pandas` library
-   one-dimensional array of indexed data
-   `pd.Series(list)`

```{python}
#| echo: true
import pandas as pd
data = pd.Series([0.25, 0.5, 0.75, 1.0])
print(data)
```

```{python}
#| echo: true
print(data[1])
```

## pandas Series vs. numpy 1d array

index

```{python}
#| echo: true
data = pd.Series([0.25, 0.5, 0.75, 1.0],
                 index=['a', 'b', 'c', 'd'])
print(data)
```

```{python}
#| echo: true
print(data['b'])
```

## pandas Series from dictionary

`pd.Series(dictionary)`

```{python}
#| echo: true
population_dict = {'California': 38332521,
                   'Texas': 26448193,
                   'New York': 19651127,
                   'Florida': 19552860,
                   'Illinois': 12882135}
population = pd.Series(population_dict)
print(population)
```

## pandas DataFrame 資料框

-   a two-dimensional array with both flexible row indices and flexible *column names*
-   a sequence of aligned Series objects
-   heterogeneous types and/or missing data

```{python}
#| echo: true
df1 = pd.DataFrame({"ID":[1, 2, 3, 4],
                    "Name":["Tom","Emma","Ryan","Amy"]})
df1
```

## pandas DataFrame 資料框

```{python}
#| echo: true
population_dict = {'California': 38332521,
                   'Texas': 26448193,
                   'New York': 19651127,
                   'Florida': 19552860,
                   'Illinois': 12882135}
population = pd.Series(population_dict)
area_dict = {'California': 423967, 
              'Texas': 695662, 
              'New York': 141297,
              'Florida': 170312,
              'Illinois': 149995}
area = pd.Series(area_dict)
states = pd.DataFrame({'population': population,
                       'area': area})
states
```

## pandas DataFrame from numpy array

```{python}
#| echo: true
pd.DataFrame(np.random.rand(3, 2),
             columns=['foo', 'bar'],
             index=['a', 'b', 'c'])
```

## pandas 資料框 - 取值

資料框物件名稱.`head(資料筆數)`：取前幾筆

```{python}
#| echo: true
df1.head(2)
```

資料框物件名稱.`head(資料筆數)`：取後幾筆

```{python}
#| echo: true
df1.tail(2)
```

## pandas 資料框 - 取值 (column)

資料框物件名稱`[`欄位名稱`]`：取出sequence

```{python}
#| echo: true
df1["Name"]
```

資料框物件名稱`[[`欄位名稱`]]`：取出資料框

```{python}
#| echo: true
df1[["Name"]]
```

## pandas 資料框 - 取值 (row)

資料框物件名稱`[`row slice`]`

```{python}
#| echo: true
df1[0:1]
```

## Hands-on

-   新增一個儲存學生學號、姓名、成績的pandas資料框，並生成5筆資料
-   試著取出學生成績欄位
-   試著取出學生姓名與成績兩個欄位
-   試著取出第3位學生的成績 (hint: sequence)

## pandas 資料框 - 設定index

預設index為0\~n的序列，可用`index`參數修改

```{python}
#| echo: true
df_index = pd.DataFrame({"ID":[1, 2, 3, 4],
                          "Name":["Tom","Emma","Ryan","Amy"]},
                          index=["a","b","c","d"])
df_index
```

## pandas 資料框 - 設定index

也可用已有的data frame設定，透過`.set_index(欄位名稱)`

```{python}
#| echo: true
df_index2 = df1.set_index("ID")
df_index2
```

## pandas 資料框 - 新增修改

`pd`.`concat([pd物件1, pd物件2])`，預設為row方向的合併

```{python}
#| echo: true
df1 = pd.DataFrame({"ID":[1, 2, 3, 4],
                    "Name":["Tom","Emma","Ryan","Amy"]})
df2 = pd.DataFrame({"ID":[5,6,7],
                    "Name":["A","B","C"]})                 
newdf = pd.concat([df1,df2])
print(newdf)
```

## Check Data Type

使用`type()`函數可查看資料結構

```{python}
#| echo: true
type(dist1)
```

## Common Data Structure 資料結構 - Recap

::::: columns
::: {.column width="50%"}
-   序列 (sequence)
    -   表 (list) `[,]`
    -   定值表 (tuple) `(,)`
    -   範圍 (range) `range(s,e,i)`
-   映射 (mapping)
    -   字典 (dist) `{key:value,key:value}`
:::

::: {.column width="50%"}
-   矩陣 (matrix, array)
    -   一般矩陣
    -   numpy array
-   資料框 (data frame)
    -   pandas 資料框
:::
:::::

## References

-   [Python for Everybody](https://www.py4e.com/)
    -   Some contents are from Python for Everybody, and these contents are Copyright 2010- Charles R. Severance (www.dr-chuck.com) of the University of Michigan School of Information and open.umich.edu and made available under a Creative Commons Attribution 4.0 License.
-   [Python Data Science Handbook](https://jakevdp.github.io/PythonDataScienceHandbook/)

# Questions?