Refactor code and structure to polars

Author: ugohuche

data-transform.ipynb

@@ -183,7 +183,7 @@
     "Note: If you're coming from **pandas**, be aware that **polars** does not use an index column and each row is indexed by its integer position in the table.\n",
     ":::\n",
     "\n",
-    "![](https://pandas.pydata.org/docs/_images/01_table_dataframe.svg)\n"
+    "![](./dataframe_illustration.svg)\n"
    ]
   },
   {
@@ -650,7 +650,7 @@
    "metadata": {},
    "source": [
     "We can use `.alias()` with an expression to assign column names, when creating new columns.\n",
-    "The other way to do this involves an 'assign()' statement and is used when you wish to chain multiple steps together (like we saw earlier). These use a special syntax called a 'lambda' statement, which (here at least) just provides a way of specifying to **pandas** that we wish to perform the operation on every row. Below is an example using the flights data. You should note though that the word 'row' below is a dummy; you could replace it with any variable name (for example, `x`) but `row` makes what is happening a little bit clearer.\n"
+    "We can chain multiple expressions together with `.with_columns()`, which would create multiple new columns with the names assigned to `.alias()`.\n"
    ]
   },
   {
@@ -900,7 +900,7 @@
     "\n",
     "The simplest way to re-order (all) columns is to create a new list of their names with them in the order that you'd like them: but be careful you don't forget any columns that you'd like to keep!\n",
     "\n",
-    "Let's see an example with a fresh version of the fake data from earlier. We'll put all of the odd-numbered columns first, in descending order, then the even similarly.\n"
+    "Let's see an example with a fresh version of the fake data from earlier. We'll put all of the odd-numbered columns first, in descending order, then the even similarly using `.select()`.\n"
    ]
   },
   {
@@ -912,20 +912,19 @@
    "source": [
     "df = pl.DataFrame(\n",
     "    data=np.reshape(range(36), (6, 6)),\n",
-    "    columns=[\"col\" + str(i) for i in range(6)],\n",
-    "    dtype=float,\n",
+    "    schema=[\"col\" + str(i) for i in range(6)]\n",
     ")\n",
     "df"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "b9a409ac",
+   "id": "3c2029cc",
    "metadata": {},
    "outputs": [],
    "source": [
-    "df = df[[\"col5\", \"col3\", \"col1\", \"col4\", \"col2\", \"col0\"]]\n",
+    "df = df.select([\"col5\", \"col3\", \"col1\", \"col4\", \"col2\", \"col0\"])\n",
     "df"
    ]
   },
@@ -934,7 +933,7 @@
    "id": "dd91d87a",
    "metadata": {},
    "source": [
-    "Of course, this is quite tedious if you have lots of columns! There are methods that can help make this easier depending on your context. Perhaps you'd just liked to sort the columns in order? This can be achieved by combining `sorted()` and the `reindex()` command (which works for rows or columns) with `axis=1`, which means the second axis (i.e. columns).\n"
+    "Of course, this is quite tedious if you have lots of columns! There are methods that can help make this easier depending on your context. Perhaps you'd just liked to sort the columns in order? This can be achieved by combining `sorted()` and the `.select()` function, passing the DataFrame `.columns`.\n"
    ]
   },
   {
@@ -944,7 +943,11 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "df.reindex(sorted(df.columns), axis=1)"
+    "# Alphabetical order\n",
+    "df.select(sorted(df.columns))\n",
+    "\n",
+    "# Reverse alphabetical order\n",
+    "df.select(sorted(df.columns, reverse=True))"
    ]
   },
   {
@@ -954,37 +957,21 @@
    "source": [
     "### Column and Row Exercises\n",
     "\n",
-    "1.  Compare `air_time` with `arr_time - dep_time`. What do you expect to see? What do you see What do you need to do to fix it?\n",
+    "1.  Compare `air_time` with `arr_time - dep_time`. What do you expect to see? What do you see? What do you need to do to fix it?\n",
     "\n",
     "2.  Compare `dep_time`, `sched_dep_time`, and `dep_delay`. How would you expect those three numbers to be related?\n",
     "\n",
     "3.  Brainstorm as many ways as possible to select `dep_time`, `dep_delay`, `arr_time`, and `arr_delay` from `flights`.\n",
     "\n",
-    "4.  What happens if you include the name of a row or column multiple times when trying to select them?\n",
-    "\n",
-    "5.  What does the `.isin()` function do in the following?\n",
-    "\n",
-    "    ```python\n",
-    "    flights.columns.isin([\"year\", \"month\", \"day\", \"dep_delay\", \"arr_delay\"])\n",
-    "    ```\n",
-    "\n",
-    "6.  Does the result of running the following code surprise you?\n",
-    "    How do functions like `str.contains` deal with case by default?\n",
-    "    How can you change that default?\n",
-    "\n",
-    "    ```python\n",
-    "    flights.loc[:, flights.columns.str.contains(\"TIME\")]\n",
-    "    ```\n",
-    "\n",
-    "    (Hint: you can use help even on functions that apply to data frames, eg use `help(flights.columns.str.contains)`)\n"
+    "4.  What happens if you include the name of a row or column multiple times when trying to select them?\n"
    ]
   },
   {
    "cell_type": "markdown",
    "id": "a3c837e4",
    "metadata": {},
    "source": [
-    "## Grouping, changing the index, and applying summary statistics\n",
+    "## Grouping and applying summary statistics\n",
     "\n",
     "So far you've learned about working with rows and columns. **polars** gets even more powerful when you add in the ability to work with groups. And because groups tend to imply an aggregation or pooling of data, they go hand-in-glove with the application of a summary statistic.\n",
     "\n",
@@ -999,50 +986,12 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 92,
+   "execution_count": null,
    "id": "62e540c0",
    "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/html": [
-       "<div><style>\n",
-       ".dataframe > thead > tr,\n",
-       ".dataframe > tbody > tr {\n",
-       "  text-align: right;\n",
-       "  white-space: pre-wrap;\n",
-       "}\n",
-       "</style>\n",
-       "<small>shape: (12, 2)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>month</th><th>dep_delay</th></tr><tr><td>i64</td><td>f64</td></tr></thead><tbody><tr><td>11</td><td>5.435362</td></tr><tr><td>12</td><td>16.576688</td></tr><tr><td>2</td><td>10.816843</td></tr><tr><td>8</td><td>12.61104</td></tr><tr><td>10</td><td>6.243988</td></tr><tr><td>&hellip;</td><td>&hellip;</td></tr><tr><td>6</td><td>20.846332</td></tr><tr><td>1</td><td>10.036665</td></tr><tr><td>7</td><td>21.727787</td></tr><tr><td>3</td><td>13.227076</td></tr><tr><td>5</td><td>12.986859</td></tr></tbody></table></div>"
-      ],
-      "text/plain": [
-       "shape: (12, 2)\n",
-       "┌───────┬───────────┐\n",
-       "│ month ┆ dep_delay │\n",
-       "│ ---   ┆ ---       │\n",
-       "│ i64   ┆ f64       │\n",
-       "╞═══════╪═══════════╡\n",
-       "│ 11    ┆ 5.435362  │\n",
-       "│ 12    ┆ 16.576688 │\n",
-       "│ 2     ┆ 10.816843 │\n",
-       "│ 8     ┆ 12.61104  │\n",
-       "│ 10    ┆ 6.243988  │\n",
-       "│ …     ┆ …         │\n",
-       "│ 6     ┆ 20.846332 │\n",
-       "│ 1     ┆ 10.036665 │\n",
-       "│ 7     ┆ 21.727787 │\n",
-       "│ 3     ┆ 13.227076 │\n",
-       "│ 5     ┆ 12.986859 │\n",
-       "└───────┴───────────┘"
-      ]
-     },
-     "execution_count": 92,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "flights.group_by(\"month\").agg(pl.mean(\"dep_delay\"))"
+   "outputs": [],
+   "source": [
+    "flights.group_by(\"month\").agg(pl.col(\"dep_delay\").mean())"
    ]
   },
   {
@@ -1062,13 +1011,14 @@
    "source": [
     "| Aggregation          | Description                     |\n",
     "| -------------------- | ------------------------------- |\n",
-    "| `count()`            | Number of items                 |\n",
+    "| `count()`            | Number of non-null items        |\n",
+    "| `len()`              | Number of all items             |\n",
     "| `first()`, `last()`  | First and last item             |\n",
     "| `mean()`, `median()` | Mean and median                 |\n",
     "| `min()`, `max()`     | Minimum and maximum             |\n",
     "| `std()`, `var()`     | Standard deviation and variance |\n",
     "| `mad()`              | Mean absolute deviation         |\n",
-    "| `prod()`             | Product of all items            |\n",
+    "| `product()`          | Product of all items            |\n",
     "| `sum()`              | Sum of all items                |\n",
     "| `value_counts()`     | Counts of unique values         |\n",
     "\n",
@@ -1082,11 +1032,10 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "(\n",
-    "    flights.group_by(\"month\").agg(\n",
-    "        mean_delay=pl.mean(\"dep_delay\"),\n",
-    "        count_flights=pl.count(\"dep_delay\"),\n",
-    "    )\n",
+    "# Multiple aggregations using polars' syntactic sugar (shorthand) for mean and count\n",
+    "flights.group_by(\"month\").agg(\n",
+    "    mean_delay=pl.mean(\"dep_delay\"),\n",
+    "    count_flights=pl.count(\"dep_delay\"),\n",
     ")"
    ]
   },
@@ -1105,7 +1054,7 @@
    "source": [
     "### Grouping by multiple variables\n",
     "\n",
-    "This is as simple as passing `.group_by()` a list representing multiple columns instead of a string representing a single column.\n"
+    "This is as simple as passing `.group_by()` a list or multiple strings representing columns instead of a single string representing a single column.\n"
    ]
   },
   {
@@ -1115,9 +1064,9 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "month_year_delay = flights.group_by([\"month\", \"year\"]).agg(\n",
-    "    mean_delay=(\"dep_delay\", \"mean\"),\n",
-    "    count_flights=(\"dep_delay\", \"count\"),\n",
+    "month_year_delay = flights.group_by(\"month\", \"year\").agg(\n",
+    "    mean_delay=pl.mean(\"dep_delay\"),\n",
+    "    count_flights=pl.count(\"dep_delay\"),\n",
     ")\n",
     "month_year_delay"
    ]
@@ -1173,7 +1122,7 @@
    "source": [
     "### Groupby Exercises\n",
     "\n",
-    "1.  Which carrier has the worst delays? Challenge: can you disentangle the effects of bad airports vs. bad carriers? Why/why not? (Hint: think about `flights.groupby([\"carrier\", \"dest\"]).count()`)\n",
+    "1.  Which carrier has the worst delays? Challenge: can you disentangle the effects of bad airports vs. bad carriers? Why/why not? (Hint: think about `flights.group_by([\"carrier\", \"dest\"])`)\n",
     "\n",
     "2.  Find the most delayed flight to each destination.\n",
     "\n",