DSc. Information Systems
MSc. Management of Information Systems
MBA
BSc. Computer Engineering
DSc. Information Systems
MSc. Management of Information Systems
MBA
BSc. Computer Engineering
During this project you will generate, process, analyze, and answer specific questions questions using algorithms and visualizations about a healthcare dataset. Each of the individual assignments will build towards the final deliverable. Individual assignments will have both must have requirements and stretch requirements. Completion of the must have requirements “Meet Expectations”. Sophistication, completeness, and style of the stretch requirements will contribute to students “Exceeding Expectations” or delivering “Superior Work”.
This course focuses on the fundamentals of computer programming using the Python interpreted programming language. Students will develop his or her problem solving skills using the divide and-conquer and top-down approaches to build real-world based software applications. Students will also learn the basics of the software development life-cycle: planning, development, testing, implementation and maintenance. Assignments will include individual coding assignments and weekly fundamental checkpoint quizzes culminating with an individual final project to test essential programming and problem solving skills.
Note: The course schedule and assignments are subject to change. Please see your Enterprise Learning Management System (e.g. Canvas, Blackboard, Desire2Learn) for the official schedule.
Lectures will contain a mixture of content form this site and others.
Quizzes will be administered approximately every week on material from prior weeks. All quizzes are closed book and no materials may be used.
The project is a collection of individual programming assignments building towards delivery of a final product.
All assignments will be evaluated by 2 criteria:
This course focuses on the fundamentals of computer programming using the Python interpreted programming language. Students will develop his or her problem solving skills using the divide and-conquer and top-down approaches to build real-world based software applications. Students will also learn the basics of the software development life-cycle: planning, development, testing, implementation and maintenance. Assignments will include individual coding assignments and weekly fundamental checkpoint quizzes culminating with an individual final project to test essential programming and problem solving skills.
Note: The course schedule and assignments are subject to change. Please see your Enterprise Learning Management System (e.g. Canvas, Blackboard, Desire2Learn) for the official schedule.
Lectures will contain a mixture of content form this site and others.
Quizzes will be administered approximately every week on material from prior weeks. All quizzes are closed book and no materials may be used.
The project is a collection of individual programming assignments building towards delivery of a final product.
All assignments will be evaluated by 2 criteria:
Welcome to Data Focused Python. In this guide we’ll walk through all the steps you need to get started with the course. Python has a robust community which is great, but it also means there’s going to be many options from python distribution to IDEs to libraries. There’s no right or wrong selection, only the one that works best for you.
Python is open source and as with many open source solutions the ecosystem can be overwhelming. If you stick with the recommended Anaconda and Pycharm setup you’ll avoid most of the complexity. However, for awareness we’ll review some of the most common tooling options used.
After you’re comfortable working with your Python IDE to create basic python programs, you’ll want to clone the course materials from GitHub. Before you can clone the repository, you’ll have to install git on your local machine. Git is a distributed version control system that’s great for managing projects with distributed team members. There’s a lot of great resources on how to use git. Here’s 2 of my favorite interactive tutorials:
We won’t be doing any branching in this course, but I think it’s important to know so I’ve included the link.
Once you’ve installed git and are somewhat comfortable with the command line tool, you can read about how to Clone a Repository from GitHub.
It’s as easy as:
git clone https://github.com/BrianKolowitz/data-focused-python.git
+However, I personally like to Add SSH Keys to GitHub and clone using this command:
git clone git@github.com:BrianKolowitz/data-focused-python.git
+
+Once the course git repository has been cloned locally you can run the course notebooks and follow along in class or try them at home.
First, change the directory to the data-focused-python directory of your clonned repository,.
cd data-focused-python
+Then start Jupyter from your terminal
jupyter-lab
+The Jupyter Notebook is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text. If you’ve installed Anaconda, Jupyter is packaged as part of the distribution.
I encourage you to review the more detailed How To Use Jupyter Notebook – An Ultimate Guide.
Python itself is an interpreted programming language which means the source code you type gets executed by the interpreter at runtime. You can start the python interpreter from any command line by executing the following in your terminal:
python
+I encourage you to try it out. You can follow along the official python docs here
IPython builds on the Python interpreter adding advanced read-eval-print-loop (REPL) functionality among other features.
You can start the ipython interpreter by executing the following command in your terminal:
ipython
+# in anaconda
+## find the location of your virtual environment
+!which python
+/Users/bk/opt/miniconda3/envs/cmu39/bin/python
+## run IDLE out of that location
+!/Users/bk/opt/miniconda3/envs/cmu39/bin/idle3.9 &
+Using the Python interpreter is great for small problems, testing libraries, etc. but all of your code lives in memory inside the python interpreter. When the interpreter stops your code is gone forever. In order to get around this limitation you can create python script files and execute the files using the interpreter.
Create a file on your harddrive called hello.py. Copy and paste this code into the file and save it.
print('Hello class')
+print('Welcome to 95-888')
+You can then execute the file by typing this command into your terminal.
python hello.py
+!python hello.py
+
+Now that you have everything setup, you’ll want to dive in and create your first program. Jetbrains has a great tutorial that describes how to Create and Running Your First Python Project.
Once you’re comfortable creating programs you might want to learn how to Debug Your First Python Application, Test Your First Python Application, or even Creating and Running Your First Django Project.
Django is a great way to build web applications in python, but debugging, testing, and creating web applications are not required components of this course. However, these are useful skills to pick up as you become more familiar with building solutions in python.
+A module allows you to logically organize your Python code. Grouping related code into a module makes the code easier to understand and use. A module is a Python object with arbitrarily named attributes that you can bind and reference.
Simply, a module is a file consisting of Python code. A module can define functions, classes and variables. A module can also include runnable code.
You can use any Python source file as a module by executing an import statement in some other Python source file. The import has the following syntax
import module1[, module2[,... moduleN]
+When the interpreter encounters an import statement, it imports the module if the module is present in the search path. A search path is a list of directories that the interpreter searches before importing a module. For example, to import the module support.py, you need to put the following command at the top of the script
# Import module support
+import support
+
+# Now you can call defined function that module as follows
+support.print_func("drk")
+Hello : drk
+A module is loaded only once, regardless of the number of times it is imported. This prevents the module execution from happening over and over again if multiple imports occur.
Python’s from statement lets you import specific attributes from a module into the current namespace. The from…import has the following syntax
from modname import name1[, name2[, ... nameN]]
+For example, to import the function fibonacci from the module fib, use the following statement −
from fib import fibonacci
+This statement does not import the entire module fib into the current namespace; it just introduces the item fibonacci from the module fib into the global symbol table of the importing module.
It is also possible to import all names from a module into the current namespace by using the following import statement
from modname import *
+This provides an easy way to import all the items from a module into the current namespace; however, this statement should be used sparingly.
When you import a module, the Python interpreter searches for the module in the following sequences −
The module search path is stored in the system module sys as the sys.path variable. The sys.path variable contains the current directory, PYTHONPATH, and the installation-dependent default.
A package is a hierarchical file directory structure that defines a single Python application environment that consists of modules and subpackages and sub-subpackages, and so on.
Consider a file Pots.py available in Phone directory. This file has following line of source code −
def Pots():
+ print "I'm Pots Phone"
+Similar way, we have another two files having different functions with the same name as above −
To make all of your functions available when you’ve imported Phone, you need to put explicit import statements in init.py as follows
from Pots import Pots
+from Isdn import Isdn
+from G3 import G3
+After you add these lines to init.py, you have all of these classes available when you import the Phone package.
# Now import your Phone Package.
+import Phone as p
+p.Pots()
+I'm a Pots Phone.
+p.Isdn()
+I'm an Isdn phone.
+p.G3()
+I'm a G3 phone.
+
+Sometimes you might want to spawn a process from python and interact with the data. You might use a process to generate some files that your python application will read, or read the output of the process within your application. Doing so is relatively simple in python using the subprocess library,.
You can start a process in Python using the Popen function call. Open a pipe to or from command. The return value is an open file object connected to the pipe, which can be read or written depending on whether mode is ‘r’ (default) or ‘w’.
from subprocess import Popen, PIPE
+process = Popen(['cat', 'scores.csv'], stdout=PIPE, stderr=PIPE)
+stdout, stderr = process.communicate()
+print(stdout.decode('utf-8'))
+The process.communicate() call reads input and output from the process. stdout is the process output. stderr will be written only if an error occurs. If you want to wait for the program to finish you can call Popen.wait().
from subprocess import Popen, PIPE
+
+process = Popen(['cat', 'scores.csv'], stdout=PIPE, stderr=PIPE)
+stdout, stderr = process.communicate()
+status = process.wait()
+print(stdout.decode('utf-8'))
+Subprocess has a method call() which can be used to start a program. This is basically just like the Popen class and takes all of the same arguments, but it simply wait until the command completes and gives us the return code.
import subprocess
+subprocess.call(['ls','-l'])
+total 192
+-rw-r--r-- 1 bk staff 7977 Oct 21 09:46 01 - Getting Started.ipynb
+-rw-r--r-- 1 bk staff 7400 Oct 17 20:44 02 - Terminal Application.ipynb
+-rw-r--r-- 1 bk staff 10581 Oct 17 20:44 03 - Argparse.ipynb
+-rw-r--r-- 1 bk staff 14711 Oct 17 20:44 04 - Start a process in Python.ipynb
+-rw-r--r-- 1 bk staff 149 Aug 26 15:35 argparse_1.py
+-rw-r--r-- 1 bk staff 202 Aug 26 15:35 argparse_2.py
+-rw-r--r-- 1 bk staff 299 Aug 26 15:35 argparse_3.py
+-rw-r--r-- 1 bk staff 309 Aug 26 15:35 argparse_4.py
+-rw-r--r-- 1 bk staff 219 Aug 26 15:35 argparse_5.py
+-rw-r--r-- 1 bk staff 580 Aug 26 15:35 argparse_6.py
+-rw-r--r-- 1 bk staff 372 Aug 26 15:35 terminal_1.py
+-rw-r--r-- 1 bk staff 370 Aug 26 15:35 terminal_2.py
+-rw-r--r-- 1 bk staff 198 Aug 26 15:35 test_class.py
+-rw-r--r-- 1 bk staff 100 Aug 26 15:35 test_sample.py
+-rw-r--r-- 1 bk staff 143 Aug 26 15:35 test_sysexit.py
+-rw-r--r-- 1 bk staff 90 Oct 17 20:44 test_tmpdir.p
+
+
+
+
+0
+
+
+
+y
+-rw-r--r-- 1 bk staff 294 Oct 17 20:44 test_tmppath.py
+total 181
+drwxr-xr-x 2 root root 4096 Mar 3 2012 bin
+drwxr-xr-x 4 root root 1024 Oct 26 2012 boot
+The command line arguments are passed as a list of strings, which avoids the need for escaping quotes or other special characters that might be interpreted by the shell.
The check_call() function works like call() except that the exit code is checked, and if it indicates an error happened then a CalledProcessError exception is raised.
import subprocess
+subprocess.check_call(['false'])
+---------------------------------------------------------------------------
+
+CalledProcessError Traceback (most recent call last)
+
+/var/folders/jd/pq0swyt521jb2424d6fvth840000gn/T/ipykernel_91211/359612823.py in <module>
+ 1 import subprocess
+----> 2 subprocess.check_call(['false'])
+
+
+~/opt/miniconda3/envs/cmu39/lib/python3.9/subprocess.py in check_call(*popenargs, **kwargs)
+ 371 if cmd is None:
+ 372 cmd = popenargs[0]
+--> 373 raise CalledProcessError(retcode, cmd)
+ 374 return 0
+ 375
+
+
+CalledProcessError: Command '['false']' returned non-zero exit status 1.
+The standard input and output channels for the process started by call() are bound to the parent’s input and output. That means the calling program cannot capture the output of the command. To capture the output, we can use check_output() for later processing.
import subprocess
+output = subprocess.check_output(['ls','-l'])
+print(output.decode('utf-8'))
+total 192
+-rw-r--r-- 1 bk staff 7977 Oct 21 09:46 01 - Getting Started.ipynb
+-rw-r--r-- 1 bk staff 7400 Oct 17 20:44 02 - Terminal Application.ipynb
+-rw-r--r-- 1 bk staff 10581 Oct 17 20:44 03 - Argparse.ipynb
+-rw-r--r-- 1 bk staff 14711 Oct 17 20:44 04 - Start a process in Python.ipynb
+-rw-r--r-- 1 bk staff 149 Aug 26 15:35 argparse_1.py
+-rw-r--r-- 1 bk staff 202 Aug 26 15:35 argparse_2.py
+-rw-r--r-- 1 bk staff 299 Aug 26 15:35 argparse_3.py
+-rw-r--r-- 1 bk staff 309 Aug 26 15:35 argparse_4.py
+-rw-r--r-- 1 bk staff 219 Aug 26 15:35 argparse_5.py
+-rw-r--r-- 1 bk staff 580 Aug 26 15:35 argparse_6.py
+-rw-r--r-- 1 bk staff 372 Aug 26 15:35 terminal_1.py
+-rw-r--r-- 1 bk staff 370 Aug 26 15:35 terminal_2.py
+-rw-r--r-- 1 bk staff 198 Aug 26 15:35 test_class.py
+-rw-r--r-- 1 bk staff 100 Aug 26 15:35 test_sample.py
+-rw-r--r-- 1 bk staff 143 Aug 26 15:35 test_sysexit.py
+-rw-r--r-- 1 bk staff 90 Oct 17 20:44 test_tmpdir.py
+-rw-r--r-- 1 bk staff 294 Oct 17 20:44 test_tmppath.py
+
+# Show a simple message
+print('hello class')
+hello class
+import os
+
+# Display a title bar.
+print("**********************************************")
+print("*** Hello to everyone in class today! ***")
+print("**********************************************")
+
+# Display a bunch of output, representing a long-running program.
+for week in range(1, 8):
+ print(f"We will learn interesting things in Week {week:02}")
+ print(f"We will learn interesting things in Week {week:02}")
+**********************************************
+*** Hello to everyone in class today! ***
+**********************************************
+We will learn interesting things in Week 01
+We will learn interesting things in Week 01
+We will learn interesting things in Week 02
+We will learn interesting things in Week 02
+We will learn interesting things in Week 03
+We will learn interesting things in Week 03
+We will learn interesting things in Week 04
+We will learn interesting things in Week 04
+We will learn interesting things in Week 05
+We will learn interesting things in Week 05
+We will learn interesting things in Week 06
+We will learn interesting things in Week 06
+We will learn interesting things in Week 07
+We will learn interesting things in Week 07
+import os
+from time import sleep
+
+# Display a title bar.
+print("**********************************************")
+print("*** Hello to everyone in class today! ***")
+print("**********************************************")
+
+# Display a bunch of output, representing a long-running program.
+for week in range(1,8):
+ sleep(1)
+ print(f"We will learn interesting things in Week {week:02}")
+**********************************************
+*** Hello to everyone in class today! ***
+**********************************************
+We will learn interesting things in Week 01
+We will learn interesting things in Week 02
+We will learn interesting things in Week 03
+We will learn interesting things in Week 04
+We will learn interesting things in Week 05
+We will learn interesting things in Week 06
+We will learn interesting things in Week 07
+You can pass arguments into your application to dynamically change the behavior at run time.
See terminal_1.py
import sys
+from time import sleep
+
+adjective = sys.argv[1]
+num_weeks = int(sys.argv[2])
+
+print("**********************************************")
+print("*** Hello to everyone in class today! ***")
+print("**********************************************")
+
+for week in range(1, num_weeks + 1):
+ sleep(1)
+ print(f"We will learn {adjective} things in Week {week:02}")
+Note:
!python terminal_1.py fun 3
+**********************************************
+*** Hello to everyone in class today! ***
+**********************************************
+We will learn fun things in Week 01
+We will learn fun things in Week 02
+We will learn fun things in Week 03
+Here we slightly modify the code to process a list of arguments. Since all arguments are strings we’ll pass in a comma separated list of values.
See terminal_2.py
import sys
+from time import sleep
+
+adjectives = sys.argv[1]
+
+print("**********************************************")
+print("*** Hello to everyone in class today! ***")
+print("**********************************************")
+
+for week, adjective in enumerate(adjectives.split(',')):
+ sleep(1)
+ print(f"We will learn {adjective} things in Week {(week + 1):02}")
+Note:
!python terminal_2.py "fun,interesting,exciting"
+**********************************************
+*** Hello to everyone in class today! ***
+**********************************************
+We will learn fun things in Week 01
+We will learn interesting things in Week 02
+We will learn exciting things in Week 03
+
+Argparse is a parser for command-line options, arguments and subcommands. This library makes it easy to write user-friendly command-line interfaces by:
The argparse module automatically generates help and usage messages and issues errors when users give the program invalid arguments.
See agrparse_1.py
# import the library
+import argparse
+
+# initialize the parser
+parser = argparse.ArgumentParser()
+
+# parse arguments from sys.argv
+parser.parse_args()
+
+# run the code and ask for help
+!python argparse_1.py --help
+usage: argparse_1.py [-h]
+
+optional arguments:
+ -h, --help show this help message and exit
+See argparse_2.py
import argparse
+
+parser = argparse.ArgumentParser()
+
+parser.add_argument("name")
+parser.add_argument("age")
+parser.add_argument("city")
+args = parser.parse_args()
+
+print(args.name, args.age, args.city)
+!python argparse_2.py --help
+usage: argparse_2.py [-h] name age city
+
+positional arguments:
+ name
+ age
+ city
+
+optional arguments:
+ -h, --help show this help message and exit
+!python argparse_2.py Ben 37 Pittsburgh
+Ben 37 Pittsburgh
+!python argparse_2.py Brian "??" Pittsburgh
+Brian ?? Pittsburgh
+See argparse_3.py
import argparse
+
+parser = argparse.ArgumentParser()
+
+parser.add_argument("name", help="the name of the person you want to find")
+parser.add_argument("age", help="the age of the person you'd like to find")
+parser.add_argument("city", help="the city you'd like to search")
+
+args = parser.parse_args()
+!python argparse_3.py --help
+usage: argparse_3.py [-h] name age city
+
+positional arguments:
+ name the name of the person you want to find
+ age the age of the person you'd like to find
+ city the city you'd like to search
+
+optional arguments:
+ -h, --help show this help message and exit
+Argparse treats all arguments as strings by default. You can change the expected data type when you add each argument.
See argparse_4.py
import argparse
+
+parser = argparse.ArgumentParser()
+
+parser.add_argument("name", help="the name of the person you want to find")
+parser.add_argument("age", help="the age of the person you'd like to find", type=int)
+parser.add_argument("city", help="the city you'd like to search")
+
+args = parser.parse_args()
+!python argparse_4.py Ben abc Pittsburgh
+usage: argparse_4.py [-h] name age city
+argparse_4.py: error: argument age: invalid int value: 'abc'
+!python argparse_4.py Ben 37 Pittsburgh
+See argparse_5.py
import argparse
+
+parser = argparse.ArgumentParser()
+
+parser.add_argument("--verbose",
+ help="increase output verbosity",
+ action="store_true")
+
+args = parser.parse_args()
+
+if args.verbose:
+ print("verbosity turned on")
+An optional argument (or option) is (by default) given None as a value when its not being used.
!python argparse_5.py --help
+usage: argparse_5.py [-h] [--verbose]
+
+optional arguments:
+ -h, --help show this help message and exit
+ --verbose increase output verbosity
+!python argparse_5.py --verbose
+verbosity turned on
+!python argparse_5.py
+See argparse_6.py
import argparse
+
+parser = argparse.ArgumentParser()
+
+parser.add_argument("-n", "--name", help="the name of the person you want to find")
+parser.add_argument("-a", "--age", help="the age of the person you'd like to find", type=int)
+parser.add_argument("-c", "--city", help="the city you'd like to search")
+parser.add_argument("-v", "--verbose", help="increase output verbosity", action="store_true")
+
+args = parser.parse_args()
+
+if args.verbose:
+ print(f"Searching for {args.name} {args.age} years of age in or around {args.city}")
+else:
+ print(f"Searching for {args.name}")
+!python argparse_6.py --help
+usage: argparse_6.py [-h] [-n NAME] [-a AGE] [-c CITY] [-v]
+
+optional arguments:
+ -h, --help show this help message and exit
+ -n NAME, --name NAME the name of the person you want to find
+ -a AGE, --age AGE the age of the person you'd like to find
+ -c CITY, --city CITY the city you'd like to search
+ -v, --verbose increase output verbosity
+!python argparse_6.py -n Ben -a 37 -c Pittsburgh
+Searching for Ben
+!python argparse_6.py --name Ben --age 37 --city Pittsburgh
+Searching for Ben
+!python argparse_6.py --name Ben --age 37 --city Pittsburgh --verbose
+Searching for Ben 37 years of age in or around Pittsburgh
+
+if condition:
+ # statement
+ # statement
+if 2 > 1: print(“2 is greater than 1”)
# boolean expressions
+1 > 2
+False
+'a' > 'b'
+False
+'a' > 'A'
+True
+chr(65)
+'A'
+ord('A')
+65
+ord('a')
+97
+balance = 119
+payment = 20
+balance == 0
+False
+payment != balance
+True
+if payment == balance:
+ print('paid in full')
+else:
+ print('you owe ' + str(balance - payment))
+you owe 99
+
+In Python programming, a tuple is similar to a list. The difference between the two is that we cannot change the elements of a tuple once it is assigned whereas in a list, elements can be changed.
Since, tuples are quite similiar to lists, both of them are used in similar situations as well.
However, there are certain advantages of implementing a tuple over a list. Below listed are some of the main advantages:
A tuple is created by placing all the items (elements) inside a parentheses (), separated by comma. The parentheses are optional but is a good practice to write it.
A tuple can have any number of items and they may be of different types (integer, float, list, string etc.).
# empty tuple
+# Output: ()
+my_tuple = ()
+print(type(my_tuple), my_tuple)
+<class 'tuple'> ()
+# tuple having integers
+# Output: (1, 2, 3)
+my_tuple = (1, 2, 3)
+print(my_tuple)
+(1, 2, 3)
+# tuple with mixed datatypes
+# Output: (1, "Hello", 3.4)
+my_tuple = (1, "Hello", 3.4)
+print(my_tuple)
+(1, 'Hello', 3.4)
+# nested tuple
+# Output: ("mouse", [8, 4, 6], (1, 2, 3))
+my_tuple = ("mouse", [8, 4, 6], (1, 2, 3))
+print(my_tuple)
+('mouse', [8, 4, 6], (1, 2, 3))
+# tuple can be created without parentheses
+# also called tuple packing
+# Output: 3, 4.6, "dog"
+
+my_tuple = 3, 4.6, "dog"
+print(my_tuple)
+(3, 4.6, 'dog')
+Creating a tuple with one element is a bit tricky.
Having one element within parentheses is not enough. We will need a trailing comma to indicate that it is in fact a tuple.
# only parentheses is not enough
+# Output: <class 'str'>
+my_tuple = ("hello")
+
+print(type(my_tuple))
+<class 'str'>
+# need a comma at the end
+# Output: <class 'tuple'>
+my_tuple = ("hello")
+
+print(type(my_tuple), my_tuple)
+<class 'str'> hello
+# what if i forget a comma
+# Output: <class 'tuple'>
+my_tuple = ("hello" "bye")
+
+print(type(my_tuple), my_tuple)
+<class 'str'> hellobye
+# parentheses is optional
+# Output: <class 'tuple'>
+my_tuple = "hello",
+
+print(type(my_tuple))
+<class 'tuple'>
+n_tuple = ("mouse", [8, 4, 6], (1, 2, 3))
+print(n_tuple[1][2])
+6
+n_tuple = ("mouse", [8, 4, 6], (1, 2, 3))
+print(type(n_tuple[0][2]), n_tuple[0][2])
+<class 'str'> u
+n_tuple = ("mouse", [8, 4, 6], (1, 2, 3))
+print(n_tuple[-1][-3], ' is the same as ', n_tuple[2][0])
+1 is the same as 1
+n_tuple = ("mouse", [8, 4, 6], (1, 2, 3))
+print(n_tuple[0][0], ' is the same as ', n_tuple[-3][-5])
+m is the same as m
+n_tuple = ("mouse", [8, 4, 6], (1, 2, 3))
+print(n_tuple[-1][1])
+2
+letter_tuple = ('a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j')
+# prints items at position 0, 1, and 2 but not 3
+print(letter_tuple[0:3])
+('a', 'b', 'c')
+# prints items at position 3, 4, and 5 but not 6
+print(letter_tuple[3:6])
+('d', 'e', 'f')
+# start at the beginning
+print(letter_tuple[:6])
+('a', 'b', 'c', 'd', 'e', 'f')
+# go until the end
+print(letter_tuple[3:])
+('d', 'e', 'f', 'g', 'h', 'i', 'j')
+# every element
+print(letter_tuple[0:10:1])
+('a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j')
+# every 3rd element
+print(letter_tuple[0:10:3])
+('a', 'd', 'g', 'j')
+# elements at even positions
+print(letter_tuple[0::2])
+('a', 'c', 'e', 'g', 'i')
+# elements at odd positions
+print(letter_tuple[1::2])
+('b', 'd', 'f', 'h', 'j')
+print(letter_tuple[len(letter_tuple)::-1])
+('j', 'i', 'h', 'g', 'f', 'e', 'd', 'c', 'b', 'a')
+print(letter_tuple[len(letter_tuple)::-2])
+('j', 'h', 'f', 'd', 'b')
+Unlike lists, tuples are immutable.
This means that elements of a tuple cannot be changed once it has been assigned.
x_tuple = (4, 2, 3, [6, 5])
+
+x_tuple[1] = 8 # throws an error
+---------------------------------------------------------------------------
+
+TypeError Traceback (most recent call last)
+
+/var/folders/jd/pq0swyt521jb2424d6fvth840000gn/T/ipykernel_91254/2014519070.py in <module>
+ 1 x_tuple = (4, 2, 3, [6, 5])
+ 2
+----> 3 x_tuple[1] = 8 # throws an error
+
+
+TypeError: 'tuple' object does not support item assignment
+But, if the element is itself a mutable datatype like list, its nested items can be changed.
x_tuple[3][0] = 9 # but item of mutable element can be changed
+print(x_tuple)
+(4, 2, 3, [9, 5])
+We cannot change the elements in a tuple. That also means we cannot delete or remove items from a tuple. But deleting a tuple entirely is possible using the keyword del.
a_tuple = (1, 2, 3)
+print(a_tuple)
+(1, 2, 3)
+del a_tuple
+print(a_tuple)
+---------------------------------------------------------------------------
+
+NameError Traceback (most recent call last)
+
+/var/folders/jd/pq0swyt521jb2424d6fvth840000gn/T/ipykernel_91254/1119046425.py in <module>
+ 1 del a_tuple
+----> 2 print(a_tuple)
+
+
+NameError: name 'a_tuple' is not defined
+count(x) - Return the number of items that is equal to xindex(x) - Return index of first item that is equal to xmy_tuple = ('r', 'e', 'd', '', 'a', 'p', 'p', 'l', 'e',)
+print(my_tuple.count('p'))
+2
+print(my_tuple.index('p'))
+5
+
+# for loops
+for i in range(1, 10, 2):
+ print(i)
+1
+3
+5
+7
+9
+# for...else loop
+for i in range(1, 5):
+ print(i)
+else:
+ print('printed all items')
+1
+2
+3
+4
+printed all items
+# for...else loop with a break
+for i in range(1, 5):
+ print(i)
+ if i > 2:
+ break
+else:
+ print('printed all items')
+1
+2
+3
+# while loop
+condition = True
+i = 0
+while condition:
+ print(i)
+ i += 1
+ if i > 3:
+ condition = False
+0
+1
+2
+3
+# while...else loop
+condition = True
+i = 0
+while condition:
+ print(i)
+ i += 1
+ if i > 3:
+ condition = False
+else:
+ print('done')
+0
+1
+2
+3
+done
+# while...else loop with a break
+condition = True
+i = 0
+while condition:
+ print(i)
+ i += 1
+ if i > 3:
+ break
+else:
+ print('done')
+0
+1
+2
+3
+width = 5
+chars = "*@"
+for char in chars:
+ for i in range(1, width):
+ print(char * i)
+ for j in range(width, 0, -1):
+ print(char * j)
+*
+**
+***
+****
+*****
+****
+***
+**
+*
+@
+@@
+@@@
+@@@@
+@@@@@
+@@@@
+@@@
+@@
+@
+width = 5
+chars = "*@"
+for char in chars:
+ for i in range(1, width):
+ for spaces in range(width - i, 0, -1):
+ print(' ', end='')
+ print(char * i)
+ for j in range(width, 0, -1):
+ for spaces in range(width - j, 0, -1):
+ print(' ', end='')
+ print(char * j)
+ *
+ **
+ ***
+ ****
+*****
+ ****
+ ***
+ **
+ *
+ @
+ @@
+ @@@
+ @@@@
+@@@@@
+ @@@@
+ @@@
+ @@
+ @
+
+Functions are used to encapsulate a set of relatred instructions that you want to use within your program to carry out a specific task. This helps with the organization of your code as well as code reusability. Oftent times functions accept parameters and return values, but that’s not always the case.
There are three types of functions in Python:
A method is a function that’s part of a class which we’ll discuss in another lecture. Keep in mind all methods are functions but not all functions are methods.
Parameters are the names used when defining a function or a method, and into which arguments will be mapped. In other words, arguments are the things which are supplied to any function or method call, while the function or method code refers to the arguments by their parameter names.
Consider the function
def sum(a, b):
+ return a + b
+sum has 2 parameters a and b. If you call the sum function with values 2 and 3 then 2 and 3 are the arguments.
The four steps to defining a function in Python are the following:
def to declare the function and follow this up with the function name.# takes no parameters, returns none
+def say_hello():
+ print('hello')
+say_hello()
+hello
+x = say_hello()
+print(type(x), x)
+hello
+<class 'NoneType'> None
+# takes parameters, returns none
+def say_something(message):
+ print(message)
+x = say_something('hello class')
+print(type(x), x)
+hello class
+<class 'NoneType'> None
+Sometimes it’s useful to reuturn values from functions. We’ll refactor our code to return values.
def get_message():
+ return 'hello class'
+
+def say_something():
+ message = get_message()
+ print(message)
+x = get_message()
+print(type(x), x)
+<class 'str'> hello class
+say_something()
+hello class
+def ask_user_to_say_something():
+ message = input('say something')
+ print(message)
+ask_user_to_say_something()
+say something welcome to class
+
+
+welcome to class
+def say_anything(fn):
+ message = fn()
+ print(message)
+fn = get_message
+say_anything(fn)
+hello class
+fn = input
+say_anything(fn)
+ python
+
+
+python
+print(type(say_something()))
+hello class
+<class 'NoneType'>
+x = say_something()
+print(type(x), x)
+hello class
+<class 'NoneType'> None
+In python you can return values in a variety of data types including primitive data structures such as integers, floats, strings, & booleans as well as non-primitive data structures such as arrays, lists, tuples, dictionaries, sets, and files.
# returning a list
+def get_messages():
+ return ['hello class', 'python is great', 'here we\'re retuning a list']
+
+messages = get_messages()
+print(type(messages), messages)
+
+for message in messages:
+ print(type(message), message)
+<class 'list'> ['hello class', 'python is great', "here we're retuning a list"]
+<class 'str'> hello class
+<class 'str'> python is great
+<class 'str'> here we're retuning a list
+# returning a tuple... more on tuples later
+def get_message():
+ return ('hello class', 3)
+
+message = get_message()
+print(type(message), message)
+
+for i in range(0, message[1]):
+ print(message[0])
+<class 'tuple'> ('hello class', 3)
+hello class
+hello class
+hello class
+def get_message():
+ return 'hello class', 3 # ('s are optional
+
+message = get_message()
+print(type(message), message)
+
+for i in range(0, message[1]):
+ print(message[0])
+<class 'tuple'> ('hello class', 3)
+hello class
+hello class
+hello class
+message, iterations = get_message()
+print(type(message), message)
+
+for i in range(0, iterations):
+ print(message)
+<class 'str'> hello class
+hello class
+hello class
+hello class
+There are four types of arguments that Python functions can take:
Default arguments are those that take a default value if no argument value is passed during the function call. You can assign this default value by with the assignment operator =, just like in the following example:
import random
+
+def get_random_numbers(n=1):
+ if n == 1:
+ return random.random()
+ elif n > 1:
+ numbers = []
+ for i in range(0, n):
+ numbers.append(random.random())
+ return numbers
+
+
+w = get_random_numbers()
+print('w:', type(w), w)
+
+x = get_random_numbers(1)
+print('x:', type(x), x)
+
+y = get_random_numbers(n=3)
+print('y:', type(y), y)
+
+z = get_random_numbers(n=-1)
+print('z:', type(z), z)
+w: <class 'float'> 0.9910565700695637
+x: <class 'float'> 0.9558550589791632
+y: <class 'list'> [0.42626170162229604, 0.5088767993123806, 0.8323423336075707]
+z: <class 'NoneType'> None
+# note : this might be a better implementation
+def get_random_numbers(n=1):
+ if n == 1:
+ return [random.random()]
+ elif n > 1:
+ numbers = []
+ for i in range(0, n):
+ numbers.append(random.random())
+ return numbers
+ else:
+ return []
+
+w = get_random_numbers()
+print('w:', type(w), w)
+
+x = get_random_numbers(1)
+print('x:', type(x), x)
+
+y = get_random_numbers(n=3)
+print('y:', type(y), y)
+
+z = get_random_numbers(n=-1)
+print('z:', type(z), z)
+w: <class 'list'> [0.1019992349979667]
+x: <class 'list'> [0.773928378946725]
+y: <class 'list'> [0.6769995705574616, 0.5620051030258161, 0.6386635457018367]
+z: <class 'list'> []
+Required arguments are mandatory and you will generate an error if they’re not present. Required arguments must be passed in precisely the right order, just like in the following example:
def say_something(message, number_of_times):
+ for i in range(0, number_of_times):
+ print(message)
+
+# arguments passed in the proper order
+say_something('hello', 3)
+hello
+hello
+hello
+# arguments passed incorrectly
+say_something(3, 'hello')
+---------------------------------------------------------------------------
+
+TypeError Traceback (most recent call last)
+
+/var/folders/jd/pq0swyt521jb2424d6fvth840000gn/T/ipykernel_91274/301183919.py in <module>
+ 1 # arguments passed incorrectly
+----> 2 say_something(3, 'hello')
+
+
+/var/folders/jd/pq0swyt521jb2424d6fvth840000gn/T/ipykernel_91274/2017276285.py in say_something(message, number_of_times)
+ 1 def say_something(message, number_of_times):
+----> 2 for i in range(0, number_of_times):
+ 3 print(message)
+ 4
+ 5 # arguments passed in the proper order
+
+
+TypeError: 'str' object cannot be interpreted as an integer
+You can use keyword arguments to make sure that you call all the parameters in the right order. You can do so by specifying their parameter name in the function call.
say_something(message='hello', number_of_times=3)
+say_something(number_of_times=3, message='hello')
+hello
+hello
+hello
+hello
+hello
+hello
+In cases where you don’t know the exact number of arguments that you want to pass to a function, you can use the following syntax with *args:
def add(*x):
+ print(type(x), x)
+ total = 0
+ for i in x:
+ total += i
+ return total
+
+total = add(1)
+print(total)
+
+total = add(1, 1)
+print(total)
+
+total = add(1, 2, 3, 4, 5)
+print(total)
+<class 'tuple'> (1,)
+1
+<class 'tuple'> (1, 1)
+2
+<class 'tuple'> (1, 2, 3, 4, 5)
+15
+The asterisk * is placed before the variable name that holds the values of all nonkeyword variable arguments. Note here that you might as well have passed *varint, *var_int_args or any other name to the plus() function.
# You can spedify any combination of required, keyword, and variable arguments.
+def add(a, b, *args):
+ total = a + b
+ for arg in args:
+ total += arg
+ return total
+
+total = add(1, 1)
+print(total)
+total = add(1, 1, 2)
+print(total)
+total = add(1, 2, 3, 4, 5)
+print(total)
+2
+4
+15
+In general, variables that are defined inside a function body have a local scope, and those defined outside have a global scope. That means that local variables are defined within a function block and can only be accessed inside that function, while global variables can be obtained by all functions that might be in your script:
# global variable
+score = 0
+
+def player_hit():
+ global score
+ hit_points = -10 # local variable
+ score += hit_points
+
+def enemy_hit():
+ global score
+ hit_points = 5 # local variable
+ score += hit_points
+
+enemy_hit()
+enemy_hit()
+enemy_hit()
+enemy_hit()
+player_hit()
+enemy_hit()
+player_hit()
+
+print(score)
+5
+Anonymous functions are also called lambda functions in Python because instead of declaring them with the standard def keyword, you use the lambda keyword.
def double(x):
+ return x * 2
+y = double(3)
+print(y)
+6
+d = lambda x: x * 2
+y = d(3)
+print(y)
+6
+sdlfjsdk = lambda x, n: x if n < 5 else 0
+result = sdlfjsdk(4, 6)
+print(result)
+0
+a = lambda x: x ** 2 if x < 0 else x
+print(a(-1))
+print(a(-2))
+print(a(3))
+1
+4
+3
+add = lambda x, y: x + y
+x = add(2, 3)
+print(x)
+5
+You use anonymous functions when you require a nameless function for a short period of time, and that is created at runtime. Specific contexts in which this would be relevant is when you’re working with filter(), map() and reduce():
filter() function filters the original input list on the basis of a criterion > 10.map() applies a function to all items of the listreduce() is part of the functools library. You use this function cumulatively to the items of the my_list list, from left to right and reduce the sequence to a single value.from functools import reduce
+
+my_list = [1,2,3,4,5,6,7,8,9,10]
+
+# Use lambda function with `filter()`
+filtered_list = list(filter(lambda x: (x * 2 > 10), my_list))
+
+# Use lambda function with `map()`
+mapped_list = list(map(lambda x: x * 2, my_list))
+
+# Use lambda function with `reduce()`
+reduced_list = reduce(lambda x, y: x + y, my_list)
+
+print(filtered_list)
+print(mapped_list)
+print(reduced_list)
+[6, 7, 8, 9, 10]
+[2, 4, 6, 8, 10, 12, 14, 16, 18, 20]
+55
+You can easily define a main() function and call it just like you have done with all of the other functions above:
# Define `main()` function
+def main():
+ print("This is a main function")
+
+main()
+This is a main function
+However, as it stands now, the code of your main() function will be called when you import it as a module. To make sure that this doesn’t happen, you call the main() function when __name__ == '__main__'.
# Define `main()` function
+def start_here():
+ print("This is a main function")
+
+# Execute `main()` function
+if __name__ == '__main__':
+ start_here()
+This is a main function
+
+It’s often useful, especially in healthcare, to generate fake data in order to test your system while protecting the privacy of your production data sets.
## let's generate a fake person
+person = {}
+print(person)
+{}
+person['age'] = 28
+print(person)
+{'age': 28}
+# let's generate a random age
+import random
+person['age'] = random.randrange(100)
+print(person)
+{'age': 2}
+person['age'] = random.randrange(100)
+print(person)
+{'age': 42}
+# in healthcare kids and elderly have special coverage so let's exclude both groups
+person['age'] = random.randint(18, 65)
+print(person)
+{'age': 32}
+person['employment'] = 'Full Time'
+print(person)
+{'age': 32, 'employment': 'Full Time'}
+# let's set employment status
+employment_status = ['Full Time', 'Part Time', 'Contract', 'Seasonal', 'Unemployed', 'Retired']
+person['employment'] = random.choice(employment_status)
+print(person)
+{'age': 32, 'employment': 'Unemployed'}
+person['employment'] = random.choice(employment_status)
+print(person)
+{'age': 32, 'employment': 'Retired'}
+person['employment'] = random.choices(employment_status, [.5, .2, .1, .1, .05, .05])[0]
+print(person)
+{'age': 32, 'employment': 'Part Time'}
+person['employment'] = random.choices(employment_status, [.5, .2, .1, .1, .05, .05])[0]
+print(person)
+{'age': 32, 'employment': 'Part Time'}
+def generate_employment():
+ employment_status = ['Full Time', 'Part Time', 'Contract', 'Seasonal', 'Unemployed', 'Retired']
+ employment = random.choices(employment_status, [.5, .2, .1, .1, .05, .05])[0]
+ return employment
+
+# let's construct some fake addresses
+def generate_address():
+ street_number = random.randint(1, 100)
+ street_name= random.choice(['Main', 'Bluff', 'Federal'])
+ city = random.choice(['Pittsburgh', 'Cleveland' ])
+
+ city_details = { 'Pittsburgh' : { 'zip': 15106, 'state': 'PA' },
+ 'Cleveland' : { 'zip': 44101, 'state': 'OH' } }
+ state = city_details[city]['zip']
+ zip_code = city_details[city]['state']
+
+ address = f"{street_number} {street_name}\n{city}, {state} {zip_code}"
+ return address
+address = generate_address()
+print(address)
+24 Federal
+Pittsburgh, 15106 PA
+def generate_name():
+ first = ['Ben', 'Jen', 'Joan', 'John']
+ last = ['Jones', 'Smith', 'Doe']
+ return f"{random.choice(first)} {random.choice(last)}"
+name = generate_name()
+print(name)
+John Doe
+def generate_person():
+ person = {}
+ person['name'] = generate_name()
+ person['address'] = generate_address()
+ person['employment'] = generate_employment()
+ return person
+person = generate_person()
+print(person)
+{'name': 'Ben Jones', 'address': '12 Bluff\nCleveland, 44101 OH', 'employment': 'Seasonal'}
+person = generate_person()
+print(person)
+{'name': 'John Jones', 'address': '64 Federal\nCleveland, 44101 OH', 'employment': 'Full Time'}
+person = generate_person()
+print(person)
+{'name': 'Ben Jones', 'address': '47 Federal\nPittsburgh, 15106 PA', 'employment': 'Part Time'}
+
+# import the library
+from faker import Faker
+# create an instance
+faker = Faker()
+# generate a name
+faker.name()
+'Vincent Allen'
+# generate another name
+faker.name()
+'Dawn Harvey'
+# generate an address
+faker.address()
+'11351 Elizabeth Canyon\nLake Sarah, NE 29707'
+# generate another address
+faker.address()
+'1006 Jessica Mills\nBrewerview, MO 39072'
+# generate some random text
+faker.text()
+'Help politics watch space study beautiful. If focus firm again.\nPay up subject. Will particularly have be article clear season. Participant huge name notice.'
+faker.name()
+'Amy Roberson'
+faker.first_name()
+'Antonio'
+faker.last_name()
+'Odom'
+faker.job()
+'Microbiologist'
+faker.job()
+'Health promotion specialist'
+for _ in range(5):
+ print(faker.job())
+Surveyor, insurance
+Engineer, chemical
+Cytogeneticist
+Drilling engineer
+Production assistant, radio
+faker = Faker('cz_CZ')
+
+for i in range(3):
+ name = faker.name()
+ address = faker.address()
+ phone = faker.phone_number()
+
+ print(f'{name}, {address}, {phone}')
+Miroslava Kratochvílová, Komenského Nám. 91
+595 83 Kojetín, 601 108 913
+Kryštof Bartoš, Lednická 347
+649 34 Horní Jelení, 723 227 473
+Bohuslav Urban, Na Staré Vinici 50
+609 26 Hostomice, 603 178 388
+faker = Faker()
+faker.currency()
+('GNF', 'Guinean franc')
+faker.currency_name()
+'Jersey pound'
+faker.currency_code()
+'AWG'
+faker.word()
+'care'
+faker.words(10)
+['its',
+ 'pattern',
+ 'concern',
+ 'road',
+ 'couple',
+ 'common',
+ 'value',
+ 'our',
+ 'either',
+ 'world']
+my_words = ['forrest', 'blue', 'cloud', 'sky', 'wood', 'falcon']
+faker.words(3, my_words, True)
+['blue', 'cloud', 'wood']
+profile1 = faker.simple_profile()
+print(profile1)
+{'username': 'churchjames', 'name': 'Victoria Ryan MD', 'sex': 'F', 'address': '90570 Hannah Stream\nSouth Shelly, TN 77015', 'mail': 'kelli28@yahoo.com', 'birthdate': datetime.date(1949, 7, 30)}
+import dumper
+dumper.dump(profile1)
+<dict at 0x7ffcc004d380>:
+ username: 'churchjames'
+ name: 'Victoria Ryan MD'
+ sex: 'F'
+ address: '90570 Hannah Stream\nSouth Shelly, TN 77015'
+ mail: 'kelli28@yahoo.com'
+ birthdate: <str at 0x7ffcc00503a0>: 'datetime.date(1949, 7, 30)'
+profile2 = faker.simple_profile('F')
+dumper.dump(profile2)
+<dict at 0x7ffcc0047200>:
+ username: 'qjohnston'
+ name: 'Jessica Gillespie'
+ sex: 'F'
+ address: '69654 Megan Rest\nPatriciafurt, MD 35261'
+ mail: 'jessicasmith@hotmail.com'
+ birthdate: <str at 0x7ffcc00509e0>: 'datetime.date(1949, 4, 3)'
+faker.random_int()
+1717
+faker.random_int(18, 64)
+31
+faker.random_digit()
+1
+faker.md5()
+'1e7bfd5eb691a36a94af457b383f375e'
+faker.sha1()
+'0c83ab884516f9ef6f877a4d077a16630354b001'
+faker.sha256()
+'b239734a398a4c8005beefefd98a772c896dc5966f6233d74a398988d98714d3'
+faker.uuid4()
+'b914121e-63e4-4610-a1df-2555719ba7b8'
+faker.email()
+'kellylester@example.org'
+faker.safe_email()
+'powellbenjamin@example.com'
+faker.free_email()
+'mphillips@gmail.com'
+faker.company_email()
+'seancarroll@murray.info'
+faker.hostname()
+'srv-25.williamson-johnson.com'
+faker.domain_name()
+'wilson.org'
+faker.domain_word()
+'clark'
+faker.tld()
+'org'
+faker.ipv4()
+'31.17.120.149'
+faker.ipv6()
+'e4d1:cfad:4a05:b049:cb63:5889:8094:c571'
+faker.slug()
+'occur-bed-work-yet'
+faker.image_url()
+'https://placekitten.com/727/516'
+faker.date_of_birth()
faker.century()
+'XV'
+faker.year()
+'1991'
+faker.month()
+'09'
+faker.month_name()
+'November'
+faker.day_of_week()
+'Sunday'
+faker.day_of_month()
+'01'
+faker.timezone()
+'America/Paramaribo'
+faker.am_pm()
+'AM'
+faker.date_time_this_century()
+datetime.datetime(2018, 2, 9, 20, 51, 37)
+faker.date_time_this_decade()
+datetime.datetime(2020, 1, 12, 23, 48, 1)
+faker.date_time_this_year()
+datetime.datetime(2021, 6, 11, 20, 16, 29)
+faker.date_time_this_month()
+datetime.datetime(2021, 10, 6, 18, 59, 38)
+faker.date_this_century()
+datetime.date(2013, 10, 11)
+faker.date_this_decade()
+datetime.date(2021, 4, 1)
+faker.date_this_year()
+datetime.date(2021, 8, 28)
+faker.date_this_month()
+datetime.date(2021, 10, 20)
+TOTAL_SECONDS = 60*60*24*2 # two days
+
+series = faker.time_series(start_date='-12d', end_date='now', precision=TOTAL_SECONDS)
+
+for val in series:
+ print(val[0])
+2021-10-16 15:13:18
+2021-10-18 15:13:18
+2021-10-20 15:13:18
+2021-10-22 15:13:18
+2021-10-24 15:13:18
+2021-10-26 15:13:18
+faker.unix_time()
+1251085387
+faker.date_time()
+datetime.datetime(1976, 7, 17, 20, 43, 28)
+faker.iso8601()
+'1982-11-03T07:46:42'
+faker.date()
+'1990-08-27'
+faker.time()
+'12:55:01'
+print(f"Datetime between: {faker.date_time_between(start_date='-15y', end_date='now')}")
+print(f"Date between: {faker.date_between()}")
+Datetime between: 2007-07-11 14:37:20
+Date between: 1993-01-12
+faker.future_datetime()
+datetime.datetime(2021, 11, 9, 6, 55, 23)
+faker.future_date()
+datetime.date(2021, 11, 18)
+faker.past_datetime()
+datetime.datetime(2021, 9, 29, 8, 18, 48)
+faker.past_date()
+datetime.date(2021, 10, 23)
+from jinja2 import Environment, FileSystemLoader
+class User:
+ def __init__(self, first_name, last_name, occupation):
+ self.first_name = first_name
+ self.last_name = last_name
+ self.occupation = occupation
+
+faker = Faker()
+users = []
+for _ in range(10):
+ first_name = faker.first_name()
+ last_name = faker.last_name()
+ occupation = faker.job()
+
+ user = User(first_name, last_name, occupation)
+
+ users.append(user)
+file_loader = FileSystemLoader('templates')
+env = Environment(loader=file_loader)
+
+template = env.get_template('users.xml.j2')
+output = template.render(users=users)
+print(output)
+<?xml version="1.0" encoding="UTF-8"?>
+<users>
+
+ <user id="1">
+ <firstname>Cheryl</firstname>
+ <lastname>Howard</lastname>
+ <occupation>Community arts worker</occupation>
+ </user>
+
+ <user id="2">
+ <firstname>Jessica</firstname>
+ <lastname>Miller</lastname>
+ <occupation>Magazine features editor</occupation>
+ </user>
+
+ <user id="3">
+ <firstname>Alexandra</firstname>
+ <lastname>Hill</lastname>
+ <occupation>Ceramics designer</occupation>
+ </user>
+
+ <user id="4">
+ <firstname>Lindsay</firstname>
+ <lastname>Holland</lastname>
+ <occupation>Economist</occupation>
+ </user>
+
+ <user id="5">
+ <firstname>Kevin</firstname>
+ <lastname>Michael</lastname>
+ <occupation>Nurse, children's</occupation>
+ </user>
+
+ <user id="6">
+ <firstname>Amanda</firstname>
+ <lastname>Bush</lastname>
+ <occupation>IT trainer</occupation>
+ </user>
+
+ <user id="7">
+ <firstname>Candice</firstname>
+ <lastname>Oliver</lastname>
+ <occupation>Quantity surveyor</occupation>
+ </user>
+
+ <user id="8">
+ <firstname>Brian</firstname>
+ <lastname>Powell</lastname>
+ <occupation>Advice worker</occupation>
+ </user>
+
+ <user id="9">
+ <firstname>Sarah</firstname>
+ <lastname>Graham</lastname>
+ <occupation>Heritage manager</occupation>
+ </user>
+
+ <user id="10">
+ <firstname>Frank</firstname>
+ <lastname>Acevedo</lastname>
+ <occupation>Archivist</occupation>
+ </user>
+
+</users>
+# write output to file
+print(output, file=open('users.xml', 'w'))
+
+One of the hardest things to do in data science is get access to high quality datasets that relate to your specific questions. There’s any number of reasons why researchers and analysts get incorrect results and misread the answers they’re getting. Bad data practice may be leading to bad research. Some bad data practice themes might include:
In this class we’re going to learn how to process and analyze data using python. Since I work in healthcare we’ll be using a tool called Synthea which will help us create consistent meaningful datasets at scale in a vareity of formats (Text, CSV, C-CDA, and FHIR). We’ll use these datasets to:
SyntheaTM is a synthetic patient generator that models the medical history of synthetic patients. Our mission is to output high-quality synthetic, realistic but not real, patient data and associated health records covering every aspect of healthcare. The resulting data is free from cost, privacy, and security restrictions. It can be used without restriction for a variety of secondary uses in academia, research, industry, and government (although a citation would be appreciated).
SyntheaTM generates synthetic patient records using an agent-based approach. Each synthetic patient is generated independently, as they progress from birth to death through modular representations of various diseases and conditions. Each patient runs through every module in the system. Once a patient dies or the simulation reaches the current day, that patient record can be exported in a number of different formats.
Synthea generates these datasets in a variety of commonly used healthcare data formats including (Text, CSV, C-CDA, and FHIR)
Text format is a quick human readable format. This format doesn’t adhere to any particular standard. Text formats are most commonly consumed by humans who may be clinicians or other application end users. The other data formats (CSV, C-CDA, and FHIR) can be easily converted to a Text format. However, converting a Text format to any of the other data formats is extremely challenging. Often times we use Natuarl Language Processing (NLP) and Regular Expressions (RegEx) as we attempt the Text to (CSV, C-CDA, FHIR) conversion.
Sample:
Mekhi724 Kemmer911
+==================
+Race: White
+Ethnicity: Non-Hispanic
+Gender: F
+Age: 33
+Birth Date: 1983-11-04
+Marital Status: M
+--------------------------------------------------------------------------------
+ALLERGIES: N/A
+--------------------------------------------------------------------------------
+MEDICATIONS:
+2013-08-22 [CURRENT] : Acetaminophen 160 MG for Acute bronchitis (disorder)
+1996-05-12 [CURRENT] : Acetaminophen 160 MG for Acute bronchitis (disorder)
+1995-04-13 [CURRENT] : Acetaminophen 160 MG for Acute bronchitis (disorder)
+1984-01-14 [CURRENT] : Penicillin V Potassium 250 MG for Streptococcal sore throat (disorder)
+--------------------------------------------------------------------------------
+CONDITIONS:
+2015-10-30 - 2015-11-07 : Fetus with chromosomal abnormality
+2015-10-30 - 2015-11-07 : Miscarriage in first trimester
+2015-10-30 - 2015-11-07 : Normal pregnancy
+2013-08-22 - 2013-09-08 : Acute bronchitis (disorder)
+1985-08-07 - : Food Allergy: Fish
+--------------------------------------------------------------------------------
+CARE PLANS:
+2013-08-22 [STOPPED] : Respiratory therapy
+ Reason: Acute bronchitis (disorder)
+ Activity: Recommendation to avoid exercise
+ Activity: Deep breathing and coughing exercises
+--------------------------------------------------------------------------------
+OBSERVATIONS:
+2014-01-14 : Body Weight 73.9 kg
+2014-01-14 : Body Height 163.7 cm
+2014-01-14 : Body Mass Index 27.6 kg/m2
+2014-01-14 : Systolic Blood Pressure 133.0 mmHg
+2014-01-14 : Diastolic Blood Pressure 76.0 mmHg
+2014-01-14 : Blood Pressure 2.0
+--------------------------------------------------------------------------------
+PROCEDURES:
+2015-10-30 : Standard pregnancy test for Normal pregnancy
+2014-01-14 : Documentation of current medications
+--------------------------------------------------------------------------------
+ENCOUNTERS:
+2015-11-07 : Encounter for Fetus with chromosomal abnormality
+2015-10-30 : Encounter for Normal pregnancy
+2014-01-14 : Outpatient Encounter
+2013-08-22 : Encounter for Acute bronchitis (disorder)
+--------------------------------------------------------------------------------
+Comma Separated Value (CSV) files are common in healthcare and one could argue one of the 3 most common data formats with the others being HL7v2 and C-CDA. Unlike the Text format generated by Synthea, which only contains a single patient per file, the CSV format contains many patients per file. However, the files themselves are “resource” based. The resources generated include:
Sample:
patients.csv
ID,BIRTHDATE,DEATHDATE,SSN,DRIVERS,PASSPORT,PREFIX,FIRST,LAST,SUFFIX,MAIDEN,MARITAL,RACE,ETHNICITY,GENDER,BIRTHPLACE,ADDRESS
+5e0d195e-1cd9-494d-8f9a-757c15da2aed,1946-12-14,2015-10-03,999-12-2377,S99962866,false,Mrs.,Miracle267,Ledner332,,Raynor597,M,white,irish,F,Millbury MA,2502 Fisher Manor Boston MA 02132
+52082709-06ce-4fde-9c93-cfb4e6542ae1,1968-05-23,,999-17-1808,S99941406,X41451685X,Mrs.,Alda869,Gorczany848,,Funk527,M,white,italian,F,Gardner MA,46973 Velda Gateway Franklin Town MA 02038
+8b4c62c8-b116-4b58-9259-466485b0345c,1967-06-22,1985-07-04,999-11-1173,S99955795,,Ms.,Moshe832,Zulauf396,,,,white,english,F,Boston MA,250 Reba Park Carver MA 02330
+965c5539-598b-4a9b-a670-e0259667deb8,1934-11-04,2015-06-19,999-63-2195,S99931866,X71888970X,Mr.,Verla554,Roberts329,,,S,white,irish,M,Fall River MA,321 Abdullah Bridge Needham MA 02492
+2b28d6c3-9e0c-48d4-99f9-292488133101,1964-08-13,,999-55-5054,S99990374,X68574707X,Ms.,Henderson277,Labadie810,,,S,black,dominican,F,North Attleborough MA,55825 Barrows Prairie Suite 144 Boston MA 02134
+conditions.csv
START,STOP,PATIENT,ENCOUNTER,CODE,DESCRIPTION
+1965-10-10,,5e0d195e-1cd9-494d-8f9a-757c15da2aed,918b17f4-e815-44ef-9eeb-41953bbcf7e9,38341003,Hypertension
+1966-09-09,,5e0d195e-1cd9-494d-8f9a-757c15da2aed,918b17f4-e815-44ef-9eeb-41953bbcf7e9,15777000,Prediabetes
+1988-09-25,,5e0d195e-1cd9-494d-8f9a-757c15da2aed,918b17f4-e815-44ef-9eeb-41953bbcf7e9,239872002,Osteoarthritis of hip
+1990-09-01,,5e0d195e-1cd9-494d-8f9a-757c15da2aed,918b17f4-e815-44ef-9eeb-41953bbcf7e9,410429000,Cardiac Arrest
+1990-09-01,,5e0d195e-1cd9-494d-8f9a-757c15da2aed,918b17f4-e815-44ef-9eeb-41953bbcf7e9,429007001,History of cardiac arrest (situation)
+We’ll talk more about each of these resources below when we discuss the FHIR data format.
Consolidated Clinical Document Architecture (C-CDA) format is an XML-based standard defined by HL7, that uses templates from a standard library to represent clinical concepts. For more information on C-CDA, see http://www.hl7.org/implement/standards/product_brief.cfm?product_id=258.
Sample:
<?xml version="1.0" encoding="UTF-8"?>
+<ClinicalDocument xmlns="urn:hl7-org:v3" xmlns:sdtc="urn:hl7-org:sdtc" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="urn:hl7-org:v3 http://xreg2.nist.gov:8080/hitspValidation/schema/cdar2c32/infrastructure/cda/C32_CDA.xsd">
+ <realmCode code="US"/>
+ <typeId root="2.16.840.1.113883.1.3" extension="POCD_HD000040"/>
+ <templateId root="2.16.840.1.113883.10.20.22.1.1" extension="2015-08-01"/>
+ <templateId root="2.16.840.1.113883.10.20.22.1.2" extension="2015-08-01"/>
+ <id root="2.16.840.1.113883.19.5" extension="47b10305-eb1b-4a47-a27c-24b7e96ee1da" assigningAuthorityName="https://github.com/synthetichealth/synthea"/>
+ <code code="34133-9" displayName="Summarization of episode note" codeSystem="2.16.840.1.113883.6.1" codeSystemName="LOINC"/>
+ <title>C-CDA R2.1 Patient Record: Augustine565 Cummings51</title>
+ <effectiveTime value="20190308211454"/>
+ <confidentialityCode code="N"/>
+ <languageCode code="en-US"/>
+ <recordTarget>
+ <patientRole>
+ <id root="2.16.840.1.113883.19.5" extension="47b10305-eb1b-4a47-a27c-24b7e96ee1da" assigningAuthorityName="https://github.com/synthetichealth/synthea"/>
+ <addr use="HP">
+ <streetAddressLine>931 Watsica Lock</streetAddressLine>
+ <city>Pittsburgh</city>
+ <state>Pennsylvania</state>
+ <postalCode>15106</postalCode>
+ </addr>
+ <telecom nullFlavor="NI"/>
+ <patient>
+ <name>
+ <given>Augustine565</given>
+ <family>Cummings51</family>
+ </name>
+ <administrativeGenderCode code="F" codeSystem="2.16.840.1.113883.5.1" codeSystemName="HL7 AdministrativeGender"/>
+ <birthTime value="19690509221454"/>
+ <raceCode code="2028-9" displayName="asian" codeSystemName="CDC Race and Ethnicity" codeSystem="2.16.840.1.113883.6.238"/>
+ <ethnicGroupCode code="2186-5" displayName="non-hispanic" codeSystemName="CDC Race and Ethnicity" codeSystem="2.16.840.1.113883.6.238"/>
+ <languageCommunication>
+ <languageCode code="en-US"/>
+ </languageCommunication>
+ </patient>
+ </patientRole>
+ </recordTarget>
+ <!-- ... -->
+</ClinicalDocument>
+HL7 FHIR is possibly the most exciting and interesting data format to deal with. Many of the formats discussed above were created in a time where the vast majority of Health IT systems ran on-premise. These data format and transport protocol standards are extremely reliable, but not so friendly to the web developer.
HL7 FHIR started as a community response to the legacy and hard to deal with Health IT standards. In just a few years the community has grown dramatically and most (if not all) EMR vendors have some support for the standard which is unheard of in the Health IT space. The primary drive for this rapid pace has been the US governement who has released a number of regulations and mandates for open data access.
FHIR® – Fast Healtcare Interoperability Resources (hl7.org/fhir) – is a next generation standards framework created by HL7. FHIR combines the best features of HL7’s Version 2, Version 3 and CDA® product lines while leveraging the latest web standards and applying a tight focus on implementability.
SyntheaTM currently supports exporting patients as Fast Healthcare Interoperability Resources (FHIR), versions 3.5.0 (R4), 3.0.1 (STU3) and 1.0.2 (DSTU2). FHIR is a standard created by HL7 for exchanging healthcare information electronically. While FHIR supports both XML and JSON, Synthea exports FHIR as JSON only.
Sample:
{
+ "resourceType": "Bundle",
+ "type": "transaction",
+ "entry": [
+ {
+ "fullUrl": "urn:uuid:4bd23de9-7d28-48a5-8093-1ac7ff1c64b7",
+ "resource": {
+ "resourceType": "Patient",
+ "id": "4bd23de9-7d28-48a5-8093-1ac7ff1c64b7",
+ "text": {
+ "status": "generated",
+ "div": "<div xmlns=\"http://www.w3.org/1999/xhtml\">Generated by <a href=\"https://github.com/synthetichealth/synthea\">Synthea</a>.Version identifier: v2.4.0-44-g6dbf88c6\n . Person seed: -1236052134575208584 Population seed: 12345</div>"
+ },
+ "name": [
+ {
+ "use": "official",
+ "family": "Cummings51",
+ "given": [
+ "Augustine565"
+ ],
+ "prefix": [
+ "Mrs."
+ ]
+ },
+ {
+ "use": "maiden",
+ "family": "Cremin516",
+ "given": [
+ "Augustine565"
+ ],
+ "prefix": [
+ "Mrs."
+ ]
+ }
+ ]
+ }]
+}
+
+\n",
+ " C-CDA R2.1 Patient Record: Augustine565 Cummings51 \n",
+ " \n",
+ " \n",
+ " \n",
+ " \n",
+ " \n",
+ " \n",
+ " \n",
+ " 931 Watsica Lock \n",
+ " Pittsburgh \n",
+ " Pennsylvania \n",
+ " 15106 \n",
+ " \n",
+ " \n",
+ " \n",
+ " \n",
+ " Augustine565 \n",
+ " Cummings51 \n",
+ " \n",
+ " \n",
+ " \n",
+ " \n",
+ " \n",
+ " \n",
+ " \n",
+ " \n",
+ " \n",
+ " \n",
+ " \n",
+ " In a termainal, change the directory to where synthea is installed and execute the run_synthea script
cd ~/Source/synthea
+./run_synthea
+cd c:\users\kolowitz\synthea
+run_synthea.bat
+Note: if running on Windows, use .\run_synthea.bat instead of ./run_synthea – this guide uses ./run_synthea for brevity
import os
+import pathlib
+
+os.chdir(os.path.join(pathlib.Path.home(), 'Source/synthea'))
+# generate a population of 5 patients
+!./run_synthea -p 5
+# generate a population of 5 patients in Pittsburgh Pennsylvania
+!./run_synthea -p 5 Pennsylvania Pittsburgh
+# generate a population of 5 patients with a seed of 12345
+!./run_synthea -p 5 -s 12345
+| Setting Name | Valid Values | Default | Description |
|---|---|---|---|
| exporter.ccda.export | true/false | false | Change this setting to true to enable exporting patients in CCDA format. |
| exporter.fhir.export | true/false | true | Change this setting to false to disable exporting patients in FHIR R4 format. |
| exporter.text.export | true/false | false | Change this setting to true to enable exporting patients in a simple text-based format. |
| exporter.csv.export | true/false | false | Change this setting to true to enable exporting patient data in a comma-separated value format. See the CSV File Data Dictionary. |
| exporter.years_of_history | Whole number | 10 | The number of years of patient history to include in patient records. For example, if set to 5, then all patient history older than 5 years old (from the time you execute the program) will not be included in the exported records. Note that conditions and medications that are currently active will still be exported, regardless of this setting. Set this to 0 to keep all history in the patient record. |
!cat ./src/main/resources/synthea.properties
+cat: ./src/main/resources/synthea.properties: No such file or directory
+
+Imagine you are tinkering with a cool machine learning algorithm like SVM or a deep neural net. What kind of dataset you should practice them on? If you are learning from scratch, the advice is to start with simple, small-scale datasets which you can plot in two dimensions to understand the patterns visually and see for yourself the working of the ML algorithm in an intuitive fashion. For example, here is an excellent article on various datasets you can try at various level of learning.
As the name suggests, quite obviously, a synthetic dataset is a repository of data that is generated programmatically. So, it is not collected by any real-life survey or experiment. Its main purpose, therefore, is to be flexible and rich enough to help an ML practitioner conduct fascinating experiments with various classification, regression, and clustering algorithms. Desired properties are,
Although in this article, we keep our discussions limited to synthetic data for better ML algorithms, its purpose can be far reaching in cases where it helps get around security and privacy concerns with real datasets, that cannot be used or acquired for learning purpose. For example, think about medical or military data. Here is an excellent summary article about such methods.
import numpy as np
+import pandas as pd
+import matplotlib.pyplot as plt
+import seaborn as sns
+Regression problem generation: Scikit-learn’s dataset.make_regression function can create random regression problem with arbitrary number of input features, output targets, and controllable degree of informative coupling between them. It can also mix Gaussian noise.
from sklearn.datasets import make_regression
+data1 = make_regression(n_samples=20, n_features=4, n_informative=2, n_targets=1,
+ bias=0.0, effective_rank=None,tail_strength=0.5,
+ noise=0.0, shuffle=True, coef=False, random_state=None)
+df1 = pd.DataFrame(data1[0],columns=['x'+str(i) for i in range(1,5)])
+df1['y'] = data1[1]
+df1.head()
+| x1 | x2 | x3 | x4 | y | |
|---|---|---|---|---|---|
| 0 | 2.326839 | 2.203741 | -0.152775 | -0.682349 | 70.815411 |
| 1 | -0.584208 | 0.097921 | -0.045379 | 1.043689 | 1.670565 |
| 2 | 0.100703 | -0.141215 | 1.752348 | 1.166683 | 62.111800 |
| 3 | 0.308948 | 1.741082 | 1.159611 | -1.625781 | 104.917778 |
| 4 | 1.320515 | 2.320092 | 1.882030 | -1.058409 | 152.690260 |
plt.figure(figsize=(15,10))
+for i in range(1,5):
+ fit = np.polyfit(df1[df1.columns[i-1]],df1['y'],1)
+ fit_fn = np.poly1d(fit)
+ plt.subplot(2,2,i)
+ plt.scatter(df1[df1.columns[i-1]],df1['y'],s=200,c='orange',edgecolor='k')
+ plt.plot(df1[df1.columns[i-1]],fit_fn(df1[df1.columns[i-1]]),'b-',lw=3)
+ plt.grid(True)
+
data2 = make_regression(n_samples=20, n_features=4, n_informative=2, n_targets=1,
+ bias=0.0, effective_rank=None,tail_strength=0.5,
+ noise=2.0, shuffle=True, coef=False, random_state=None)
+df2 = pd.DataFrame(data2[0],columns=['x'+str(i) for i in range(1,5)])
+df2['y'] = data2[1]
+plt.figure(figsize=(15,10))
+for i in range(1,5):
+ fit = np.polyfit(df2[df2.columns[i-1]],df2['y'],1)
+ fit_fn = np.poly1d(fit)
+ plt.subplot(2,2,i)
+ plt.scatter(df2[df2.columns[i-1]],df2['y'],s=200,c='orange',edgecolor='k')
+ plt.plot(df2[df2.columns[i-1]],fit_fn(df2[df2.columns[i-1]]),'b-',lw=3)
+ plt.grid(True)
+
plt.figure(figsize=(15,6))
+df2 = pd.DataFrame(data=np.zeros((20,1)))
+for i in range(3):
+ data2 = make_regression(n_samples=20, n_features=1, n_informative=1, n_targets=1,
+ bias=0.0, effective_rank=None,tail_strength=0.5,
+ noise=i*10, shuffle=True, coef=False, random_state=None)
+ df2['x'+str(i+1)]=data2[0]
+ df2['y'+str(i+1)] = data2[1]
+
+for i in range(3):
+ fit = np.polyfit(df2['x'+str(i+1)],df2['y'+str(i+1)],1)
+ fit_fn = np.poly1d(fit)
+ plt.subplot(1,3,i+1)
+ plt.scatter(df2['x'+str(i+1)],df2['y'+str(i+1)],s=200,c='orange',edgecolor='k')
+ plt.plot(df2['x'+str(i+1)],fit_fn(df2['x'+str(i+1)]),'b-',lw=3)
+ plt.grid(True)
+
Classification problem generation: Similar to the regression function above, dataset.make_classification generates a random multi-class classification problem (dataset) with controllable class separation and added noise. You can also randomly flip any percentage of output signs to create a harder classification dataset if you want.
from sklearn.datasets import make_classification
+data3 = make_classification(n_samples=20, n_features=4, n_informative=4, n_redundant=0, n_repeated=0,
+ n_classes=2, n_clusters_per_class=1, weights=None, flip_y=0.01, class_sep=1.0,
+ hypercube=True, shift=0.0, scale=1.0, shuffle=True, random_state=None)
+df3 = pd.DataFrame(data3[0],columns=['x'+str(i) for i in range(1,5)])
+df3['y'] = data3[1]
+df3.head()
+| x1 | x2 | x3 | x4 | y | |
|---|---|---|---|---|---|
| 0 | -0.937800 | 0.620428 | 1.710371 | -1.335103 | 0 |
| 1 | -1.843944 | 1.498890 | -0.950427 | -0.315925 | 0 |
| 2 | -0.670669 | 0.493982 | -0.687241 | -1.206348 | 0 |
| 3 | 3.140552 | -2.195994 | 1.571735 | -0.247905 | 1 |
| 4 | 1.602336 | -2.151781 | 4.427198 | -4.207194 | 0 |
from itertools import combinations
+from math import ceil
+lst_var=list(combinations(df3.columns[:-1],2))
+len_var = len(lst_var)
+plt.figure(figsize=(18,10))
+for i in range(1,len_var+1):
+ plt.subplot(2,ceil(len_var/2),i)
+ var1 = lst_var[i-1][0]
+ var2 = lst_var[i-1][1]
+ plt.scatter(df3[var1],df3[var2],s=200,c=df3['y'],edgecolor='k')
+ plt.xlabel(var1,fontsize=14)
+ plt.ylabel(var2,fontsize=14)
+ plt.grid(True)
+
class_sep data3 = make_classification(n_samples=20, n_features=4, n_informative=4, n_redundant=0, n_repeated=0,
+ n_classes=2, n_clusters_per_class=1, weights=None, flip_y=0.01, class_sep=3.0,
+ hypercube=True, shift=0.0, scale=1.0, shuffle=True, random_state=None)
+df3 = pd.DataFrame(data3[0],columns=['x'+str(i) for i in range(1,5)])
+df3['y'] = data3[1]
+from itertools import combinations
+from math import ceil
+lst_var=list(combinations(df3.columns[:-1],2))
+len_var = len(lst_var)
+plt.figure(figsize=(18,10))
+for i in range(1,len_var+1):
+ plt.subplot(2,ceil(len_var/2),i)
+ var1 = lst_var[i-1][0]
+ var2 = lst_var[i-1][1]
+ plt.scatter(df3[var1],df3[var2],s=200,c=df3['y'],edgecolor='k')
+ plt.xlabel(var1,fontsize=14)
+ plt.ylabel(var2,fontsize=14)
+ plt.grid(True)
+
class_sep data3 = make_classification(n_samples=20, n_features=4, n_informative=4, n_redundant=0, n_repeated=0,
+ n_classes=2, n_clusters_per_class=1, weights=None, flip_y=0.01, class_sep=0.5,
+ hypercube=True, shift=0.0, scale=1.0, shuffle=True, random_state=None)
+df3 = pd.DataFrame(data3[0],columns=['x'+str(i) for i in range(1,5)])
+df3['y'] = data3[1]
+from itertools import combinations
+from math import ceil
+lst_var=list(combinations(df3.columns[:-1],2))
+len_var = len(lst_var)
+plt.figure(figsize=(18,10))
+for i in range(1,len_var+1):
+ plt.subplot(2,ceil(len_var/2),i)
+ var1 = lst_var[i-1][0]
+ var2 = lst_var[i-1][1]
+ plt.scatter(df3[var1],df3[var2],s=200,c=df3['y'],edgecolor='k')
+ plt.xlabel(var1,fontsize=14)
+ plt.ylabel(var2,fontsize=14)
+ plt.grid(True)
+
flip_y plt.figure(figsize=(18,10))
+for i in range(6):
+ data3 = make_classification(n_samples=20, n_features=4, n_informative=4, n_redundant=0, n_repeated=0,
+ n_classes=2, n_clusters_per_class=1, weights=None, flip_y=0.1*i, class_sep=1.0,
+ hypercube=True, shift=0.0, scale=1.0, shuffle=False, random_state=101)
+ df3 = pd.DataFrame(data3[0],columns=['x'+str(i) for i in range(1,5)])
+ df3['y'] = data3[1]
+ plt.subplot(2,3,i+1)
+ plt.title(f"Plot for flip_y={round(0.1*i,2)}")
+ plt.scatter(df3['x1'],df3['x2'],s=200,c=df3['y'],edgecolor='k')
+ plt.xlabel('x1',fontsize=14)
+ plt.ylabel('x2',fontsize=14)
+ plt.grid(True)
+
plt.figure(figsize=(18,5))
+df2 = pd.DataFrame(data=np.zeros((20,1)))
+for i in range(3):
+ data2 = make_classification(n_samples=20, n_features=2, n_informative=2, n_redundant=0, n_repeated=0,
+ n_classes=2, n_clusters_per_class=1, weights=None, flip_y=0, class_sep=i+0.5,
+ hypercube=True, shift=0.0, scale=1.0, shuffle=False, random_state=101)
+ df2['x'+str(i+1)+'1']=data2[0][:,0]
+ df2['x'+str(i+1)+'2']=data2[0][:,1]
+ df2['y'+str(i+1)] = data2[1]
+
+for i in range(3):
+ plt.subplot(1,3,i+1)
+ plt.scatter(df2['x'+str(i+1)+'1'],df2['x'+str(i+1)+'2'],s=200,c=df2['y'+str(i+1)],edgecolor='k')
+ plt.grid(True)
+
Clustering problem generation: There are quite a few functions for generating interesting clusters. The most straightforward one is datasets.make_blobs, which generates arbitrary number of clusters with controllable distance parameters.
from sklearn.datasets import make_blobs
+data4 = make_blobs(n_samples=60, n_features=4, centers=3, cluster_std=1.0,
+ center_box=(-5.0, 5.0), shuffle=True, random_state=None)
+df4 = pd.DataFrame(data4[0],columns=['x'+str(i) for i in range(1,5)])
+df4['y'] = data4[1]
+from itertools import combinations
+from math import ceil
+lst_var=list(combinations(df4.columns[:-1],2))
+len_var = len(lst_var)
+plt.figure(figsize=(18,10))
+for i in range(1,len_var+1):
+ plt.subplot(2,ceil(len_var/2),i)
+ var1 = lst_var[i-1][0]
+ var2 = lst_var[i-1][1]
+ plt.scatter(df4[var1],df4[var2],s=200,c=df4['y'],edgecolor='k')
+ plt.xlabel(var1,fontsize=14)
+ plt.ylabel(var2,fontsize=14)
+ plt.grid(True)
+
cluster_std data4 = make_blobs(n_samples=60, n_features=4, centers=3, cluster_std=0.3,
+ center_box=(-5.0, 5.0), shuffle=True, random_state=None)
+df4 = pd.DataFrame(data4[0],columns=['x'+str(i) for i in range(1,5)])
+df4['y'] = data4[1]
+from itertools import combinations
+from math import ceil
+lst_var=list(combinations(df4.columns[:-1],2))
+len_var = len(lst_var)
+plt.figure(figsize=(18,10))
+for i in range(1,len_var+1):
+ plt.subplot(2,ceil(len_var/2),i)
+ var1 = lst_var[i-1][0]
+ var2 = lst_var[i-1][1]
+ plt.scatter(df4[var1],df4[var2],s=200,c=df4['y'],edgecolor='k')
+ plt.xlabel(var1,fontsize=14)
+ plt.ylabel(var2,fontsize=14)
+ plt.grid(True)
+
cluster_std data4 = make_blobs(n_samples=60, n_features=4, centers=3, cluster_std=2.5,
+ center_box=(-5.0, 5.0), shuffle=True, random_state=None)
+df4 = pd.DataFrame(data4[0],columns=['x'+str(i) for i in range(1,5)])
+df4['y'] = data4[1]
+from itertools import combinations
+from math import ceil
+lst_var=list(combinations(df4.columns[:-1],2))
+len_var = len(lst_var)
+plt.figure(figsize=(18,10))
+for i in range(1,len_var+1):
+ plt.subplot(2,ceil(len_var/2),i)
+ var1 = lst_var[i-1][0]
+ var2 = lst_var[i-1][1]
+ plt.scatter(df4[var1],df4[var2],s=200,c=df4['y'],edgecolor='k')
+ plt.xlabel(var1,fontsize=14)
+ plt.ylabel(var2,fontsize=14)
+ plt.grid(True)
+
Anisotropic cluster generation: With a simple transformation using matrix multiplication, you can generate clusters which is aligned along certain axis or anisotropically distributed.
data5 = make_blobs(n_samples=50, n_features=2, centers=3,cluster_std=1.5)
+transformation = [[0.5, -0.5], [-0.4, 0.8]]
+data5_0=np.dot(data5[0],transformation)
+df5 = pd.DataFrame(data5_0,columns=['x'+str(i) for i in range(1,3)])
+df5['y'] = data5[1]
+plt.figure(figsize=(8,5))
+plt.scatter(df5['x1'],df5['x2'],c=df5['y'],s=200,edgecolors='k')
+plt.xlabel('x1',fontsize=14)
+plt.ylabel('x2',fontsize=14)
+plt.grid(True)
+plt.show()
+
Concentric ring cluster data generation: For testing affinity based clustering algorithm or Gaussian mixture models, it is useful to have clusters generated in a special shape. We can use datasets.make_circles function to accomplish that.
from sklearn.datasets import make_circles
+data6 = make_circles(n_samples=50, shuffle=True, noise=None, random_state=None, factor=0.6)
+df6 = pd.DataFrame(data6[0],columns=['x'+str(i) for i in range(1,3)])
+df6['y'] = data6[1]
+plt.figure(figsize=(8,5))
+plt.scatter(df6['x1'],df6['x2'],c=df6['y'],s=200,edgecolors='k')
+plt.xlabel('x1',fontsize=14)
+plt.ylabel('x2',fontsize=14)
+plt.grid(True)
+plt.show()
+
data6 = make_circles(n_samples=50, shuffle=True, noise=0.15, random_state=None, factor=0.6)
+df6 = pd.DataFrame(data6[0],columns=['x'+str(i) for i in range(1,3)])
+df6['y'] = data6[1]
+plt.figure(figsize=(8,5))
+plt.scatter(df6['x1'],df6['x2'],c=df6['y'],s=200,edgecolors='k')
+plt.xlabel('x1',fontsize=14)
+plt.ylabel('x2',fontsize=14)
+plt.grid(True)
+plt.show()
+
Moon-shaped cluster data generation: We can also generate moon-shaped cluster data for testing algorithms, with controllable noise using datasets.make_moons function.
from sklearn.datasets import make_moons
+data7 = make_moons(n_samples=50, shuffle=True, noise=None, random_state=None)
+df7 = pd.DataFrame(data7[0],columns=['x'+str(i) for i in range(1,3)])
+df7['y'] = data7[1]
+plt.figure(figsize=(8,5))
+plt.scatter(df7['x1'],df7['x2'],c=df7['y'],s=200,edgecolors='k')
+plt.xlabel('x1',fontsize=14)
+plt.ylabel('x2',fontsize=14)
+plt.grid(True)
+plt.show()
+
data7 = make_moons(n_samples=50, shuffle=True, noise=0.1, random_state=None)
+df7 = pd.DataFrame(data7[0],columns=['x'+str(i) for i in range(1,3)])
+df7['y'] = data7[1]
+plt.figure(figsize=(8,5))
+plt.scatter(df7['x1'],df7['x2'],c=df7['y'],s=200,edgecolors='k')
+plt.xlabel('x1',fontsize=14)
+plt.ylabel('x2',fontsize=14)
+plt.grid(True)
+plt.show()
+
from Symbolic_regression_classification_generator import gen_regression_symbolic, gen_classification_symbolic
+\(\frac{x_1^2}{2}-3x_2+20.\text{sin}(x_3)\)
data8 = gen_regression_symbolic(m='((x1^2)/2-3*x2)+20*sin(x3)',n_samples=50,noise=0.01)
+df8=pd.DataFrame(data8, columns=['x'+str(i) for i in range(1,4)]+['y'])
+df8.head()
+| x1 | x2 | x3 | y | |
|---|---|---|---|---|
| 0 | -1.995143 | 1.969439 | -6.585953 | -9.86940209041576 |
| 1 | 6.694431 | 10.646075 | -9.944827 | 0.416231121214657 |
| 2 | 0.824618 | -2.154371 | 1.861316 | 25.9776280245960 |
| 3 | -3.778498 | 4.509513 | 11.273044 | -25.6259760456950 |
| 4 | -1.664324 | 3.435822 | -4.291008 | 9.33521803074928 |
plt.figure(figsize=(18,5))
+for i in range(1,4):
+ plt.subplot(1,3,i)
+ plt.scatter(df8[df8.columns[i-1]],df8['y'],s=200,c='orange',edgecolor='k')
+ plt.grid(True)
+
\(x_1^2*sin(x_1)\)
data8 = 0.1*gen_regression_symbolic(m='x1^2*sin(x1)',n_samples=200,noise=0.05)
+df8=pd.DataFrame(data8, columns=['x'+str(i) for i in range(1,2)]+['y'])
+plt.figure(figsize=(8,5))
+plt.scatter(df8['x1'],df8['y'],s=100,c='orange',edgecolor='k')
+plt.grid(True)
+
\(\frac{x_1^2}{3}-\frac{x_2^2}{15}\)
data9 = gen_classification_symbolic(m='((x1^2)/3-(x2^2)/15)',n_samples=500,flip_y=0.01)
+df9=pd.DataFrame(data9, columns=['x'+str(i) for i in range(1,3)]+['y'])
+df9.head()
+| x1 | x2 | y | |
|---|---|---|---|
| 0 | 0.602170 | -6.029703 | 0.0 |
| 1 | 3.255014 | 0.452398 | 1.0 |
| 2 | 7.986863 | 8.692522 | 1.0 |
| 3 | 4.936118 | 3.508951 | 1.0 |
| 4 | 4.838671 | 4.002539 | 1.0 |
plt.figure(figsize=(8,5))
+plt.scatter(df9['x1'],df9['x2'],c=df9['y'],s=100,edgecolors='k')
+plt.xlabel('x1',fontsize=14)
+plt.ylabel('x2',fontsize=14)
+plt.grid(True)
+plt.show()
+
\(x_1-3.\text{sin}\frac{x_2}{2}\)
data9 = gen_classification_symbolic(m='x1-3*sin(x2/2)',n_samples=500,flip_y=0.01)
+df9=pd.DataFrame(data9, columns=['x'+str(i) for i in range(1,3)]+['y'])
+plt.figure(figsize=(8,5))
+plt.scatter(df9['x1'],df9['x2'],c=df9['y'],s=100,edgecolors='k')
+plt.xlabel('x1',fontsize=14)
+plt.ylabel('x2',fontsize=14)
+plt.grid(True)
+plt.show()
+
While many high-quality real-life datasets are available on the web for trying out cool machine learning techniques, from my personal experience, I found that the same is not true when it comes to learning SQL.
For data science expertise, having a basic familiarity of SQL is almost as important as knowing how to write code in Python or R. But access to a large enough database with real categorical data (such as name, age, credit card, SSN, address, birthday, etc.) is not nearly as common as access to toy datasets on Kaggle, specifically designed or curated for machine learning task.
Apart from the beginners in data science, even seasoned software testers may find it useful to have a simple tool where with a few lines of code they can generate arbitrarily large data sets with random (fake) yet meaningful entries.
Enter pydbgen. Read the docs here.
It is a lightweight, pure-python library to generate random useful entries (e.g. name, address, credit card number, date, time, company name, job title, license plate number, etc.) and save them in either Pandas dataframe object, or as a SQLite table in a database file, or in a MS Excel file.
Faker package This is a package you need to install first with pip install faker.
from faker import Faker
+fake = Faker()
+fake.name()
+'Valerie Lynch'
+fake.license_plate()
+'577-EQG'
+df = pd.DataFrame()
+for i in range(20):
+ data = {
+ 'name': fake.name(),
+ 'street_address': fake.street_address(),
+ 'city': fake.city(),
+ 'zip_code': fake.zipcode(),
+ 'ssn': fake.ssn(),
+ 'email': fake.email(),
+ 'date': fake.date()
+ }
+ df = df.append(data, True)
+df.head(5)
+| name | street_address | city | zip_code | ssn | date | ||
|---|---|---|---|---|---|---|---|
| 0 | Molly Cook | 5539 Gray Inlet Suite 416 | Whitestad | 23319 | 524-03-0773 | anoble@example.com | 1987-03-17 |
| 1 | Katherine Jones | 070 Christina Wells Suite 498 | North Daniel | 04028 | 586-43-5250 | esolis@example.net | 1984-05-21 |
| 2 | Julia Reese | 790 Teresa Cove Suite 479 | Espinozaton | 17207 | 848-13-7165 | jhanson@example.org | 1974-11-02 |
| 3 | Christine Rogers | 0797 Gibson Cove Suite 290 | Silvachester | 52803 | 406-94-4118 | williamschad@example.net | 1983-05-08 |
| 4 | Dennis Mccoy | 109 Michelle Rest | Gomezville | 39863 | 572-81-2883 | fosterjames@example.org | 1982-05-23 |
+| \n", + " | x1 | \n", + "x2 | \n", + "x3 | \n", + "x4 | \n", + "y | \n", + "
|---|---|---|---|---|---|
| 0 | \n", + "2.326839 | \n", + "2.203741 | \n", + "-0.152775 | \n", + "-0.682349 | \n", + "70.815411 | \n", + "
| 1 | \n", + "-0.584208 | \n", + "0.097921 | \n", + "-0.045379 | \n", + "1.043689 | \n", + "1.670565 | \n", + "
| 2 | \n", + "0.100703 | \n", + "-0.141215 | \n", + "1.752348 | \n", + "1.166683 | \n", + "62.111800 | \n", + "
| 3 | \n", + "0.308948 | \n", + "1.741082 | \n", + "1.159611 | \n", + "-1.625781 | \n", + "104.917778 | \n", + "
| 4 | \n", + "1.320515 | \n", + "2.320092 | \n", + "1.882030 | \n", + "-1.058409 | \n", + "152.690260 | \n", + "
| \n", + " | x1 | \n", + "x2 | \n", + "x3 | \n", + "x4 | \n", + "y | \n", + "
|---|---|---|---|---|---|
| 0 | \n", + "-0.937800 | \n", + "0.620428 | \n", + "1.710371 | \n", + "-1.335103 | \n", + "0 | \n", + "
| 1 | \n", + "-1.843944 | \n", + "1.498890 | \n", + "-0.950427 | \n", + "-0.315925 | \n", + "0 | \n", + "
| 2 | \n", + "-0.670669 | \n", + "0.493982 | \n", + "-0.687241 | \n", + "-1.206348 | \n", + "0 | \n", + "
| 3 | \n", + "3.140552 | \n", + "-2.195994 | \n", + "1.571735 | \n", + "-0.247905 | \n", + "1 | \n", + "
| 4 | \n", + "1.602336 | \n", + "-2.151781 | \n", + "4.427198 | \n", + "-4.207194 | \n", + "0 | \n", + "
| \n", + " | x1 | \n", + "x2 | \n", + "x3 | \n", + "y | \n", + "
|---|---|---|---|---|
| 0 | \n", + "-1.995143 | \n", + "1.969439 | \n", + "-6.585953 | \n", + "-9.86940209041576 | \n", + "
| 1 | \n", + "6.694431 | \n", + "10.646075 | \n", + "-9.944827 | \n", + "0.416231121214657 | \n", + "
| 2 | \n", + "0.824618 | \n", + "-2.154371 | \n", + "1.861316 | \n", + "25.9776280245960 | \n", + "
| 3 | \n", + "-3.778498 | \n", + "4.509513 | \n", + "11.273044 | \n", + "-25.6259760456950 | \n", + "
| 4 | \n", + "-1.664324 | \n", + "3.435822 | \n", + "-4.291008 | \n", + "9.33521803074928 | \n", + "
| \n", + " | x1 | \n", + "x2 | \n", + "y | \n", + "
|---|---|---|---|
| 0 | \n", + "0.602170 | \n", + "-6.029703 | \n", + "0.0 | \n", + "
| 1 | \n", + "3.255014 | \n", + "0.452398 | \n", + "1.0 | \n", + "
| 2 | \n", + "7.986863 | \n", + "8.692522 | \n", + "1.0 | \n", + "
| 3 | \n", + "4.936118 | \n", + "3.508951 | \n", + "1.0 | \n", + "
| 4 | \n", + "4.838671 | \n", + "4.002539 | \n", + "1.0 | \n", + "
| \n", + " | name | \n", + "street_address | \n", + "city | \n", + "zip_code | \n", + "ssn | \n", + "date | \n", + "|
|---|---|---|---|---|---|---|---|
| 0 | \n", + "Molly Cook | \n", + "5539 Gray Inlet Suite 416 | \n", + "Whitestad | \n", + "23319 | \n", + "524-03-0773 | \n", + "anoble@example.com | \n", + "1987-03-17 | \n", + "
| 1 | \n", + "Katherine Jones | \n", + "070 Christina Wells Suite 498 | \n", + "North Daniel | \n", + "04028 | \n", + "586-43-5250 | \n", + "esolis@example.net | \n", + "1984-05-21 | \n", + "
| 2 | \n", + "Julia Reese | \n", + "790 Teresa Cove Suite 479 | \n", + "Espinozaton | \n", + "17207 | \n", + "848-13-7165 | \n", + "jhanson@example.org | \n", + "1974-11-02 | \n", + "
| 3 | \n", + "Christine Rogers | \n", + "0797 Gibson Cove Suite 290 | \n", + "Silvachester | \n", + "52803 | \n", + "406-94-4118 | \n", + "williamschad@example.net | \n", + "1983-05-08 | \n", + "
| 4 | \n", + "Dennis Mccoy | \n", + "109 Michelle Rest | \n", + "Gomezville | \n", + "39863 | \n", + "572-81-2883 | \n", + "fosterjames@example.org | \n", + "1982-05-23 | \n", + "
Tools and Best Practices Source
Of course you want quality code, who wouldn’t? But to improve code quality, we have to define what it is.
A quick Google search yields many results defining code quality. As it turns out, the term can mean many different things to people.
One way of trying to define code quality is to look at one end of the spectrum: high-quality code. Hopefully, you can agree on the following high-quality code identifiers:
These three identifiers, while simplistic, seem to be generally agreed upon. In an effort to expand these ideas further, let’s delve into why each one matters in the realm of software.
To determine why high-quality code is important, let’s revisit those identifiers. We’ll see what happens when code doesn’t meet them.
Meeting requirements is the basis of any product, software or otherwise. We make software to do something. If in the end, it doesn’t do it… well it’s definitely not high quality. If it doesn’t meet basic requirements, it’s hard to even call it low quality.
If something you’re using has issues or causes you problems, you probably wouldn’t call it high-quality. In fact, if it’s bad enough, you may stop using it altogether.
For the sake of not using software as an example, let’s say your vacuum works great on regular carpet. It cleans up all the dust and cat hair. One fateful night the cat knocks over a plant, spilling dirt everywhere. When you try to use the vacuum to clean the pile of dirt, it breaks, spewing the dirt everywhere.
While the vacuum worked under some circumstances, it didn’t efficiently handle the occasional extra load. Thus, you wouldn’t call it a high-quality vacuum cleaner.
That is a problem we want to avoid in our code. If things break on edge cases and defects cause unwanted behavior, we don’t have a high-quality product.
Imagine this: a customer requests a new feature. The person who wrote the original code is gone. The person who has replaced them now has to make sense of the code that’s already there. That person is you.
If the code is easy to comprehend, you’ll be able to analyze the problem and come up with a solution much quicker. If the code is complex and convoluted, you’ll probably take longer and possibly make some wrong assumptions.
It’s also nice if it’s easy to add the new feature without disrupting previous features. If the code is not easy to extend, your new feature could break other things.
No one wants to be in the position where they have to read, maintain, or extend low-quality code. It means more headaches and more work for everyone.
It’s bad enough that you have to deal with low-quality code, but don’t put someone else in the same situation. You can improve the quality of code that you write.
If you work with a team of developers, you can start putting into place methods to ensure better overall code quality. Assuming that you have their support, of course.
There are a few things to consider on our journey for high-quality code. First, this journey is not one of pure objectivity. There are some strong feelings of what high-quality code looks like.
While everyone can hopefully agree on the identifiers mentioned above, the way they get achieved is a subjective road. The most opinionated topics usually come up when you talk about achieving readability, maintenance, and extensibility.
So keep in mind that while this article will try to stay objective throughout, there is a very-opinionated world out there when it comes to code.
So, let’s start with the most opinionated topic: code style.
Ah, yes. The age-old question: spaces or tabs?
Regardless of your personal view on how to represent whitespace, it’s safe to assume that you at least want consistency in code.
A style guide serves the purpose of defining a consistent way to write your code. Typically this is all cosmetic, meaning it doesn’t change the logical outcome of the code. Although, some stylistic choices do avoid common logical mistakes.
Style guides serve to help facilitate the goal of making code easy to read, maintain, and extend.
As far as Python goes, there is a well-accepted standard. It was written, in part, by the author of the Python programming language itself.
PEP 8 provides coding conventions for Python code. It is fairly common for Python code to follow this style guide. It’s a great place to start since it’s already well-defined.
A sister Python Enhancement Proposal, PEP 257 describes conventions for Python’s docstrings, which are strings intended to document modules, classes, functions, and methods. As an added bonus, if docstrings are consistent, there are tools capable of generating documentation directly from the code.
All these guides do is define a way to style code. But how do you enforce it? And what about defects and problems in the code, how can you detect those? That’s where linters come in.
First, let’s talk about lint. Those tiny, annoying little defects that somehow get all over your clothes. Clothes look and feel much better without all that lint. Your code is no different. Little mistakes, stylistic inconsistencies, and dangerous logic don’t make your code feel great.
But we all make mistakes. You can’t expect yourself to always catch them in time. Mistyped variable names, forgetting a closing bracket, incorrect tabbing in Python, calling a function with the wrong number of arguments, the list goes on and on. Linters help to identify those problem areas.
Additionally, most editors and IDEs have the ability to run linters in the background as you type. This results in an environment capable of highlighting, underlining, or otherwise identifying problem areas in the code before you run it. It is like an advanced spell-check for code. It underlines issues in squiggly red lines much like your favorite word processor does.
Linters analyze code to detect various categories of lint. Those categories can be broadly defined as the following:
There are also code analysis tools that provide other insights into your code. While maybe not linters by definition, these tools are usually used side-by-side with linters. They too hope to improve the quality of the code.
Finally, there are tools that automatically format code to some specification. These automated tools ensure that our inferior human minds don’t mess up conventions.
Before delving into your options, it’s important to recognize that some “linters” are just multiple linters packaged nicely together. Some popular examples of those combo-linters are the following:
Flake8: Capable of detecting both logical and stylistic lint. It adds the style and complexity checks of pycodestyle to the logical lint detection of PyFlakes. It combines the following linters:
Pylama: A code audit tool composed of a large number of linters and other tools for analyzing code. It combines the following:
Here are some stand-alone linters categorized with brief descriptions:
| Linter | Category | Description |
|---|---|---|
| Pylint | Logical & Stylistic | Checks for errors, tries to enforce a coding standard, looks for code smells |
| PyFlakes | Logical | Analyzes programs and detects various errors |
| pycodestyle | Stylistic | Checks against some of the style conventions in PEP 8 |
| pydocstyle | Stylistic | Checks compliance with Python docstring conventions |
| Bandit | Logical | Analyzes code to find common security issues |
| MyPy | Logical | Checks for optionally-enforced static types |
And here are some code analysis and formatting tools:
| Tool | Category | Description |
|---|---|---|
| Mccabe | Analytical | Checks McCabe complexity |
| Radon | Analytical | Analyzes code for various metrics (lines of code, complexity, and so on) |
| Black | Formatter | Formats Python code without compromise |
| Isort | Formatter | Formats imports by sorting alphabetically and separating into sections |
Let’s get a better idea of what different linters are capable of catching and what the output looks like. To do this, I ran the same code through a handful of different linters with the default settings.
See code_with_lint.py for an example of a file with multiple stylistic issues.
The comparison below shows the linters I used and their runtime for analyzing the above file. I should point out that these aren’t all entirely comparable as they serve different purposes. PyFlakes, for example, does not identify stylistic errors like Pylint does.
Pylint is one of the oldest linters (circa 2006) and is still well-maintained. Some might call this software battle-hardened. It’s been around long enough that contributors have fixed most major bugs and the core features are well-developed.
The common complaints against Pylint are that it is slow, too verbose by default, and takes a lot of configuration to get it working the way you want. Slowness aside, the other complaints are somewhat of a double-edged sword. Verbosity can be because of thoroughness. Lots of configuration can mean lots of adaptability to your preferences.
!pylint code_with_lint.py
+************* Module code_with_lint
+code_with_lint.py:12:0: W0311: Bad indentation. Found 3 spaces, expected 4 (bad-indentation)
+code_with_lint.py:15:0: W0311: Bad indentation. Found 3 spaces, expected 4 (bad-indentation)
+code_with_lint.py:16:0: W0311: Bad indentation. Found 3 spaces, expected 4 (bad-indentation)
+code_with_lint.py:17:0: W0311: Bad indentation. Found 3 spaces, expected 4 (bad-indentation)
+code_with_lint.py:18:0: W0311: Bad indentation. Found 3 spaces, expected 4 (bad-indentation)
+code_with_lint.py:19:0: W0311: Bad indentation. Found 7 spaces, expected 8 (bad-indentation)
+code_with_lint.py:22:0: W0311: Bad indentation. Found 3 spaces, expected 4 (bad-indentation)
+code_with_lint.py:27:0: W0311: Bad indentation. Found 3 spaces, expected 4 (bad-indentation)
+code_with_lint.py:28:0: W0311: Bad indentation. Found 7 spaces, expected 8 (bad-indentation)
+code_with_lint.py:29:0: W0311: Bad indentation. Found 11 spaces, expected 12 (bad-indentation)
+code_with_lint.py:29:0: W0301: Unnecessary semicolon (unnecessary-semicolon)
+code_with_lint.py:30:0: W0311: Bad indentation. Found 11 spaces, expected 12 (bad-indentation)
+code_with_lint.py:31:0: W0311: Bad indentation. Found 15 spaces, expected 16 (bad-indentation)
+code_with_lint.py:32:0: W0311: Bad indentation. Found 11 spaces, expected 12 (bad-indentation)
+code_with_lint.py:33:0: W0311: Bad indentation. Found 15 spaces, expected 16 (bad-indentation)
+code_with_lint.py:33:0: C0325: Unnecessary parens after 'return' keyword (superfluous-parens)
+code_with_lint.py:35:0: W0311: Bad indentation. Found 11 spaces, expected 12 (bad-indentation)
+code_with_lint.py:35:0: C0325: Unnecessary parens after 'return' keyword (superfluous-parens)
+code_with_lint.py:38:0: W0311: Bad indentation. Found 3 spaces, expected 4 (bad-indentation)
+code_with_lint.py:39:0: W0311: Bad indentation. Found 7 spaces, expected 8 (bad-indentation)
+code_with_lint.py:40:0: W0311: Bad indentation. Found 7 spaces, expected 8 (bad-indentation)
+code_with_lint.py:41:0: W0311: Bad indentation. Found 7 spaces, expected 8 (bad-indentation)
+code_with_lint.py:42:0: W0311: Bad indentation. Found 7 spaces, expected 8 (bad-indentation)
+code_with_lint.py:43:0: W0311: Bad indentation. Found 7 spaces, expected 8 (bad-indentation)
+code_with_lint.py:44:0: W0311: Bad indentation. Found 7 spaces, expected 8 (bad-indentation)
+code_with_lint.py:45:0: C0304: Final newline missing (missing-final-newline)
+code_with_lint.py:45:0: W0311: Bad indentation. Found 7 spaces, expected 8 (bad-indentation)
+code_with_lint.py:6:0: W0622: Redefining built-in 'pow' (redefined-builtin)
+code_with_lint.py:6:0: W0401: Wildcard import math (wildcard-import)
+code_with_lint.py:9:0: C0103: Constant name "some_global_var" doesn't conform to UPPER_CASE naming style (invalid-name)
+code_with_lint.py:15:3: W0621: Redefining name 'some_global_var' from outer scope (line 9) (redefined-outer-name)
+code_with_lint.py:11:0: C0103: Argument name "x" doesn't conform to snake_case naming style (invalid-name)
+code_with_lint.py:11:0: C0103: Argument name "y" doesn't conform to snake_case naming style (invalid-name)
+code_with_lint.py:18:3: W0101: Unreachable code (unreachable)
+code_with_lint.py:15:3: W0612: Unused variable 'some_global_var' (unused-variable)
+code_with_lint.py:21:0: C0103: Argument name "x" doesn't conform to snake_case naming style (invalid-name)
+code_with_lint.py:21:0: C0103: Argument name "y" doesn't conform to snake_case naming style (invalid-name)
+code_with_lint.py:27:6: C0121: Comparison 'x != None' should be 'x is not None' (singleton-comparison)
+code_with_lint.py:30:11: R1705: Unnecessary "else" after "return" (no-else-return)
+code_with_lint.py:21:0: R1710: Either all return statements in a function should return an expression, or none of them should. (inconsistent-return-statements)
+code_with_lint.py:37:0: C0115: Missing class docstring (missing-class-docstring)
+code_with_lint.py:42:7: W0621: Redefining name 'time' from outer scope (line 7) (redefined-outer-name)
+code_with_lint.py:42:14: E0601: Using variable 'time' before assignment (used-before-assignment)
+code_with_lint.py:43:7: C0415: Import outside toplevel (datetime.datetime) (import-outside-toplevel)
+code_with_lint.py:38:3: R1711: Useless return at end of function or method (useless-return)
+code_with_lint.py:38:49: W0613: Unused argument 'verbose' (unused-argument)
+code_with_lint.py:41:7: W0612: Unused variable 'list_comprehension' (unused-variable)
+code_with_lint.py:44:7: W0612: Unused variable 'date_and_time' (unused-variable)
+code_with_lint.py:37:0: R0903: Too few public methods (0/2) (too-few-public-methods)
+code_with_lint.py:5:0: W0611: Unused import io (unused-import)
+code_with_lint.py:6:0: W0614: Unused import acos from wildcard import (unused-wildcard-import)
+code_with_lint.py:6:0: W0614: Unused import acosh from wildcard import (unused-wildcard-import)
+code_with_lint.py:6:0: W0614: Unused import asin from wildcard import (unused-wildcard-import)
+code_with_lint.py:6:0: W0614: Unused import asinh from wildcard import (unused-wildcard-import)
+code_with_lint.py:6:0: W0614: Unused import atan from wildcard import (unused-wildcard-import)
+code_with_lint.py:6:0: W0614: Unused import atan2 from wildcard import (unused-wildcard-import)
+code_with_lint.py:6:0: W0614: Unused import atanh from wildcard import (unused-wildcard-import)
+code_with_lint.py:6:0: W0614: Unused import ceil from wildcard import (unused-wildcard-import)
+code_with_lint.py:6:0: W0614: Unused import comb from wildcard import (unused-wildcard-import)
+code_with_lint.py:6:0: W0614: Unused import copysign from wildcard import (unused-wildcard-import)
+code_with_lint.py:6:0: W0614: Unused import cos from wildcard import (unused-wildcard-import)
+code_with_lint.py:6:0: W0614: Unused import cosh from wildcard import (unused-wildcard-import)
+code_with_lint.py:6:0: W0614: Unused import degrees from wildcard import (unused-wildcard-import)
+code_with_lint.py:6:0: W0614: Unused import dist from wildcard import (unused-wildcard-import)
+code_with_lint.py:6:0: W0614: Unused import e from wildcard import (unused-wildcard-import)
+code_with_lint.py:6:0: W0614: Unused import erf from wildcard import (unused-wildcard-import)
+code_with_lint.py:6:0: W0614: Unused import erfc from wildcard import (unused-wildcard-import)
+code_with_lint.py:6:0: W0614: Unused import exp from wildcard import (unused-wildcard-import)
+code_with_lint.py:6:0: W0614: Unused import expm1 from wildcard import (unused-wildcard-import)
+code_with_lint.py:6:0: W0614: Unused import fabs from wildcard import (unused-wildcard-import)
+code_with_lint.py:6:0: W0614: Unused import factorial from wildcard import (unused-wildcard-import)
+code_with_lint.py:6:0: W0614: Unused import floor from wildcard import (unused-wildcard-import)
+code_with_lint.py:6:0: W0614: Unused import fmod from wildcard import (unused-wildcard-import)
+code_with_lint.py:6:0: W0614: Unused import frexp from wildcard import (unused-wildcard-import)
+code_with_lint.py:6:0: W0614: Unused import fsum from wildcard import (unused-wildcard-import)
+code_with_lint.py:6:0: W0614: Unused import gamma from wildcard import (unused-wildcard-import)
+code_with_lint.py:6:0: W0614: Unused import gcd from wildcard import (unused-wildcard-import)
+code_with_lint.py:6:0: W0614: Unused import hypot from wildcard import (unused-wildcard-import)
+code_with_lint.py:6:0: W0614: Unused import inf from wildcard import (unused-wildcard-import)
+code_with_lint.py:6:0: W0614: Unused import isclose from wildcard import (unused-wildcard-import)
+code_with_lint.py:6:0: W0614: Unused import isfinite from wildcard import (unused-wildcard-import)
+code_with_lint.py:6:0: W0614: Unused import isinf from wildcard import (unused-wildcard-import)
+code_with_lint.py:6:0: W0614: Unused import isnan from wildcard import (unused-wildcard-import)
+code_with_lint.py:6:0: W0614: Unused import isqrt from wildcard import (unused-wildcard-import)
+code_with_lint.py:6:0: W0614: Unused import lcm from wildcard import (unused-wildcard-import)
+code_with_lint.py:6:0: W0614: Unused import ldexp from wildcard import (unused-wildcard-import)
+code_with_lint.py:6:0: W0614: Unused import lgamma from wildcard import (unused-wildcard-import)
+code_with_lint.py:6:0: W0614: Unused import log from wildcard import (unused-wildcard-import)
+code_with_lint.py:6:0: W0614: Unused import log10 from wildcard import (unused-wildcard-import)
+code_with_lint.py:6:0: W0614: Unused import log1p from wildcard import (unused-wildcard-import)
+code_with_lint.py:6:0: W0614: Unused import log2 from wildcard import (unused-wildcard-import)
+code_with_lint.py:6:0: W0614: Unused import modf from wildcard import (unused-wildcard-import)
+code_with_lint.py:6:0: W0614: Unused import nan from wildcard import (unused-wildcard-import)
+code_with_lint.py:6:0: W0614: Unused import nextafter from wildcard import (unused-wildcard-import)
+code_with_lint.py:6:0: W0614: Unused import perm from wildcard import (unused-wildcard-import)
+code_with_lint.py:6:0: W0614: Unused import pow from wildcard import (unused-wildcard-import)
+code_with_lint.py:6:0: W0614: Unused import prod from wildcard import (unused-wildcard-import)
+code_with_lint.py:6:0: W0614: Unused import radians from wildcard import (unused-wildcard-import)
+code_with_lint.py:6:0: W0614: Unused import remainder from wildcard import (unused-wildcard-import)
+code_with_lint.py:6:0: W0614: Unused import sin from wildcard import (unused-wildcard-import)
+code_with_lint.py:6:0: W0614: Unused import sinh from wildcard import (unused-wildcard-import)
+code_with_lint.py:6:0: W0614: Unused import sqrt from wildcard import (unused-wildcard-import)
+code_with_lint.py:6:0: W0614: Unused import tan from wildcard import (unused-wildcard-import)
+code_with_lint.py:6:0: W0614: Unused import tanh from wildcard import (unused-wildcard-import)
+code_with_lint.py:6:0: W0614: Unused import tau from wildcard import (unused-wildcard-import)
+code_with_lint.py:6:0: W0614: Unused import trunc from wildcard import (unused-wildcard-import)
+code_with_lint.py:6:0: W0614: Unused import ulp from wildcard import (unused-wildcard-import)
+code_with_lint.py:7:0: W0611: Unused time imported from time (unused-import)
+
+----------------------------------------------------------------------
+Your code has been rated at -31.48/10 (previous run: -31.48/10, +0.00)
+
+[0m
+Pyflakes “makes a simple promise: it will never complain about style, and it will try very, very hard to never emit false positives”. This means that Pyflakes won’t tell you about missing docstrings or argument names not conforming to a naming style. It focuses on logical code issues and potential errors.
The benefit here is speed. PyFlakes runs in a fraction of the time Pylint takes.
!python -m pyflakes code_with_lint.py
+code_with_lint.py:5:1 'io' imported but unused
+code_with_lint.py:6:1 'from math import *' used; unable to detect undefined names
+code_with_lint.py:15:4 local variable 'some_global_var' is assigned to but never used
+code_with_lint.py:41:43 'pi' may be undefined, or defined from star imports: math
+code_with_lint.py:41:8 local variable 'list_comprehension' is assigned to but never used
+code_with_lint.py:42:15 local variable 'time' defined in enclosing scope on line 7 referenced before assignment
+code_with_lint.py:42:8 local variable 'time' is assigned to but never used
+code_with_lint.py:44:8 local variable 'date_and_time' is assigned to but never used
+Used to check some style conventions from PEP8. Naming conventions are not checked and neither are docstrings. The errors and warnings it does catch are categorized in this table.
!pycodestyle code_with_lint.py
+code_with_lint.py:11:1: E302 expected 2 blank lines, found 1
+code_with_lint.py:12:4: E111 indentation is not a multiple of 4
+code_with_lint.py:15:4: E111 indentation is not a multiple of 4
+code_with_lint.py:16:4: E111 indentation is not a multiple of 4
+code_with_lint.py:16:14: E225 missing whitespace around operator
+code_with_lint.py:17:4: E111 indentation is not a multiple of 4
+code_with_lint.py:18:4: E111 indentation is not a multiple of 4
+code_with_lint.py:19:8: E111 indentation is not a multiple of 4
+code_with_lint.py:21:1: E302 expected 2 blank lines, found 1
+code_with_lint.py:22:4: E111 indentation is not a multiple of 4
+code_with_lint.py:27:4: E111 indentation is not a multiple of 4
+code_with_lint.py:27:9: E711 comparison to None should be 'if cond is not None:'
+code_with_lint.py:28:8: E111 indentation is not a multiple of 4
+code_with_lint.py:29:12: E111 indentation is not a multiple of 4
+code_with_lint.py:29:24: E703 statement ends with a semicolon
+code_with_lint.py:30:12: E111 indentation is not a multiple of 4
+code_with_lint.py:31:16: E111 indentation is not a multiple of 4
+code_with_lint.py:32:12: E111 indentation is not a multiple of 4
+code_with_lint.py:33:16: E111 indentation is not a multiple of 4
+code_with_lint.py:33:23: E201 whitespace after '('
+code_with_lint.py:35:12: E111 indentation is not a multiple of 4
+code_with_lint.py:37:1: E302 expected 2 blank lines, found 1
+code_with_lint.py:38:4: E111 indentation is not a multiple of 4
+code_with_lint.py:38:57: E251 unexpected spaces around keyword / parameter equals
+code_with_lint.py:38:59: E251 unexpected spaces around keyword / parameter equals
+code_with_lint.py:39:8: E111 indentation is not a multiple of 4
+code_with_lint.py:39:27: E221 multiple spaces before operator
+code_with_lint.py:39:30: E222 multiple spaces after operator
+code_with_lint.py:40:8: E111 indentation is not a multiple of 4
+code_with_lint.py:40:21: E221 multiple spaces before operator
+code_with_lint.py:40:30: E222 multiple spaces after operator
+code_with_lint.py:41:8: E111 indentation is not a multiple of 4
+code_with_lint.py:41:80: E501 line too long (82 > 79 characters)
+code_with_lint.py:42:8: E111 indentation is not a multiple of 4
+code_with_lint.py:43:8: E111 indentation is not a multiple of 4
+code_with_lint.py:44:8: E111 indentation is not a multiple of 4
+code_with_lint.py:45:8: E111 indentation is not a multiple of 4
+code_with_lint.py:45:14: W292 no newline at end of file
+Flake8 is a Python library that wraps PyFlakes, pycodestyle and Ned Batchelder’s McCabe script. It is a great toolkit for checking your code base against coding style (PEP8), programming errors (like “library imported but unused” and “Undefined name”) and to check cyclomatic complexity.
!flake8 code_with_lint.py
+code_with_lint.py:5:1: F401 'io' imported but unused
+code_with_lint.py:6:1: F403 'from math import *' used; unable to detect undefined names
+code_with_lint.py:11:1: E302 expected 2 blank lines, found 1
+code_with_lint.py:12:4: E111 indentation is not a multiple of 4
+code_with_lint.py:15:4: F841 local variable 'some_global_var' is assigned to but never used
+code_with_lint.py:15:4: E111 indentation is not a multiple of 4
+code_with_lint.py:16:4: E111 indentation is not a multiple of 4
+code_with_lint.py:16:14: E225 missing whitespace around operator
+code_with_lint.py:17:4: E111 indentation is not a multiple of 4
+code_with_lint.py:18:4: E111 indentation is not a multiple of 4
+code_with_lint.py:19:8: E111 indentation is not a multiple of 4
+code_with_lint.py:21:1: E302 expected 2 blank lines, found 1
+code_with_lint.py:22:4: E111 indentation is not a multiple of 4
+code_with_lint.py:27:4: E111 indentation is not a multiple of 4
+code_with_lint.py:27:9: E711 comparison to None should be 'if cond is not None:'
+code_with_lint.py:28:8: E111 indentation is not a multiple of 4
+code_with_lint.py:29:12: E111 indentation is not a multiple of 4
+code_with_lint.py:29:24: E703 statement ends with a semicolon
+code_with_lint.py:30:12: E111 indentation is not a multiple of 4
+code_with_lint.py:31:16: E111 indentation is not a multiple of 4
+code_with_lint.py:32:12: E111 indentation is not a multiple of 4
+code_with_lint.py:33:16: E111 indentation is not a multiple of 4
+code_with_lint.py:33:23: E201 whitespace after '('
+code_with_lint.py:35:12: E111 indentation is not a multiple of 4
+code_with_lint.py:37:1: E302 expected 2 blank lines, found 1
+code_with_lint.py:38:4: E111 indentation is not a multiple of 4
+code_with_lint.py:38:57: E251 unexpected spaces around keyword / parameter equals
+code_with_lint.py:38:59: E251 unexpected spaces around keyword / parameter equals
+code_with_lint.py:39:8: E111 indentation is not a multiple of 4
+code_with_lint.py:39:27: E221 multiple spaces before operator
+code_with_lint.py:39:30: E222 multiple spaces after operator
+code_with_lint.py:40:8: E111 indentation is not a multiple of 4
+code_with_lint.py:40:21: E221 multiple spaces before operator
+code_with_lint.py:40:30: E222 multiple spaces after operator
+code_with_lint.py:41:8: F841 local variable 'list_comprehension' is assigned to but never used
+code_with_lint.py:41:8: E111 indentation is not a multiple of 4
+code_with_lint.py:41:43: F405 'pi' may be undefined, or defined from star imports: math
+code_with_lint.py:41:80: E501 line too long (82 > 79 characters)
+code_with_lint.py:42:8: F841 local variable 'time' is assigned to but never used
+code_with_lint.py:42:8: E111 indentation is not a multiple of 4
+code_with_lint.py:42:15: F823 local variable 'time' defined in enclosing scope on line 7 referenced before assignment
+code_with_lint.py:43:8: E111 indentation is not a multiple of 4
+code_with_lint.py:44:8: F841 local variable 'date_and_time' is assigned to but never used
+code_with_lint.py:44:8: E111 indentation is not a multiple of 4
+code_with_lint.py:45:8: E111 indentation is not a multiple of 4
+code_with_lint.py:45:14: W292 no newline at end of file
+You can check your code’s quality:
It’s useful to have linters run against your code frequently. If automation and consistency aren’t there, it’s easy for a large team or project to lose sight of the goal and start creating lower quality code. It happens slowly, of course. Some poorly written logic or maybe some code with formatting that doesn’t match the neighboring code. Over time, all that lint piles up. Eventually, you can get stuck with something that’s buggy, hard to read, hard to fix, and a pain to maintain.
To avoid that, check code quality often!
You can use linters as you write code, but configuring your environment to do so may take some extra work. It’s generally a matter of finding the plugin for your IDE or editor of choice. In fact, most IDEs will already have linters built in.
Here’s some general info on Python linting for various editors:
If you’re using Git, Git hooks can be set up to run your linters before committing. Other version control systems have similar methods to run scripts before or after some action in the system. You can use these methods to block any new code that doesn’t meet quality standards.
While this may seem drastic, forcing every bit of code through a screening for lint is an important step towards ensuring continued quality. Automating that screening at the front gate to your code may be the best way to avoid lint-filled code.
You can also place linters directly into whatever system you may use for continuous integration. The linters can be set up to fail the build if the code doesn’t meet quality standards.
Again, this may seem like a drastic step, especially if there are already lots of linter errors in the existing code. To combat this, some continuous integration systems will allow you the option of only failing the build if the new code increases the number of linter errors that were already present. That way you can start improving quality without doing a whole rewrite of your existing code base.
High-quality code does what it’s supposed to do without breaking. It is easy to read, maintain, and extend. It functions without problems or defects and is written so that it’s easy for the next person to work with.
Hopefully it goes without saying that you should strive to have such high-quality code. Luckily, there are methods and tools to help improve code quality.
Style guides will bring consistency to your code. PEP8 is a great starting point for Python. Linters will help you identify problem areas and inconsistencies. You can use linters throughout the development process, even automating them to flag lint-filled code before it gets too far.
Having linters complain about style also avoids the need for style discussions during code reviews. Some people may find it easier to receive candid feedback from these tools instead of a team member. Additionally, some team members may not want to “nitpick” style during code reviews. Linters avoid the politics, save time, and complain about any inconsistency.
In addition, all the linters mentioned in this article have various command line options and configurations that let you tailor the tool to your liking. You can be as strict or as loose as you want, which is an important thing to realize.
Improving code quality is a process. You can take steps towards improving it without completely disallowing all nonconformant code. Awareness is a great first step. It just takes a person, like you, to first realize how important high-quality code is.
+The goal of unit testing is to segregate each part of the program and test that the individual parts are working correctly.
One of the main benefits of unit testing is that it makes the coding process more Agile. When you add more and more features to a software, you sometimes need to change old design and code. However, changing already-tested code is both risky and costly. If we have unit tests in place, then we can proceed for refactoring confidently.
Unit testing really goes hand-in-hand with agile programming of all flavors because it builds in tests that allow you to make changes more easily. In other words, unit tests facilitate safe refactoring.
Unit testing improves the quality of the code. It identifies every defect that may have come up before code is sent further for integration testing. Writing tests before actual coding makes you think harder about the problem. It exposes the edge cases and makes you write better code.
Issues are found at an early stage. Since unit testing is carried out by developers who test individual code before integration, issues can be found very early and can be resolved then and there without impacting the other pieces of the code. This includes both bugs in the programmer’s implementation and flaws or missing parts of the specification for the unit.
Unit testing allows the programmer to refactor code or upgrade system libraries at a later date and make sure the module still works correctly. Unit tests detect changes that may break a design contract. They help with maintaining and changing the code.
Unit testing reduces defects in the newly developed features or reduces bugs when changing the existing functionality.
Unit testing verifies the accuracy of the each unit. Afterward, the units are integrated into an application by testing parts of the application via unit testing. Later testing of the application during the integration process is easier due to the verification of the individual units.
Unit testing provides documentation of the system. Developers looking to learn what functionality is provided by a unit and how to use it can look at the unit tests to gain a basic understanding of the unit’s interface (API).
Unit testing helps simplify the debugging process. If a test fails, then only the latest changes made in the code need to be debugged.
Writing the test first forces you to think through your design and what it must accomplish before you write the code. This not only keeps you focused; it makes you create better designs. Testing a piece of code forces you to define what that code is responsible for. If you can do this easily, that means the code’s responsibility is well-defined and therefore that it has high cohesion.
Since the bugs are found early, unit testing helps reduce the cost of bug fixes. Just imagine the cost of a bug found during the later stages of development, like during system testing or during acceptance testing. Of course, bugs detected earlier are easier to fix because bugs detected later are usually the result of many changes, and you don’t really know which one caused the bug.
The world of testing has no shortage of terminology, and now that you know the difference between automated and manual testing, it’s time to go a level deeper.
Think of how you might test the lights on a car. You would turn on the lights (known as the test step) and go outside the car or ask a friend to check that the lights are on (known as the test assertion). Testing multiple components is known as integration testing.
Think of all the things that need to work correctly in order for a simple task to give the right result. These components are like the parts to your application, all of those classes, functions, and modules you’ve written.
A major challenge with integration testing is when an integration test doesn’t give the right result. It’s very hard to diagnose the issue without being able to isolate which part of the system is failing. If the lights didn’t turn on, then maybe the bulbs are broken. Is the battery dead? What about the alternator? Is the car’s computer failing?
If you have a fancy modern car, it will tell you when your light bulbs have gone. It does this using a form of unit test.
A unit test is a smaller test, one that checks that a single component operates in the right way. A unit test helps you to isolate what is broken in your application and fix it faster.
You have just seen two types of tests:
You can write both integration tests and unit tests in Python. To write a unit test for the built-in function sum(), you would check the output of sum() against a known output.
For example, here’s how you check that the sum() of the numbers (1, 2, 3) equals 6:
To write a unit test for the built-in function sum(), you would check the output of sum() against a known output.
assert sum([1, 2, 3]) == 6, "Should be 6"
+If the result from sum() is incorrect, this will fail with an AssertionError and the message “Should be 6”.
assert sum([1, 1, 1]) == 6, "Should be 6"
+---------------------------------------------------------------------------
+
+AssertionError Traceback (most recent call last)
+
+/var/folders/jd/pq0swyt521jb2424d6fvth840000gn/T/ipykernel_3247/17110154.py in <module>
+----> 1 assert sum([1, 1, 1]) == 6, "Should be 6"
+
+
+AssertionError: Should be 6
+We can write a test function to encapsualte the test.
def test_sum():
+ assert sum([1, 2, 3]) == 6, "Should be 6"
+… and execute it as follows.
test_sum()
+print("Everything passed")
+Everything passed
+However, instead of testing on the REPL, you’ll want to put this into a new Python file called test_sum.py and execute it again:
!python test_sum.py
+Everything passed
+Let’s expand out test suite to include summing a tuple.
def test_sum_tuple():
+ assert sum((1, 2, 2)) == 6, "Should be 6"
+test_sum()
+test_sum_tuple()
+print("Everything passed")
+---------------------------------------------------------------------------
+
+AssertionError Traceback (most recent call last)
+
+/var/folders/jd/pq0swyt521jb2424d6fvth840000gn/T/ipykernel_3247/2054533739.py in <module>
+ 1 test_sum()
+----> 2 test_sum_tuple()
+ 3 print("Everything passed")
+
+
+/var/folders/jd/pq0swyt521jb2424d6fvth840000gn/T/ipykernel_3247/2526289347.py in test_sum_tuple()
+ 1 def test_sum_tuple():
+----> 2 assert sum((1, 2, 2)) == 6, "Should be 6"
+
+
+AssertionError: Should be 6
+We’ve updated test_sum_2.py with the new code which can be executed as follows:
!python test_sum_2.py
+Traceback (most recent call last):
+ File "/Users/bk/Source/data-focused-python/lectures/Topic 04 - Writing Testable Code/test_sum_2.py", line 9, in <module>
+ test_sum_tuple()
+ File "/Users/bk/Source/data-focused-python/lectures/Topic 04 - Writing Testable Code/test_sum_2.py", line 5, in test_sum_tuple
+ assert sum((1, 2, 2)) == 6, "Should be 6"
+AssertionError: Should be 6
+There are many test runners available for Python. The one built into the Python standard library is called unittest. In this tutorial, you will be using unittest test cases and the unittest test runner. The principles of unittest are easily portable to other frameworks. The three most popular test runners are:
Choosing the best test runner for your requirements and level of experience is important.
unittest has been built into the Python standard library since version 2.1. You’ll probably see it in commercial Python applications and open-source projects.
unittest contains both a testing framework and a test runner. unittest has some important requirements for writing and executing tests.
unittest requires that:
To convert the earlier example to a unittest test case, you would have to:
# test_sum_unittest.py
+import unittest
+
+
+class TestSum(unittest.TestCase):
+
+ def test_sum(self):
+ self.assertEqual(sum([1, 2, 3]), 6, "Should be 6")
+
+ def test_sum_tuple(self):
+ self.assertEqual(sum((1, 2, 2)), 6, "Should be 6")
+
+if __name__ == '__main__':
+ unittest.main()
+If you execute this at the command line, you’ll see one success (indicated with .) and one failure (indicated with F):
!python test_sum_unittest.py
+.F
+======================================================================
+FAIL: test_sum_tuple (__main__.TestSum)
+----------------------------------------------------------------------
+Traceback (most recent call last):
+ File "/Users/bk/Source/data-focused-python/lectures/Topic 04 - Writing Testable Code/test_sum_unittest.py", line 10, in test_sum_tuple
+ self.assertEqual(sum((1, 2, 2)), 6, "Should be 6")
+AssertionError: 5 != 6 : Should be 6
+
+----------------------------------------------------------------------
+Ran 2 tests in 0.000s
+
+FAILED (failures=1)
+You may find that over time, as you write hundreds or even thousands of tests for your application, it becomes increasingly hard to understand and use the output from unittest.
nose is compatible with any tests written using the unittest framework and can be used as a drop-in replacement for the unittest test runner. The development of nose as an open-source application fell behind, and a fork called nose2 was created. If you’re starting from scratch, it is recommended that you use nose2 instead of nose.
To get started with nose2, install nose2 from PyPI and execute it on the command line. nose2 will try to discover all test scripts named test*.py and test cases inheriting from unittest.TestCase in your current directory:
pip install nose2
+python -m nose2
+!python -m nose2 test_sum_unittest
+.F
+======================================================================
+FAIL: test_sum_tuple (test_sum_unittest.TestSum)
+----------------------------------------------------------------------
+Traceback (most recent call last):
+ File "/Users/bk/Source/data-focused-python/lectures/Topic 04 - Writing Testable Code/test_sum_unittest.py", line 10, in test_sum_tuple
+ self.assertEqual(sum((1, 2, 2)), 6, "Should be 6")
+AssertionError: 5 != 6 : Should be 6
+
+----------------------------------------------------------------------
+Ran 2 tests in 0.000s
+
+FAILED (failures=1)
+You have just executed the test you created in test_sum_unittest.py from the nose2 test runner. nose2 offers many command-line flags for filtering the tests that you execute. For more information, you can explore the Nose 2 documentation.
pytest supports execution of unittest test cases. The real advantage of pytest comes by writing pytest test cases. pytest test cases are a series of functions in a Python file starting with the name test_.
pytest has some other great features:
self.assert*() methodsWriting the TestSum test case example for pytest would look like this:
def test_sum():
+ assert sum([1, 2, 3]) == 6, "Should be 6"
+
+def test_sum_tuple():
+ assert sum((1, 2, 2)) == 6, "Should be 6"
+You have dropped the TestCase, any use of classes, and the command-line entry point.
More information can be found at the Pytest Documentation Website.
Let’s bring together what you’ve learned so far and, instead of testing the built-in sum() function, test a simple implementation of the same requirement.
Let’s bring together what you’ve learned so far and, instead of testing the built-in sum() function, test a simple implementation of the same requirement.
Create a new project folder and, inside that, create a new folder called my_sum. Inside my_sum, create an empty file called __init__.py. Creating the __init__.py file means that the my_sum folder can be imported as a module from the parent directory.
Your project folder should look like this:
project/
+│
+└── my_sum/
+ └── __init__.py
+Open up my_sum/__init__.py and create a new function called sum(), which takes an iterable (a list, tuple, or set) and adds the values together:
# contents of my_sum/__init__.py
+def sum(arg):
+ total = 0
+ for val in arg:
+ total += val
+ return total
+To get started writing tests, you can simply create a file called test.py, which will contain your first test case. Because the file will need to be able to import your application to be able to test it, you want to place test.py above the package folder, so your directory tree will look something like this:
project/
+│
+├── my_sum/
+│ └── __init__.py
+|
+└── test.py
+You’ll find that, as you add more and more tests, your single file will become cluttered and hard to maintain, so you can create a folder called tests/ and split the tests into multiple files. It is convention to ensure each file starts with test_ so all test runners will assume that Python file contains tests to be executed. Some very large projects split tests into more subdirectories based on their purpose or usage.
Note: What if your application is a single script?
You can import any attributes of the script, such as classes, functions, and variables by using the built-in __import__() function. Instead of from my_sum import sum, you can write the following:
target = __import__("my_sum.py")
+sum = target.sum
+The benefit of using __import__() is that you don’t have to turn your project folder into a package, and you can specify the file name. This is also useful if your filename collides with any standard library packages. For example, math.py would collide with the math module.
Before you dive into writing tests, you’ll want to first make a couple of decisions:
Then the structure of a test should loosely follow this workflow:
For this application, you’re testing sum(). There are many behaviors in sum() you could check, such as:
The most simple test would be a list of integers. Create a file, test.py with the following Python code:
import unittest
+
+from fractions import Fraction
+from my_sum import sum
+
+class TestSum(unittest.TestCase):
+ def test_list_int(self):
+ """
+ Test that it can sum a list of integers
+ """
+ data = [1, 2, 3]
+ result = sum(data)
+ self.assertEqual(result, 6)
+
+ def test_list_fraction(self):
+ """
+ Test that it can sum a list of fractions
+ """
+ data = [Fraction(1, 4), Fraction(1, 4), Fraction(1, 2)]
+ result = sum(data)
+ self.assertEqual(result, 1)
+
+ def test_bad_type(self):
+ data = "banana"
+ with self.assertRaises(TypeError):
+ result = sum(data)
+
+if __name__ == '__main__':
+ unittest.main()
+!python -m nose2 test
+...
+----------------------------------------------------------------------
+Ran 3 tests in 0.000s
+
+OK
+Side effects make unit testing harder since, each time a test is run, it might give a different result, or even worse, one test could impact the state of the application and cause another test to fail!
There are some simple techniques you can use to test parts of your application that have many side effects:
If you’re not familiar with mocking, see Python CLI Testing for some great examples.
+So far, you’ve been learning mainly about unit testing. Unit testing is a great way to build predictable and stable code. But at the end of the day, your application needs to work when it starts!
Integration testing is the testing of multiple components of the application to check that they work together. Integration testing might require acting like a consumer or user of the application by:
Each of these types of integration tests can be written in the same way as a unit test, following the Input, Execute, and Assert pattern. The most significant difference is that integration tests are checking more components at once and therefore will have more side effects than a unit test. Also, integration tests will require more fixtures to be in place, like a database, a network socket, or a configuration file.
This is why it’s good practice to separate your unit tests and your integration tests. The creation of fixtures required for an integration like a test database and the test cases themselves often take a lot longer to execute than unit tests, so you may only want to run integration tests before you push to production instead of once on every commit.
A simple way to separate unit and integration tests is simply to put them in different folders:
project/
+│
+├── my_app/
+│ └── __init__.py
+│
+└── tests/
+ |
+ ├── unit/
+ | ├── __init__.py
+ | └── test_sum.py
+ |
+ └── integration/
+ ├── __init__.py
+ └── test_integration.py
+There are many ways to execute only a select group of tests. The specify source directory flag, -s, can be added to unittest discover with the path containing the tests:
!python -m unittest discover -s project/tests/integration
+----------------------------------------------------------------------
+Ran 0 tests in 0.000s
+
+OK
+Many integration tests will require backend data like a database to exist with certain values. For example, you might want to have a test that checks that the application displays correctly with more than 100 customers in the database, or the order page works even if the product names are displayed in Japanese.
These types of integration tests will depend on different test fixtures to make sure they are repeatable and predictable.
A good technique to use is to store the test data in a folder within your integration testing folder called fixtures to indicate that it contains test data. Then, within your tests, you can load the data and run the test.
Here’s an example of that structure if the data consisted of JSON files:
project/
+│
+├── my_app/
+│ └── __init__.py
+│
+└── tests/
+ |
+ └── unit/
+ | ├── __init__.py
+ | └── test_sum.py
+ |
+ └── integration/
+ |
+ ├── fixtures/
+ | └── test_basic.json
+ |
+ |
+ ├── __init__.py
+ └── test_integration.py
+Within your test case, you can use the .setUp() method to load the test data from a fixture file in a known path and execute many tests against that test data. Remember you can have multiple test cases in a single Python file, and the unittest discovery will execute both. You can have one test case for each set of test data:
# test_integration.py
+import unittest
+
+from my_app.app import App
+
+
+class TestBasic(unittest.TestCase):
+ def setUp(self):
+ # Load test data
+ self.app = App(database='tests/integration/fixtures/test_basic.json')
+
+ def test_customer_count(self):
+ self.assertEqual(len(self.app.customers), 3)
+
+ def test_existence_of_customer(self):
+ customer = self.app.get_customer(id=10)
+ self.assertEqual(customer['name'], "Org XYZ")
+ self.assertEqual(customer['address'], "10 Red Road, Reading")
+
+if __name__ == '__main__':
+ unittest.main()
+You can run the tests from the project folder by typing the following command from the terminal:
python -m unittest discover -s tests/integration
+
+So far, you’ve been testing against a single version of Python using a virtual environment with a specific set of dependencies. You might want to check that your application works on multiple versions of Python, or multiple versions of a package. Tox is an application that automates testing in multiple environments.
Tox is available on PyPI as a package to install via pip:
pip install tox
+Now that you have Tox installed, it needs to be configured.
Tox is configured via a configuration file in your project directory. The Tox configuration file contains the following:
Instead of having to learn the Tox configuration syntax, you can get a head start by running the quickstart application:
tox-quickstart
+The Tox configuration tool will ask you those questions and create a file similar to the following in tox.ini:
[tox]
+envlist = py27, py36
+
+[testenv]
+deps =
+
+commands =
+ python -m unittest discover
+Before you can run Tox, it requires that you have a setup.py file in your application folder containing the steps to install your package. If you don’t have one, you can follow this guide on how to create a setup.py before you continue.
Alternatively, if your project is not for distribution on PyPI, you can skip this requirement by adding the following line in the tox.ini file under the [tox] heading:
[tox]
+envlist = py27, py36
+skipsdist=True
+If you don’t create a setup.py, and your application has some dependencies from PyPI, you’ll need to specify those on a number of lines under the [testenv] section. For example, Django would require the following:
[testenv]
+deps = django
+Once you have completed that stage, you’re ready to run the tests.
You can now execute Tox, and it will create two virtual environments: one for Python 2.7 and one for Python 3.6. The Tox directory is called .tox/. Within the .tox/ directory, Tox will execute python -m unittest discover against each virtual environment.
You can run this process by calling Tox at the command line:
tox
+Tox will output the results of your tests against each environment. The first time it runs, Tox takes a little bit of time to create the virtual environments, but once it has, the second execution will be a lot faster.
The output of Tox is quite straightforward. It creates an environment for each version, installs your dependencies, and then runs the test commands.
There are some additional command line options that are great to remember.
Run only a single environment, such as Python 3.6:
tox -e py36
+Recreate the virtual environments, in case your dependencies have changed or site-packages is corrupt:
tox -r
+Run Tox with less verbose output:
tox -q
+Running Tox with more verbose output:
tox -v
+More information on Tox can be found at the Tox Documentation Website.
See the tox_example folder for a complete example.
+A linter will look at your code and comment on it. It could give you tips about mistakes you’ve made, correct trailing spaces, and even predict bugs you may have introduced.
A popular linter that comments on the style of your code in relation to the PEP 8 specification is flake8.
You can install flake8 using pip:
pip install flake8
+You can then run flake8 over a single file, a folder, or a pattern:
!flake8 lint_example.py
+lint_example.py:2:3: E111 indentation is not a multiple of 4
+lint_example.py:4:1: E302 expected 2 blank lines, found 1
+lint_example.py:7:1: W293 blank line contains whitespace
+lint_example.py:10:4: E111 indentation is not a multiple of 4
+lint_example.py:11:4: E111 indentation is not a multiple of 4
+lint_example.py:11:44: W292 no newline at end of file
+flake8 is configurable on the command line or inside a configuration file in your project. If you wanted to ignore certain rules, like E302 shown above, you can set them in the configuration. flake8 will inspect a .flake8 file in the project folder or a setup.cfg file. If you decided to use Tox, you can put the flake8 configuration section inside tox.ini.
This example ignores the .git and __pycache__ directories as well as the E302 rule. Also, it sets the max line length to 90 instead of 80 characters. You will likely find that the default constraint of 79 characters for line-width is very limiting for tests, as they contain long method names, string literals with test values, and other pieces of data that can be longer. It is common to set the line length for tests to up to 120 characters:
[flake8]
+ignore = E302
+exclude = .git,__pycache__
+max-line-length = 90
+Alternatively, you can provide these options on the command line:
!flake8 --ignore E302 --exclude .git,__pycache__ --max-line-length=90 lint_example.py
+lint_example.py:2:3: E111 indentation is not a multiple of 4
+lint_example.py:7:1: W293 blank line contains whitespace
+lint_example.py:10:4: E111 indentation is not a multiple of 4
+lint_example.py:11:4: E111 indentation is not a multiple of 4
+lint_example.py:11:44: W292 no newline at end of file
+A full list of configuration options is available on the Documentation Website.
flake8 is a passive linter: it recommends changes, but you have to go and change the code. A more aggressive approach is a code formatter. Code formatters will change your code automatically to meet a collection of style and layout practices.
black is a very unforgiving formatter. It doesn’t have any configuration options, and it has a very specific style. This makes it great as a drop-in tool to put in your test pipeline.
You can install black via pip:
pip install black
+Then to run black at the command line, provide the file or directory you want to format:
black test.py
+
+There are many ways to benchmark code in Python. The standard library provides the timeit module, which can time functions a number of times and give you the distribution. This example will execute test() 100 times and print() the output:
import random
+from functools import reduce
+
+def add(numbers):
+ total = reduce(lambda x, y: x + y, numbers)
+ return total
+
+if __name__ == '__main__':
+ import timeit
+ numbers = [random.random() for i in range(100000)]
+ print(timeit.timeit(f"add({numbers})", setup="from __main__ import add", number=100))
+0.5682077080000454
+Another option, if you decided to use pytest as a test runner, is the pytest-benchmark plugin. This provides a pytest fixture called benchmark. You can pass benchmark() any callable, and it will log the timing of the callable to the results of pytest.
You can install pytest-benchmark from PyPI using pip:
pip install pytest-benchmark
+Then, you can add a test that uses the fixture and passes the callable to be executed:
def test_my_function(benchmark):
+ result = benchmark(test)
+More information is available at the Documentation Website
See the benchmark folder for a runnable example.
+Another test you will want to run on your application is checking for common security mistakes or vulnerabilities.
You can install bandit from PyPI using pip:
pip install bandit
+You can then pass the name of your application module with the -r flag, and it will give you a summary:
bandit -r my_sum
+As with flake8, the rules that bandit flags are configurable, and if there are any you wish to ignore, you can add the following section to your setup.cfg file with the options:
[bandit]
+exclude: /test
+tests: B101,B102,B301
+You can run some of the examples from https://github.com/PyCQA/bandit/tree/master/examples by executing:
bandit -r bandit_examples
+
+!bandit -r bandit_examples
+[main] INFO profile include tests: None
+[main] INFO profile exclude tests: None
+[main] INFO cli include tests: None
+[main] INFO cli exclude tests: None
+[main] INFO running on Python 3.9.7
+[tester] ERROR Bandit internal error running: blacklist on file bandit_examples/imports-with-importlib.py at line 12: list index out of rangeTraceback (most recent call last):
+ File "/Users/bk/opt/miniconda3/envs/cmu39/lib/python3.9/site-packages/bandit/core/tester.py", line 52, in run_tests
+ result = test(context, test._config)
+ File "/Users/bk/opt/miniconda3/envs/cmu39/lib/python3.9/site-packages/bandit/core/blacklisting.py", line 50, in blacklist
+ name = context.call_args[0]
+IndexError: list index out of range
+
+[tester] ERROR Bandit internal error running: blacklist on file bandit_examples/imports-with-importlib.py at line 13: list index out of rangeTraceback (most recent call last):
+ File "/Users/bk/opt/miniconda3/envs/cmu39/lib/python3.9/site-packages/bandit/core/tester.py", line 52, in run_tests
+ result = test(context, test._config)
+ File "/Users/bk/opt/miniconda3/envs/cmu39/lib/python3.9/site-packages/bandit/core/blacklisting.py", line 50, in blacklist
+ name = context.call_args[0]
+IndexError: list index out of range
+
+[tester] ERROR Bandit internal error running: blacklist on file bandit_examples/imports-with-importlib.py at line 14: list index out of rangeTraceback (most recent call last):
+ File "/Users/bk/opt/miniconda3/envs/cmu39/lib/python3.9/site-packages/bandit/core/tester.py", line 52, in run_tests
+ result = test(context, test._config)
+ File "/Users/bk/opt/miniconda3/envs/cmu39/lib/python3.9/site-packages/bandit/core/blacklisting.py", line 50, in blacklist
+ name = context.call_args[0]
+IndexError: list index out of range
+
+[tester] ERROR Bandit internal error running: blacklist on file bandit_examples/imports-with-importlib.py at line 15: list index out of rangeTraceback (most recent call last):
+ File "/Users/bk/opt/miniconda3/envs/cmu39/lib/python3.9/site-packages/bandit/core/tester.py", line 52, in run_tests
+ result = test(context, test._config)
+ File "/Users/bk/opt/miniconda3/envs/cmu39/lib/python3.9/site-packages/bandit/core/blacklisting.py", line 50, in blacklist
+ name = context.call_args[0]
+IndexError: list index out of range
+
+[95mRun started:2021-10-28 19:11:44.040296[0m
+[95m
+Test results:[0m
+[93m>> Issue: [B307:blacklist] Use of possibly insecure function - consider using safer ast.literal_eval.
+ Severity: Medium Confidence: High
+ Location: bandit_examples/eval.py:3
+ More Info: https://bandit.readthedocs.io/en/latest/blacklists/blacklist_calls.html#b307-eval[0m
+2
+3 print(eval("1+1"))
+4 print(eval("os.getcwd()"))
+
+--------------------------------------------------
+[93m>> Issue: [B307:blacklist] Use of possibly insecure function - consider using safer ast.literal_eval.
+ Severity: Medium Confidence: High
+ Location: bandit_examples/eval.py:4
+ More Info: https://bandit.readthedocs.io/en/latest/blacklists/blacklist_calls.html#b307-eval[0m
+3 print(eval("1+1"))
+4 print(eval("os.getcwd()"))
+5 print(eval("os.chmod('%s', 0777)" % 'test.txt'))
+
+--------------------------------------------------
+[93m>> Issue: [B307:blacklist] Use of possibly insecure function - consider using safer ast.literal_eval.
+ Severity: Medium Confidence: High
+ Location: bandit_examples/eval.py:5
+ More Info: https://bandit.readthedocs.io/en/latest/blacklists/blacklist_calls.html#b307-eval[0m
+4 print(eval("os.getcwd()"))
+5 print(eval("os.chmod('%s', 0777)" % 'test.txt'))
+6
+
+--------------------------------------------------
+[93m>> Issue: [B309:blacklist] Use of HTTPSConnection on older versions of Python prior to 2.7.9 and 3.4.3 do not provide security, see https://wiki.openstack.org/wiki/OSSN/OSSN-0033
+ Severity: Medium Confidence: High
+ Location: bandit_examples/httplib_https.py:2
+ More Info: https://bandit.readthedocs.io/en/latest/blacklists/blacklist_calls.html#b309-httpsconnection[0m
+1 import httplib
+2 c = httplib.HTTPSConnection("example.com")
+3
+
+--------------------------------------------------
+[93m>> Issue: [B309:blacklist] Use of HTTPSConnection on older versions of Python prior to 2.7.9 and 3.4.3 do not provide security, see https://wiki.openstack.org/wiki/OSSN/OSSN-0033
+ Severity: Medium Confidence: High
+ Location: bandit_examples/httplib_https.py:5
+ More Info: https://bandit.readthedocs.io/en/latest/blacklists/blacklist_calls.html#b309-httpsconnection[0m
+4 import http.client
+5 c = http.client.HTTPSConnection("example.com")
+6
+
+--------------------------------------------------
+[93m>> Issue: [B309:blacklist] Use of HTTPSConnection on older versions of Python prior to 2.7.9 and 3.4.3 do not provide security, see https://wiki.openstack.org/wiki/OSSN/OSSN-0033
+ Severity: Medium Confidence: High
+ Location: bandit_examples/httplib_https.py:8
+ More Info: https://bandit.readthedocs.io/en/latest/blacklists/blacklist_calls.html#b309-httpsconnection[0m
+7 import six
+8 six.moves.http_client.HTTPSConnection("example.com")
+
+--------------------------------------------------
+[94m>> Issue: [B403:blacklist] Consider possible security implications associated with pickle module.
+ Severity: Low Confidence: High
+ Location: bandit_examples/imports-with-importlib.py:3
+ More Info: https://bandit.readthedocs.io/en/latest/blacklists/blacklist_imports.html#b403-import-pickle[0m
+2 a = importlib.import_module('os')
+3 b = importlib.import_module('pickle')
+4 c = importlib.__import__('sys')
+
+--------------------------------------------------
+[94m>> Issue: [B404:blacklist] Consider possible security implications associated with subprocess module.
+ Severity: Low Confidence: High
+ Location: bandit_examples/imports-with-importlib.py:5
+ More Info: https://bandit.readthedocs.io/en/latest/blacklists/blacklist_imports.html#b404-import-subprocess[0m
+4 c = importlib.__import__('sys')
+5 d = importlib.__import__('subprocess')
+6
+
+--------------------------------------------------
+[94m>> Issue: [B403:blacklist] Consider possible security implications associated with pickle module.
+ Severity: Low Confidence: High
+ Location: bandit_examples/imports.py:2
+ More Info: https://bandit.readthedocs.io/en/latest/blacklists/blacklist_imports.html#b403-import-pickle[0m
+1 import os
+2 import pickle
+3 import sys
+
+--------------------------------------------------
+[94m>> Issue: [B404:blacklist] Consider possible security implications associated with subprocess module.
+ Severity: Low Confidence: High
+ Location: bandit_examples/imports.py:4
+ More Info: https://bandit.readthedocs.io/en/latest/blacklists/blacklist_imports.html#b404-import-subprocess[0m
+3 import sys
+4 import subprocess
+
+--------------------------------------------------
+[93m>> Issue: [B608:hardcoded_sql_expressions] Possible SQL injection vector through string-based query construction.
+ Severity: Medium Confidence: Low
+ Location: bandit_examples/sql_statements-py36.py:4
+ More Info: https://bandit.readthedocs.io/en/latest/plugins/b608_hardcoded_sql_expressions.html[0m
+3 # bad
+4 query = "SELECT * FROM foo WHERE id = '%s'" % identifier
+5 query = "INSERT INTO foo VALUES ('a', 'b', '%s')" % value
+
+--------------------------------------------------
+[93m>> Issue: [B608:hardcoded_sql_expressions] Possible SQL injection vector through string-based query construction.
+ Severity: Medium Confidence: Low
+ Location: bandit_examples/sql_statements-py36.py:5
+ More Info: https://bandit.readthedocs.io/en/latest/plugins/b608_hardcoded_sql_expressions.html[0m
+4 query = "SELECT * FROM foo WHERE id = '%s'" % identifier
+5 query = "INSERT INTO foo VALUES ('a', 'b', '%s')" % value
+6 query = "DELETE FROM foo WHERE id = '%s'" % identifier
+
+--------------------------------------------------
+[93m>> Issue: [B608:hardcoded_sql_expressions] Possible SQL injection vector through string-based query construction.
+ Severity: Medium Confidence: Low
+ Location: bandit_examples/sql_statements-py36.py:6
+ More Info: https://bandit.readthedocs.io/en/latest/plugins/b608_hardcoded_sql_expressions.html[0m
+5 query = "INSERT INTO foo VALUES ('a', 'b', '%s')" % value
+6 query = "DELETE FROM foo WHERE id = '%s'" % identifier
+7 query = "UPDATE foo SET value = 'b' WHERE id = '%s'" % identifier
+
+--------------------------------------------------
+[93m>> Issue: [B608:hardcoded_sql_expressions] Possible SQL injection vector through string-based query construction.
+ Severity: Medium Confidence: Low
+ Location: bandit_examples/sql_statements-py36.py:7
+ More Info: https://bandit.readthedocs.io/en/latest/plugins/b608_hardcoded_sql_expressions.html[0m
+6 query = "DELETE FROM foo WHERE id = '%s'" % identifier
+7 query = "UPDATE foo SET value = 'b' WHERE id = '%s'" % identifier
+8 query = """WITH cte AS (SELECT x FROM foo)
+
+--------------------------------------------------
+[93m>> Issue: [B608:hardcoded_sql_expressions] Possible SQL injection vector through string-based query construction.
+ Severity: Medium Confidence: Low
+ Location: bandit_examples/sql_statements-py36.py:8
+ More Info: https://bandit.readthedocs.io/en/latest/plugins/b608_hardcoded_sql_expressions.html[0m
+7 query = "UPDATE foo SET value = 'b' WHERE id = '%s'" % identifier
+8 query = """WITH cte AS (SELECT x FROM foo)
+9 SELECT x FROM cte WHERE x = '%s'""" % identifier
+10 # bad alternate forms
+
+--------------------------------------------------
+[93m>> Issue: [B608:hardcoded_sql_expressions] Possible SQL injection vector through string-based query construction.
+ Severity: Medium Confidence: Low
+ Location: bandit_examples/sql_statements-py36.py:11
+ More Info: https://bandit.readthedocs.io/en/latest/plugins/b608_hardcoded_sql_expressions.html[0m
+10 # bad alternate forms
+11 query = "SELECT * FROM foo WHERE id = '" + identifier + "'"
+12 query = "SELECT * FROM foo WHERE id = '{}'".format(identifier)
+
+--------------------------------------------------
+[93m>> Issue: [B608:hardcoded_sql_expressions] Possible SQL injection vector through string-based query construction.
+ Severity: Medium Confidence: Low
+ Location: bandit_examples/sql_statements-py36.py:12
+ More Info: https://bandit.readthedocs.io/en/latest/plugins/b608_hardcoded_sql_expressions.html[0m
+11 query = "SELECT * FROM foo WHERE id = '" + identifier + "'"
+12 query = "SELECT * FROM foo WHERE id = '{}'".format(identifier)
+13 query = f"SELECT * FROM foo WHERE id = {tmp}"
+
+--------------------------------------------------
+[93m>> Issue: [B608:hardcoded_sql_expressions] Possible SQL injection vector through string-based query construction.
+ Severity: Medium Confidence: Low
+ Location: bandit_examples/sql_statements-py36.py:13
+ More Info: https://bandit.readthedocs.io/en/latest/plugins/b608_hardcoded_sql_expressions.html[0m
+12 query = "SELECT * FROM foo WHERE id = '{}'".format(identifier)
+13 query = f"SELECT * FROM foo WHERE id = {tmp}"
+14
+
+--------------------------------------------------
+[93m>> Issue: [B608:hardcoded_sql_expressions] Possible SQL injection vector through string-based query construction.
+ Severity: Medium Confidence: Medium
+ Location: bandit_examples/sql_statements-py36.py:16
+ More Info: https://bandit.readthedocs.io/en/latest/plugins/b608_hardcoded_sql_expressions.html[0m
+15 # bad
+16 cur.execute("SELECT * FROM foo WHERE id = '%s'" % identifier)
+17 cur.execute("INSERT INTO foo VALUES ('a', 'b', '%s')" % value)
+
+--------------------------------------------------
+[93m>> Issue: [B608:hardcoded_sql_expressions] Possible SQL injection vector through string-based query construction.
+ Severity: Medium Confidence: Medium
+ Location: bandit_examples/sql_statements-py36.py:17
+ More Info: https://bandit.readthedocs.io/en/latest/plugins/b608_hardcoded_sql_expressions.html[0m
+16 cur.execute("SELECT * FROM foo WHERE id = '%s'" % identifier)
+17 cur.execute("INSERT INTO foo VALUES ('a', 'b', '%s')" % value)
+18 cur.execute("DELETE FROM foo WHERE id = '%s'" % identifier)
+
+--------------------------------------------------
+[93m>> Issue: [B608:hardcoded_sql_expressions] Possible SQL injection vector through string-based query construction.
+ Severity: Medium Confidence: Medium
+ Location: bandit_examples/sql_statements-py36.py:18
+ More Info: https://bandit.readthedocs.io/en/latest/plugins/b608_hardcoded_sql_expressions.html[0m
+17 cur.execute("INSERT INTO foo VALUES ('a', 'b', '%s')" % value)
+18 cur.execute("DELETE FROM foo WHERE id = '%s'" % identifier)
+19 cur.execute("UPDATE foo SET value = 'b' WHERE id = '%s'" % identifier)
+
+--------------------------------------------------
+[93m>> Issue: [B608:hardcoded_sql_expressions] Possible SQL injection vector through string-based query construction.
+ Severity: Medium Confidence: Medium
+ Location: bandit_examples/sql_statements-py36.py:19
+ More Info: https://bandit.readthedocs.io/en/latest/plugins/b608_hardcoded_sql_expressions.html[0m
+18 cur.execute("DELETE FROM foo WHERE id = '%s'" % identifier)
+19 cur.execute("UPDATE foo SET value = 'b' WHERE id = '%s'" % identifier)
+20 # bad alternate forms
+
+--------------------------------------------------
+[93m>> Issue: [B608:hardcoded_sql_expressions] Possible SQL injection vector through string-based query construction.
+ Severity: Medium Confidence: Medium
+ Location: bandit_examples/sql_statements-py36.py:21
+ More Info: https://bandit.readthedocs.io/en/latest/plugins/b608_hardcoded_sql_expressions.html[0m
+20 # bad alternate forms
+21 cur.execute("SELECT * FROM foo WHERE id = '" + identifier + "'")
+22 cur.execute("SELECT * FROM foo WHERE id = '{}'".format(identifier))
+
+--------------------------------------------------
+[93m>> Issue: [B608:hardcoded_sql_expressions] Possible SQL injection vector through string-based query construction.
+ Severity: Medium Confidence: Medium
+ Location: bandit_examples/sql_statements-py36.py:22
+ More Info: https://bandit.readthedocs.io/en/latest/plugins/b608_hardcoded_sql_expressions.html[0m
+21 cur.execute("SELECT * FROM foo WHERE id = '" + identifier + "'")
+22 cur.execute("SELECT * FROM foo WHERE id = '{}'".format(identifier))
+23 cur.execute(f"SELECT * FROM foo WHERE id {tmp}")
+
+--------------------------------------------------
+[93m>> Issue: [B608:hardcoded_sql_expressions] Possible SQL injection vector through string-based query construction.
+ Severity: Medium Confidence: Medium
+ Location: bandit_examples/sql_statements-py36.py:23
+ More Info: https://bandit.readthedocs.io/en/latest/plugins/b608_hardcoded_sql_expressions.html[0m
+22 cur.execute("SELECT * FROM foo WHERE id = '{}'".format(identifier))
+23 cur.execute(f"SELECT * FROM foo WHERE id {tmp}")
+24
+
+--------------------------------------------------
+[93m>> Issue: [B608:hardcoded_sql_expressions] Possible SQL injection vector through string-based query construction.
+ Severity: Medium Confidence: Low
+ Location: bandit_examples/sql_statements-py36.py:37
+ More Info: https://bandit.readthedocs.io/en/latest/plugins/b608_hardcoded_sql_expressions.html[0m
+36
+37 a()("SELECT %s FROM foo" % val)
+38
+
+--------------------------------------------------
+[95m
+Code scanned:[0m
+ Total lines of code: 59
+ Total lines skipped (#nosec): 0
+[95m
+Run metrics:[0m
+ Total issues (by severity):
+ Undefined: 0.0
+ Low: 4.0
+ Medium: 22.0
+ High: 0.0
+ Total issues (by confidence):
+ Undefined: 0.0
+ Low: 9.0
+ Medium: 7.0
+ High: 10.0
+[95mFiles skipped (0):[0m
+
+f = open('file.txt')
+| Mode | Description |
|---|---|
| ‘r’ | Open a file for reading. (default) |
| ‘w’ | Open a file for writing. Creates a new file if it does not exist or truncates the file if it exists. |
| ‘x’ | Open a file for exclusive creation. If the file already exists, the operation fails. |
| ‘a’ | Open for appending at the end of the file without truncating it. Creates a new file if it does not exist. |
| ‘t’ | Open in text mode. (default) |
| ‘b’ | Open in binary mode. |
| ’+’ | Open a file for updating (reading and writing) |
f = open("test.txt") # equivalent to 'r' or 'rt'
+f = open("test.txt", 'w') # write in text mode
+f = open("python.png",'r+b') # read and write in binary mode
+Unlike other languages, the character ‘a’ does not imply the number 97 until it is encoded using ASCII (or other equivalent encodings). The default encoding is platform dependent.
You cannot rely on the default encoding or else our code will behave differently in different platforms. It is highly recommended to specify the encoding type when working with text files.
f = open("file.txt", mode='r', encoding='utf-8')
+f = open("file.txt", encoding='utf-8')
+# perform file operations
+f.close()
+In the last example exceptions could crash your program. Exception handlig is a topic in itself, but here we’ll simply wrap the code in a try…catch block. Now the file handle will be closed regardless of whether or not there was an exception. In the example above, an exeption might result in the file remaining open (and potentially locked) from other applications in the event of an exception.
try:
+ f = open("file.txt",encoding = 'utf-8')
+ # perform file operations
+finally:
+ f.close()
+The best way to ensure that you close your files when the code block terminates is by using the with statement. When using the with statement you do not have to explicitly call the file’s close() method.
with open('file.txt') as f:
+ text = f.read()
+
+print(text)
+# f.close() # not necessary
+hello class
+python is fun
+
+let's get started
+with open("file.txt", 'w', encoding='utf-8') as f:
+ print(type(f))
+ f.write("hello class\n" )
+ f.write("python is fun\n\n")
+ f.write("let's get started\n")
+<class '_io.TextIOWrapper'>
+f = open("file.txt", 'r', encoding='utf-8')
+print('1:', f.read(3)) # read the 1st 3 characters
+print('2:', f.read(3)) # read the next 3 characters
+print('3:', f.read()) # read until the end
+print('4:', f.read()) # return's an empty string
+1: hel
+2: lo
+3: class
+python is fun
+
+let's get started
+
+4:
+f.tell() # get the current file position
+45
+f.seek(0) # bring file cursor to initial position
+0
+f.read() # read the entire file
+"hello class\npython is fun\n\nlet's get started\n"
+f.seek(0) # reset the cursor
+0
+# read line by line
+for line in f:
+ print(line, end='')
+hello class
+python is fun
+
+let's get started
+f.seek(0) # reset the cursor
+0
+# read line by line
+f.readline()
+'hello class\n'
+f.readline()
+'python is fun\n'
+f.readline()
+'\n'
+line = f.readline()
+print(type(line), line)
+<class 'str'> let's get started
+f.readline()
+''
+f.seek(0) # reset the cursor
+0
+# read all lines at once
+f.readlines()
+['hello class\n', 'python is fun\n', '\n', "let's get started\n"]
+print('abcd')
+print('efgh')
+abcd
+efgh
+print('abcd', end='')
+print('efgh')
+abcdefgh
+text = ' abcd '
+print(text, end='')
+print(text)
+ abcd abcd
+text = ' abcd '
+text = text.strip()
+print(text, end='')
+print(text)
+abcdabcd
+print?
+[0;31mDocstring:[0m
+print(value, ..., sep=' ', end='\n', file=sys.stdout, flush=False)
+
+Prints the values to a stream, or to sys.stdout by default.
+Optional keyword arguments:
+file: a file-like object (stream); defaults to the current sys.stdout.
+sep: string inserted between values, default a space.
+end: string appended after the last value, default a newline.
+flush: whether to forcibly flush the stream.
+[0;31mType:[0m builtin_function_or_method
+print('abc', 'def', 'xyz', sep=',')
+abc,def,xyz
+
+# open and print line by line
+with open('mock.csv') as f:
+ for line in f:
+ print(line)
+id,first_name,last_name,email,gender,ip_address
+
+1,Finn,Burchmore,fburchmore0@epa.gov,Male,74.20.172.212
+
+2,Coral,Grigorey,cgrigorey1@cdc.gov,Female,151.91.212.169
+
+3,Harold,McCormack,hmccormack2@marketwatch.com,Male,72.36.41.150
+
+4,Thacher,Woodvine,twoodvine3@marketwatch.com,Male,66.50.122.48
+
+5,Querida,Allmann,qallmann4@people.com.cn,Female,236.145.83.165
+
+6,Irving,Loughman,iloughman5@globo.com,Male,19.142.188.98
+
+7,Meghann,Vittet,mvittet6@cyberchimps.com,Female,235.59.102.124
+
+8,Wolfgang,Mishow,wmishow7@gnu.org,Male,189.36.37.25
+
+9,Babbie,Reide,breide8@jiathis.com,Female,129.226.63.38
+
+10,Carson,Vowdon,cvowdon9@slashdot.org,Male,51.228.124.19
+with open('mock.csv') as f:
+ # use enumerate to get the index as you iterate
+ for index, line in enumerate(f):
+ print(index, line)
+0 id,first_name,last_name,email,gender,ip_address
+
+1 1,Finn,Burchmore,fburchmore0@epa.gov,Male,74.20.172.212
+
+2 2,Coral,Grigorey,cgrigorey1@cdc.gov,Female,151.91.212.169
+
+3 3,Harold,McCormack,hmccormack2@marketwatch.com,Male,72.36.41.150
+
+4 4,Thacher,Woodvine,twoodvine3@marketwatch.com,Male,66.50.122.48
+
+5 5,Querida,Allmann,qallmann4@people.com.cn,Female,236.145.83.165
+
+6 6,Irving,Loughman,iloughman5@globo.com,Male,19.142.188.98
+
+7 7,Meghann,Vittet,mvittet6@cyberchimps.com,Female,235.59.102.124
+
+8 8,Wolfgang,Mishow,wmishow7@gnu.org,Male,189.36.37.25
+
+9 9,Babbie,Reide,breide8@jiathis.com,Female,129.226.63.38
+
+10 10,Carson,Vowdon,cvowdon9@slashdot.org,Male,51.228.124.19
+index = 0
+processed_header = False
+with open('mock.csv') as f:
+ for line in f:
+ # handle the header differently
+ if processed_header == False:
+ print("header", index, line)
+ processed_header = True
+ else:
+ print("row", index, line)
+ index += 1
+header 0 id,first_name,last_name,email,gender,ip_address
+
+row 1 1,Finn,Burchmore,fburchmore0@epa.gov,Male,74.20.172.212
+
+row 2 2,Coral,Grigorey,cgrigorey1@cdc.gov,Female,151.91.212.169
+
+row 3 3,Harold,McCormack,hmccormack2@marketwatch.com,Male,72.36.41.150
+
+row 4 4,Thacher,Woodvine,twoodvine3@marketwatch.com,Male,66.50.122.48
+
+row 5 5,Querida,Allmann,qallmann4@people.com.cn,Female,236.145.83.165
+
+row 6 6,Irving,Loughman,iloughman5@globo.com,Male,19.142.188.98
+
+row 7 7,Meghann,Vittet,mvittet6@cyberchimps.com,Female,235.59.102.124
+
+row 8 8,Wolfgang,Mishow,wmishow7@gnu.org,Male,189.36.37.25
+
+row 9 9,Babbie,Reide,breide8@jiathis.com,Female,129.226.63.38
+
+row 10 10,Carson,Vowdon,cvowdon9@slashdot.org,Male,51.228.124.19
+with open('mock.csv') as f:
+ for index, line in enumerate(f):
+ # handle the header differently
+ if index == 0:
+ print("header", index, line)
+ else:
+ print("row", index, line)
+header 0 id,first_name,last_name,email,gender,ip_address
+
+row 1 1,Finn,Burchmore,fburchmore0@epa.gov,Male,74.20.172.212
+
+row 2 2,Coral,Grigorey,cgrigorey1@cdc.gov,Female,151.91.212.169
+
+row 3 3,Harold,McCormack,hmccormack2@marketwatch.com,Male,72.36.41.150
+
+row 4 4,Thacher,Woodvine,twoodvine3@marketwatch.com,Male,66.50.122.48
+
+row 5 5,Querida,Allmann,qallmann4@people.com.cn,Female,236.145.83.165
+
+row 6 6,Irving,Loughman,iloughman5@globo.com,Male,19.142.188.98
+
+row 7 7,Meghann,Vittet,mvittet6@cyberchimps.com,Female,235.59.102.124
+
+row 8 8,Wolfgang,Mishow,wmishow7@gnu.org,Male,189.36.37.25
+
+row 9 9,Babbie,Reide,breide8@jiathis.com,Female,129.226.63.38
+
+row 10 10,Carson,Vowdon,cvowdon9@slashdot.org,Male,51.228.124.19
+headers = []
+with open('mock.csv') as f:
+ for index, line in enumerate(f):
+ line = line.strip()
+ if index == 0:
+ # split the headers
+ headers = line.split(',')
+ else:
+ pass
+print('header fields', headers)
+header fields ['id', 'first_name', 'last_name', 'email', 'gender', 'ip_address']
+headers = []
+rows = []
+with open('mock.csv') as f:
+ for index, line in enumerate(f):
+ line = line.strip()
+ if index == 0:
+ headers = line.split(',')
+ else:
+ # append the rows
+ rows.append(line.split(','))
+print('header fields', headers)
+print('rows fields', rows)
+header fields ['id', 'first_name', 'last_name', 'email', 'gender', 'ip_address']
+rows fields [['1', 'Finn', 'Burchmore', 'fburchmore0@epa.gov', 'Male', '74.20.172.212'], ['2', 'Coral', 'Grigorey', 'cgrigorey1@cdc.gov', 'Female', '151.91.212.169'], ['3', 'Harold', 'McCormack', 'hmccormack2@marketwatch.com', 'Male', '72.36.41.150'], ['4', 'Thacher', 'Woodvine', 'twoodvine3@marketwatch.com', 'Male', '66.50.122.48'], ['5', 'Querida', 'Allmann', 'qallmann4@people.com.cn', 'Female', '236.145.83.165'], ['6', 'Irving', 'Loughman', 'iloughman5@globo.com', 'Male', '19.142.188.98'], ['7', 'Meghann', 'Vittet', 'mvittet6@cyberchimps.com', 'Female', '235.59.102.124'], ['8', 'Wolfgang', 'Mishow', 'wmishow7@gnu.org', 'Male', '189.36.37.25'], ['9', 'Babbie', 'Reide', 'breide8@jiathis.com', 'Female', '129.226.63.38'], ['10', 'Carson', 'Vowdon', 'cvowdon9@slashdot.org', 'Male', '51.228.124.19']]
+Reading and Writing CSV Files in Python
import csv
+with open('mock.csv') as f:
+ reader = csv.reader(f, delimiter=',')
+ for row in reader:
+ print(row)
+['id', 'first_name', 'last_name', 'email', 'gender', 'ip_address']
+['1', 'Finn', 'Burchmore', 'fburchmore0@epa.gov', 'Male', '74.20.172.212']
+['2', 'Coral', 'Grigorey', 'cgrigorey1@cdc.gov', 'Female', '151.91.212.169']
+['3', 'Harold', 'McCormack', 'hmccormack2@marketwatch.com', 'Male', '72.36.41.150']
+['4', 'Thacher', 'Woodvine', 'twoodvine3@marketwatch.com', 'Male', '66.50.122.48']
+['5', 'Querida', 'Allmann', 'qallmann4@people.com.cn', 'Female', '236.145.83.165']
+['6', 'Irving', 'Loughman', 'iloughman5@globo.com', 'Male', '19.142.188.98']
+['7', 'Meghann', 'Vittet', 'mvittet6@cyberchimps.com', 'Female', '235.59.102.124']
+['8', 'Wolfgang', 'Mishow', 'wmishow7@gnu.org', 'Male', '189.36.37.25']
+['9', 'Babbie', 'Reide', 'breide8@jiathis.com', 'Female', '129.226.63.38']
+['10', 'Carson', 'Vowdon', 'cvowdon9@slashdot.org', 'Male', '51.228.124.19']
+with open('mock.csv') as f:
+ reader = csv.reader(f, delimiter=',')
+ for index, row in enumerate(reader):
+ if index == 0:
+ print(f'{", ".join(row)}')
+ else:
+ print(f'id: {row[0]} first_name: {row[1]} last_name: {row[2]} email: {row[3]}')
+id, first_name, last_name, email, gender, ip_address
+id: 1 first_name: Finn last_name: Burchmore email: fburchmore0@epa.gov
+id: 2 first_name: Coral last_name: Grigorey email: cgrigorey1@cdc.gov
+id: 3 first_name: Harold last_name: McCormack email: hmccormack2@marketwatch.com
+id: 4 first_name: Thacher last_name: Woodvine email: twoodvine3@marketwatch.com
+id: 5 first_name: Querida last_name: Allmann email: qallmann4@people.com.cn
+id: 6 first_name: Irving last_name: Loughman email: iloughman5@globo.com
+id: 7 first_name: Meghann last_name: Vittet email: mvittet6@cyberchimps.com
+id: 8 first_name: Wolfgang last_name: Mishow email: wmishow7@gnu.org
+id: 9 first_name: Babbie last_name: Reide email: breide8@jiathis.com
+id: 10 first_name: Carson last_name: Vowdon email: cvowdon9@slashdot.org
+with open('mock.csv') as f:
+ reader = csv.DictReader(f)
+ for index, row in enumerate(reader):
+ print(f'id: {row["id"]} first_name: {row["first_name"]} last_name: {row["last_name"]} email: {row["email"]}')
+id: 1 first_name: Finn last_name: Burchmore email: fburchmore0@epa.gov
+id: 2 first_name: Coral last_name: Grigorey email: cgrigorey1@cdc.gov
+id: 3 first_name: Harold last_name: McCormack email: hmccormack2@marketwatch.com
+id: 4 first_name: Thacher last_name: Woodvine email: twoodvine3@marketwatch.com
+id: 5 first_name: Querida last_name: Allmann email: qallmann4@people.com.cn
+id: 6 first_name: Irving last_name: Loughman email: iloughman5@globo.com
+id: 7 first_name: Meghann last_name: Vittet email: mvittet6@cyberchimps.com
+id: 8 first_name: Wolfgang last_name: Mishow email: wmishow7@gnu.org
+id: 9 first_name: Babbie last_name: Reide email: breide8@jiathis.com
+id: 10 first_name: Carson last_name: Vowdon email: cvowdon9@slashdot.org
+with open('scores.csv', mode='w') as employee_file:
+ writer = csv.writer(employee_file, delimiter=',', quotechar='"', quoting=csv.QUOTE_MINIMAL)
+ writer.writerow(['2019-05-01', 'Pirates', 0, 'Cubs', 10])
+ writer.writerow(['2019-05-15', 'Reds', 7, 'Pirates', 0])
+with open('scores.csv', mode='w') as csv_file:
+ fieldnames = ['date', 'home_team', 'home_score', 'away_team', 'away_score']
+ writer = csv.DictWriter(csv_file, fieldnames=fieldnames)
+
+ writer.writeheader()
+ writer.writerow({'date': '2019-05-01', 'home_team': 'Pirates', 'home_score': 0, 'away_team': 'Cubs', 'away_score': 10})
+ writer.writerow({'date': '2019-05-15', 'home_team': 'Reds', 'home_score': 7, 'away_team': 'Pirates', 'away_score': 0})
+x = "1"
+print(x + str(3))
+13
+x = "1"
+n = float(x)
+print(n + n)
+print(n * 3)
+2.0
+3.0
+x = "blah"
+print(x)
+x = 125
+print(x)
+blah
+125
+
+Working With JSON Data in Python
from pprint import pprint
+# scores dictionary
+games = {
+ 'games':[
+ {
+ 'date': '2019-05-01',
+ 'home_team': 'Pirates',
+ 'home_score': 0,
+ 'away_team': 'Cubs',
+ 'away_score': 10
+ },
+ {
+ 'date': '2019-05-15',
+ 'home_team': 'Reds',
+ 'home_score': 7,
+ 'away_team': 'Pirates',
+ 'away_score': 0
+ },
+ {
+ 'date': '2019-05-17',
+ 'home_team': 'Reds',
+ 'home_score': 12,
+ 'away_team': 'Pirates',
+ 'away_score': 0
+ },
+ {
+ 'date': '2019-05-18',
+ 'home_team': 'Reds',
+ 'home_score': 8,
+ 'away_team': 'Pirates',
+ 'away_score': 0
+ },
+ {
+ 'date': '2019-05-21',
+ 'home_team': 'Pirates',
+ 'home_score': 1,
+ 'away_team': 'Brewers',
+ 'away_score': 11
+ }
+ ]
+}
+pprint(games)
+{'games': [{'away_score': 10,
+ 'away_team': 'Cubs',
+ 'date': '2019-05-01',
+ 'home_score': 0,
+ 'home_team': 'Pirates'},
+ {'away_score': 0,
+ 'away_team': 'Pirates',
+ 'date': '2019-05-15',
+ 'home_score': 7,
+ 'home_team': 'Reds'},
+ {'away_score': 0,
+ 'away_team': 'Pirates',
+ 'date': '2019-05-17',
+ 'home_score': 12,
+ 'home_team': 'Reds'},
+ {'away_score': 0,
+ 'away_team': 'Pirates',
+ 'date': '2019-05-18',
+ 'home_score': 8,
+ 'home_team': 'Reds'},
+ {'away_score': 11,
+ 'away_team': 'Brewers',
+ 'date': '2019-05-21',
+ 'home_score': 1,
+ 'home_team': 'Pirates'}]}
+In this instance, serialization is the process of converting the python objects to JSON. Python object types get serialized to the following JSON types.
| Python | JSON |
|---|---|
| dict | object |
| list, tuple | array |
| str | string |
| int, long, float | number |
| True | true |
| False | false |
| None | null |
import json
+pprint(games)
+print()
+
+games_json = json.dumps(games)
+pprint(games_json)
+{'games': [{'away_score': 10,
+ 'away_team': 'Cubs',
+ 'date': '2019-05-01',
+ 'home_score': 0,
+ 'home_team': 'Pirates'},
+ {'away_score': 0,
+ 'away_team': 'Pirates',
+ 'date': '2019-05-15',
+ 'home_score': 7,
+ 'home_team': 'Reds'},
+ {'away_score': 0,
+ 'away_team': 'Pirates',
+ 'date': '2019-05-17',
+ 'home_score': 12,
+ 'home_team': 'Reds'},
+ {'away_score': 0,
+ 'away_team': 'Pirates',
+ 'date': '2019-05-18',
+ 'home_score': 8,
+ 'home_team': 'Reds'},
+ {'away_score': 11,
+ 'away_team': 'Brewers',
+ 'date': '2019-05-21',
+ 'home_score': 1,
+ 'home_team': 'Pirates'}]}
+
+('{"games": [{"date": "2019-05-01", "home_team": "Pirates", "home_score": 0, '
+ '"away_team": "Cubs", "away_score": 10}, {"date": "2019-05-15", "home_team": '
+ '"Reds", "home_score": 7, "away_team": "Pirates", "away_score": 0}, {"date": '
+ '"2019-05-17", "home_team": "Reds", "home_score": 12, "away_team": "Pirates", '
+ '"away_score": 0}, {"date": "2019-05-18", "home_team": "Reds", "home_score": '
+ '8, "away_team": "Pirates", "away_score": 0}, {"date": "2019-05-21", '
+ '"home_team": "Pirates", "home_score": 1, "away_team": "Brewers", '
+ '"away_score": 11}]}')
+# changing whitespace
+games_json = json.dumps(games, indent=4)
+print(games_json)
+{
+ "games": [
+ {
+ "date": "2019-05-01",
+ "home_team": "Pirates",
+ "home_score": 0,
+ "away_team": "Cubs",
+ "away_score": 10
+ },
+ {
+ "date": "2019-05-15",
+ "home_team": "Reds",
+ "home_score": 7,
+ "away_team": "Pirates",
+ "away_score": 0
+ },
+ {
+ "date": "2019-05-17",
+ "home_team": "Reds",
+ "home_score": 12,
+ "away_team": "Pirates",
+ "away_score": 0
+ },
+ {
+ "date": "2019-05-18",
+ "home_team": "Reds",
+ "home_score": 8,
+ "away_team": "Pirates",
+ "away_score": 0
+ },
+ {
+ "date": "2019-05-21",
+ "home_team": "Pirates",
+ "home_score": 1,
+ "away_team": "Brewers",
+ "away_score": 11
+ }
+ ]
+}
+# compacting
+games_json = json.dumps(games, separators=(',', ':'))
+print(games_json)
+{"games":[{"date":"2019-05-01","home_team":"Pirates","home_score":0,"away_team":"Cubs","away_score":10},{"date":"2019-05-15","home_team":"Reds","home_score":7,"away_team":"Pirates","away_score":0},{"date":"2019-05-17","home_team":"Reds","home_score":12,"away_team":"Pirates","away_score":0},{"date":"2019-05-18","home_team":"Reds","home_score":8,"away_team":"Pirates","away_score":0},{"date":"2019-05-21","home_team":"Pirates","home_score":1,"away_team":"Brewers","away_score":11}]}
+with open("games.json", "w") as f:
+ json.dump(games, f)
+# open and print line by line
+with open('games.json', "r") as f:
+ games = json.load(f)
+ print(games)
+{'games': [{'date': '2019-05-01', 'home_team': 'Pirates', 'home_score': 0, 'away_team': 'Cubs', 'away_score': 10}, {'date': '2019-05-15', 'home_team': 'Reds', 'home_score': 7, 'away_team': 'Pirates', 'away_score': 0}, {'date': '2019-05-17', 'home_team': 'Reds', 'home_score': 12, 'away_team': 'Pirates', 'away_score': 0}, {'date': '2019-05-18', 'home_team': 'Reds', 'home_score': 8, 'away_team': 'Pirates', 'away_score': 0}, {'date': '2019-05-21', 'home_team': 'Pirates', 'home_score': 1, 'away_team': 'Brewers', 'away_score': 11}]}
+One of the biggest strengths of XML is XPath, the query-oriented language to query subsections of an XML document. In the same line, JSONPath is a query language with features similar to XPath that lets you extract just the bits of a JSON document your application needs.
As XPath, JSONPath also has syntax to follow:
$ – symbol refers to the root object or element.@ – symbol refers to the current object or element.. – operator is the dot-child operator, which you use to denote a child element of the current element.[ ] – is the subscript operator, which you use to denote a child element of the current element (by name or index).* – operator is a wildcard, returning all objects or elements regardless of their names., – operator is the union operator, which returns the union of the children or indexes indicated.: – operator is the array slice operator, so you can slice collections using the syntax [start:end:step] to return a subcollection of a collection.( ) – operator lets you pass a script expression in the underlying implementation’s script language. It’s not supported by every implementation of JSONPath, however.? ( ) – to query all items that meet a certain criteria.There are many online jsonpath validators out there, and I encourage you to try a few.
To use JSONPath, you will need to include its dependency and then use it.
The library we’ll use is jsonpath-rw-ext
pip install jsonpath-rw-ext
+Once installed we’re ready to use it.
Lets try to parse a JSON file a few different ways. First, we need to load the data as a JSON object into memory.
with open('games.json', "r") as f:
+ games = json.load(f)
+ pprint(games)
+{'games': [{'away_score': 10,
+ 'away_team': 'Cubs',
+ 'date': '2019-05-01',
+ 'home_score': 0,
+ 'home_team': 'Pirates'},
+ {'away_score': 0,
+ 'away_team': 'Pirates',
+ 'date': '2019-05-15',
+ 'home_score': 7,
+ 'home_team': 'Reds'},
+ {'away_score': 0,
+ 'away_team': 'Pirates',
+ 'date': '2019-05-17',
+ 'home_score': 12,
+ 'home_team': 'Reds'},
+ {'away_score': 0,
+ 'away_team': 'Pirates',
+ 'date': '2019-05-18',
+ 'home_score': 8,
+ 'home_team': 'Reds'},
+ {'away_score': 11,
+ 'away_team': 'Brewers',
+ 'date': '2019-05-21',
+ 'home_score': 1,
+ 'home_team': 'Pirates'}]}
+Now that the data is loaded, we can try to answer a few questions.
# json parsing
+game_dates = []
+for game in games['games']:
+ game_date = game['date']
+ game_dates.append(game_date)
+print(game_dates)
+['2019-05-01', '2019-05-15', '2019-05-17', '2019-05-18', '2019-05-21']
+# jsonpath parsing
+# import the library
+import jsonpath_rw_ext as jp
+# create a jsonpath expression and match
+game_dates = jp.match("$.games[*].date", games)
+print(game_dates)
+['2019-05-01', '2019-05-15', '2019-05-17', '2019-05-18', '2019-05-21']
+# this also works
+game_dates = jp.match("$.games..date", games)
+print(game_dates)
+['2019-05-01', '2019-05-15', '2019-05-17', '2019-05-18', '2019-05-21']
+# json parsing
+home_scores = []
+for game in games['games']:
+ if game['home_team'] == 'Pirates':
+ home_scores.append(game)
+
+pprint(home_scores)
+[{'away_score': 10,
+ 'away_team': 'Cubs',
+ 'date': '2019-05-01',
+ 'home_score': 0,
+ 'home_team': 'Pirates'},
+ {'away_score': 11,
+ 'away_team': 'Brewers',
+ 'date': '2019-05-21',
+ 'home_score': 1,
+ 'home_team': 'Pirates'}]
+# jsonpath parsing
+home_scores = jp.match("$.games[?(@.home_team=='Pirates')]", games)
+pprint(home_scores)
+[{'away_score': 10,
+ 'away_team': 'Cubs',
+ 'date': '2019-05-01',
+ 'home_score': 0,
+ 'home_team': 'Pirates'},
+ {'away_score': 11,
+ 'away_team': 'Brewers',
+ 'date': '2019-05-21',
+ 'home_score': 1,
+ 'home_team': 'Pirates'}]
+# json parsing
+game_scores = []
+for game in games['games']:
+ if game['date'] > '2019-05-15':
+ game_scores.append(game)
+
+pprint(game_scores)
+[{'away_score': 0,
+ 'away_team': 'Pirates',
+ 'date': '2019-05-17',
+ 'home_score': 12,
+ 'home_team': 'Reds'},
+ {'away_score': 0,
+ 'away_team': 'Pirates',
+ 'date': '2019-05-18',
+ 'home_score': 8,
+ 'home_team': 'Reds'},
+ {'away_score': 11,
+ 'away_team': 'Brewers',
+ 'date': '2019-05-21',
+ 'home_score': 1,
+ 'home_team': 'Pirates'}]
+# jsonpath parsing
+game_scores = jp.match("$.games[?(@.date > '2019-05-15')]", games)
+pprint(game_scores)
+[{'away_score': 0,
+ 'away_team': 'Pirates',
+ 'date': '2019-05-17',
+ 'home_score': 12,
+ 'home_team': 'Reds'},
+ {'away_score': 0,
+ 'away_team': 'Pirates',
+ 'date': '2019-05-18',
+ 'home_score': 8,
+ 'home_team': 'Reds'},
+ {'away_score': 11,
+ 'away_team': 'Brewers',
+ 'date': '2019-05-21',
+ 'home_score': 1,
+ 'home_team': 'Pirates'}]
+
+Python XML with ElementTree: Beginner’s Guide
# scores dictionary
+scores_xml = """
+<?xml version="1.0"?>
+<scores>
+ <score>
+ <date>2019-05-01</date>
+ <home-team>Pirates</home-team>
+ <home-score>0</home-score>
+ <away-team>Cubs</away-team>
+ <away-score>10</away-score>
+ </score>
+ <score>
+ <date>2019-05-15</date>
+ <home-team>Reds</home-team>
+ <home-score>7</home-score>
+ <away-team>Pirates</away-team>
+ <away-score>0</away-score>
+ </score>
+</scores>
+"""
+print(scores_xml)
+<?xml version="1.0"?>
+<scores>
+ <score>
+ <date>2019-05-01</date>
+ <home-team>Pirates</home-team>
+ <home-score>0</home-score>
+ <away-team>Cubs</away-team>
+ <away-score>10</away-score>
+ </score>
+ <score>
+ <date>2019-05-15</date>
+ <home-team>Reds</home-team>
+ <home-score>7</home-score>
+ <away-team>Pirates</away-team>
+ <away-score>0</away-score>
+ </score>
+</scores>
+scores_xsd = """
+<xs:schema attributeFormDefault="unqualified" elementFormDefault="qualified" xmlns:xs="http://www.w3.org/2001/XMLSchema">
+ <xs:element name="scores">
+ <xs:complexType>
+ <xs:sequence>
+ <xs:element name="score" maxOccurs="unbounded" minOccurs="0">
+ <xs:complexType>
+ <xs:sequence>
+ <xs:element type="xs:date" name="date"/>
+ <xs:element type="xs:string" name="home-team"/>
+ <xs:element type="xs:int" name="home-score"/>
+ <xs:element type="xs:string" name="away-team"/>
+ <xs:element type="xs:int" name="away-score"/>
+ </xs:sequence>
+ </xs:complexType>
+ </xs:element>
+ </xs:sequence>
+ </xs:complexType>
+ </xs:element>
+</xs:schema>
+"""
+print(scores_xsd)
+<xs:schema attributeFormDefault="unqualified" elementFormDefault="qualified" xmlns:xs="http://www.w3.org/2001/XMLSchema">
+ <xs:element name="scores">
+ <xs:complexType>
+ <xs:sequence>
+ <xs:element name="score" maxOccurs="unbounded" minOccurs="0">
+ <xs:complexType>
+ <xs:sequence>
+ <xs:element type="xs:date" name="date"/>
+ <xs:element type="xs:string" name="home-team"/>
+ <xs:element type="xs:int" name="home-score"/>
+ <xs:element type="xs:string" name="away-team"/>
+ <xs:element type="xs:int" name="away-score"/>
+ </xs:sequence>
+ </xs:complexType>
+ </xs:element>
+ </xs:sequence>
+ </xs:complexType>
+ </xs:element>
+</xs:schema>
+import xml.etree.ElementTree as ET
+tree = ET.parse('scores.xml')
+root = tree.getroot()
+print(root.tag)
+scores
+print(root.attrib)
+{'{http://www.w3.org/2001/XMLSchema-instance}noNamespaceSchemaLocation': 'scores.xsd'}
+for child in root:
+ print(child.tag, child.attrib)
+score {}
+score {}
+Typically it is helpful to know all the elements in the entire tree. A useful function for doing that is root.iter(). You can put this function into a “for” loop and it will iterate over the entire tree.
for elem in root.iter():
+ print(elem.tag)
+scores
+score
+date
+home-team
+home-score
+away-team
+away-score
+score
+date
+home-team
+home-score
+away-team
+away-score
+If you need to print the entire string you can use ElementTree.tostring(). However, you must specify both the encoding and decoding of the document you are displaying as the string because ElementTree is a powerful library that can interpret more than just XML, For XMLs, use utf8 - this is the typical document format type for an XML.
print(ET.tostring(root, encoding='utf8').decode('utf8'))
+<?xml version='1.0' encoding='utf8'?>
+<scores xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="scores.xsd">
+ <score>
+ <date>2019-05-01</date>
+ <home-team>Pirates</home-team>
+ <home-score>0</home-score>
+ <away-team>Cubs</away-team>
+ <away-score>10</away-score>
+ </score>
+ <score>
+ <date>2019-05-15</date>
+ <home-team>Reds</home-team>
+ <home-score>7</home-score>
+ <away-team>Pirates</away-team>
+ <away-score>0</away-score>
+ </score>
+</scores>
+for game_date in root.iter('date'):
+ print(game_date.text)
+2019-05-01
+2019-05-15
+for score in root.findall('./score[date="2019-05-01"]/date'):
+ print(score.text)
+2019-05-01
+tree.write("scores2.xml")
+
+Note: this example uses FHIR DSTU3 whereas synthea now supports FHIR R4 so you’ll need a different version of the fhirclient library to deal with each dataset
The fhirclient, a flexible Python client for FHIR servers supporting the SMART on FHIR protocol.
fhirclient versioning is not identical to FHIR versioning, see the full table for reference.
| Version | FHIR | |
|---|---|---|
| 4.0.0 | 4.0.0 | (R4) |
| 3.0.0 | 3.0.0 | (STU-3) |
| 1.0.3 | 1.0.2 | (DSTU 2) |
| 1.0 | 1.0.1 | (DSTU 2) |
For FHIR version 3
pip install fhirclient
+or for FHIR version 4
pip install git+https://github.com/smart-on-fhir/client-py.git
+NOTE: We’ll use FHIR R4 here so you need to install from GitHub
Technical documentation is available at docs.
To connect to a SMART on FHIR server (or any open FHIR server), you can use the FHIRClient class. It will initialize and handle a FHIRServer instance, your actual handle to the FHIR server you’d like to access.
To read a given patient from an open FHIR server, you can use:
from pprint import pprint
+from fhirclient import client
+
+settings = {
+'app_id': 'my_web_app',
+'api_base': 'http://hapi.fhir.org/baseR4'
+}
+
+smart = client.FHIRClient(settings=settings)
+
+import fhirclient.models.patient as p
+
+patient = p.Patient.read('2543713', smart.server)
+print(patient.birthDate.isostring)
+print(smart.human_name(patient.name[0]))
+1974-12-25
+Peter James Chalmers
+If this is a protected server, you will first have to send your user to the authorize endpoint to log in.
smart.authorize_url to obtain the correct URL. smart.prepare(), which will return False if the server is protected and you need to authorize.smart.ready property has the same purpose, it will however not retrieve the server’s CapabilityStatement resource and hence is only useful as a quick check whether the server instance is ready.smart = client.FHIRClient(settings=settings)
+smart.ready
+
+# prints False
+print(smart.prepare())
+
+# prints True after fetching CapabilityStatement
+print(smart.ready)
+
+# prints True
+print(smart.prepare())
+
+# prints True immediately
+print(smart.authorize_url)
+True
+True
+True
+None
+You can work with the FHIRServer class directly, without using FHIRClient, but this is not recommended:
You can also search for resources matching a particular set of criteria:
smart = client.FHIRClient(settings=settings)
+
+import fhirclient.models.patient as p
+
+search = p.Patient.where(struct={'family': 'Cushing'})
+patients = search.perform_resources(smart.server)
+# print(observations)
+
+pprint(patients[0].as_json())
+{'id': '2581341',
+ 'identifier': [{'type': {'coding': [{'code': 'MR',
+ 'system': 'http://hl7.org/fhir/v2/0203'}]},
+ 'value': 'dbee78a3-a72d-41a2-b705-0023498f5228'}],
+ 'meta': {'lastUpdated': '2021-10-07T05:30:35.359+00:00',
+ 'source': '#ghjs1LZGhTvv0mp2',
+ 'versionId': '1'},
+ 'name': [{'family': 'Cushing', 'given': ['Caleb']}],
+ 'resourceType': 'Patient',
+ 'text': {'div': '<div xmlns="http://www.w3.org/1999/xhtml"><div '
+ 'class="hapiHeaderText">Caleb <b>CUSHING </b></div><table '
+ 'class="hapiPropertyTable"><tbody><tr><td>Identifier</td><td>dbee78a3-a72d-41a2-b705-0023498f5228</td></tr></tbody></table></div>',
+ 'status': 'generated'}}
+# to get the raw Bundle instead of resources only, you can use:
+bundle = search.perform(smart.server)
+pprint(bundle.as_json())
+{'entry': [{'fullUrl': 'http://hapi.fhir.org/baseR4/Patient/2581341',
+ 'resource': {'id': '2581341',
+ 'identifier': [{'type': {'coding': [{'code': 'MR',
+ 'system': 'http://hl7.org/fhir/v2/0203'}]},
+ 'value': 'dbee78a3-a72d-41a2-b705-0023498f5228'}],
+ 'meta': {'lastUpdated': '2021-10-07T05:30:35.359+00:00',
+ 'source': '#ghjs1LZGhTvv0mp2',
+ 'versionId': '1'},
+ 'name': [{'family': 'Cushing', 'given': ['Caleb']}],
+ 'resourceType': 'Patient',
+ 'text': {'div': '<div '
+ 'xmlns="http://www.w3.org/1999/xhtml"><div '
+ 'class="hapiHeaderText">Caleb '
+ '<b>CUSHING </b></div><table '
+ 'class="hapiPropertyTable"><tbody><tr><td>Identifier</td><td>dbee78a3-a72d-41a2-b705-0023498f5228</td></tr></tbody></table></div>',
+ 'status': 'generated'}},
+ 'search': {'mode': 'match'}},
+ {'fullUrl': 'http://hapi.fhir.org/baseR4/Patient/2581393',
+ 'resource': {'id': '2581393',
+ 'identifier': [{'type': {'coding': [{'code': 'MR',
+ 'system': 'http://hl7.org/fhir/v2/0203'}]},
+ 'value': '2575e281-136c-489b-9b4e-4baba209258e'}],
+ 'meta': {'lastUpdated': '2021-10-07T05:30:55.659+00:00',
+ 'source': '#CHofcK6KSc6z9CD3',
+ 'versionId': '1'},
+ 'name': [{'family': 'Cushing', 'given': ['Caleb']}],
+ 'resourceType': 'Patient',
+ 'text': {'div': '<div '
+ 'xmlns="http://www.w3.org/1999/xhtml"><div '
+ 'class="hapiHeaderText">Caleb '
+ '<b>CUSHING </b></div><table '
+ 'class="hapiPropertyTable"><tbody><tr><td>Identifier</td><td>2575e281-136c-489b-9b4e-4baba209258e</td></tr></tbody></table></div>',
+ 'status': 'generated'}},
+ 'search': {'mode': 'match'}},
+ {'fullUrl': 'http://hapi.fhir.org/baseR4/Patient/2581353',
+ 'resource': {'id': '2581353',
+ 'identifier': [{'type': {'coding': [{'code': 'MR',
+ 'system': 'http://hl7.org/fhir/v2/0203'}]},
+ 'value': '174ee1ec-c239-4b82-84a6-f0354aa6ee5f'}],
+ 'meta': {'lastUpdated': '2021-10-07T05:30:39.301+00:00',
+ 'source': '#vB6DyRxG5VKe1DAW',
+ 'versionId': '1'},
+ 'name': [{'family': 'Cushing', 'given': ['Caleb']}],
+ 'resourceType': 'Patient',
+ 'text': {'div': '<div '
+ 'xmlns="http://www.w3.org/1999/xhtml"><div '
+ 'class="hapiHeaderText">Caleb '
+ '<b>CUSHING </b></div><table '
+ 'class="hapiPropertyTable"><tbody><tr><td>Identifier</td><td>174ee1ec-c239-4b82-84a6-f0354aa6ee5f</td></tr></tbody></table></div>',
+ 'status': 'generated'}},
+ 'search': {'mode': 'match'}},
+ {'fullUrl': 'http://hapi.fhir.org/baseR4/Patient/2581361',
+ 'resource': {'id': '2581361',
+ 'identifier': [{'type': {'coding': [{'code': 'MR',
+ 'system': 'http://hl7.org/fhir/v2/0203'}]},
+ 'value': '7c411bcd-c418-42d5-98d1-15aac35c7be0'}],
+ 'meta': {'lastUpdated': '2021-10-07T05:30:41.751+00:00',
+ 'source': '#bmgIW3uJ6beOv2VV',
+ 'versionId': '1'},
+ 'name': [{'family': 'Cushing', 'given': ['Caleb']}],
+ 'resourceType': 'Patient',
+ 'text': {'div': '<div '
+ 'xmlns="http://www.w3.org/1999/xhtml"><div '
+ 'class="hapiHeaderText">Caleb '
+ '<b>CUSHING </b></div><table '
+ 'class="hapiPropertyTable"><tbody><tr><td>Identifier</td><td>7c411bcd-c418-42d5-98d1-15aac35c7be0</td></tr></tbody></table></div>',
+ 'status': 'generated'}},
+ 'search': {'mode': 'match'}},
+ {'fullUrl': 'http://hapi.fhir.org/baseR4/Patient/2581369',
+ 'resource': {'id': '2581369',
+ 'identifier': [{'type': {'coding': [{'code': 'MR',
+ 'system': 'http://hl7.org/fhir/v2/0203'}]},
+ 'value': '9972a064-bebe-424c-9093-3c9a159ef810'}],
+ 'meta': {'lastUpdated': '2021-10-07T05:30:47.767+00:00',
+ 'source': '#Gv3zhMPhdfY3kmZV',
+ 'versionId': '1'},
+ 'name': [{'family': 'Cushing', 'given': ['Caleb']}],
+ 'resourceType': 'Patient',
+ 'text': {'div': '<div '
+ 'xmlns="http://www.w3.org/1999/xhtml"><div '
+ 'class="hapiHeaderText">Caleb '
+ '<b>CUSHING </b></div><table '
+ 'class="hapiPropertyTable"><tbody><tr><td>Identifier</td><td>9972a064-bebe-424c-9093-3c9a159ef810</td></tr></tbody></table></div>',
+ 'status': 'generated'}},
+ 'search': {'mode': 'match'}},
+ {'fullUrl': 'http://hapi.fhir.org/baseR4/Patient/2581377',
+ 'resource': {'id': '2581377',
+ 'identifier': [{'type': {'coding': [{'code': 'MR',
+ 'system': 'http://hl7.org/fhir/v2/0203'}]},
+ 'value': 'f5f32d1f-4cb7-46ca-aa9e-2cf0ceddcf2d'}],
+ 'meta': {'lastUpdated': '2021-10-07T05:30:50.334+00:00',
+ 'source': '#H1FwlSuIbTH8HoQO',
+ 'versionId': '1'},
+ 'name': [{'family': 'Cushing', 'given': ['Caleb']}],
+ 'resourceType': 'Patient',
+ 'text': {'div': '<div '
+ 'xmlns="http://www.w3.org/1999/xhtml"><div '
+ 'class="hapiHeaderText">Caleb '
+ '<b>CUSHING </b></div><table '
+ 'class="hapiPropertyTable"><tbody><tr><td>Identifier</td><td>f5f32d1f-4cb7-46ca-aa9e-2cf0ceddcf2d</td></tr></tbody></table></div>',
+ 'status': 'generated'}},
+ 'search': {'mode': 'match'}},
+ {'fullUrl': 'http://hapi.fhir.org/baseR4/Patient/2581385',
+ 'resource': {'id': '2581385',
+ 'identifier': [{'type': {'coding': [{'code': 'MR',
+ 'system': 'http://hl7.org/fhir/v2/0203'}]},
+ 'value': '6354a4f3-d14c-46fd-935b-061f12ae61cf'}],
+ 'meta': {'lastUpdated': '2021-10-07T05:30:52.897+00:00',
+ 'source': '#PY6zZl3CLaJBv1iE',
+ 'versionId': '1'},
+ 'name': [{'family': 'Cushing', 'given': ['Caleb']}],
+ 'resourceType': 'Patient',
+ 'text': {'div': '<div '
+ 'xmlns="http://www.w3.org/1999/xhtml"><div '
+ 'class="hapiHeaderText">Caleb '
+ '<b>CUSHING </b></div><table '
+ 'class="hapiPropertyTable"><tbody><tr><td>Identifier</td><td>6354a4f3-d14c-46fd-935b-061f12ae61cf</td></tr></tbody></table></div>',
+ 'status': 'generated'}},
+ 'search': {'mode': 'match'}},
+ {'fullUrl': 'http://hapi.fhir.org/baseR4/Patient/1710186',
+ 'resource': {'id': '1710186',
+ 'identifier': [{'type': {'coding': [{'code': 'MR',
+ 'system': 'http://hl7.org/fhir/v2/0203'}]},
+ 'value': '4221bcac-4dc9-4bc4-9998-c926228c82b6'}],
+ 'meta': {'lastUpdated': '2020-12-12T05:30:20.585+00:00',
+ 'source': '#MW9t13hB5Sot1wr8',
+ 'versionId': '1'},
+ 'name': [{'family': 'Cushing', 'given': ['Caleb']}],
+ 'resourceType': 'Patient',
+ 'text': {'div': '<div '
+ 'xmlns="http://www.w3.org/1999/xhtml"><div '
+ 'class="hapiHeaderText">Caleb '
+ '<b>CUSHING </b></div><table '
+ 'class="hapiPropertyTable"><tbody><tr><td>Identifier</td><td>4221bcac-4dc9-4bc4-9998-c926228c82b6</td></tr></tbody></table></div>',
+ 'status': 'generated'}},
+ 'search': {'mode': 'match'}},
+ {'fullUrl': 'http://hapi.fhir.org/baseR4/Patient/1710194',
+ 'resource': {'id': '1710194',
+ 'identifier': [{'type': {'coding': [{'code': 'MR',
+ 'system': 'http://hl7.org/fhir/v2/0203'}]},
+ 'value': '8c01f1ab-b1bd-49ed-8d2b-d05604d3b647'}],
+ 'meta': {'lastUpdated': '2020-12-12T05:30:25.227+00:00',
+ 'source': '#oFl3LmIy0jJBy2uW',
+ 'versionId': '1'},
+ 'name': [{'family': 'Cushing', 'given': ['Caleb']}],
+ 'resourceType': 'Patient',
+ 'text': {'div': '<div '
+ 'xmlns="http://www.w3.org/1999/xhtml"><div '
+ 'class="hapiHeaderText">Caleb '
+ '<b>CUSHING </b></div><table '
+ 'class="hapiPropertyTable"><tbody><tr><td>Identifier</td><td>8c01f1ab-b1bd-49ed-8d2b-d05604d3b647</td></tr></tbody></table></div>',
+ 'status': 'generated'}},
+ 'search': {'mode': 'match'}},
+ {'fullUrl': 'http://hapi.fhir.org/baseR4/Patient/1710202',
+ 'resource': {'id': '1710202',
+ 'identifier': [{'type': {'coding': [{'code': 'MR',
+ 'system': 'http://hl7.org/fhir/v2/0203'}]},
+ 'value': 'c7fdceca-5c46-43a3-8612-f05b653d8d50'}],
+ 'meta': {'lastUpdated': '2020-12-12T05:30:26.105+00:00',
+ 'source': '#PObX9KJkPSlqEaqq',
+ 'versionId': '1'},
+ 'name': [{'family': 'Cushing', 'given': ['Caleb']}],
+ 'resourceType': 'Patient',
+ 'text': {'div': '<div '
+ 'xmlns="http://www.w3.org/1999/xhtml"><div '
+ 'class="hapiHeaderText">Caleb '
+ '<b>CUSHING </b></div><table '
+ 'class="hapiPropertyTable"><tbody><tr><td>Identifier</td><td>c7fdceca-5c46-43a3-8612-f05b653d8d50</td></tr></tbody></table></div>',
+ 'status': 'generated'}},
+ 'search': {'mode': 'match'}},
+ {'fullUrl': 'http://hapi.fhir.org/baseR4/Patient/1710210',
+ 'resource': {'id': '1710210',
+ 'identifier': [{'type': {'coding': [{'code': 'MR',
+ 'system': 'http://hl7.org/fhir/v2/0203'}]},
+ 'value': '524ba1f0-4d0b-4156-a027-090271ad0b9d'}],
+ 'meta': {'lastUpdated': '2020-12-12T05:30:27.026+00:00',
+ 'source': '#QJBnM09DMDikgg58',
+ 'versionId': '1'},
+ 'name': [{'family': 'Cushing', 'given': ['Caleb']}],
+ 'resourceType': 'Patient',
+ 'text': {'div': '<div '
+ 'xmlns="http://www.w3.org/1999/xhtml"><div '
+ 'class="hapiHeaderText">Caleb '
+ '<b>CUSHING </b></div><table '
+ 'class="hapiPropertyTable"><tbody><tr><td>Identifier</td><td>524ba1f0-4d0b-4156-a027-090271ad0b9d</td></tr></tbody></table></div>',
+ 'status': 'generated'}},
+ 'search': {'mode': 'match'}},
+ {'fullUrl': 'http://hapi.fhir.org/baseR4/Patient/1710218',
+ 'resource': {'id': '1710218',
+ 'identifier': [{'type': {'coding': [{'code': 'MR',
+ 'system': 'http://hl7.org/fhir/v2/0203'}]},
+ 'value': '670ff5c0-1bb8-49b6-afa4-ba1ea116dfcd'}],
+ 'meta': {'lastUpdated': '2020-12-12T05:30:27.852+00:00',
+ 'source': '#fr8pH05YdV3GcXaX',
+ 'versionId': '1'},
+ 'name': [{'family': 'Cushing', 'given': ['Caleb']}],
+ 'resourceType': 'Patient',
+ 'text': {'div': '<div '
+ 'xmlns="http://www.w3.org/1999/xhtml"><div '
+ 'class="hapiHeaderText">Caleb '
+ '<b>CUSHING </b></div><table '
+ 'class="hapiPropertyTable"><tbody><tr><td>Identifier</td><td>670ff5c0-1bb8-49b6-afa4-ba1ea116dfcd</td></tr></tbody></table></div>',
+ 'status': 'generated'}},
+ 'search': {'mode': 'match'}},
+ {'fullUrl': 'http://hapi.fhir.org/baseR4/Patient/1710226',
+ 'resource': {'id': '1710226',
+ 'identifier': [{'type': {'coding': [{'code': 'MR',
+ 'system': 'http://hl7.org/fhir/v2/0203'}]},
+ 'value': '71083472-9e8e-4850-9e71-3e0ea9eb72bf'}],
+ 'meta': {'lastUpdated': '2020-12-12T05:30:28.868+00:00',
+ 'source': '#n8oLKEzzwtyWtIq0',
+ 'versionId': '1'},
+ 'name': [{'family': 'Cushing', 'given': ['Caleb']}],
+ 'resourceType': 'Patient',
+ 'text': {'div': '<div '
+ 'xmlns="http://www.w3.org/1999/xhtml"><div '
+ 'class="hapiHeaderText">Caleb '
+ '<b>CUSHING </b></div><table '
+ 'class="hapiPropertyTable"><tbody><tr><td>Identifier</td><td>71083472-9e8e-4850-9e71-3e0ea9eb72bf</td></tr></tbody></table></div>',
+ 'status': 'generated'}},
+ 'search': {'mode': 'match'}},
+ {'fullUrl': 'http://hapi.fhir.org/baseR4/Patient/1710238',
+ 'resource': {'id': '1710238',
+ 'identifier': [{'type': {'coding': [{'code': 'MR',
+ 'system': 'http://hl7.org/fhir/v2/0203'}]},
+ 'value': 'b0f4dc2f-62b2-480f-a9de-9330e20bcf73'}],
+ 'meta': {'lastUpdated': '2020-12-12T05:30:30.304+00:00',
+ 'source': '#qTLtPryVVRvxQWJx',
+ 'versionId': '1'},
+ 'name': [{'family': 'Cushing', 'given': ['Caleb']}],
+ 'resourceType': 'Patient',
+ 'text': {'div': '<div '
+ 'xmlns="http://www.w3.org/1999/xhtml"><div '
+ 'class="hapiHeaderText">Caleb '
+ '<b>CUSHING </b></div><table '
+ 'class="hapiPropertyTable"><tbody><tr><td>Identifier</td><td>b0f4dc2f-62b2-480f-a9de-9330e20bcf73</td></tr></tbody></table></div>',
+ 'status': 'generated'}},
+ 'search': {'mode': 'match'}},
+ {'fullUrl': 'http://hapi.fhir.org/baseR4/Patient/1567009',
+ 'resource': {'id': '1567009',
+ 'identifier': [{'type': {'coding': [{'code': 'MR',
+ 'system': 'http://hl7.org/fhir/v2/0203'}]},
+ 'value': '835855bd-e369-4dad-b29f-88b3eac35886'}],
+ 'meta': {'lastUpdated': '2020-10-13T05:30:33.394+00:00',
+ 'source': '#v4eEjiC3mU1IFuh1',
+ 'versionId': '1'},
+ 'name': [{'family': 'Cushing', 'given': ['Caleb']}],
+ 'resourceType': 'Patient',
+ 'text': {'div': '<div '
+ 'xmlns="http://www.w3.org/1999/xhtml"><div '
+ 'class="hapiHeaderText">Caleb '
+ '<b>CUSHING </b></div><table '
+ 'class="hapiPropertyTable"><tbody><tr><td>Identifier</td><td>835855bd-e369-4dad-b29f-88b3eac35886</td></tr></tbody></table></div>',
+ 'status': 'generated'}},
+ 'search': {'mode': 'match'}},
+ {'fullUrl': 'http://hapi.fhir.org/baseR4/Patient/1567021',
+ 'resource': {'id': '1567021',
+ 'identifier': [{'type': {'coding': [{'code': 'MR',
+ 'system': 'http://hl7.org/fhir/v2/0203'}]},
+ 'value': 'b769ce39-54b0-42cd-bfc1-b2bc00bc8503'}],
+ 'meta': {'lastUpdated': '2020-10-13T05:30:35.498+00:00',
+ 'source': '#PPFZF3sf3RtNCdg3',
+ 'versionId': '1'},
+ 'name': [{'family': 'Cushing', 'given': ['Caleb']}],
+ 'resourceType': 'Patient',
+ 'text': {'div': '<div '
+ 'xmlns="http://www.w3.org/1999/xhtml"><div '
+ 'class="hapiHeaderText">Caleb '
+ '<b>CUSHING </b></div><table '
+ 'class="hapiPropertyTable"><tbody><tr><td>Identifier</td><td>b769ce39-54b0-42cd-bfc1-b2bc00bc8503</td></tr></tbody></table></div>',
+ 'status': 'generated'}},
+ 'search': {'mode': 'match'}},
+ {'fullUrl': 'http://hapi.fhir.org/baseR4/Patient/1567029',
+ 'resource': {'id': '1567029',
+ 'identifier': [{'type': {'coding': [{'code': 'MR',
+ 'system': 'http://hl7.org/fhir/v2/0203'}]},
+ 'value': '67687225-1f93-49fc-a269-f33b1dea0fc2'}],
+ 'meta': {'lastUpdated': '2020-10-13T05:30:36.314+00:00',
+ 'source': '#YKCsK4p3FGH5uptc',
+ 'versionId': '1'},
+ 'name': [{'family': 'Cushing', 'given': ['Caleb']}],
+ 'resourceType': 'Patient',
+ 'text': {'div': '<div '
+ 'xmlns="http://www.w3.org/1999/xhtml"><div '
+ 'class="hapiHeaderText">Caleb '
+ '<b>CUSHING </b></div><table '
+ 'class="hapiPropertyTable"><tbody><tr><td>Identifier</td><td>67687225-1f93-49fc-a269-f33b1dea0fc2</td></tr></tbody></table></div>',
+ 'status': 'generated'}},
+ 'search': {'mode': 'match'}},
+ {'fullUrl': 'http://hapi.fhir.org/baseR4/Patient/1567037',
+ 'resource': {'id': '1567037',
+ 'identifier': [{'type': {'coding': [{'code': 'MR',
+ 'system': 'http://hl7.org/fhir/v2/0203'}]},
+ 'value': 'c345df12-ff3a-431a-b149-77c4e20d1e84'}],
+ 'meta': {'lastUpdated': '2020-10-13T05:30:40.008+00:00',
+ 'source': '#7xIEqCRuJyGrEyWX',
+ 'versionId': '1'},
+ 'name': [{'family': 'Cushing', 'given': ['Caleb']}],
+ 'resourceType': 'Patient',
+ 'text': {'div': '<div '
+ 'xmlns="http://www.w3.org/1999/xhtml"><div '
+ 'class="hapiHeaderText">Caleb '
+ '<b>CUSHING </b></div><table '
+ 'class="hapiPropertyTable"><tbody><tr><td>Identifier</td><td>c345df12-ff3a-431a-b149-77c4e20d1e84</td></tr></tbody></table></div>',
+ 'status': 'generated'}},
+ 'search': {'mode': 'match'}},
+ {'fullUrl': 'http://hapi.fhir.org/baseR4/Patient/1567045',
+ 'resource': {'id': '1567045',
+ 'identifier': [{'type': {'coding': [{'code': 'MR',
+ 'system': 'http://hl7.org/fhir/v2/0203'}]},
+ 'value': 'e4d1beaa-5034-4f52-9489-69c7a4af355d'}],
+ 'meta': {'lastUpdated': '2020-10-13T05:30:40.845+00:00',
+ 'source': '#1ig80EBKlB6QubTN',
+ 'versionId': '1'},
+ 'name': [{'family': 'Cushing', 'given': ['Caleb']}],
+ 'resourceType': 'Patient',
+ 'text': {'div': '<div '
+ 'xmlns="http://www.w3.org/1999/xhtml"><div '
+ 'class="hapiHeaderText">Caleb '
+ '<b>CUSHING </b></div><table '
+ 'class="hapiPropertyTable"><tbody><tr><td>Identifier</td><td>e4d1beaa-5034-4f52-9489-69c7a4af355d</td></tr></tbody></table></div>',
+ 'status': 'generated'}},
+ 'search': {'mode': 'match'}},
+ {'fullUrl': 'http://hapi.fhir.org/baseR4/Patient/1567053',
+ 'resource': {'id': '1567053',
+ 'identifier': [{'type': {'coding': [{'code': 'MR',
+ 'system': 'http://hl7.org/fhir/v2/0203'}]},
+ 'value': '02d67065-0107-4f99-b967-ee4396a037ba'}],
+ 'meta': {'lastUpdated': '2020-10-13T05:30:41.748+00:00',
+ 'source': '#Jrmnbq6HWRAMwvQ6',
+ 'versionId': '1'},
+ 'name': [{'family': 'Cushing', 'given': ['Caleb']}],
+ 'resourceType': 'Patient',
+ 'text': {'div': '<div '
+ 'xmlns="http://www.w3.org/1999/xhtml"><div '
+ 'class="hapiHeaderText">Caleb '
+ '<b>CUSHING </b></div><table '
+ 'class="hapiPropertyTable"><tbody><tr><td>Identifier</td><td>02d67065-0107-4f99-b967-ee4396a037ba</td></tr></tbody></table></div>',
+ 'status': 'generated'}},
+ 'search': {'mode': 'match'}}],
+ 'id': '80e8c67f-800d-4fce-8299-5b674b257a5b',
+ 'link': [{'relation': 'self',
+ 'url': 'http://hapi.fhir.org/baseR4/Patient?family=Cushing'},
+ {'relation': 'next',
+ 'url': 'http://hapi.fhir.org/baseR4?_getpages=80e8c67f-800d-4fce-8299-5b674b257a5b&_getpagesoffset=20&_count=20&_pretty=true&_bundletype=searchset'}],
+ 'meta': {'lastUpdated': '2021-10-21T14:34:50.512+00:00'},
+ 'resourceType': 'Bundle',
+ 'type': 'searchset'}
+The client contains data model classes, built using fhir-parser, that handle (de)serialization and allow to work with FHIR data in a Pythonic way.
import fhirclient.models.patient
+import fhirclient.models.humanname
+
+data = {'id': 'patient-1'}
+patient = fhirclient.models.patient.Patient(data)
+
+print(patient.id)
+patient-1
+name = fhirclient.models.humanname.HumanName()
+name.given = ['Peter']
+name.family = 'Parker'
+
+patient.name = [name]
+pprint(patient.as_json())
+{'id': 'patient-1',
+ 'name': [{'family': 'Parker', 'given': ['Peter']}],
+ 'resourceType': 'Patient'}
+name.given = 'Peter'
+patient.as_json() # throws FHIRValidationError: because we incorrectly set the name to a string
+---------------------------------------------------------------------------
+
+FHIRValidationError Traceback (most recent call last)
+
+/var/folders/jd/pq0swyt521jb2424d6fvth840000gn/T/ipykernel_93131/1295677727.py in <module>
+ 1 name.given = 'Peter'
+----> 2 patient.as_json() # throws FHIRValidationError: because we incorrectly set the name to a string
+
+
+~/opt/miniconda3/envs/cmu39/lib/python3.9/site-packages/fhirclient/models/fhirabstractresource.py in as_json(self)
+ 40
+ 41 def as_json(self):
+---> 42 js = super(FHIRAbstractResource, self).as_json()
+ 43 js['resourceType'] = self.resource_type
+ 44 return js
+
+
+~/opt/miniconda3/envs/cmu39/lib/python3.9/site-packages/fhirclient/models/fhirabstractbase.py in as_json(self)
+ 295
+ 296 if len(errs) > 0:
+--> 297 raise FHIRValidationError(errs)
+ 298 return js
+ 299
+
+
+FHIRValidationError: {root}:
+ name.0:
+ given:
+ Expecting property "given" on <class 'fhirclient.models.humanname.HumanName'> to be list, but is <class 'str'>
+import json
+import fhirclient.models.patient
+
+with open('patient.json', 'r') as h:
+ pjs = json.load(h)
+ patient = fhirclient.models.patient.Patient(pjs)
+ print(patient.name[0].family)
+ print(patient.name[0].given[0])
+ print(patient.gender)
+ print(patient.birthDate.isostring)
+Sky
+Luc
+male
+1980-01-01
+
+| Identifier | dbee78a3-a72d-41a2-b705-0023498f5228 |
| Identifier | dbee78a3-a72d-41a2-b705-0023498f5228 |
| Identifier | 2575e281-136c-489b-9b4e-4baba209258e |
| Identifier | 174ee1ec-c239-4b82-84a6-f0354aa6ee5f |
| Identifier | 7c411bcd-c418-42d5-98d1-15aac35c7be0 |
| Identifier | 9972a064-bebe-424c-9093-3c9a159ef810 |
| Identifier | f5f32d1f-4cb7-46ca-aa9e-2cf0ceddcf2d |
| Identifier | 6354a4f3-d14c-46fd-935b-061f12ae61cf |
| Identifier | 4221bcac-4dc9-4bc4-9998-c926228c82b6 |
| Identifier | 8c01f1ab-b1bd-49ed-8d2b-d05604d3b647 |
| Identifier | c7fdceca-5c46-43a3-8612-f05b653d8d50 |
| Identifier | 524ba1f0-4d0b-4156-a027-090271ad0b9d |
| Identifier | 670ff5c0-1bb8-49b6-afa4-ba1ea116dfcd |
| Identifier | 71083472-9e8e-4850-9e71-3e0ea9eb72bf |
| Identifier | b0f4dc2f-62b2-480f-a9de-9330e20bcf73 |
| Identifier | 835855bd-e369-4dad-b29f-88b3eac35886 |
| Identifier | b769ce39-54b0-42cd-bfc1-b2bc00bc8503 |
| Identifier | 67687225-1f93-49fc-a269-f33b1dea0fc2 |
| Identifier | c345df12-ff3a-431a-b149-77c4e20d1e84 |
| Identifier | e4d1beaa-5034-4f52-9489-69c7a4af355d |
| Identifier | 02d67065-0107-4f99-b967-ee4396a037ba |
Here we’ll walk through the different data files from the synthea data generator. Synthea can generate data in a number of formats including:
with open('../data/text/Abe604_Veum823_e841a5e8-9ace-437b-be32-b37d006aef87.txt') as f:
+ for idx, line in enumerate(f.readlines()):
+ print(f'{idx}: {line}')
+ if idx >= 5:
+ break
+0: Abe604 Veum823
+
+1: ==============
+
+2: Race: Asian
+
+3: Ethnicity: Non-Hispanic
+
+4: Gender: M
+
+5: Age: 69
+# find all lines
+with open('../data/csv/providers.csv') as f:
+ for idx, line in enumerate(f.readlines()):
+ print(line)
+ if idx >= 5:
+ break
+
+Id,ORGANIZATION,NAME,GENDER,SPECIALITY,ADDRESS,CITY,STATE,ZIP,UTILIZATION
+
+4f073dcc-c92a-455b-8b0c-be967da311b8,ef58ea08-d883-3957-8300-150554edc8fb,Noe500 Dibbert990,M,GENERAL PRACTICE,60 HOSPITAL ROAD,LEOMINSTER,MA,01453,362
+
+7066c8e7-c63a-4de5-a6ed-2fe78ba2d484,69176529-fd1f-3b3f-abce-a0a3626769eb,Mariam937 Gleason633,F,GENERAL PRACTICE,330 MOUNT AUBURN STREET,CAMBRIDGE,MA,02138,334
+
+2d6d2a74-e052-4546-8173-ac72a39b7365,5e765f2b-e908-3888-9fc7-df2cb87beb58,Dagny669 Schoen8,F,GENERAL PRACTICE,211 PARK STREET,ATTLEBORO,MA,02703,77
+
+66ab043d-06d1-4f21-b837-2d74448feea7,f1fbcbfb-fcfa-3bd2-b7f4-df20f1b3c3a4,Tyron580 Torphy630,M,GENERAL PRACTICE,ONE GENERAL STREET,LAWRENCE,MA,01842,359
+
+4e37e414-41b9-467f-be47-4293b6dea918,e002090d-4e92-300e-b41e-7d1f21dee4c6,Loren192 Fay398,M,GENERAL PRACTICE,1493 CAMBRIDGE STREET,CAMBRIDGE,MA,02138,7
+with open('../data/ccda/Abe604_Veum823_e841a5e8-9ace-437b-be32-b37d006aef87.xml') as f:
+ for idx, line in enumerate(f.readlines()):
+ print(f'{idx}: {line}')
+ if idx >= 5:
+ break
+0: <?xml version="1.0" encoding="UTF-8"?>
+
+1: <ClinicalDocument xmlns="urn:hl7-org:v3" xmlns:sdtc="urn:hl7-org:sdtc" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="urn:hl7-org:v3 http://xreg2.nist.gov:8080/hitspValidation/schema/cdar2c32/infrastructure/cda/C32_CDA.xsd">
+
+2: <realmCode code="US"/>
+
+3: <typeId root="2.16.840.1.113883.1.3" extension="POCD_HD000040"/>
+
+4: <templateId root="2.16.840.1.113883.10.20.22.1.1" extension="2015-08-01"/>
+
+5: <templateId root="2.16.840.1.113883.10.20.22.1.2" extension="2015-08-01"/>
+with open('../data/fhir/Abe604_Veum823_e841a5e8-9ace-437b-be32-b37d006aef87.json') as f:
+ for idx, line in enumerate(f.readlines()):
+ print(f'{idx}: {line}')
+ if idx >= 5:
+ break
+0: {
+
+1: "resourceType": "Bundle",
+
+2: "type": "transaction",
+
+3: "entry": [
+
+4: {
+
+5: "fullUrl": "urn:uuid:df5f01e0-810b-4379-be90-bf53a6b3563d",
+
+Python List Comprehension Tutorial
When doing data science, you might find yourself wanting to read lists of lists, filtering column names, removing vowels from a list or flattening a matrix. You can easily use a lambda function or a for loop; As you well know, there are multiple ways to go about this. One other way to do this is by using list comprehensions.
List comprehensions are used to create lists programatically. List comprehensions in Python are constructed as follows:
list_variable = [x for x in iterable]
+Remember in maths, the common ways to describe lists (or sets, or tuples, or vectors) are:
S = {x² : x in {0 ... 9}}
+V = (1, 2, 4, 8, ..., 2¹²)
+M = {x | x in S and x even}
+In other words, you’ll find that the above definitions actually tell you the following:
If the above definitions give you a headache, take a look at the actual lists that these definitions would produce:
S = {0, 1, 4, 9, 16, 25, 36, 49, 64, 81}
+V = {1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096}
+M = {0, 4, 16, 36, 64}
+# S = {x² : x in {0 ... 9}}
+S = []
+for x in range(10):
+ S.append(x**2)
+print(S)
+[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
+# S = {x² : x in {0 ... 9}}
+S = [x**2 for x in range(10)]
+print(S)
+[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
+The list S is built up with the square brackets that you read above in the first section. In those brackets, you see that there is an element x, which is raised to the power of 10. Now, you just need to know for how many values (and which values!) you need to raise to the power of 2. This is determined in range(10). Considering all of this, you can derive that you’ll raise all numbers, going from 0 to 9, to the power of 2.
# V = (1, 2, 4, 8, ..., 2¹²)
+V = []
+for i in range(13):
+ V.append(2**i)
+print(V)
+[1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096]
+# V = (1, 2, 4, 8, ..., 2¹²)
+V = [2**i for i in range(13)]
+print(V)
+[1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096]
+The list V contains the base value 2, which is raised to a certain power. Just like before, now you need to know which power or i is exactly going to be used to do this. You see that i in this case is part of range(13), which means that you start from 0 and go until 12. All of this means that your list is going to have 13 values - those values will be 2 raised to the power 0, 1, 2, … all the way up to 12.
# M = {x | x in S and x even}
+M = []
+for x in S:
+ if x % 2 == 0:
+ M.append(x)
+print(M)
+[0, 4, 16, 36, 64]
+# M = {x | x in S and x even}
+M = [x for x in S if x % 2 == 0]
+print(M)
+[0, 4, 16, 36, 64]
+Lastly, the list M contains elements that are part of S if -and only if- they can be divided by 2 without having any leftovers. The modulo needs to be 0. In other words, the list M is built up with the equal values that are stored in list S.
List comprehension is a complete substitute to for loops, lambda function as well as the functions map(), filter() and reduce().
List comprehensions are actually good alternatives to for loops, as they are more compact.
numbers = range(0, 10)
+
+# Initialize `new_list`
+new_list = []
+
+# Add values to `new_list`
+for n in numbers:
+ if n % 2 == 0:
+ new_list.append(n**2)
+
+# Print `new_list`
+print(new_list)
+[0, 4, 16, 36, 64]
+numbers = range(0, 10)
+
+# Create `new_list`
+new_list = [n**2 for n in numbers if n%2==0]
+
+# Print `new_list`
+print(new_list)
+[0, 4, 16, 36, 64]
+Let’s study the difference in performance between the list comprehension and the for loop with a small test: you can set this up very quickly with the timeit library, which you can use to time small bits of Python code in a simple way. In this case, the small pieces of code that you will test are the for loop, which you will put in a function called power_two() for your convenience, and the exact list comprehension which you have formulated above.
# Import `timeit`
+import timeit
+# Print the execution time
+print(timeit.timeit('[n**2 for n in range(10) if n%2==0]', number=10000))
+0.04181958400000063
+# Define `power_two()`
+def power_two(numbers):
+ for n in numbers:
+ if n%2==0:
+ new_list.append(n**2)
+ return new_list
+print(timeit.timeit('power_two(numbers)', globals=globals(), number=10000))
+0.044310624999999604
+Lambda functions are also called “anonymous functions” or “functions without name”. That means that you only use this type of functions when they are created. Lambda functions borrow their name from the lambda keyword in Python, which is used to declare these functions instead of the standard def keyword.
You usually use these functions together with the map(), filter(), and reduce() functions.
# Initialize the `kilometer` list
+kilometer = [39.2, 36.5, 37.3, 37.8]
+
+# Construct `feet` with `map()`
+feet = map(lambda x: 3280.8399*x, kilometer)
+
+# Print `feet` as a list
+print(type(feet), list(feet))
+<class 'map'> [128608.92408000001, 119750.65635, 122375.32826999998, 124015.74822]
+# Convert `kilometer` to `feet`
+feet = []
+for x in kilometer:
+ feet.append(3280.8399*x)
+print(feet)
+[128608.92408000001, 119750.65635, 122375.32826999998, 124015.74822]
+# Convert `kilometer` to `feet`
+feet = [3280.8399*x for x in kilometer]
+
+# Print `feet`
+print(feet)
+[128608.92408000001, 119750.65635, 122375.32826999998, 124015.74822]
+# Filter `feet` to only include uneven distances
+uneven = filter(lambda x: x % 2, feet)
+
+# Check the type of `uneven`
+type(uneven)
+
+# Print `uneven` as a list
+print(list(uneven))
+[128608.92408000001, 119750.65635, 122375.32826999998, 124015.74822]
+# Constructing `feet`
+feet = [int(x) for x in feet]
+
+# Print `feet`
+print(feet)
+
+# Get all uneven distances
+uneven = [True if x%2 else False for x in feet]
+
+# Print `uneven`
+print(uneven)
+[128608, 119750, 122375, 124015]
+[False, False, True, True]
+# Import `reduce` from `functools`
+from functools import reduce
+
+# Reduce `feet` to `reduced_feet`
+reduced_feet = reduce(lambda x,y: x+y, feet)
+
+# Print `reduced_feet`
+print(reduced_feet)
+494748
+# Construct `reduced_feet`
+reduced_feet = sum([float(3280.8399)*x for x in kilometer])
+
+# Print `reduced_feet`
+print(reduced_feet)
+494750.65692000004
+# Define `uneven`
+uneven = [int(x) for x in feet if x%2]
+
+# Print `uneven`
+print(uneven)
+[122375, 124015]
+divided = []
+
+for x in range(100):
+ if x%2 == 0 :
+ if x%6 == 0:
+ divided.append(x)
+
+print(divided)
+[0, 6, 12, 18, 24, 30, 36, 42, 48, 54, 60, 66, 72, 78, 84, 90, 96]
+divided = [x for x in range(100) if x % 2 == 0 if x % 6 == 0]
+
+print(divided)
+[0, 6, 12, 18, 24, 30, 36, 42, 48, 54, 60, 66, 72, 78, 84, 90, 96]
+values = []
+for x in feet:
+ if x >= 120000:
+ x + 1
+ else:
+ x + 5
+ values.append(x)
+
+print(values)
+[128608, 119750, 122375, 124015]
+values = [x+1 if x >= 120000 else x+5 for x in feet]
+print(values)
+[128609, 119755, 122376, 124016]
+Apart from conditionals, you can also adjust your list comprehensions by nesting them within other list comprehensions. This is handy when you want to work with lists of lists: generating lists of lists, transposing lists of lists or flattening lists of lists to regular lists, for example, becomes extremely easy with nested list comprehensions.
You see that most of the keywords and elements that are used in the example of the nested list comprehension are similar to the ones that you used in the simple list comprehension examples:
list_of_list = [[1,2,3],[4,5,6],[7,8]]
+print(list_of_list)
+print(list_of_list[0])
+print(list_of_list[1])
+print(list_of_list[1][1])
+print(list_of_list[2])
+[[1, 2, 3], [4, 5, 6], [7, 8]]
+[1, 2, 3]
+[4, 5, 6]
+5
+[7, 8]
+# Flatten `list_of_list`
+flat = []
+for x in list_of_list:
+ for y in x:
+ flat.append(y)
+print(flat)
+[1, 2, 3, 4, 5, 6, 7, 8]
+# Flatten `list_of_list`
+flat = [y for x in list_of_list for y in x]
+print(flat)
+[1, 2, 3, 4, 5, 6, 7, 8]
+Let’s now consider another example, where you see that you can also use two pairs of square brackets to change the logic of your nested list comprehension:
matrix = [[1,2,3], [4,5,6],[7,8,9]]
+
+[[row[i] for row in matrix] for i in range(len(matrix))]
+[[1, 4, 7], [2, 5, 8], [3, 6, 9]]
+Now practice: rewrite the code chunk above to a nested for loop. If you need some pointers on how to tackle this exercise, go to one of the previous sections of this tutorial.
transposed = []
+for i in range(len(matrix)):
+ transposed_row = []
+ for row in matrix:
+ transposed_row.append(row[i])
+ transposed.append(transposed_row)
+print(transposed)
+[[1, 4, 7], [2, 5, 8], [3, 6, 9]]
+You can also use nested list comprehensions when you need to create a list of lists that is actually a matrix. Check out the following example:
matrix = []
+for x in range(3):
+ nested = []
+ matrix.append(nested)
+ for row in range(4):
+ nested.append(0)
+print(matrix)
+[[0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0]]
+# rewritten as a list comprehension
+matrix = [[0 for col in range(4)] for row in range(3)]
+matrix
+[[0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0]]
+
+Python Sets Set Comprehensions
Perhaps you recall learning about sets and set theory at some point in your mathematical education. Maybe you even remember Venn diagrams:
In mathematics, a rigorous definition of a set can be abstract and difficult to grasp. Practically though, a set can be thought of simply as a well-defined collection of distinct objects, typically called elements or members.
Grouping objects into a set can be useful in programming as well, and Python provides a built-in set type to do so. Sets are distinguished from other object types by the unique operations that can be performed on them.
Here’s what you’ll learn in this tutorial:
Python’s built-in set type has the following characteristics:
A set can be created in two ways. First, you can define a set with the built-in set() function:
x = set(<iter>)
+x = set(['foo', 'bar', 'baz', 'foo', 'qux'])
+print(x)
+{'bar', 'qux', 'baz', 'foo'}
+y = set(('foo', 'bar', 'baz', 'foo', 'qux'))
+print(y)
+{'bar', 'qux', 'baz', 'foo'}
+z = set()
+print(type(z), z)
+<class 'set'> set()
+Strings are also iterable, so a string can be passed to set() as well. You have already seen that list(s) generates a list of the characters in the string s. Similarly, set(s) generates a set of the characters in s:
s = 'data focused python'
+print(s)
+data focused python
+for c in s:
+ print(c)
+d
+a
+t
+a
+
+f
+o
+c
+u
+s
+e
+d
+
+p
+y
+t
+h
+o
+n
+print(list(s))
+['d', 'a', 't', 'a', ' ', 'f', 'o', 'c', 'u', 's', 'e', 'd', ' ', 'p', 'y', 't', 'h', 'o', 'n']
+print(set(s))
+{'h', 'y', ' ', 'u', 'c', 'e', 'p', 'd', 'o', 't', 'a', 'n', 's', 'f'}
+s = 'data focused python is cool because we learn python and work with data'
+words = s.split(' ')
+print(words)
+print(set(words))
+['data', 'focused', 'python', 'is', 'cool', 'because', 'we', 'learn', 'python', 'and', 'work', 'with', 'data']
+{'because', 'python', 'we', 'work', 'is', 'with', 'data', 'focused', 'learn', 'cool', 'and'}
+You can see that the resulting sets are unordered: the original order, as specified in the definition, is not necessarily preserved. Additionally, duplicate values are only represented in the set once, as with the string ‘foo’ in the first two examples and the letter ‘u’ in the third.
Alternately, a set can be defined with curly braces ({}):
x = {<obj>, <obj>, ..., <obj>}
+When a set is defined this way, each
Thus, the sets shown above can also be defined like this:
x = { 'foo', 'bar', 'baz', 'foo', 'qux' }
+print(type(x), x)
+<class 'set'> {'bar', 'qux', 'baz', 'foo'}
+y = {'q', 'u', 'u', 'x'}
+print(type(y), y)
+<class 'set'> {'x', 'u', 'q'}
+To recap:
Observe the difference between these two set definitions:
x = {'foo'}
+print(type(x), x)
+<class 'set'> {'foo'}
+y = set('foo')
+print(type(y), y)
+<class 'set'> {'f', 'o'}
+A set can be empty. However, recall that Python interprets empty curly braces ({}) as an empty dictionary, so the only way to define an empty set is with the set() function:
x = {}
+print(type(x))
+<class 'dict'>
+y = set()
+print(type(y))
+<class 'set'>
+An empty set is falsy in Boolean context:
x = set()
+bool(x)
+False
+x or 1
+1
+x and 1
+set()
+You might think the most intuitive sets would contain similar objects—for example, even numbers or surnames:
s1 = {2, 4, 6, 8, 10}
+s2 = {'Smith', 'McArthur', 'Wilson', 'Johansson'}
+Python does not require this, though. The elements in a set can be objects of different types:
x = {42, 'foo', 3.14159, None}
+print(x)
+{42, 3.14159, 'foo', None}
+Don’t forget that set elements must be immutable. For example, a tuple may be included in a set:
x = {42, 'foo', (1, 2, 3), 3.14159}
+print(x)
+{42, 3.14159, 'foo', (1, 2, 3)}
+But lists and dictionaries are mutable, so they can’t be set elements:
a = [1, 2, 3]
+x = {a}
+---------------------------------------------------------------------------
+
+TypeError Traceback (most recent call last)
+
+/var/folders/jd/pq0swyt521jb2424d6fvth840000gn/T/ipykernel_93351/1659420868.py in <module>
+ 1 a = [1, 2, 3]
+----> 2 x = {a}
+
+
+TypeError: unhashable type: 'list'
+x = {'foo', 'bar', 'baz'}
+len(x)
+3
+'bar' in x
+True
+'qux' in x
+False
+Many of the operations that can be used for Python’s other composite data types don’t make sense for sets. For example, sets can’t be indexed or sliced. However, Python provides a whole host of operations on set objects that generally mimic the operations that are defined for mathematical sets.
Most, though not quite all, set operations in Python can be performed in two different ways: by operator or by method. Let’s take a look at how these operators and methods work, using set union as an example.
Given two sets, x1 and x2, the union of x1 and x2 is a set consisting of all elements in either set.
Consider these two sets:
x1 = {'foo', 'bar', 'baz'}
+x2 = {'baz', 'qux', 'quux'}
+The union of x1 and x2 is {'foo', 'bar', 'baz', 'qux', 'quux'}.
In Python, set union can be performed with the | operator:
union_x = x1 | x2
+print(union_x)
+{'baz', 'foo', 'bar', 'qux', 'quux'}
+# note : there aren't any duplicates
+print(len(union_x), len(x1), len(x2))
+5 3 3
+Set union can also be obtained with the .union() method. The method is invoked on one of the sets, and the other is passed as an argument:
x1.union(x2)
+{'bar', 'baz', 'foo', 'quux', 'qux'}
+| The way they are used in the examples above, the operator and method behave identically. But there is a subtle difference between them. When you use the | operator, both operands must be sets. The .union() method, on the other hand, will take any iterable as an argument, convert it to a set, and then perform the union. |
Observe the difference between these two statements:
x1 | ('baz', 'qux', 'quux')
+---------------------------------------------------------------------------
+
+TypeError Traceback (most recent call last)
+
+/var/folders/jd/pq0swyt521jb2424d6fvth840000gn/T/ipykernel_93351/1141666621.py in <module>
+----> 1 x1 | ('baz', 'qux', 'quux')
+
+
+TypeError: unsupported operand type(s) for |: 'set' and 'tuple'
+x1.union(('baz', 'qux', 'quux'))
+{'bar', 'baz', 'foo', 'quux', 'qux'}
+Below is a list of the set operations available in Python. Some are performed by operator, some by method, and some by both. The principle outlined above generally applies: where a set is expected, methods will typically accept any iterable as an argument, but operators require actual sets as operands.
x1 = {'foo', 'bar', 'baz'}
+x2 = {'baz', 'qux', 'quux'}
+
+print(x1.union(x2))
+print(x1 | x2)
+{'baz', 'foo', 'bar', 'qux', 'quux'}
+{'baz', 'foo', 'bar', 'qux', 'quux'}
+More than two sets may be specified with either the operator or the method:
a = {1, 2, 3, 4}
+b = {2, 3, 4, 5}
+c = {3, 4, 5, 6}
+d = {4, 5, 6, 7}
+
+print(a.union(b, c, d))
+print(a | b | c | d)
+{1, 2, 3, 4, 5, 6, 7}
+{1, 2, 3, 4, 5, 6, 7}
+
x1 = {'foo', 'bar', 'baz'}
+x2 = {'baz', 'qux', 'quux'}
+
+print(x1.intersection(x2))
+print(x1 & x2)
+{'baz'}
+{'baz'}
+You can specify multiple sets with the intersection method and operator, just like you can with set union:
a = {1, 2, 3, 4}
+b = {2, 3, 4, 5}
+c = {3, 4, 5, 6}
+d = {4, 5, 6, 7}
+
+print(a.intersection(b, c, d))
+print(a & b & c & d)
+{4}
+{4}
+
x1 = {'foo', 'bar', 'baz'}
+x2 = {'baz', 'qux', 'quux'}
+
+print(x1.difference(x2))
+print(x1 - x2)
+{'bar', 'foo'}
+{'bar', 'foo'}
+Once again, you can specify more than two sets:
a = {1, 2, 3, 30, 300}
+b = {10, 20, 30, 40}
+c = {100, 200, 300, 400}
+
+print(a.difference(b, c))
+print(a - b - c)
+{1, 2, 3}
+{1, 2, 3}
+When multiple sets are specified, the operation is performed from left to right. In the example above, a - b is computed first, resulting in {1, 2, 3, 300}. Then c is subtracted from that set, leaving {1, 2, 3}:
x1 = {'foo', 'bar', 'baz'}
+x2 = {'baz', 'qux', 'quux'}
+
+print(x1.symmetric_difference(x2))
+print(x1 ^ x2)
+{'foo', 'bar', 'qux', 'quux'}
+{'foo', 'bar', 'qux', 'quux'}
+The ^ operator also allows more than two sets:
a = {1, 2, 3, 4, 5}
+b = {10, 2, 3, 4, 50}
+c = {1, 50, 100}
+
+print(a ^ b ^ c)
+{100, 5, 10}
+Determines whether or not two sets have any elements in common.
x1.isdisjoint(x2) returns True if x1 and x2 have no elements in common:
x1 = {'foo', 'bar', 'baz'}
+x2 = {'baz', 'qux', 'quux'}
+
+print(x1.isdisjoint(x2))
+False
+x3 = x2 - {'baz'}
+
+print(x1.isdisjoint(x3))
+True
+If x1.isdisjoint(x2) is True, then x1 & x2 is the empty set:
x1 = {1, 3, 5}
+x2 = {2, 4, 6}
+
+print(x1.isdisjoint(x2))
+True
+print(x1 & x2)
+set()
+Determine whether one set is a subset of the other.
In set theory, a set x1 is considered a subset of another set x2 if every element of x1 is in x2.
x1.issubset(x2) and x1 <= x2 return True if x1 is a subset of x2:
x1 = {'foo', 'bar', 'baz'}
+print(x1.issubset({'foo', 'bar', 'baz', 'qux', 'quux'}))
+True
+x2 = {'baz', 'qux', 'quux'}
+print(x1 <= x2)
+False
+A proper subset is the same as a subset, except that the sets can’t be identical. A set x1 is considered a proper subset of another set x2 if every element of x1 is in x2, and x1 and x2 are not equal.
x1 = {'foo', 'bar'}
+x2 = {'foo', 'bar', 'baz'}
+print(x1 < x2)
+True
+x1 = {'foo', 'bar', 'baz'}
+x2 = {'foo', 'bar', 'baz'}
+print(x1 < x2)
+False
+A superset is the reverse of a subset. A set x1 is considered a superset of another set x2 if x1 contains every element of x2.
x1.issuperset(x2) and x1 >= x2 return True if x1 is a superset of x2:
x1 = {'foo', 'bar', 'baz'}
+print(x1.issuperset({'foo', 'bar'}))
+True
+x2 = {'baz', 'qux', 'quux'}
+print(x1 >= x2)
+False
+A proper superset is the same as a superset, except that the sets can’t be identical. A set x1 is considered a proper superset of another set x2 if x1 contains every element of x2, and x1 and x2 are not equal.
x1 = {'foo', 'bar', 'baz'}
+x2 = {'foo', 'bar'}
+print(x1 > x2)
+True
+x1 = {'foo', 'bar', 'baz'}
+x2 = {'foo', 'bar', 'baz'}
+print(x1 > x2)
+False
+Python provides another built-in type called a frozenset, which is in all respects exactly like a set, except that a frozenset is immutable. You can perform non-modifying operations on a frozenset:
x = frozenset(['foo', 'bar', 'baz'])
+print(x)
+frozenset({'bar', 'baz', 'foo'})
+print(len(x))
+3
+print(x & {'baz', 'qux', 'quux'})
+frozenset({'baz'})
+But methods that attempt to modify a frozenset fail:
x.add('quux')
+---------------------------------------------------------------------------
+
+AttributeError Traceback (most recent call last)
+
+/var/folders/jd/pq0swyt521jb2424d6fvth840000gn/T/ipykernel_93351/3158252394.py in <module>
+----> 1 x.add('quux')
+
+
+AttributeError: 'frozenset' object has no attribute 'add'
+import random
+from random import randint
+seed = 1234
+random.seed(seed)
+x = 0
+y = 5
+a = [randint(x, y) for i in range(0, 10)]
+print(a)
+[3, 0, 0, 0, 4, 0, 5, 5, 0, 0]
+random.seed(seed)
+x = 0
+y = 5
+b = {randint(x, y) for i in range(0, 10)}
+print(b)
+{0, 3, 4, 5}
+random.seed(seed)
+a = ['Even' if i % 2 else 'Odd' for i in range(10)]
+print(a)
+['Odd', 'Even', 'Odd', 'Even', 'Odd', 'Even', 'Odd', 'Even', 'Odd', 'Even']
+random.seed(seed)
+b = {'Even' if i % 2 else 'Odd' for i in range(10)}
+print(b)
+{'Even', 'Odd'}
+
+![](\"https://files.realpython.com/media/t.8b7abb515ae8.png\")
\n",
+ "\n",
+ "In mathematics, a rigorous definition of a set can be abstract and difficult to grasp. Practically though, a set can be thought of simply as a well-defined collection of distinct objects, typically called elements or members.\n",
+ "\n",
+ "Grouping objects into a set can be useful in programming as well, and Python provides a built-in set type to do so. Sets are distinguished from other object types by the unique operations that can be performed on them.\n",
+ "\n",
+ "Here’s what you’ll learn in this tutorial: \n",
+ "* You’ll see how to define set objects in Python and discover the operations that they support.\n",
+ "* You'll also learn about frozen sets, which are similar to sets except for one important detail."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Defining a Set\n",
+ "\n",
+ "Python’s built-in set type has the following characteristics:\n",
+ "* Sets are unordered.\n",
+ "* Set elements are unique. Duplicate elements are not allowed.\n",
+ "* A set itself may be modified, but the elements contained in the set must be of an immutable type.\n",
+ "\n",
+ "A set can be created in two ways. First, you can define a set with the built-in set() function:\n",
+ "\n",
+ "```python\n",
+ "x = set(![](\"https://files.realpython.com/media/t.ca57b915cec6.png\")
"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 33,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "{'baz', 'foo', 'bar', 'qux', 'quux'}\n",
+ "{'baz', 'foo', 'bar', 'qux', 'quux'}\n"
+ ]
+ }
+ ],
+ "source": [
+ "x1 = {'foo', 'bar', 'baz'}\n",
+ "x2 = {'baz', 'qux', 'quux'}\n",
+ "\n",
+ "print(x1.union(x2))\n",
+ "print(x1 | x2)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "More than two sets may be specified with either the operator or the method:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 34,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "{1, 2, 3, 4, 5, 6, 7}\n",
+ "{1, 2, 3, 4, 5, 6, 7}\n"
+ ]
+ }
+ ],
+ "source": [
+ "a = {1, 2, 3, 4}\n",
+ "b = {2, 3, 4, 5}\n",
+ "c = {3, 4, 5, 6}\n",
+ "d = {4, 5, 6, 7}\n",
+ "\n",
+ "print(a.union(b, c, d))\n",
+ "print(a | b | c | d)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "#### Intersection\n",
+ "\n",
+ "![](\"https://files.realpython.com/media/t.9c6d33717cdc.png\")
"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 35,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "{'baz'}\n",
+ "{'baz'}\n"
+ ]
+ }
+ ],
+ "source": [
+ "x1 = {'foo', 'bar', 'baz'}\n",
+ "x2 = {'baz', 'qux', 'quux'}\n",
+ "\n",
+ "print(x1.intersection(x2))\n",
+ "print(x1 & x2)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "You can specify multiple sets with the intersection method and operator, just like you can with set union:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 36,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "{4}\n",
+ "{4}\n"
+ ]
+ }
+ ],
+ "source": [
+ "a = {1, 2, 3, 4}\n",
+ "b = {2, 3, 4, 5}\n",
+ "c = {3, 4, 5, 6}\n",
+ "d = {4, 5, 6, 7}\n",
+ "\n",
+ "print(a.intersection(b, c, d))\n",
+ "print(a & b & c & d)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "#### Difference\n",
+ "\n",
+ "![](\"https://files.realpython.com/media/t.a90b4c323d99.png\")
"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 37,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "{'bar', 'foo'}\n",
+ "{'bar', 'foo'}\n"
+ ]
+ }
+ ],
+ "source": [
+ "x1 = {'foo', 'bar', 'baz'}\n",
+ "x2 = {'baz', 'qux', 'quux'}\n",
+ "\n",
+ "print(x1.difference(x2))\n",
+ "print(x1 - x2)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Once again, you can specify more than two sets:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 38,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "{1, 2, 3}\n",
+ "{1, 2, 3}\n"
+ ]
+ }
+ ],
+ "source": [
+ "a = {1, 2, 3, 30, 300}\n",
+ "b = {10, 20, 30, 40}\n",
+ "c = {100, 200, 300, 400}\n",
+ "\n",
+ "print(a.difference(b, c))\n",
+ "print(a - b - c)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "When multiple sets are specified, the operation is performed from left to right. In the example above, ```a - b``` is computed first, resulting in ```{1, 2, 3, 300}```. Then ```c``` is subtracted from that set, leaving ```{1, 2, 3}```:\n",
+ "\n",
+ "![](\"https://files.realpython.com/media/t.b37d6f78f99a.png\")
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "#### Symmetric Difference\n",
+ "\n",
+ "![](\"https://files.realpython.com/media/t.604de51646cc.png\")
"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 39,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "{'foo', 'bar', 'qux', 'quux'}\n",
+ "{'foo', 'bar', 'qux', 'quux'}\n"
+ ]
+ }
+ ],
+ "source": [
+ "x1 = {'foo', 'bar', 'baz'}\n",
+ "x2 = {'baz', 'qux', 'quux'}\n",
+ "\n",
+ "print(x1.symmetric_difference(x2))\n",
+ "print(x1 ^ x2)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "The ```^``` operator also allows more than two sets:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 40,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "{100, 5, 10}\n"
+ ]
+ }
+ ],
+ "source": [
+ "a = {1, 2, 3, 4, 5}\n",
+ "b = {10, 2, 3, 4, 50}\n",
+ "c = {1, 50, 100}\n",
+ "\n",
+ "print(a ^ b ^ c)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "#### Disjoint\n",
+ "\n",
+ "Determines whether or not two sets have any elements in common.\n",
+ "\n",
+ "```x1.isdisjoint(x2)``` returns ```True``` if ```x1``` and ```x2``` have no elements in common:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 41,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "False\n"
+ ]
+ }
+ ],
+ "source": [
+ "x1 = {'foo', 'bar', 'baz'}\n",
+ "x2 = {'baz', 'qux', 'quux'}\n",
+ "\n",
+ "print(x1.isdisjoint(x2))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 42,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "True\n"
+ ]
+ }
+ ],
+ "source": [
+ "x3 = x2 - {'baz'}\n",
+ "\n",
+ "print(x1.isdisjoint(x3))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "If ```x1.isdisjoint(x2)``` is ```True```, then ```x1 & x2``` is the empty set:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 43,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "True\n"
+ ]
+ }
+ ],
+ "source": [
+ "x1 = {1, 3, 5}\n",
+ "x2 = {2, 4, 6}\n",
+ "\n",
+ "print(x1.isdisjoint(x2))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 44,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "set()\n"
+ ]
+ }
+ ],
+ "source": [
+ "print(x1 & x2)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "#### Is Subset\n",
+ "\n",
+ "Determine whether one set is a subset of the other.\n",
+ "\n",
+ "In set theory, a set ```x1``` is considered a subset of another set ```x2``` if every element of ```x1``` is in ```x2```.\n",
+ "\n",
+ "```x1.issubset(x2)``` and ```x1 <= x2``` return ```True``` if ```x1``` is a subset of ```x2```:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 45,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "True\n"
+ ]
+ }
+ ],
+ "source": [
+ "x1 = {'foo', 'bar', 'baz'}\n",
+ "print(x1.issubset({'foo', 'bar', 'baz', 'qux', 'quux'}))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 46,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "False\n"
+ ]
+ }
+ ],
+ "source": [
+ "x2 = {'baz', 'qux', 'quux'}\n",
+ "print(x1 <= x2)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "#### Is Proper Subset\n",
+ "\n",
+ "A proper subset is the same as a subset, except that the sets can’t be identical. A set ```x1``` is considered a proper subset of another set ```x2``` if every element of ```x1``` is in ```x2```, and ```x1``` and ```x2``` are not equal."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 47,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "True\n"
+ ]
+ }
+ ],
+ "source": [
+ "x1 = {'foo', 'bar'}\n",
+ "x2 = {'foo', 'bar', 'baz'}\n",
+ "print(x1 < x2)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 48,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "False\n"
+ ]
+ }
+ ],
+ "source": [
+ "x1 = {'foo', 'bar', 'baz'}\n",
+ "x2 = {'foo', 'bar', 'baz'}\n",
+ "print(x1 < x2)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "#### Is Superset\n",
+ "\n",
+ "A superset is the reverse of a subset. A set ```x1``` is considered a superset of another set ```x2``` if ```x1``` contains every element of ```x2```.\n",
+ "\n",
+ "```x1.issuperset(x2)``` and ```x1 >= x2``` return True if ```x1``` is a superset of ```x2```:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 49,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "True\n"
+ ]
+ }
+ ],
+ "source": [
+ "x1 = {'foo', 'bar', 'baz'}\n",
+ "print(x1.issuperset({'foo', 'bar'}))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 50,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "False\n"
+ ]
+ }
+ ],
+ "source": [
+ "x2 = {'baz', 'qux', 'quux'}\n",
+ "print(x1 >= x2)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "#### Is Proper Superset\n",
+ "\n",
+ "A proper superset is the same as a superset, except that the sets can’t be identical. A set ```x1``` is considered a proper superset of another set ```x2``` if ```x1``` contains every element of ```x2```, and ```x1``` and ```x2``` are not equal."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 51,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "True\n"
+ ]
+ }
+ ],
+ "source": [
+ "x1 = {'foo', 'bar', 'baz'}\n",
+ "x2 = {'foo', 'bar'}\n",
+ "print(x1 > x2)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 52,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "False\n"
+ ]
+ }
+ ],
+ "source": [
+ "x1 = {'foo', 'bar', 'baz'}\n",
+ "x2 = {'foo', 'bar', 'baz'}\n",
+ "print(x1 > x2)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Frozen Sets\n",
+ "\n",
+ "Python provides another built-in type called a frozenset, which is in all respects exactly like a set, except that a frozenset is immutable. You can perform non-modifying operations on a frozenset:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 53,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "frozenset({'bar', 'baz', 'foo'})\n"
+ ]
+ }
+ ],
+ "source": [
+ "x = frozenset(['foo', 'bar', 'baz'])\n",
+ "print(x)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 54,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "3\n"
+ ]
+ }
+ ],
+ "source": [
+ "print(len(x))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 55,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "frozenset({'baz'})\n"
+ ]
+ }
+ ],
+ "source": [
+ "print(x & {'baz', 'qux', 'quux'})"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "But methods that attempt to modify a frozenset fail:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 56,
+ "metadata": {},
+ "outputs": [
+ {
+ "ename": "AttributeError",
+ "evalue": "'frozenset' object has no attribute 'add'",
+ "output_type": "error",
+ "traceback": [
+ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
+ "\u001b[0;31mAttributeError\u001b[0m Traceback (most recent call last)",
+ "\u001b[0;32m/var/folders/jd/pq0swyt521jb2424d6fvth840000gn/T/ipykernel_93351/3158252394.py\u001b[0m in \u001b[0;36mA dictionary in Python is a collection of items accessed by a specific key rather than by index. What does this mean?
Imagine a dictionary in the real world… when you need to look up the meaning of a word, you try to find the meaning using the word itself and not the possible index of the word. Python dictionaries work with the same concept, the word whose meaning you are looking for is the key and the meaning of the word is the value, you do not need to know the index of the word in a dictionary to find its meaning.
Note: The keys in a dictionary have to be hashable.
Hashing is the process of running an item through a specific kind of function. This function is called a “hash function”. This hash function returns a unique output for a unique input value. Integers, floating point numbers, strings, tuples, and frozensets are hashable. While lists, dictionaries, and sets other than frozensets are not. Hashing is a somewhat complex topic and this is only the basic concept behind hashing.
You can initialize a dictionary in Python a variety of ways:
# Creating an empty dict using empty brackets
+word_frequency = {}
+
+# Creating an empty dict using dict()
+word_frequency = dict()
+
+print(word_frequency)
+{}
+# Creating Dictionaries with literals
+word_frequency = {
+"Hello" : 7,
+"hi" : 10,
+"there" : 45,
+"at" : 23,
+"this" : 77
+}
+print(word_frequency)
+{'Hello': 7, 'hi': 10, 'there': 45, 'at': 23, 'this': 77}
+# Creating Dictionaries by passing parametrs in dict constructor
+word_frequency = dict(Hello = 7,
+ hi = 10,
+ there = 45,
+ at = 23,
+ this = 77)
+print(word_frequency)
+{'Hello': 7, 'hi': 10, 'there': 45, 'at': 23, 'this': 77}
+# Creating dictionaries with a List of tuples
+list_of_tuples = [("Hello" , 7), ("hi" , 10), ("there" , 45),("at" , 23),("this" , 77)]
+word_frequency = dict(list_of_tuples)
+print(word_frequency)
+{'Hello': 7, 'hi': 10, 'there': 45, 'at': 23, 'this': 77}
+# create and Initialize a dictionary by this list elements as keys and with same value 0
+list_of_strings = ["Hello", "hi", "there", "at", "this"]
+word_frequency = dict.fromkeys(list_of_strings,0 )
+print(word_frequency)
+{'Hello': 0, 'hi': 0, 'there': 0, 'at': 0, 'this': 0}
+# Creating a Dictionary by a two lists
+list_of_strings = ["Hello", "hi", "there", "at", "this"]
+list_of_ints = [7, 10, 45, 23, 77, 99, 100, 101]
+# Merge the two lists to create a dictionary
+word_frequency = dict(zip(list_of_strings, list_of_ints))
+print(word_frequency)
+{'Hello': 7, 'hi': 10, 'there': 45, 'at': 23, 'this': 77}
+You can then access the values by specifying the dictionary key
# or this way
+a = {'apple': 'fruit', 'beetroot': 'vegetable', 'cake': 'dessert'}
+print(a['apple'])
+fruit
+The items in a dictionary can have any data type. Check out some more examples of a dictionary to get a hang of it:
a = {'one': 1, 'two': 'to', 'three': 3.0, 'four': [4,4.0]}
+print(a)
+{'one': 1, 'two': 'to', 'three': 3.0, 'four': [4, 4.0]}
+# Update a dictionary
+a['one'] = 1.0
+print(a)
+{'one': 1.0, 'two': 'to', 'three': 3.0, 'four': [4, 4.0]}
+# Delete a single element
+del a['one']
+print(a)
+{'two': 'to', 'three': 3.0, 'four': [4, 4.0]}
+# Delete all elements in the dictionary
+a.clear()
+print(a)
+{}
+# Delete the dictionary
+del a
+print(a)
+---------------------------------------------------------------------------
+
+NameError Traceback (most recent call last)
+
+/var/folders/jd/pq0swyt521jb2424d6fvth840000gn/T/ipykernel_93358/2899747350.py in <module>
+ 1 # Delete the dictionary
+ 2 del a
+----> 3 print(a)
+
+
+NameError: name 'a' is not defined
+It is important to remember is that a key has to be unique in a dictionary, no duplicates are allowed. However, in case of duplicate keys rather than giving an error, Python will take the last instance of the key to be valid and simply ignore the first key-value pair.
sweet_dict = {'a1': 'cake', 'a2':'cookie', 'a1': 'icecream'}
+print(sweet_dict)
+print(sweet_dict['a1'])
+{'a1': 'icecream', 'a2': 'cookie'}
+icecream
+
+Python Dictionary Comprehension Tutorial
You’ll learn:
Dictionary comprehension is a method for transforming one dictionary into another dictionary. During this transformation, items within the original dictionary can be conditionally included in the new dictionary and each item can be transformed as needed.
A good list comprehension can make your code more expressive and thus, easier to read. The key with creating comprehensions is to not let them get so complex that your head spins when you try to decipher what they are actually doing. Keeping the idea of “easy to read” alive.
The way to do dictionary comprehension in Python is to be able to access the key objects and the value objects of a dictionary.
dict1 = {'a': 1, 'b': 2, 'c': 3, 'd': 4}
+# Put all keys of `dict1` in a list and returns the list
+keys = dict1.keys()
+print(type(keys), keys)
+<class 'dict_keys'> dict_keys(['a', 'b', 'c', 'd'])
+# Put all values saved in `dict1` in a list and returns the list
+vals = dict1.values()
+print(type(vals), vals)
+<class 'dict_values'> dict_values([1, 2, 3, 4])
+z = { 'x': 1, 'y': None, None: 'a'}
+print(z)
+print(type(None))
+{'x': 1, 'y': None, None: 'a'}
+<class 'NoneType'>
+So, now that you know how to access all the keys and their values in a dictionary. You can also access each key-value pair within a dictionary using the items() method:
items = dict1.items()
+print(type(items), items)
+<class 'dict_items'> dict_items([('a', 1), ('b', 2), ('c', 3), ('d', 4)])
+dict1 = {'a': 1, 'b': 2, 'c': 3, 'd': 4, 'e': 5}
+This is the general template you can follow for dictionary comprehension. This can serve as the basic and the most simple template which can get more and more complex as you add conditionalities to it.
dict_variable = {key:value for (key,value) in dictonary.items()}
+# Double each value in the dictionary
+double_dict1 = {}
+for (k,v) in dict1.items():
+ double_dict1[k] = v*2
+print(double_dict1)
+{'a': 2, 'b': 4, 'c': 6, 'd': 8, 'e': 10}
+# Double each value in the dictionary
+double_dict1 = {k:v*2 for (k,v) in dict1.items()}
+print(double_dict1)
+{'a': 2, 'b': 4, 'c': 6, 'd': 8, 'e': 10}
+# Double each value in the dictionary
+double_dict1 = {x[0]:x[1]*2 for x in dict1.items()}
+print(double_dict1)
+{'a': 2, 'b': 4, 'c': 6, 'd': 8, 'e': 10}
+double_dict1 = {k:dict1[k]*2 for k in dict1.keys()}
+print(double_dict1)
+{'a': 2, 'b': 4, 'c': 6, 'd': 8, 'e': 10}
+You can also make changes to the key values. For example, let’s create the same dictionary as above but also change the names of the key.
dict1_keys = {k*4:v for (k,v) in dict1.items()}
+print(dict1_keys)
+{'aaaa': 1, 'bbbb': 2, 'cccc': 3, 'dddd': 4, 'eeee': 5}
+Dictionary comprehension is a powerful concept and can be used to substitute for loops and lambda functions. However, not all for loop can be written as a dictionary comprehension but all dictionary comprehension can be written with a for loop.
Consider the following problem, where you want to create a new dictionary where the key is a number divisible by 2 in a range of 0-10 and it’s value is the square of the number.
For loops are used to repeat a certain operation or a block of instructions in a program for a given number of times. However, nested for loops (for loop inside another for loop) can get confusing and complex. Dictionary comprehensions are better in such situations and can simplify the readability and your understanding of the code.
Let’s see how the same probem can be solved using a for loop and dictionary comprehension:
numbers = range(10)
+new_dict_for = {}
+
+# Add values to `new_dict` using for loop
+for n in numbers:
+ if n % 2 == 0:
+ new_dict_for[n] = n**2
+
+print(new_dict_for)
+{0: 0, 2: 4, 4: 16, 6: 36, 8: 64}
+# Use dictionary comprehension
+new_dict_comp = {n:n**2 for n in numbers if n % 2 == 0}
+
+print(new_dict_comp)
+{0: 0, 2: 4, 4: 16, 6: 36, 8: 64}
+Lambda functions are a way of creating small anonymous functions. They are functions without a name. These functions are throw-away functions, which are only needed where they have been created. Lambda functions are mainly used in combination with the functions filter(), map() and reduce().
Let’s look at the lambda function along with the map() function:
# Initialize `fahrenheit` dictionary
+fahrenheit = {'t1':-30, 't2':-20, 't3':-10, 't4':0}
+
+#Get the corresponding `celsius` values
+celsius = map(lambda x: (float(5)/9)*(x-32), fahrenheit.values())
+celsius = list(celsius)
+
+#Create the `celsius` dictionary
+celsius_dict = dict(zip(fahrenheit.keys(), celsius))
+
+print(celsius_dict)
+{'t1': -34.44444444444444, 't2': -28.88888888888889, 't3': -23.333333333333336, 't4': -17.77777777777778}
+Now, let’s try to solve the same problem using dictionary comprehension:
# Initialize the `fahrenheit` dictionary
+fahrenheit = {'t1': -30,'t2': -20,'t3': -10,'t4': 0}
+
+# Get the corresponding `celsius` values and create the new dictionary
+celsius = {k:(float(5)/9)*(v-32) for (k,v) in fahrenheit.items()}
+
+print(celsius_dict)
+{'t1': -34.44444444444444, 't2': -28.88888888888889, 't3': -23.333333333333336, 't4': -17.77777777777778}
+You often need to add conditions to a solution while tackling problems. Let’s explore how you can add conditionals into dictionary comprehension to make it more powerful.
Let’s suppose you need to create a new dictionary from a given dictionary but with items that are greater than 2. This means that you need to add a condition to the original template you saw above…
dict1 = {'a': 1, 'b': 2, 'c': 3, 'd': 4, 'e': 5}
+
+# Check for items greater than 2
+dict1_cond = {k:v for (k,v) in dict1.items() if v > 2}
+
+print(dict1_cond)
+{'c': 3, 'd': 4, 'e': 5}
+In the problem above, what if you have to not only get the items greater than 2 but also need to check if they are multiples of 2 at the same time.
dict1_doubleCond = {k:v for (k,v) in dict1.items() if v>2 if v%2 == 0}
+
+print(dict1_doubleCond)
+{'d': 4}
+dict1_doubleCond = {k:v for (k,v) in dict1.items() if v>2 and v%2 == 0}
+
+print(dict1_doubleCond)
+{'d': 4}
+dict1_doubleCond = {k:v for (k,v) in dict1.items() if v>2 or v%2 == 0}
+
+print(dict1_doubleCond)
+{'b': 2, 'c': 3, 'd': 4, 'e': 5}
+The solution to adding multiple conditionals is as easy as simply adding the conditions one after another in your comprehension. However, you need to be careful about what you are trying to do in the problem. Remember, that the consecutive if statements work as if they had and clauses between them.
Lets see one more example with three conditionals:
dict1 = {'a': 1, 'b': 2, 'c': 3, 'd': 4, 'e': 5, 'f':6}
+
+dict1_tripleCond = {k:v for (k,v) in dict1.items() if v>2 if v%2 == 0 if v%3 == 0}
+
+print(dict1_tripleCond)
+{'f': 6}
+In a for loop, this will correspond to:
dict1_tripleCond = {}
+
+for (k,v) in dict1.items():
+ if (v>=2 and v%2 == 0 and v%3 == 0):
+ dict1_tripleCond[k] = v
+
+print(dict1_tripleCond)
+{'f': 6}
+dict1_tripleCond = {}
+
+for (k,v) in dict1.items():
+ if v>=2:
+ if v%2 == 0:
+ if v%3 == 0:
+ dict1_tripleCond[k] = v
+
+print(dict1_tripleCond)
+{'f': 6}
+Dealing with an if-else condition is also easy with dictionary comprehension. Check out the following example to see it for yourself:
dict1 = {'a': 1, 'b': 2, 'c': 3, 'd': 4, 'e': 5, 'f':6}
+
+# Identify odd and even entries
+dict1_tripleCond = {k:('even' if v%2==0 else 'odd') for (k,v) in dict1.items()}
+
+print(dict1_tripleCond)
+{'a': 'odd', 'b': 'even', 'c': 'odd', 'd': 'even', 'e': 'odd', 'f': 'even'}
+Nesting is a programming concept where data is organized in layers, or where objects contain other similar objects. You must have often seen a nested ‘if’ structure, which is an if condition inside another if condition.
Similarly, dictionaries can be nested and thus their comprehensions can be nested as well. Let’s see what this means:
nested_dict = {'first':{'a':1}, 'second':{'b':2}}
+float_dict = { outer_k: {inner_k:float(inner_v) for (inner_k, inner_v) in outer_v.items()}
+ for (outer_k, outer_v) in nested_dict.items() }
+
+print(float_dict)
+{'first': {'a': 1.0}, 'second': {'b': 2.0}}
+This is an example of a nested dictionary. The nested_dict is a dictionary with the keys: first and second, which hold dictionary objects in their values. The code works with the inner dictionary values and converts them to float and then combines the outer keys with the new float inner values into a new dictionary.
The code also has a nested dictionary comprehension, which is dictionary comprehension inside another one. The dictionary comprehension when nested as you can see can get pretty hard to read as well as understand, which takes away the whole point of using comprehensions in the first place. As the structure of the dictionary you are working with gets complicated, the dictionary comprehension starts to get complicated as well. For such situations, you might be better off not using complicated comprehensions in your code.
Note that you can rewrite the above code chunk also with a nested for loop:
nested_dict = {'first':{'a':1}, 'second':{'b':2}}
+
+for (outer_k, outer_v) in nested_dict.items():
+ for (inner_k, inner_v) in outer_v.items():
+ outer_v.update({inner_k: float(inner_v)})
+nested_dict.update({outer_k:outer_v})
+
+print(nested_dict)
+{'first': {'a': 1.0}, 'second': {'b': 2.0}}
+
+todo : add content
def infinite_sequence():
+ num = 0
+ while True:
+ num += 1
+ return num
+x = infinite_sequence()
+print(x)
+---------------------------------------------------------------------------
+
+KeyboardInterrupt Traceback (most recent call last)
+
+/var/folders/jd/pq0swyt521jb2424d6fvth840000gn/T/ipykernel_93378/1323376258.py in <module>
+----> 1 x = infinite_sequence()
+ 2 print(x)
+
+
+/var/folders/jd/pq0swyt521jb2424d6fvth840000gn/T/ipykernel_93378/3775151451.py in infinite_sequence()
+ 2 num = 0
+ 3 while True:
+----> 4 num += 1
+ 5 return num
+
+
+KeyboardInterrupt:
+def infinite_sequence():
+ num = 0
+ while True:
+ yield num
+ num += 1
+seq = infinite_sequence()
+print(next(seq))
+for i in range(0, 10):
+ print(next(seq))
+for index, value in enumerate(seq):
+ print(value)
+ if index > 10:
+ break
+print(next(seq))
+import random
+def my_sequence():
+ num = 0
+ while True:
+ yield num
+ num += random.randint(0, 10)
+ if num > 20:
+ break
+seq = my_sequence()
+print(next(seq))
+print(next(seq))
+print(next(seq))
+print(next(seq))
+12
+print(next(seq))
+14
+print(next(seq))
+20
+print(next(seq))
+---------------------------------------------------------------------------
+
+StopIteration Traceback (most recent call last)
+
+<ipython-input-20-d9775cae3656> in <module>
+----> 1 print(next(seq))
+
+
+StopIteration:
+seq2 = my_sequence()
+print(next(seq2))
+0
+image_db = [ 1, 2, 5, 7, 10]
+meds_db = [ 3, 5, 7, 9 ]
+
+i = get_next_image_patient()
+m = get_next_med_patient()
+
+def get_next_patient()
+ while no_match:
+ if i == m:
+ yield i
+ elif i > m:
+ i = get_next_image_patient()
+ elif m < i:
+ m = get_next_med_patient()
+
+for patient in get_next_patient():
+ # do something
+Object-Oriented Programming (OOP) in Python
Focusing first on the data, each thing or object is an instance of some class.
Classes are used to create new user-defined data structures that contain arbitrary information about something. In the case of an person, we could create a Person() class to track properties about the Person like the name and age.
It’s important to note that a class just provides structure—it’s a blueprint for how something should be defined, but it doesn’t actually provide any real content itself. The Person() class may specify that the name and age are necessary for defining an person, but it will not actually state what a specific person’s name or age is.
It may help to think of a class as an idea for how something should be defined.
While the class is the blueprint, an instance is a copy of the class with actual values, literally an object belonging to a specific class. It’s not an idea anymore; it’s an actual person, like a person named Bill who’s twenty years old.
Put another way, a class is like a form or questionnaire. It defines the needed information. After you fill out the form, your specific copy is an instance of the class; it contains actual information relevant to you.
You can fill out multiple copies to create many different instances, but without the form as a guide, you would be lost, not knowing what information is required. Thus, before you can create individual instances of an object, we must first specify what is needed by defining a class.
class Person:
+ pass
+All classes create objects, and all objects contain characteristics called attributes (referred to as properties in the opening paragraph). Use the __init__() method to initialize (e.g., specify) an object’s initial attributes by giving them their default value (or state). This method must have at least one argument as well as the self variable, which refers to the object itself (e.g., Person).
class Person:
+ # Initializer / Instance Attributes
+ def __init__(self, name, age):
+ self.name = name
+ self.age = age
+In the case of our Person() class:
self variable is also an instance of the class. Since instances of a class have varying values we could state Person.name = name rather than self.name = name. But since not all persons share the same name, we need to be able to assign different values to different instances. Hence the need for the special self variable, which will help to keep track of individual instances of each class.Note: You will never have to call the __init__() method; it gets called automatically when you create a new ‘Person’ instance.
While instance attributes are specific to each object, class attributes are the same for all instances—which in this case is all persons. So while each person has a unique name and age, every person is a homo sapien.
class Person:
+
+ # Class Attribute
+ species = 'homo sapien'
+
+ # Initializer / Instance Attributes
+ def __init__(self, name, age):
+ self.name = name
+ self.age = age
+Instantiating is a fancy term for creating a new, unique instance of a class.
class Animal:
+ pass
+a = Animal()
+print('a is type', type(a), 'value', a)
+
+b = Animal()
+print('b is type', type(b), 'value', b)
+
+c = b
+a is type <class '__main__.Animal'> value <__main__.Animal object at 0x7ff4b06b02b0>
+b is type <class '__main__.Animal'> value <__main__.Animal object at 0x7ff4b06b0a30>
+a == b
+False
+print(a == c)
+print(b == c)
+False
+True
+# Instantiate the Person object
+ben = Person("Ben", 37)
+geno = Person("Geno", 32)
+
+# Access the instance attributes
+print(f"{ben.name} is {ben.age} and {geno.name} is {geno.age}.")
+
+# Is Ben a mammal?
+if ben.species == "homo sapien":
+ print(f"{ben.name} is a {ben.species}!")
+Ben is 37 and Geno is 32.
+Ben is a homo sapien!
+Instance methods are defined inside a class and are used to get the contents of an instance. They can also be used to perform operations with the attributes of our objects. Like the __init__ method, the first argument is always self:
class Dog:
+ # Class Attribute
+ species = 'mammal'
+
+ # Initializer / Instance Attributes
+ def __init__(self, name, age):
+ self.name = name
+ self.age = age
+
+ # instance method
+ def description(self):
+ return "{} is {} years old".format(self.name, self.age)
+
+ # instance method
+ def speak(self, sound):
+ return "{} says {}".format(self.name, sound)
+# Instantiate the Dog object
+mikey = Dog("Mikey", 6)
+# call our instance methods
+print(mikey.description())
+print(mikey.speak("Gruff Gruff"))
+Mikey is 6 years old
+Mikey says Gruff Gruff
+You can change the value of attributes based on some behavior:
class Email:
+ def __init__(self):
+ self.is_sent = False
+
+ def send_email(self):
+ self.is_sent = True
+my_email = Email()
+my_email.is_sent
+False
+my_email.send_email()
+my_email.is_sent
+True
+Inheritance is the process by which one class takes on the attributes and methods of another. Newly formed classes are called child classes, and the classes that child classes are derived from are called parent classes.
It’s important to note that child classes override or extend the functionality (e.g., attributes and behaviors) of parent classes. In other words, child classes inherit all of the parent’s attributes and behaviors but can also specify different behavior to follow. The most basic type of class is an object, which generally all other classes inherit as their parent.
Let’s pretend that we’re at a dog park. There are multiple Dog objects engaging in Dog behaviors, each with different attributes. In regular-speak that means some dogs are running, while some are stretching and some are just watching other dogs. Furthermore, each dog has been named by its owner and, since each dog is living and breathing, each ages.
class Dog:
+ def __init__(self, breed):
+ self.breed = breed
+spencer = Dog("German Shepard")
+spencer.breed
+'German Shepard'
+sara = Dog("Boston Terrier")
+sara.breed
+'Boston Terrier'
+# Parent class
+class Dog:
+
+ # Class attribute
+ species = 'mammal'
+
+ # Initializer / Instance attributes
+ def __init__(self, name, age):
+ self.name = name
+ self.age = age
+
+ # instance method
+ def description(self):
+ return "{} is {} years old".format(self.name, self.age)
+
+ # instance method
+ def speak(self, sound):
+ return "{} says {}".format(self.name, sound)
+# Child class (inherits from Dog class)
+class RussellTerrier(Dog):
+ def run(self, speed):
+ return "{} runs {}".format(self.name, speed)
+# Child class (inherits from Dog class)
+class Bulldog(Dog):
+ def run(self, speed):
+ return "{} runs {}".format(self.name, speed)
+# Child classes inherit attributes and
+# behaviors from the parent class
+jim = Bulldog("Jim", 12)
+print(jim.description())
+Jim is 12 years old
+# Child classes have specific attributes
+# and behaviors as well
+print(jim.run("slowly"))
+Jim runs slowly
+The isinstance() function is used to determine if an instance is also an instance of a certain parent class.
# Child classes inherit attributes and
+# behaviors from the parent class
+jim = Bulldog("Jim", 12)
+
+# Is jim an instance of Dog()?
+print(isinstance(jim, Dog))
+True
+print(isinstance(jim, Bulldog))
+True
+print(isinstance(jim, RussellTerrier))
+False
+Remember that child classes can also override attributes and behaviors from the parent class. For examples:
class FrenchBulldog(Bulldog):
+ species = 'french bulldog'
+sleepy = FrenchBulldog('sleepy', 2)
+print(sleepy.species)
+french bulldog
+print(isinstance(sleepy, Bulldog))
+True
+print(isinstance(sleepy, FrenchBulldog))
+True
+
+import requests
+import json
+response = requests.get("http://hapi.fhir.org/baseR4/Patient/2543713")
+response.status_code
+200
+data = json.loads(response.content.decode('utf-8'))
+data['name']
+[{'use': 'official', 'family': 'Chalmers', 'given': ['Peter', 'James']}]
+data['name'][0]['family']
+'Chalmers'
+data['name'][0]['given']
+['Peter', 'James']
+data['name'][0]['given'][0]
+'Peter'
+response.text
+'{\n "resourceType": "Patient",\n "id": "2543713",\n "meta": {\n "versionId": "1",\n "lastUpdated": "2021-09-06T03:21:51.345+00:00",\n "source": "#tE3DOhkavDLHnkhZ"\n },\n "text": {\n "status": "generated",\n "div": "<div xmlns=\\"http://www.w3.org/1999/xhtml\\">Some narrative</div>"\n },\n "active": true,\n "name": [ {\n "use": "official",\n "family": "Chalmers",\n "given": [ "Peter", "James" ]\n } ],\n "gender": "male",\n "birthDate": "1974-12-25"\n}'
+response.json()
+{'resourceType': 'Patient',
+ 'id': '2543713',
+ 'meta': {'versionId': '1',
+ 'lastUpdated': '2021-09-06T03:21:51.345+00:00',
+ 'source': '#tE3DOhkavDLHnkhZ'},
+ 'text': {'status': 'generated',
+ 'div': '<div xmlns="http://www.w3.org/1999/xhtml">Some narrative</div>'},
+ 'active': True,
+ 'name': [{'use': 'official',
+ 'family': 'Chalmers',
+ 'given': ['Peter', 'James']}],
+ 'gender': 'male',
+ 'birthDate': '1974-12-25'}
+
+Application Program Interfaces, or APIs, are commonly used to retrieve data from remote websites. Sites like Reddit, Twitter, and Facebook all offer certain data through their APIs. To use an API, you make a request to a remote web server, and retrieve the data you need.
But why use an API instead of a static dataset you can download? APIs are useful in the following cases:
In cases like the ones above, an API is the right solution. In this blog post, we’ll be querying a simple API to retrieve data about the International Space Station (ISS). Using an API will save us time and effort over doing all the computation ourselves.
APIs are hosted on web servers. When you type www.google.com in your browser’s address bar, your computer is actually asking the www.google.com server for a webpage, which it then returns to your browser.
APIs work much the same way, except instead of your web browser asking for a webpage, your program asks for data. This data is usually returned in JSON format (for more on this, checkout our tutorial on working with JSON data).
In order to get the data, we make a request to a webserver. The server then replies with our data. In Python, we’ll use the requests library to do this. In this Python API tutorial we’ll be using Python 3.4 for all of our examples.
There are many different types of requests. The most commonly used one, a GET request, is used to retrieve data.
We can use a simple GET request to retrieve information from the OpenNotify API.
OpenNotify has several API endpoints. An endpoint is a server route that is used to retrieve different data from the API. For example, the /comments endpoint on the Reddit API might retrieve information about comments, whereas the /users endpoint might retrieve data about users. To access them, you would add the endpoint to the base url of the API.
The first endpoint we’ll look at on OpenNotify is the iss-now.json endpoint. This endpoint gets the current latitude and longitude of the International Space Station. As you can see, retrieving this data isn’t a great fit for a dataset, because it involves some calculation on the server, and changes quickly.
You can see a listing of all the endpoints on OpenNotify here.
The base url for the OpenNotify API is http://api.open-notify.org, so we’ll add this to the beginning of all of our endpoints.
import requests
+
+# Make a get request to get the latest position of the international space station from the opennotify api.
+response = requests.get("http://api.open-notify.org/iss-now.json")
+
+# Print the status code of the response.
+print(response.status_code)
+print(response.content)
+200
+b'{"iss_position": {"latitude": "32.4451", "longitude": "-148.5181"}, "timestamp": 1634828637, "message": "success"}'
+The request we just made had a status code of 200. Status codes are returned with every request that is made to a web server. Status codes indicate information about what happened with a request. Here are some codes that are relevant to GET requests:
We’ll now make a GET request to http://api.open-notify.org/iss-pass, an endpoint that doesn’t exist, per the API documentation.
response = requests.get("http://api.open-notify.org/iss-pass")
+
+print(response.status_code)
+404
+iss-pass wasn’t a valid endpoint, so we got a 404 status code in response. We forgot to add .json at the end, as the API documentation states.
We’ll now make a GET request to http://api.open-notify.org/iss-pass.json.
response = requests.get("http://api.open-notify.org/iss-pass.json")
+print(response.status_code)
+400
+You’ll see that in the last example, we got a 400 status code, which indicates a bad request. If you look at the documentation for the OpenNotify API, we see that the ISS Pass endpoint requires two parameters.
The ISS Pass endpoint returns when the ISS will next pass over a given location on earth. In order to compute this, we need to pass the coordinates of the location to the API. We do this by passing two parameters – latitude and longitude.
We can do this by adding an optional keyword argument, params, to our request. In this case, there are two parameters we need to pass:
We can make a dictionary with these parameters, and then pass them into the requests.get function.
We can also do the same thing directly by adding the query parameters to the url, like this: http://api.open-notify.org/iss-pass.json?lat=40.71&lon=-74.
It’s almost always preferable to setup the parameters as a dictionary, because requests takes care of some things that come up, like properly formatting the query parameters.
We’ll make a request using the coordinates of New York City, and see what response we get.
# Set up the parameters we want to pass to the API.
+# This is the latitude and longitude of New York City.
+parameters = {"lat": 40.71, "lon": -74}
+
+# Make a get request with the parameters.
+response = requests.get("http://api.open-notify.org/iss-pass.json", params=parameters)
+# response = requests.get("http://api.open-notify.org/iss-pass.json?lat={0}&lon={1}".format(parameters['lat'], parameters['lon']))
+
+# Print the content of the response (the data the server returned)
+print(response.json())
+
+# This gets the same data as the command above
+# response = requests.get("http://api.open-notify.org/iss-pass.json?lat=40.71&lon=-74")
+# print(response.content)
+{'message': 'success', 'request': {'altitude': 100, 'datetime': 1634828641, 'latitude': 40.71, 'longitude': -74.0, 'passes': 5}, 'response': [{'duration': 591, 'risetime': 1634829288}, {'duration': 652, 'risetime': 1634835096}, {'duration': 558, 'risetime': 1634840931}, {'duration': 614, 'risetime': 1634895303}, {'duration': 644, 'risetime': 1634901090}]}
+You may have noticed that the content of the response earlier was a string. Although it was shown as a bytes object, we can easily convert the content to a string using response.content.decode("utf-8").
Strings are the way that we pass information back and forth to APIs, but it’s hard to get the information we want out of them. How do we know how to decode the string that we get back and work with it in Python? How do we figure out the altitude of the ISS from the string response?
Luckily, there’s a format called JavaScript Object Notation (JSON). JSON is a way to encode data structures like lists and dictionaries to strings that ensures that they are easily readable by machines. JSON is the primary format in which data is passed back and forth to APIs, and most API servers will send their responses in JSON format.
Python has great JSON support, with the json package. The json package is part of the standard library, so we don’t have to install anything to use it. We can both convert lists and dictionaries to JSON, and convert strings to lists and dictionaries. In the case of our ISS Pass data, it is a dictionary encoded to a string in JSON format.
The json library has two main methods:
# Make a list of fast food chains.
+best_food_chains = ["Taco Bell", "Shake Shack", "Chipotle"]
+
+# This is a list.
+print(type(best_food_chains))
+<class 'list'>
+# Import the json library
+import json
+# Use json.dumps to convert best_food_chains to a string.
+best_food_chains_string = json.dumps(best_food_chains)
+print('***', best_food_chains_string)
+*** ["Taco Bell", "Shake Shack", "Chipotle"]
+# We've successfully converted our list to a string.
+print(type(best_food_chains_string))
+<class 'str'>
+# Convert best_food_chains_string back into a list
+print(type(json.loads(best_food_chains_string)))
+<class 'list'>
+# Make a dictionary
+fast_food_franchise = {
+ "Subway": 24722,
+ "McDonalds": 14098,
+ "Starbucks": 10821,
+ "Pizza Hut": 7600
+}
+# We can also dump a dictionary to a string and load it.
+fast_food_franchise_string = json.dumps(fast_food_franchise)
+print(type(fast_food_franchise_string))
+<class 'str'>
+You can get the content of a response as a python object by using the .json() method on the response.
# Make the same request we did earlier, but with the coordinates of San Francisco instead.
+parameters = {"lat": 37.78, "lon": -122.41}
+response = requests.get("http://api.open-notify.org/iss-pass.json", params=parameters)
+print(type(response.content))
+<class 'bytes'>
+# Get the response data as a python object. Verify that it's a dictionary.
+data = response.json()
+print(type(data))
+print(data)
+print('message is ', data['message'])
+print('altitude is ', data['request']['altitude'])
+<class 'dict'>
+{'message': 'success', 'request': {'altitude': 100, 'datetime': 1634828645, 'latitude': 37.78, 'longitude': -122.41, 'passes': 5}, 'response': [{'duration': 603, 'risetime': 1634828665}, {'duration': 496, 'risetime': 1634834593}, {'duration': 531, 'risetime': 1634840465}, {'duration': 639, 'risetime': 1634846267}, {'duration': 600, 'risetime': 1634852086}]}
+message is success
+altitude is 100
+The server doesn’t just send a status code and the data when it generates a response. It also sends metadata containing information on how the data was generated and how to decode it. This is stored in the response headers. In Python, we can access this with the headers property of a response object.
The headers will be shown as a dictionary. Within the headers, content-type is the most important key for now. It tells us the format of the response, and how to decode it. For the OpenNotify API, the format is JSON, which is why we could decode it with the json package earlier.
# Headers is a dictionary
+print(response.headers)
+
+# Get the content-type from the dictionary.
+print(response.headers["content-type"])
+{'Server': 'nginx/1.10.3', 'Date': 'Thu, 21 Oct 2021 15:04:05 GMT', 'Content-Type': 'application/json', 'Content-Length': '521', 'Connection': 'keep-alive', 'Via': '1.1 vegur'}
+application/json
+OpenNotify has one more API endpoint, astros.json. It tells you how many people are currently in space. The format of the responses can be found here.
# Get the response from the API endpoint.
+response = requests.get("http://api.open-notify.org/astros.json")
+data = response.json()
+# people are currently in space.
+print(data["number"])
+print(data)
+10
+{'number': 10, 'people': [{'craft': 'ISS', 'name': 'Mark Vande Hei'}, {'craft': 'ISS', 'name': 'Pyotr Dubrov'}, {'craft': 'ISS', 'name': 'Thomas Pesquet'}, {'craft': 'ISS', 'name': 'Megan McArthur'}, {'craft': 'ISS', 'name': 'Shane Kimbrough'}, {'craft': 'ISS', 'name': 'Akihiko Hoshide'}, {'craft': 'ISS', 'name': 'Anton Shkaplerov'}, {'craft': 'Shenzhou 13', 'name': 'Zhai Zhigang'}, {'craft': 'Shenzhou 13', 'name': 'Wang Yaping'}, {'craft': 'Shenzhou 13', 'name': 'Ye Guangfu'}], 'message': 'success'}
+
+import csv
+import json
+from pprint import pprint
+with open('../data/csv/patients.csv') as f:
+ reader = csv.reader(f)
+ header = next(reader, None)
+ rows = [row for row in reader]
+
+print(header)
+print(rows[0])
+['Id', 'BIRTHDATE', 'DEATHDATE', 'SSN', 'DRIVERS', 'PASSPORT', 'PREFIX', 'FIRST', 'LAST', 'SUFFIX', 'MAIDEN', 'MARITAL', 'RACE', 'ETHNICITY', 'GENDER', 'BIRTHPLACE', 'ADDRESS', 'CITY', 'STATE', 'ZIP']
+['3287bb9c-e395-4146-8dd5-1fc3d887d220', '2015-07-12', '', '999-82-9751', '', '', '', 'Victor265', 'Kilback373', '', '', '', 'white', 'polish', 'M', 'Newton Massachusetts US', '657 Heathcote Divide', 'Fitchburg', 'Massachusetts', '01420']
+import datetime
+from datetime import date, timedelta
+from dateutil import parser
+
+def get_age(birth_date):
+ if isinstance(birth_date, str):
+ birth_date = parser.parse(birth_date).date()
+
+ age = (date.today() - birth_date) // timedelta(days=365)
+ return age
+date.today()
+datetime.date(2021, 10, 21)
+datetime.datetime.now()
+datetime.datetime(2021, 10, 21, 11, 4, 20, 310225)
+print(get_age('2011-09-10'))
+10
+print(get_age('2014-01-10'))
+7
+ages = [get_age(row[1]) for row in rows]
+print(ages)
+[6, 18, 13, 17, 8, 9, 55, 84, 14, 20, 53, 23, 18, 64, 11, 39, 34, 5, 82, 19, 18, 15, 26, 28, 29, 8, 71, 32, 82, 41, 51, 17, 40, 53, 60, 65, 11, 21, 32, 52, 3, 52, 82, 30, 69, 24, 99, 12, 51, 99, 27, 76, 78, 29, 16, 69, 34, 78, 58, 76, 99, 65, 70, 60, 50, 32, 4, 78, 65, 48, 3, 66, 77, 99, 25, 18, 47, 79, 99, 32, 10, 74, 61, 99, 62, 59, 67, 2, 50, 99, 3, 70, 61, 62, 43, 18, 71, 29, 99, 52, 71, 26, 28, 56, 32, 69, 23, 31, 46, 99, 106, 30, 21, 67, 67, 106, 99, 106, 99, 106, 106, 99, 106, 99, 99, 99, 99, 90, 90]
+patients = [ {'birth_date': row[1], 'age': get_age(row[1])} for row in rows ]
+pprint(patients)
+[{'age': 6, 'birth_date': '2015-07-12'},
+ {'age': 18, 'birth_date': '2003-04-15'},
+ {'age': 13, 'birth_date': '2008-06-26'},
+ {'age': 17, 'birth_date': '2004-06-22'},
+ {'age': 8, 'birth_date': '2012-12-12'},
+ {'age': 9, 'birth_date': '2012-10-04'},
+ {'age': 55, 'birth_date': '1966-10-02'},
+ {'age': 84, 'birth_date': '1937-04-18'},
+ {'age': 14, 'birth_date': '2007-01-01'},
+ {'age': 20, 'birth_date': '2001-09-12'},
+ {'age': 53, 'birth_date': '1967-11-05'},
+ {'age': 23, 'birth_date': '1998-05-27'},
+ {'age': 18, 'birth_date': '2003-06-22'},
+ {'age': 64, 'birth_date': '1957-02-03'},
+ {'age': 11, 'birth_date': '2009-12-15'},
+ {'age': 39, 'birth_date': '1981-11-24'},
+ {'age': 34, 'birth_date': '1987-04-23'},
+ {'age': 5, 'birth_date': '2016-02-26'},
+ {'age': 82, 'birth_date': '1939-05-17'},
+ {'age': 19, 'birth_date': '2002-05-04'},
+ {'age': 18, 'birth_date': '2003-07-13'},
+ {'age': 15, 'birth_date': '2006-09-05'},
+ {'age': 26, 'birth_date': '1995-04-13'},
+ {'age': 28, 'birth_date': '1993-08-19'},
+ {'age': 29, 'birth_date': '1992-02-09'},
+ {'age': 8, 'birth_date': '2013-09-10'},
+ {'age': 71, 'birth_date': '1950-02-06'},
+ {'age': 32, 'birth_date': '1989-05-19'},
+ {'age': 82, 'birth_date': '1939-05-17'},
+ {'age': 41, 'birth_date': '1980-08-01'},
+ {'age': 51, 'birth_date': '1970-10-17'},
+ {'age': 17, 'birth_date': '2004-09-02'},
+ {'age': 40, 'birth_date': '1981-09-17'},
+ {'age': 53, 'birth_date': '1967-11-06'},
+ {'age': 60, 'birth_date': '1961-07-15'},
+ {'age': 65, 'birth_date': '1956-04-05'},
+ {'age': 11, 'birth_date': '2010-04-02'},
+ {'age': 21, 'birth_date': '2000-09-13'},
+ {'age': 32, 'birth_date': '1989-04-14'},
+ {'age': 52, 'birth_date': '1969-08-30'},
+ {'age': 3, 'birth_date': '2018-02-15'},
+ {'age': 52, 'birth_date': '1968-11-10'},
+ {'age': 82, 'birth_date': '1939-05-17'},
+ {'age': 30, 'birth_date': '1991-04-14'},
+ {'age': 69, 'birth_date': '1951-12-13'},
+ {'age': 24, 'birth_date': '1997-09-25'},
+ {'age': 99, 'birth_date': '1922-04-03'},
+ {'age': 12, 'birth_date': '2009-06-25'},
+ {'age': 51, 'birth_date': '1970-01-22'},
+ {'age': 99, 'birth_date': '1922-04-03'},
+ {'age': 27, 'birth_date': '1994-10-02'},
+ {'age': 76, 'birth_date': '1945-05-24'},
+ {'age': 78, 'birth_date': '1943-02-09'},
+ {'age': 29, 'birth_date': '1992-01-30'},
+ {'age': 16, 'birth_date': '2004-11-05'},
+ {'age': 69, 'birth_date': '1951-12-13'},
+ {'age': 34, 'birth_date': '1987-04-26'},
+ {'age': 78, 'birth_date': '1943-02-09'},
+ {'age': 58, 'birth_date': '1963-03-09'},
+ {'age': 76, 'birth_date': '1945-05-24'},
+ {'age': 99, 'birth_date': '1922-04-03'},
+ {'age': 65, 'birth_date': '1955-11-30'},
+ {'age': 70, 'birth_date': '1950-12-18'},
+ {'age': 60, 'birth_date': '1961-05-19'},
+ {'age': 50, 'birth_date': '1971-07-29'},
+ {'age': 32, 'birth_date': '1988-11-14'},
+ {'age': 4, 'birth_date': '2017-05-08'},
+ {'age': 78, 'birth_date': '1943-02-09'},
+ {'age': 65, 'birth_date': '1955-11-30'},
+ {'age': 48, 'birth_date': '1973-09-21'},
+ {'age': 3, 'birth_date': '2018-03-25'},
+ {'age': 66, 'birth_date': '1955-03-13'},
+ {'age': 77, 'birth_date': '1944-05-19'},
+ {'age': 99, 'birth_date': '1922-04-03'},
+ {'age': 25, 'birth_date': '1996-09-26'},
+ {'age': 18, 'birth_date': '2002-11-08'},
+ {'age': 47, 'birth_date': '1974-11-01'},
+ {'age': 79, 'birth_date': '1942-06-19'},
+ {'age': 99, 'birth_date': '1922-09-27'},
+ {'age': 32, 'birth_date': '1989-07-17'},
+ {'age': 10, 'birth_date': '2011-10-02'},
+ {'age': 74, 'birth_date': '1947-05-03'},
+ {'age': 61, 'birth_date': '1960-06-11'},
+ {'age': 99, 'birth_date': '1922-04-03'},
+ {'age': 62, 'birth_date': '1958-11-22'},
+ {'age': 59, 'birth_date': '1962-06-28'},
+ {'age': 67, 'birth_date': '1954-08-17'},
+ {'age': 2, 'birth_date': '2019-03-24'},
+ {'age': 50, 'birth_date': '1971-07-25'},
+ {'age': 99, 'birth_date': '1922-09-27'},
+ {'age': 3, 'birth_date': '2018-09-20'},
+ {'age': 70, 'birth_date': '1951-09-17'},
+ {'age': 61, 'birth_date': '1960-06-11'},
+ {'age': 62, 'birth_date': '1959-03-18'},
+ {'age': 43, 'birth_date': '1978-06-04'},
+ {'age': 18, 'birth_date': '2003-10-09'},
+ {'age': 71, 'birth_date': '1949-11-25'},
+ {'age': 29, 'birth_date': '1992-08-04'},
+ {'age': 99, 'birth_date': '1922-09-27'},
+ {'age': 52, 'birth_date': '1968-11-11'},
+ {'age': 71, 'birth_date': '1950-03-23'},
+ {'age': 26, 'birth_date': '1995-01-25'},
+ {'age': 28, 'birth_date': '1993-10-21'},
+ {'age': 56, 'birth_date': '1965-08-30'},
+ {'age': 32, 'birth_date': '1989-08-11'},
+ {'age': 69, 'birth_date': '1951-12-24'},
+ {'age': 23, 'birth_date': '1998-02-14'},
+ {'age': 31, 'birth_date': '1990-02-24'},
+ {'age': 46, 'birth_date': '1975-03-04'},
+ {'age': 99, 'birth_date': '1922-09-27'},
+ {'age': 106, 'birth_date': '1915-04-02'},
+ {'age': 30, 'birth_date': '1991-09-07'},
+ {'age': 21, 'birth_date': '2000-10-13'},
+ {'age': 67, 'birth_date': '1953-12-24'},
+ {'age': 67, 'birth_date': '1954-07-17'},
+ {'age': 106, 'birth_date': '1915-04-02'},
+ {'age': 99, 'birth_date': '1922-09-27'},
+ {'age': 106, 'birth_date': '1915-04-02'},
+ {'age': 99, 'birth_date': '1922-09-27'},
+ {'age': 106, 'birth_date': '1915-04-02'},
+ {'age': 106, 'birth_date': '1915-04-02'},
+ {'age': 99, 'birth_date': '1922-09-27'},
+ {'age': 106, 'birth_date': '1915-04-02'},
+ {'age': 99, 'birth_date': '1922-09-27'},
+ {'age': 99, 'birth_date': '1922-09-27'},
+ {'age': 99, 'birth_date': '1922-09-27'},
+ {'age': 99, 'birth_date': '1922-09-27'},
+ {'age': 90, 'birth_date': '1931-03-16'},
+ {'age': 90, 'birth_date': '1931-03-16'}]
+# filtering based on age
+min_age = 18
+max_age = 79
+patients = [ {'birth_date': row[1], 'age': get_age(row[1])}
+ for row in rows if get_age(row[1]) >= min_age and get_age(row[1]) <= max_age ]
+pprint(patients)
+[{'age': 18, 'birth_date': '2003-04-15'},
+ {'age': 55, 'birth_date': '1966-10-02'},
+ {'age': 20, 'birth_date': '2001-09-12'},
+ {'age': 53, 'birth_date': '1967-11-05'},
+ {'age': 23, 'birth_date': '1998-05-27'},
+ {'age': 18, 'birth_date': '2003-06-22'},
+ {'age': 64, 'birth_date': '1957-02-03'},
+ {'age': 39, 'birth_date': '1981-11-24'},
+ {'age': 34, 'birth_date': '1987-04-23'},
+ {'age': 19, 'birth_date': '2002-05-04'},
+ {'age': 18, 'birth_date': '2003-07-13'},
+ {'age': 26, 'birth_date': '1995-04-13'},
+ {'age': 28, 'birth_date': '1993-08-19'},
+ {'age': 29, 'birth_date': '1992-02-09'},
+ {'age': 71, 'birth_date': '1950-02-06'},
+ {'age': 32, 'birth_date': '1989-05-19'},
+ {'age': 41, 'birth_date': '1980-08-01'},
+ {'age': 51, 'birth_date': '1970-10-17'},
+ {'age': 40, 'birth_date': '1981-09-17'},
+ {'age': 53, 'birth_date': '1967-11-06'},
+ {'age': 60, 'birth_date': '1961-07-15'},
+ {'age': 65, 'birth_date': '1956-04-05'},
+ {'age': 21, 'birth_date': '2000-09-13'},
+ {'age': 32, 'birth_date': '1989-04-14'},
+ {'age': 52, 'birth_date': '1969-08-30'},
+ {'age': 52, 'birth_date': '1968-11-10'},
+ {'age': 30, 'birth_date': '1991-04-14'},
+ {'age': 69, 'birth_date': '1951-12-13'},
+ {'age': 24, 'birth_date': '1997-09-25'},
+ {'age': 51, 'birth_date': '1970-01-22'},
+ {'age': 27, 'birth_date': '1994-10-02'},
+ {'age': 76, 'birth_date': '1945-05-24'},
+ {'age': 78, 'birth_date': '1943-02-09'},
+ {'age': 29, 'birth_date': '1992-01-30'},
+ {'age': 69, 'birth_date': '1951-12-13'},
+ {'age': 34, 'birth_date': '1987-04-26'},
+ {'age': 78, 'birth_date': '1943-02-09'},
+ {'age': 58, 'birth_date': '1963-03-09'},
+ {'age': 76, 'birth_date': '1945-05-24'},
+ {'age': 65, 'birth_date': '1955-11-30'},
+ {'age': 70, 'birth_date': '1950-12-18'},
+ {'age': 60, 'birth_date': '1961-05-19'},
+ {'age': 50, 'birth_date': '1971-07-29'},
+ {'age': 32, 'birth_date': '1988-11-14'},
+ {'age': 78, 'birth_date': '1943-02-09'},
+ {'age': 65, 'birth_date': '1955-11-30'},
+ {'age': 48, 'birth_date': '1973-09-21'},
+ {'age': 66, 'birth_date': '1955-03-13'},
+ {'age': 77, 'birth_date': '1944-05-19'},
+ {'age': 25, 'birth_date': '1996-09-26'},
+ {'age': 18, 'birth_date': '2002-11-08'},
+ {'age': 47, 'birth_date': '1974-11-01'},
+ {'age': 79, 'birth_date': '1942-06-19'},
+ {'age': 32, 'birth_date': '1989-07-17'},
+ {'age': 74, 'birth_date': '1947-05-03'},
+ {'age': 61, 'birth_date': '1960-06-11'},
+ {'age': 62, 'birth_date': '1958-11-22'},
+ {'age': 59, 'birth_date': '1962-06-28'},
+ {'age': 67, 'birth_date': '1954-08-17'},
+ {'age': 50, 'birth_date': '1971-07-25'},
+ {'age': 70, 'birth_date': '1951-09-17'},
+ {'age': 61, 'birth_date': '1960-06-11'},
+ {'age': 62, 'birth_date': '1959-03-18'},
+ {'age': 43, 'birth_date': '1978-06-04'},
+ {'age': 18, 'birth_date': '2003-10-09'},
+ {'age': 71, 'birth_date': '1949-11-25'},
+ {'age': 29, 'birth_date': '1992-08-04'},
+ {'age': 52, 'birth_date': '1968-11-11'},
+ {'age': 71, 'birth_date': '1950-03-23'},
+ {'age': 26, 'birth_date': '1995-01-25'},
+ {'age': 28, 'birth_date': '1993-10-21'},
+ {'age': 56, 'birth_date': '1965-08-30'},
+ {'age': 32, 'birth_date': '1989-08-11'},
+ {'age': 69, 'birth_date': '1951-12-24'},
+ {'age': 23, 'birth_date': '1998-02-14'},
+ {'age': 31, 'birth_date': '1990-02-24'},
+ {'age': 46, 'birth_date': '1975-03-04'},
+ {'age': 30, 'birth_date': '1991-09-07'},
+ {'age': 21, 'birth_date': '2000-10-13'},
+ {'age': 67, 'birth_date': '1953-12-24'},
+ {'age': 67, 'birth_date': '1954-07-17'}]
+# another way to do it, add the age to the end
+for row in rows:
+ row.append(get_age(row[1]))
+print(rows[0])
+['3287bb9c-e395-4146-8dd5-1fc3d887d220', '2015-07-12', '', '999-82-9751', '', '', '', 'Victor265', 'Kilback373', '', '', '', 'white', 'polish', 'M', 'Newton Massachusetts US', '657 Heathcote Divide', 'Fitchburg', 'Massachusetts', '01420', 6]
+min_age = 18
+max_age = 79
+patients = [ {'birth_date': row[1], 'age': row[-1]}
+ for row in rows if row[20] >= min_age and row[-1] <= max_age ]
+pprint(patients)
+[{'age': 18, 'birth_date': '2003-04-15'},
+ {'age': 55, 'birth_date': '1966-10-02'},
+ {'age': 20, 'birth_date': '2001-09-12'},
+ {'age': 53, 'birth_date': '1967-11-05'},
+ {'age': 23, 'birth_date': '1998-05-27'},
+ {'age': 18, 'birth_date': '2003-06-22'},
+ {'age': 64, 'birth_date': '1957-02-03'},
+ {'age': 39, 'birth_date': '1981-11-24'},
+ {'age': 34, 'birth_date': '1987-04-23'},
+ {'age': 19, 'birth_date': '2002-05-04'},
+ {'age': 18, 'birth_date': '2003-07-13'},
+ {'age': 26, 'birth_date': '1995-04-13'},
+ {'age': 28, 'birth_date': '1993-08-19'},
+ {'age': 29, 'birth_date': '1992-02-09'},
+ {'age': 71, 'birth_date': '1950-02-06'},
+ {'age': 32, 'birth_date': '1989-05-19'},
+ {'age': 41, 'birth_date': '1980-08-01'},
+ {'age': 51, 'birth_date': '1970-10-17'},
+ {'age': 40, 'birth_date': '1981-09-17'},
+ {'age': 53, 'birth_date': '1967-11-06'},
+ {'age': 60, 'birth_date': '1961-07-15'},
+ {'age': 65, 'birth_date': '1956-04-05'},
+ {'age': 21, 'birth_date': '2000-09-13'},
+ {'age': 32, 'birth_date': '1989-04-14'},
+ {'age': 52, 'birth_date': '1969-08-30'},
+ {'age': 52, 'birth_date': '1968-11-10'},
+ {'age': 30, 'birth_date': '1991-04-14'},
+ {'age': 69, 'birth_date': '1951-12-13'},
+ {'age': 24, 'birth_date': '1997-09-25'},
+ {'age': 51, 'birth_date': '1970-01-22'},
+ {'age': 27, 'birth_date': '1994-10-02'},
+ {'age': 76, 'birth_date': '1945-05-24'},
+ {'age': 78, 'birth_date': '1943-02-09'},
+ {'age': 29, 'birth_date': '1992-01-30'},
+ {'age': 69, 'birth_date': '1951-12-13'},
+ {'age': 34, 'birth_date': '1987-04-26'},
+ {'age': 78, 'birth_date': '1943-02-09'},
+ {'age': 58, 'birth_date': '1963-03-09'},
+ {'age': 76, 'birth_date': '1945-05-24'},
+ {'age': 65, 'birth_date': '1955-11-30'},
+ {'age': 70, 'birth_date': '1950-12-18'},
+ {'age': 60, 'birth_date': '1961-05-19'},
+ {'age': 50, 'birth_date': '1971-07-29'},
+ {'age': 32, 'birth_date': '1988-11-14'},
+ {'age': 78, 'birth_date': '1943-02-09'},
+ {'age': 65, 'birth_date': '1955-11-30'},
+ {'age': 48, 'birth_date': '1973-09-21'},
+ {'age': 66, 'birth_date': '1955-03-13'},
+ {'age': 77, 'birth_date': '1944-05-19'},
+ {'age': 25, 'birth_date': '1996-09-26'},
+ {'age': 18, 'birth_date': '2002-11-08'},
+ {'age': 47, 'birth_date': '1974-11-01'},
+ {'age': 79, 'birth_date': '1942-06-19'},
+ {'age': 32, 'birth_date': '1989-07-17'},
+ {'age': 74, 'birth_date': '1947-05-03'},
+ {'age': 61, 'birth_date': '1960-06-11'},
+ {'age': 62, 'birth_date': '1958-11-22'},
+ {'age': 59, 'birth_date': '1962-06-28'},
+ {'age': 67, 'birth_date': '1954-08-17'},
+ {'age': 50, 'birth_date': '1971-07-25'},
+ {'age': 70, 'birth_date': '1951-09-17'},
+ {'age': 61, 'birth_date': '1960-06-11'},
+ {'age': 62, 'birth_date': '1959-03-18'},
+ {'age': 43, 'birth_date': '1978-06-04'},
+ {'age': 18, 'birth_date': '2003-10-09'},
+ {'age': 71, 'birth_date': '1949-11-25'},
+ {'age': 29, 'birth_date': '1992-08-04'},
+ {'age': 52, 'birth_date': '1968-11-11'},
+ {'age': 71, 'birth_date': '1950-03-23'},
+ {'age': 26, 'birth_date': '1995-01-25'},
+ {'age': 28, 'birth_date': '1993-10-21'},
+ {'age': 56, 'birth_date': '1965-08-30'},
+ {'age': 32, 'birth_date': '1989-08-11'},
+ {'age': 69, 'birth_date': '1951-12-24'},
+ {'age': 23, 'birth_date': '1998-02-14'},
+ {'age': 31, 'birth_date': '1990-02-24'},
+ {'age': 46, 'birth_date': '1975-03-04'},
+ {'age': 30, 'birth_date': '1991-09-07'},
+ {'age': 21, 'birth_date': '2000-10-13'},
+ {'age': 67, 'birth_date': '1953-12-24'},
+ {'age': 67, 'birth_date': '1954-07-17'}]
+# todo we might want to shift all dates (e.g. birthday, encounter date)
+fake_names = '''Veronique Tippetts
+Sarah Santiago
+Eustolia Bushard
+Emanuel Riker
+Maybelle Denney
+Lilia Gobel
+Clarine Vandermeer
+Felicidad Joynes
+Rod Pixley
+Rashad Fukushima
+Marci Bakley
+Melvina Cichon
+Susy Sibert
+Oma Hoskins
+Lance Curnutte
+Mei Wooldridge
+Jillian Mccroy
+Darby Castellon
+Raul Pickney
+Loni Kaur'''
+print(fake_names)
+Veronique Tippetts
+Sarah Santiago
+Eustolia Bushard
+Emanuel Riker
+Maybelle Denney
+Lilia Gobel
+Clarine Vandermeer
+Felicidad Joynes
+Rod Pixley
+Rashad Fukushima
+Marci Bakley
+Melvina Cichon
+Susy Sibert
+Oma Hoskins
+Lance Curnutte
+Mei Wooldridge
+Jillian Mccroy
+Darby Castellon
+Raul Pickney
+Loni Kaur
+fake_names.split('\n')
+['Veronique Tippetts',
+ 'Sarah Santiago',
+ 'Eustolia Bushard',
+ 'Emanuel Riker',
+ 'Maybelle Denney',
+ 'Lilia Gobel',
+ 'Clarine Vandermeer',
+ 'Felicidad Joynes',
+ 'Rod Pixley',
+ 'Rashad Fukushima',
+ 'Marci Bakley',
+ 'Melvina Cichon',
+ 'Susy Sibert',
+ 'Oma Hoskins',
+ 'Lance Curnutte',
+ 'Mei Wooldridge',
+ 'Jillian Mccroy',
+ 'Darby Castellon',
+ 'Raul Pickney',
+ 'Loni Kaur']
+# deidentify the name
+import random
+
+def fake_name_generator():
+ fake_names = \
+"""Veronique Tippetts
+Sarah Santiago
+Eustolia Bushard
+Emanuel Riker
+Maybelle Denney
+Lilia Gobel
+Clarine Vandermeer
+Felicidad Joynes
+Rod Pixley
+Rashad Fukushima
+Marci Bakley
+Melvina Cichon
+Susy Sibert
+Oma Hoskins
+Lance Curnutte
+Mei Wooldridge
+Jillian Mccroy
+Darby Castellon
+Raul Pickney
+Loni Kaur""".split('\n')
+ return random.choice(fake_names).strip()
+print(fake_name_generator())
+print(fake_name_generator())
+print(fake_name_generator())
+Lance Curnutte
+Darby Castellon
+Loni Kaur
+# don't reuse names
+fake_names = \
+"""Veronique Tippetts
+Sarah Santiago
+Eustolia Bushard
+Emanuel Riker
+Maybelle Denney
+Lilia Gobel
+Clarine Vandermeer
+Felicidad Joynes
+Rod Pixley
+Rashad Fukushima
+Marci Bakley
+Melvina Cichon
+Susy Sibert
+Oma Hoskins
+Lance Curnutte
+Mei Wooldridge
+Jillian Mccroy
+Darby Castellon
+Raul Pickney
+Loni Kaur""".split('\n')
+
+def get_fake_name():
+ if len(fake_names) == 0:
+ return None
+ r = random.randint(0, len(fake_names) - 1)
+ n = fake_names[r].strip()
+ del fake_names[r]
+ return n
+for i in range(21):
+ print(get_fake_name())
+Rashad Fukushima
+Raul Pickney
+Marci Bakley
+Melvina Cichon
+Mei Wooldridge
+Maybelle Denney
+Emanuel Riker
+Veronique Tippetts
+Rod Pixley
+Lance Curnutte
+Susy Sibert
+Loni Kaur
+Sarah Santiago
+Clarine Vandermeer
+Jillian Mccroy
+Felicidad Joynes
+Darby Castellon
+Oma Hoskins
+Eustolia Bushard
+Lilia Gobel
+None
+min_age = 18
+max_age = 79
+patients = [ {'birth_date': row[1], 'age': row[20], 'name': get_fake_name() }
+ for row in rows if row[20] >= min_age and row[20] <= max_age ]
+pprint(patients)
+[{'age': 18, 'birth_date': '2003-04-15', 'name': None},
+ {'age': 55, 'birth_date': '1966-10-02', 'name': None},
+ {'age': 20, 'birth_date': '2001-09-12', 'name': None},
+ {'age': 53, 'birth_date': '1967-11-05', 'name': None},
+ {'age': 23, 'birth_date': '1998-05-27', 'name': None},
+ {'age': 18, 'birth_date': '2003-06-22', 'name': None},
+ {'age': 64, 'birth_date': '1957-02-03', 'name': None},
+ {'age': 39, 'birth_date': '1981-11-24', 'name': None},
+ {'age': 34, 'birth_date': '1987-04-23', 'name': None},
+ {'age': 19, 'birth_date': '2002-05-04', 'name': None},
+ {'age': 18, 'birth_date': '2003-07-13', 'name': None},
+ {'age': 26, 'birth_date': '1995-04-13', 'name': None},
+ {'age': 28, 'birth_date': '1993-08-19', 'name': None},
+ {'age': 29, 'birth_date': '1992-02-09', 'name': None},
+ {'age': 71, 'birth_date': '1950-02-06', 'name': None},
+ {'age': 32, 'birth_date': '1989-05-19', 'name': None},
+ {'age': 41, 'birth_date': '1980-08-01', 'name': None},
+ {'age': 51, 'birth_date': '1970-10-17', 'name': None},
+ {'age': 40, 'birth_date': '1981-09-17', 'name': None},
+ {'age': 53, 'birth_date': '1967-11-06', 'name': None},
+ {'age': 60, 'birth_date': '1961-07-15', 'name': None},
+ {'age': 65, 'birth_date': '1956-04-05', 'name': None},
+ {'age': 21, 'birth_date': '2000-09-13', 'name': None},
+ {'age': 32, 'birth_date': '1989-04-14', 'name': None},
+ {'age': 52, 'birth_date': '1969-08-30', 'name': None},
+ {'age': 52, 'birth_date': '1968-11-10', 'name': None},
+ {'age': 30, 'birth_date': '1991-04-14', 'name': None},
+ {'age': 69, 'birth_date': '1951-12-13', 'name': None},
+ {'age': 24, 'birth_date': '1997-09-25', 'name': None},
+ {'age': 51, 'birth_date': '1970-01-22', 'name': None},
+ {'age': 27, 'birth_date': '1994-10-02', 'name': None},
+ {'age': 76, 'birth_date': '1945-05-24', 'name': None},
+ {'age': 78, 'birth_date': '1943-02-09', 'name': None},
+ {'age': 29, 'birth_date': '1992-01-30', 'name': None},
+ {'age': 69, 'birth_date': '1951-12-13', 'name': None},
+ {'age': 34, 'birth_date': '1987-04-26', 'name': None},
+ {'age': 78, 'birth_date': '1943-02-09', 'name': None},
+ {'age': 58, 'birth_date': '1963-03-09', 'name': None},
+ {'age': 76, 'birth_date': '1945-05-24', 'name': None},
+ {'age': 65, 'birth_date': '1955-11-30', 'name': None},
+ {'age': 70, 'birth_date': '1950-12-18', 'name': None},
+ {'age': 60, 'birth_date': '1961-05-19', 'name': None},
+ {'age': 50, 'birth_date': '1971-07-29', 'name': None},
+ {'age': 32, 'birth_date': '1988-11-14', 'name': None},
+ {'age': 78, 'birth_date': '1943-02-09', 'name': None},
+ {'age': 65, 'birth_date': '1955-11-30', 'name': None},
+ {'age': 48, 'birth_date': '1973-09-21', 'name': None},
+ {'age': 66, 'birth_date': '1955-03-13', 'name': None},
+ {'age': 77, 'birth_date': '1944-05-19', 'name': None},
+ {'age': 25, 'birth_date': '1996-09-26', 'name': None},
+ {'age': 18, 'birth_date': '2002-11-08', 'name': None},
+ {'age': 47, 'birth_date': '1974-11-01', 'name': None},
+ {'age': 79, 'birth_date': '1942-06-19', 'name': None},
+ {'age': 32, 'birth_date': '1989-07-17', 'name': None},
+ {'age': 74, 'birth_date': '1947-05-03', 'name': None},
+ {'age': 61, 'birth_date': '1960-06-11', 'name': None},
+ {'age': 62, 'birth_date': '1958-11-22', 'name': None},
+ {'age': 59, 'birth_date': '1962-06-28', 'name': None},
+ {'age': 67, 'birth_date': '1954-08-17', 'name': None},
+ {'age': 50, 'birth_date': '1971-07-25', 'name': None},
+ {'age': 70, 'birth_date': '1951-09-17', 'name': None},
+ {'age': 61, 'birth_date': '1960-06-11', 'name': None},
+ {'age': 62, 'birth_date': '1959-03-18', 'name': None},
+ {'age': 43, 'birth_date': '1978-06-04', 'name': None},
+ {'age': 18, 'birth_date': '2003-10-09', 'name': None},
+ {'age': 71, 'birth_date': '1949-11-25', 'name': None},
+ {'age': 29, 'birth_date': '1992-08-04', 'name': None},
+ {'age': 52, 'birth_date': '1968-11-11', 'name': None},
+ {'age': 71, 'birth_date': '1950-03-23', 'name': None},
+ {'age': 26, 'birth_date': '1995-01-25', 'name': None},
+ {'age': 28, 'birth_date': '1993-10-21', 'name': None},
+ {'age': 56, 'birth_date': '1965-08-30', 'name': None},
+ {'age': 32, 'birth_date': '1989-08-11', 'name': None},
+ {'age': 69, 'birth_date': '1951-12-24', 'name': None},
+ {'age': 23, 'birth_date': '1998-02-14', 'name': None},
+ {'age': 31, 'birth_date': '1990-02-24', 'name': None},
+ {'age': 46, 'birth_date': '1975-03-04', 'name': None},
+ {'age': 30, 'birth_date': '1991-09-07', 'name': None},
+ {'age': 21, 'birth_date': '2000-10-13', 'name': None},
+ {'age': 67, 'birth_date': '1953-12-24', 'name': None},
+ {'age': 67, 'birth_date': '1954-07-17', 'name': None}]
+
+
+In this tutorial we’re going to learn how to use basic python funtionality to group datasets.
import csv
+from pprint import pprint
+# recall, opening the file and reading the data
+with open('../data/csv/patients.csv') as f:
+ reader = csv.DictReader(f)
+ for i, row in enumerate(reader):
+ pprint(row)
+ if i >= 1:
+ break
+{'ADDRESS': '657 Heathcote Divide',
+ 'BIRTHDATE': '2015-07-12',
+ 'BIRTHPLACE': 'Newton Massachusetts US',
+ 'CITY': 'Fitchburg',
+ 'DEATHDATE': '',
+ 'DRIVERS': '',
+ 'ETHNICITY': 'polish',
+ 'FIRST': 'Victor265',
+ 'GENDER': 'M',
+ 'Id': '3287bb9c-e395-4146-8dd5-1fc3d887d220',
+ 'LAST': 'Kilback373',
+ 'MAIDEN': '',
+ 'MARITAL': '',
+ 'PASSPORT': '',
+ 'PREFIX': '',
+ 'RACE': 'white',
+ 'SSN': '999-82-9751',
+ 'STATE': 'Massachusetts',
+ 'SUFFIX': '',
+ 'ZIP': '01420'}
+{'ADDRESS': '419 Mraz Ranch Apt 13',
+ 'BIRTHDATE': '2003-04-15',
+ 'BIRTHPLACE': 'Ipswich Massachusetts US',
+ 'CITY': 'Saugus',
+ 'DEATHDATE': '',
+ 'DRIVERS': 'S99990332',
+ 'ETHNICITY': 'puerto_rican',
+ 'FIRST': 'Reena181',
+ 'GENDER': 'F',
+ 'Id': 'e118c014-fef7-4fbd-8c5d-fe42e9e16d84',
+ 'LAST': 'Dooley940',
+ 'MAIDEN': '',
+ 'MARITAL': '',
+ 'PASSPORT': '',
+ 'PREFIX': '',
+ 'RACE': 'hispanic',
+ 'SSN': '999-57-7583',
+ 'STATE': 'Massachusetts',
+ 'SUFFIX': '',
+ 'ZIP': '01906'}
+# then we're going to parse out a group, e.g. Gender
+# recall, opening the file and reading the data
+with open('../data/csv/patients.csv') as f:
+ reader = csv.DictReader(f)
+ for i, row in enumerate(reader):
+ patient_gender = row['GENDER']
+ print(patient_gender)
+ if i >= 2:
+ break
+M
+F
+F
+# how do we know what the unique genders are?
+# let's iterate over them and create a set
+patient_genders = set()
+
+with open('../data/csv/patients.csv') as f:
+ reader = csv.DictReader(f)
+ for i, row in enumerate(reader):
+ patient_gender = row['GENDER']
+ patient_genders.add(patient_gender)
+print(patient_genders)
+{'F', 'M'}
+# okay, we have 2 genders let's create 2 lists
+male_patients = []
+female_patients = []
+
+with open('../data/csv/patients.csv') as f:
+ reader = csv.DictReader(f)
+ for i, row in enumerate(reader):
+ patient_gender = row['GENDER']
+ if patient_gender == 'M':
+ male_patients.append(row)
+ elif patient_gender == 'F':
+ female_patients.append(row)
+ else:
+ raise Exception('Unknown Gender')
+
+pprint(male_patients[0])
+pprint(female_patients[0])
+{'ADDRESS': '657 Heathcote Divide',
+ 'BIRTHDATE': '2015-07-12',
+ 'BIRTHPLACE': 'Newton Massachusetts US',
+ 'CITY': 'Fitchburg',
+ 'DEATHDATE': '',
+ 'DRIVERS': '',
+ 'ETHNICITY': 'polish',
+ 'FIRST': 'Victor265',
+ 'GENDER': 'M',
+ 'Id': '3287bb9c-e395-4146-8dd5-1fc3d887d220',
+ 'LAST': 'Kilback373',
+ 'MAIDEN': '',
+ 'MARITAL': '',
+ 'PASSPORT': '',
+ 'PREFIX': '',
+ 'RACE': 'white',
+ 'SSN': '999-82-9751',
+ 'STATE': 'Massachusetts',
+ 'SUFFIX': '',
+ 'ZIP': '01420'}
+{'ADDRESS': '419 Mraz Ranch Apt 13',
+ 'BIRTHDATE': '2003-04-15',
+ 'BIRTHPLACE': 'Ipswich Massachusetts US',
+ 'CITY': 'Saugus',
+ 'DEATHDATE': '',
+ 'DRIVERS': 'S99990332',
+ 'ETHNICITY': 'puerto_rican',
+ 'FIRST': 'Reena181',
+ 'GENDER': 'F',
+ 'Id': 'e118c014-fef7-4fbd-8c5d-fe42e9e16d84',
+ 'LAST': 'Dooley940',
+ 'MAIDEN': '',
+ 'MARITAL': '',
+ 'PASSPORT': '',
+ 'PREFIX': '',
+ 'RACE': 'hispanic',
+ 'SSN': '999-57-7583',
+ 'STATE': 'Massachusetts',
+ 'SUFFIX': '',
+ 'ZIP': '01906'}
+What’s wrong with the code above?
What can we do?
# patients by gender
+patients_by_gender = {}
+
+patients_by_gender['F'] = ['patient1', 'patient3']
+patients_by_gender['M'] = ['patient2', 'patient4']
+
+pprint(patients_by_gender)
+{'F': ['patient1', 'patient3'], 'M': ['patient2', 'patient4']}
+print(patients_by_gender.keys())
+dict_keys(['F', 'M'])
+print(patients_by_gender.values())
+dict_values([['patient1', 'patient3'], ['patient2', 'patient4']])
+print(patients_by_gender.items())
+dict_items([('F', ['patient1', 'patient3']), ('M', ['patient2', 'patient4'])])
+# check if a gender is in the dictionary
+print('F' in patients_by_gender)
+True
+print('f' in patients_by_gender)
+False
+# let's group using a dictionary
+patients_by_gender = {}
+
+with open('../data/csv/patients.csv') as f:
+ reader = csv.DictReader(f)
+
+ for row in reader:
+ patient_gender = row['GENDER'].upper() # let's store the keys as uppercase
+
+ # check to see if the key exists, if not
+ if patient_gender not in patients_by_gender:
+ # add the key
+ patients_by_gender[patient_gender] = [] # create an empty list
+
+ # append the patient as a new row to the correct grouping
+ patients_by_gender[patient_gender].append(row)
+
+print(patients_by_gender.keys())
+dict_keys(['M', 'F'])
+pprint(patients_by_gender['F'][0:2])
+[{'ADDRESS': '419 Mraz Ranch Apt 13',
+ 'BIRTHDATE': '2003-04-15',
+ 'BIRTHPLACE': 'Ipswich Massachusetts US',
+ 'CITY': 'Saugus',
+ 'DEATHDATE': '',
+ 'DRIVERS': 'S99990332',
+ 'ETHNICITY': 'puerto_rican',
+ 'FIRST': 'Reena181',
+ 'GENDER': 'F',
+ 'Id': 'e118c014-fef7-4fbd-8c5d-fe42e9e16d84',
+ 'LAST': 'Dooley940',
+ 'MAIDEN': '',
+ 'MARITAL': '',
+ 'PASSPORT': '',
+ 'PREFIX': '',
+ 'RACE': 'hispanic',
+ 'SSN': '999-57-7583',
+ 'STATE': 'Massachusetts',
+ 'SUFFIX': '',
+ 'ZIP': '01906'},
+ {'ADDRESS': '570 Gleichner Branch Suite 81',
+ 'BIRTHDATE': '2008-06-26',
+ 'BIRTHPLACE': 'Danvers Massachusetts US',
+ 'CITY': 'Lynn',
+ 'DEATHDATE': '',
+ 'DRIVERS': '',
+ 'ETHNICITY': 'english',
+ 'FIRST': 'Brinda322',
+ 'GENDER': 'F',
+ 'Id': '4a2a4a0b-0c4a-47b3-9b93-ce82048a41e6',
+ 'LAST': 'Harber290',
+ 'MAIDEN': '',
+ 'MARITAL': '',
+ 'PASSPORT': '',
+ 'PREFIX': '',
+ 'RACE': 'white',
+ 'SSN': '999-57-3145',
+ 'STATE': 'Massachusetts',
+ 'SUFFIX': '',
+ 'ZIP': '01901'}]
+print(patients_by_gender['F'][0]['LAST'])
+Dooley940
+What’s common?
What’s different?
# define the function
+def group_patient(file_name, groupby):
+ pass
+# call the function
+patients_by_gender = group_patient('../data/csv/patients.csv', 'GENDER')
+print(patients_by_gender)
+None
+# now let's implement the function
+def group_patient(file_name, groupby):
+ patients_by_group = {}
+ with open(file_name) as f:
+ reader = csv.DictReader(f)
+ for row in reader:
+ # note : we renamed the variable to _attribute
+ patient_attribute = row[groupby].upper()
+
+ # check to see if the key exists, if not
+ if patient_attribute not in patients_by_group:
+ # add the key
+ patients_by_group[patient_attribute] = [] # create an empty list
+
+ # append the patient as a new row to the correct grouping
+ patients_by_group[patient_attribute].append(row)
+ return patients_by_group
+patients_by_gender = group_patient('../data/csv/patients.csv', 'GENDER')
+pprint(patients_by_gender['F'][0:2])
+[{'ADDRESS': '419 Mraz Ranch Apt 13',
+ 'BIRTHDATE': '2003-04-15',
+ 'BIRTHPLACE': 'Ipswich Massachusetts US',
+ 'CITY': 'Saugus',
+ 'DEATHDATE': '',
+ 'DRIVERS': 'S99990332',
+ 'ETHNICITY': 'puerto_rican',
+ 'FIRST': 'Reena181',
+ 'GENDER': 'F',
+ 'Id': 'e118c014-fef7-4fbd-8c5d-fe42e9e16d84',
+ 'LAST': 'Dooley940',
+ 'MAIDEN': '',
+ 'MARITAL': '',
+ 'PASSPORT': '',
+ 'PREFIX': '',
+ 'RACE': 'hispanic',
+ 'SSN': '999-57-7583',
+ 'STATE': 'Massachusetts',
+ 'SUFFIX': '',
+ 'ZIP': '01906'},
+ {'ADDRESS': '570 Gleichner Branch Suite 81',
+ 'BIRTHDATE': '2008-06-26',
+ 'BIRTHPLACE': 'Danvers Massachusetts US',
+ 'CITY': 'Lynn',
+ 'DEATHDATE': '',
+ 'DRIVERS': '',
+ 'ETHNICITY': 'english',
+ 'FIRST': 'Brinda322',
+ 'GENDER': 'F',
+ 'Id': '4a2a4a0b-0c4a-47b3-9b93-ce82048a41e6',
+ 'LAST': 'Harber290',
+ 'MAIDEN': '',
+ 'MARITAL': '',
+ 'PASSPORT': '',
+ 'PREFIX': '',
+ 'RACE': 'white',
+ 'SSN': '999-57-3145',
+ 'STATE': 'Massachusetts',
+ 'SUFFIX': '',
+ 'ZIP': '01901'}]
+# but wait... is there anything patient specific???
+# let's refactor the code to make it more extensible
+# and easier to read
+def group(file_name, groupby):
+ grouped_data = {}
+ with open(file_name) as f:
+ reader = csv.DictReader(f)
+ for row in reader:
+ attribute = row[groupby].upper()
+ if attribute not in grouped_data:
+ grouped_data[attribute] = []
+ grouped_data[attribute].append(row)
+ return grouped_data
+patients_by_gender = group('../data/csv/patients.csv', 'GENDER')
+pprint(patients_by_gender['F'][0])
+{'ADDRESS': '419 Mraz Ranch Apt 13',
+ 'BIRTHDATE': '2003-04-15',
+ 'BIRTHPLACE': 'Ipswich Massachusetts US',
+ 'CITY': 'Saugus',
+ 'DEATHDATE': '',
+ 'DRIVERS': 'S99990332',
+ 'ETHNICITY': 'puerto_rican',
+ 'FIRST': 'Reena181',
+ 'GENDER': 'F',
+ 'Id': 'e118c014-fef7-4fbd-8c5d-fe42e9e16d84',
+ 'LAST': 'Dooley940',
+ 'MAIDEN': '',
+ 'MARITAL': '',
+ 'PASSPORT': '',
+ 'PREFIX': '',
+ 'RACE': 'hispanic',
+ 'SSN': '999-57-7583',
+ 'STATE': 'Massachusetts',
+ 'SUFFIX': '',
+ 'ZIP': '01906'}
+# single level grouping by gender
+{
+ "F": [ 'patient1', 'patient2'],
+ "M": [ 'patient3', 'patient4']
+}
+{'F': ['patient1', 'patient2'], 'M': ['patient3', 'patient4']}
+# multi level grouping by gender and race
+{
+ "F": {
+ "WHITE": ['patient1', 'patient2'],
+ "HISPANIC": ['patient3']
+ },
+ "M": {
+ "WHITE": [],
+ "HISPANIC": ['patient4', 'patient5']
+ }
+}
+{'F': {'WHITE': ['patient1', 'patient2'], 'HISPANIC': ['patient3']},
+ 'M': {'WHITE': [], 'HISPANIC': ['patient4', 'patient5']}}
+def group_file(file_name, groupby):
+ with open(file_name) as f:
+ reader = csv.DictReader(f)
+ return group(reader, groupby)
+
+def group(iterable, groupby):
+ grouped_data = {}
+ for item in iterable:
+ attribute = item[groupby].upper()
+ if attribute not in grouped_data:
+ grouped_data[attribute] = []
+ grouped_data[attribute].append(item)
+ return grouped_data
+patients_by_gender = group_file('../data/csv/patients.csv', 'GENDER')
+pprint(patients_by_gender['F'][0])
+{'ADDRESS': '419 Mraz Ranch Apt 13',
+ 'BIRTHDATE': '2003-04-15',
+ 'BIRTHPLACE': 'Ipswich Massachusetts US',
+ 'CITY': 'Saugus',
+ 'DEATHDATE': '',
+ 'DRIVERS': 'S99990332',
+ 'ETHNICITY': 'puerto_rican',
+ 'FIRST': 'Reena181',
+ 'GENDER': 'F',
+ 'Id': 'e118c014-fef7-4fbd-8c5d-fe42e9e16d84',
+ 'LAST': 'Dooley940',
+ 'MAIDEN': '',
+ 'MARITAL': '',
+ 'PASSPORT': '',
+ 'PREFIX': '',
+ 'RACE': 'hispanic',
+ 'SSN': '999-57-7583',
+ 'STATE': 'Massachusetts',
+ 'SUFFIX': '',
+ 'ZIP': '01906'}
+# patients by gender and race
+# here we'll use a dictionary to represent the 2nd level
+patients_by_gender_and_race = group_file('../data/csv/patients.csv', 'GENDER')
+for gender in patients_by_gender_and_race.keys():
+ patients_by_gender_and_race[gender] = {}
+pprint(patients_by_gender_and_race)
+{'F': {}, 'M': {}}
+# now let's perform the groupings
+patients_by_gender_and_race = group_file('../data/csv/patients.csv', 'GENDER')
+# print(patients_by_gender_and_race)
+for gender in patients_by_gender_and_race.keys():
+ patients_by_gender_and_race[gender] = group(patients_by_gender_and_race[gender], 'RACE')
+# print(patients_by_gender)
+# finally let's print the unique genders and races
+for gender in patients_by_gender_and_race.keys():
+ for race in patients_by_gender_and_race[gender].keys():
+ print(gender, race)
+M WHITE
+M ASIAN
+M NATIVE
+M HISPANIC
+M BLACK
+F HISPANIC
+F WHITE
+F BLACK
+F NATIVE
+F ASIAN
+# as a sanity check let's look over the dataset again
+gender = set()
+race = set()
+
+with open('../data/csv/patients.csv') as f:
+ reader = csv.DictReader(f)
+ for row in reader:
+ gender.add(row['GENDER'])
+ race.add(row['RACE'])
+print(gender, race)
+{'F', 'M'} {'black', 'white', 'asian', 'native', 'hispanic'}
+# what about 3 levels of grouping?
+grouped_patients = group_file('../data/csv/patients.csv', 'GENDER')
+for gender in grouped_patients.keys():
+ grouped_patients[gender] = group(grouped_patients[gender], 'RACE')
+ for race in grouped_patients[gender].keys():
+ grouped_patients[gender][race] = group(grouped_patients[gender][race], 'ETHNICITY')
+
+# print(grouped_patients)
+# let's print the unique genders and races
+for gender in grouped_patients.keys():
+ for race in grouped_patients[gender].keys():
+ for ethnicity in grouped_patients[gender][race].keys():
+ print(gender, race, ethnicity)
+M WHITE POLISH
+M WHITE PORTUGUESE
+M WHITE IRISH
+M WHITE FRENCH_CANADIAN
+M WHITE ITALIAN
+M WHITE GREEK
+M WHITE SCOTTISH
+M WHITE ENGLISH
+M WHITE AMERICAN
+M WHITE FRENCH
+M WHITE GERMAN
+M ASIAN CHINESE
+M ASIAN ASIAN_INDIAN
+M NATIVE AMERICAN_INDIAN
+M HISPANIC PUERTO_RICAN
+M HISPANIC SOUTH_AMERICAN
+M HISPANIC MEXICAN
+M BLACK DOMINICAN
+M BLACK WEST_INDIAN
+F HISPANIC PUERTO_RICAN
+F WHITE ENGLISH
+F WHITE FRENCH
+F WHITE ITALIAN
+F WHITE GERMAN
+F WHITE FRENCH_CANADIAN
+F WHITE IRISH
+F WHITE PORTUGUESE
+F WHITE SCOTTISH
+F WHITE POLISH
+F BLACK DOMINICAN
+F NATIVE AMERICAN_INDIAN
+F ASIAN CHINESE
+I think we should refactor again. This time using recursion. This time we’ll support any number of groupings.
def group_file_by_list(file_name, groupings):
+ with open(file_name) as f:
+ reader = csv.DictReader(f)
+ grouped_data = group(reader, groupings[0])
+ if len(groupings) > 1:
+ group_by_list(grouped_data.keys(), grouped_data, groupings[1:])
+ return grouped_data
+
+def group_by_list(iterable, grouped_data, groupings):
+ for item in iterable:
+ grouped_data[item] = group(grouped_data[item], groupings[0])
+ if len(groupings) > 1:
+ group_by_list(grouped_data[item].keys(), grouped_data[item], groupings[1:])
+ return grouped_data
+groupings = ['GENDER', 'RACE', 'ETHNICITY', 'BIRTHPLACE']
+grouped_patients = group_file_by_list('../data/csv/patients.csv', groupings)
+for gender in grouped_patients.keys():
+ for race in grouped_patients[gender].keys():
+ for ethnicity in grouped_patients[gender][race].keys():
+ print(gender, race, ethnicity)
+M WHITE POLISH
+M WHITE PORTUGUESE
+M WHITE IRISH
+M WHITE FRENCH_CANADIAN
+M WHITE ITALIAN
+M WHITE GREEK
+M WHITE SCOTTISH
+M WHITE ENGLISH
+M WHITE AMERICAN
+M WHITE FRENCH
+M WHITE GERMAN
+M ASIAN CHINESE
+M ASIAN ASIAN_INDIAN
+M NATIVE AMERICAN_INDIAN
+M HISPANIC PUERTO_RICAN
+M HISPANIC SOUTH_AMERICAN
+M HISPANIC MEXICAN
+M BLACK DOMINICAN
+M BLACK WEST_INDIAN
+F HISPANIC PUERTO_RICAN
+F WHITE ENGLISH
+F WHITE FRENCH
+F WHITE ITALIAN
+F WHITE GERMAN
+F WHITE FRENCH_CANADIAN
+F WHITE IRISH
+F WHITE PORTUGUESE
+F WHITE SCOTTISH
+F WHITE POLISH
+F BLACK DOMINICAN
+F NATIVE AMERICAN_INDIAN
+F ASIAN CHINESE
+# let's create a recursive print
+def print_keys(d):
+ for k, v in d.items():
+ print_keys2(v, [k])
+
+def print_keys2(d, l):
+ if isinstance(d, dict):
+ for k, v in d.items():
+ print_keys2(v, l + [k])
+ else:
+ print(' '.join(l))
+ l = []
+
+print_keys(grouped_patients)
+M WHITE POLISH NEWTON MASSACHUSETTS US
+M WHITE POLISH ADAMS MASSACHUSETTS US
+M WHITE POLISH BOSTON MASSACHUSETTS US
+M WHITE POLISH HOLYOKE MASSACHUSETTS US
+M WHITE PORTUGUESE PORTO DOURO LITORAL PT
+M WHITE PORTUGUESE BRAGA MINHO PT
+M WHITE PORTUGUESE YARMOUTH MASSACHUSETTS US
+M WHITE IRISH CHICOPEE MASSACHUSETTS US
+M WHITE IRISH WORCESTER MASSACHUSETTS US
+M WHITE IRISH WESTPORT MASSACHUSETTS US
+M WHITE IRISH WEYMOUTH TOWN MASSACHUSETTS US
+M WHITE IRISH MASHPEE MASSACHUSETTS US
+M WHITE IRISH PITTSFIELD MASSACHUSETTS US
+M WHITE FRENCH_CANADIAN LENOX MASSACHUSETTS US
+M WHITE FRENCH_CANADIAN BOSTON MASSACHUSETTS US
+M WHITE FRENCH_CANADIAN SHERBORN MASSACHUSETTS US
+M WHITE FRENCH_CANADIAN HUDSON MASSACHUSETTS US
+M WHITE FRENCH_CANADIAN SANDWICH MASSACHUSETTS US
+M WHITE FRENCH_CANADIAN WORCESTER MASSACHUSETTS US
+M WHITE ITALIAN FALL RIVER MASSACHUSETTS US
+M WHITE ITALIAN HAVERHILL MASSACHUSETTS US
+M WHITE ITALIAN BOSTON MASSACHUSETTS US
+M WHITE ITALIAN SUDBURY MASSACHUSETTS US
+M WHITE ITALIAN SOMERVILLE MASSACHUSETTS US
+M WHITE ITALIAN MANSFIELD MASSACHUSETTS US
+M WHITE ITALIAN BELCHERTOWN MASSACHUSETTS US
+M WHITE ITALIAN NORTH ATTLEBOROUGH MASSACHUSETTS US
+M WHITE ITALIAN FAIRHAVEN MASSACHUSETTS US
+M WHITE ITALIAN MIDDLETON MASSACHUSETTS US
+M WHITE ITALIAN BEVERLY MASSACHUSETTS US
+M WHITE GREEK ARLINGTON MASSACHUSETTS US
+M WHITE GREEK WORCESTER MASSACHUSETTS US
+M WHITE SCOTTISH PITTSFIELD MASSACHUSETTS US
+M WHITE ENGLISH HAVERHILL MASSACHUSETTS US
+M WHITE ENGLISH BOSTON MASSACHUSETTS US
+M WHITE ENGLISH SHARON MASSACHUSETTS US
+M WHITE ENGLISH BEVERLY MASSACHUSETTS US
+M WHITE ENGLISH ATTLEBORO MASSACHUSETTS US
+M WHITE ENGLISH NEW BEDFORD MASSACHUSETTS US
+M WHITE ENGLISH WORCESTER MASSACHUSETTS US
+M WHITE ENGLISH SOMERVILLE MASSACHUSETTS US
+M WHITE ENGLISH CHICOPEE MASSACHUSETTS US
+M WHITE ENGLISH REVERE MASSACHUSETTS US
+M WHITE ENGLISH RUTLAND MASSACHUSETTS US
+M WHITE ENGLISH PLYMOUTH MASSACHUSETTS US
+M WHITE ENGLISH NORTHAMPTON MASSACHUSETTS US
+M WHITE ENGLISH DARTMOUTH MASSACHUSETTS US
+M WHITE ENGLISH NATICK MASSACHUSETTS US
+M WHITE ENGLISH WHATELY MASSACHUSETTS US
+M WHITE ENGLISH SOUTHWICK MASSACHUSETTS US
+M WHITE AMERICAN SPRINGFIELD MASSACHUSETTS US
+M WHITE FRENCH SHREWSBURY MASSACHUSETTS US
+M WHITE GERMAN WALTHAM MASSACHUSETTS US
+M ASIAN CHINESE QUINCY MASSACHUSETTS US
+M ASIAN ASIAN_INDIAN ROCKLAND MASSACHUSETTS US
+M NATIVE AMERICAN_INDIAN NEW BEDFORD MASSACHUSETTS US
+M HISPANIC PUERTO_RICAN TEMPLETON MASSACHUSETTS US
+M HISPANIC PUERTO_RICAN LOWELL MASSACHUSETTS US
+M HISPANIC PUERTO_RICAN BAYAMON PUERTO RICO PR
+M HISPANIC PUERTO_RICAN PONCE PUERTO RICO PR
+M HISPANIC PUERTO_RICAN CAROLINA PUERTO RICO PR
+M HISPANIC SOUTH_AMERICAN RIO DE JANERIO RIO DE JANERIO BR
+M HISPANIC MEXICAN LA PAZ BAJA CALIFORNIA MX
+M BLACK DOMINICAN UXBRIDGE MASSACHUSETTS US
+M BLACK DOMINICAN DARTMOUTH MASSACHUSETTS US
+M BLACK WEST_INDIAN SANTIAGO DE LOS CABALLEROS SANTIAGO DO
+M BLACK WEST_INDIAN KINGSTON KINGSTON JM
+F HISPANIC PUERTO_RICAN IPSWICH MASSACHUSETTS US
+F HISPANIC PUERTO_RICAN BAYAMON PUERTO RICO PR
+F HISPANIC PUERTO_RICAN CAROLINA PUERTO RICO PR
+F WHITE ENGLISH DANVERS MASSACHUSETTS US
+F WHITE ENGLISH BELCHERTOWN MASSACHUSETTS US
+F WHITE ENGLISH HARVARD MASSACHUSETTS US
+F WHITE ENGLISH STERLING MASSACHUSETTS US
+F WHITE ENGLISH NATICK MASSACHUSETTS US
+F WHITE ENGLISH HOPKINTON MASSACHUSETTS US
+F WHITE FRENCH CHICOPEE MASSACHUSETTS US
+F WHITE FRENCH STOUGHTON MASSACHUSETTS US
+F WHITE FRENCH BOSTON MASSACHUSETTS US
+F WHITE FRENCH SALEM MASSACHUSETTS US
+F WHITE FRENCH CONCORD MASSACHUSETTS US
+F WHITE ITALIAN MILFORD MASSACHUSETTS US
+F WHITE ITALIAN SPRINGFIELD MASSACHUSETTS US
+F WHITE ITALIAN MILTON MASSACHUSETTS US
+F WHITE ITALIAN DRACUT MASSACHUSETTS US
+F WHITE ITALIAN PALERMO SICILY IT
+F WHITE ITALIAN NEW BEDFORD MASSACHUSETTS US
+F WHITE ITALIAN SOUTHWICK MASSACHUSETTS US
+F WHITE GERMAN NORTH ADAMS MASSACHUSETTS US
+F WHITE GERMAN ATTLEBORO MASSACHUSETTS US
+F WHITE GERMAN EASTHAMPTON TOWN MASSACHUSETTS US
+F WHITE GERMAN QUINCY MASSACHUSETTS US
+F WHITE GERMAN REVERE MASSACHUSETTS US
+F WHITE FRENCH_CANADIAN BILLERICA MASSACHUSETTS US
+F WHITE FRENCH_CANADIAN STONEHAM MASSACHUSETTS US
+F WHITE FRENCH_CANADIAN HOLYOKE MASSACHUSETTS US
+F WHITE FRENCH_CANADIAN ROCKLAND MASSACHUSETTS US
+F WHITE FRENCH_CANADIAN AGAWAM TOWN MASSACHUSETTS US
+F WHITE FRENCH_CANADIAN WEST SPRINGFIELD TOWN MASSACHUSETTS US
+F WHITE FRENCH_CANADIAN BOSTON MASSACHUSETTS US
+F WHITE FRENCH_CANADIAN LOWELL MASSACHUSETTS US
+F WHITE FRENCH_CANADIAN PALMER TOWN MASSACHUSETTS US
+F WHITE IRISH LYNN MASSACHUSETTS US
+F WHITE IRISH FRAMINGHAM MASSACHUSETTS US
+F WHITE IRISH PALMER TOWN MASSACHUSETTS US
+F WHITE IRISH BOSTON MASSACHUSETTS US
+F WHITE IRISH BRAINTREE TOWN MASSACHUSETTS US
+F WHITE IRISH SOMERSET MASSACHUSETTS US
+F WHITE PORTUGUESE FUNCHAL MADEIRA PT
+F WHITE PORTUGUESE BELMONT MASSACHUSETTS US
+F WHITE PORTUGUESE PORTO DOURO LITORAL PT
+F WHITE PORTUGUESE MARSHFIELD MASSACHUSETTS US
+F WHITE PORTUGUESE FARO ALGARVE PT
+F WHITE SCOTTISH EAST LONGMEADOW MASSACHUSETTS US
+F WHITE SCOTTISH WINDSOR MASSACHUSETTS US
+F WHITE SCOTTISH BOSTON MASSACHUSETTS US
+F WHITE POLISH TAUNTON MASSACHUSETTS US
+F WHITE POLISH CONCORD MASSACHUSETTS US
+F BLACK DOMINICAN SALISBURY SAINT JOSEPH PARISH DM
+F BLACK DOMINICAN FALL RIVER MASSACHUSETTS US
+F BLACK DOMINICAN MARIGOT SAINT ANDREW PARISH DM
+F NATIVE AMERICAN_INDIAN STONEHAM MASSACHUSETTS US
+F ASIAN CHINESE WENHAM MASSACHUSETTS US
+# lets try it out on a few examples
+grouped_patients = group_file_by_list('../data/csv/patients.csv', ['GENDER'])
+print_keys(grouped_patients)
+M
+F
+grouped_patients = group_file_by_list('../data/csv/patients.csv', ['RACE'])
+print_keys(grouped_patients)
+WHITE
+HISPANIC
+ASIAN
+NATIVE
+BLACK
+grouped_patients = group_file_by_list('../data/csv/patients.csv', ['GENDER', 'RACE'])
+print_keys(grouped_patients)
+M WHITE
+M ASIAN
+M NATIVE
+M HISPANIC
+M BLACK
+F HISPANIC
+F WHITE
+F BLACK
+F NATIVE
+F ASIAN
+grouped_patients = group_file_by_list('../data/csv/patients.csv', ['RACE', 'GENDER'])
+print_keys(grouped_patients)
+WHITE M
+WHITE F
+HISPANIC F
+HISPANIC M
+ASIAN M
+ASIAN F
+NATIVE M
+NATIVE F
+BLACK M
+BLACK F
+grouped_patients = group_file_by_list('../data/csv/patients.csv', ['ETHNICITY'])
+print_keys(grouped_patients)
+POLISH
+PUERTO_RICAN
+ENGLISH
+FRENCH
+ITALIAN
+PORTUGUESE
+IRISH
+GERMAN
+CHINESE
+FRENCH_CANADIAN
+AMERICAN_INDIAN
+SCOTTISH
+DOMINICAN
+SOUTH_AMERICAN
+GREEK
+AMERICAN
+WEST_INDIAN
+MEXICAN
+ASIAN_INDIAN
+grouped_patients = group_file_by_list('../data/csv/patients.csv', ['RACE', 'ETHNICITY'])
+print_keys(grouped_patients)
+WHITE POLISH
+WHITE ENGLISH
+WHITE FRENCH
+WHITE ITALIAN
+WHITE PORTUGUESE
+WHITE IRISH
+WHITE GERMAN
+WHITE FRENCH_CANADIAN
+WHITE SCOTTISH
+WHITE GREEK
+WHITE AMERICAN
+HISPANIC PUERTO_RICAN
+HISPANIC SOUTH_AMERICAN
+HISPANIC MEXICAN
+ASIAN CHINESE
+ASIAN ASIAN_INDIAN
+NATIVE AMERICAN_INDIAN
+BLACK DOMINICAN
+BLACK WEST_INDIAN
+grouped_patients = group_file_by_list('../data/csv/patients.csv', ['RACE', 'ETHNICITY', 'BIRTHPLACE', 'GENDER'])
+print_keys(grouped_patients)
+WHITE POLISH NEWTON MASSACHUSETTS US M
+WHITE POLISH TAUNTON MASSACHUSETTS US F
+WHITE POLISH ADAMS MASSACHUSETTS US M
+WHITE POLISH BOSTON MASSACHUSETTS US M
+WHITE POLISH CONCORD MASSACHUSETTS US F
+WHITE POLISH HOLYOKE MASSACHUSETTS US M
+WHITE ENGLISH DANVERS MASSACHUSETTS US F
+WHITE ENGLISH BELCHERTOWN MASSACHUSETTS US F
+WHITE ENGLISH HAVERHILL MASSACHUSETTS US M
+WHITE ENGLISH BOSTON MASSACHUSETTS US M
+WHITE ENGLISH SHARON MASSACHUSETTS US M
+WHITE ENGLISH BEVERLY MASSACHUSETTS US M
+WHITE ENGLISH HARVARD MASSACHUSETTS US F
+WHITE ENGLISH ATTLEBORO MASSACHUSETTS US M
+WHITE ENGLISH NEW BEDFORD MASSACHUSETTS US M
+WHITE ENGLISH STERLING MASSACHUSETTS US F
+WHITE ENGLISH WORCESTER MASSACHUSETTS US M
+WHITE ENGLISH SOMERVILLE MASSACHUSETTS US M
+WHITE ENGLISH NATICK MASSACHUSETTS US F
+WHITE ENGLISH NATICK MASSACHUSETTS US M
+WHITE ENGLISH HOPKINTON MASSACHUSETTS US F
+WHITE ENGLISH CHICOPEE MASSACHUSETTS US M
+WHITE ENGLISH REVERE MASSACHUSETTS US M
+WHITE ENGLISH RUTLAND MASSACHUSETTS US M
+WHITE ENGLISH PLYMOUTH MASSACHUSETTS US M
+WHITE ENGLISH NORTHAMPTON MASSACHUSETTS US M
+WHITE ENGLISH DARTMOUTH MASSACHUSETTS US M
+WHITE ENGLISH WHATELY MASSACHUSETTS US M
+WHITE ENGLISH SOUTHWICK MASSACHUSETTS US M
+WHITE FRENCH CHICOPEE MASSACHUSETTS US F
+WHITE FRENCH STOUGHTON MASSACHUSETTS US F
+WHITE FRENCH SHREWSBURY MASSACHUSETTS US M
+WHITE FRENCH BOSTON MASSACHUSETTS US F
+WHITE FRENCH SALEM MASSACHUSETTS US F
+WHITE FRENCH CONCORD MASSACHUSETTS US F
+WHITE ITALIAN MILFORD MASSACHUSETTS US F
+WHITE ITALIAN SPRINGFIELD MASSACHUSETTS US F
+WHITE ITALIAN FALL RIVER MASSACHUSETTS US M
+WHITE ITALIAN HAVERHILL MASSACHUSETTS US M
+WHITE ITALIAN BOSTON MASSACHUSETTS US M
+WHITE ITALIAN SUDBURY MASSACHUSETTS US M
+WHITE ITALIAN SOMERVILLE MASSACHUSETTS US M
+WHITE ITALIAN MANSFIELD MASSACHUSETTS US M
+WHITE ITALIAN BELCHERTOWN MASSACHUSETTS US M
+WHITE ITALIAN NORTH ATTLEBOROUGH MASSACHUSETTS US M
+WHITE ITALIAN FAIRHAVEN MASSACHUSETTS US M
+WHITE ITALIAN MIDDLETON MASSACHUSETTS US M
+WHITE ITALIAN MILTON MASSACHUSETTS US F
+WHITE ITALIAN BEVERLY MASSACHUSETTS US M
+WHITE ITALIAN DRACUT MASSACHUSETTS US F
+WHITE ITALIAN PALERMO SICILY IT F
+WHITE ITALIAN NEW BEDFORD MASSACHUSETTS US F
+WHITE ITALIAN SOUTHWICK MASSACHUSETTS US F
+WHITE PORTUGUESE PORTO DOURO LITORAL PT M
+WHITE PORTUGUESE PORTO DOURO LITORAL PT F
+WHITE PORTUGUESE BRAGA MINHO PT M
+WHITE PORTUGUESE FUNCHAL MADEIRA PT F
+WHITE PORTUGUESE BELMONT MASSACHUSETTS US F
+WHITE PORTUGUESE MARSHFIELD MASSACHUSETTS US F
+WHITE PORTUGUESE FARO ALGARVE PT F
+WHITE PORTUGUESE YARMOUTH MASSACHUSETTS US M
+WHITE IRISH CHICOPEE MASSACHUSETTS US M
+WHITE IRISH WORCESTER MASSACHUSETTS US M
+WHITE IRISH LYNN MASSACHUSETTS US F
+WHITE IRISH WESTPORT MASSACHUSETTS US M
+WHITE IRISH FRAMINGHAM MASSACHUSETTS US F
+WHITE IRISH PALMER TOWN MASSACHUSETTS US F
+WHITE IRISH BOSTON MASSACHUSETTS US F
+WHITE IRISH WEYMOUTH TOWN MASSACHUSETTS US M
+WHITE IRISH MASHPEE MASSACHUSETTS US M
+WHITE IRISH PITTSFIELD MASSACHUSETTS US M
+WHITE IRISH BRAINTREE TOWN MASSACHUSETTS US F
+WHITE IRISH SOMERSET MASSACHUSETTS US F
+WHITE GERMAN NORTH ADAMS MASSACHUSETTS US F
+WHITE GERMAN ATTLEBORO MASSACHUSETTS US F
+WHITE GERMAN EASTHAMPTON TOWN MASSACHUSETTS US F
+WHITE GERMAN QUINCY MASSACHUSETTS US F
+WHITE GERMAN REVERE MASSACHUSETTS US F
+WHITE GERMAN WALTHAM MASSACHUSETTS US M
+WHITE FRENCH_CANADIAN BILLERICA MASSACHUSETTS US F
+WHITE FRENCH_CANADIAN LENOX MASSACHUSETTS US M
+WHITE FRENCH_CANADIAN STONEHAM MASSACHUSETTS US F
+WHITE FRENCH_CANADIAN BOSTON MASSACHUSETTS US M
+WHITE FRENCH_CANADIAN BOSTON MASSACHUSETTS US F
+WHITE FRENCH_CANADIAN SHERBORN MASSACHUSETTS US M
+WHITE FRENCH_CANADIAN HUDSON MASSACHUSETTS US M
+WHITE FRENCH_CANADIAN SANDWICH MASSACHUSETTS US M
+WHITE FRENCH_CANADIAN WORCESTER MASSACHUSETTS US M
+WHITE FRENCH_CANADIAN HOLYOKE MASSACHUSETTS US F
+WHITE FRENCH_CANADIAN ROCKLAND MASSACHUSETTS US F
+WHITE FRENCH_CANADIAN AGAWAM TOWN MASSACHUSETTS US F
+WHITE FRENCH_CANADIAN WEST SPRINGFIELD TOWN MASSACHUSETTS US F
+WHITE FRENCH_CANADIAN LOWELL MASSACHUSETTS US F
+WHITE FRENCH_CANADIAN PALMER TOWN MASSACHUSETTS US F
+WHITE SCOTTISH EAST LONGMEADOW MASSACHUSETTS US F
+WHITE SCOTTISH PITTSFIELD MASSACHUSETTS US M
+WHITE SCOTTISH WINDSOR MASSACHUSETTS US F
+WHITE SCOTTISH BOSTON MASSACHUSETTS US F
+WHITE GREEK ARLINGTON MASSACHUSETTS US M
+WHITE GREEK WORCESTER MASSACHUSETTS US M
+WHITE AMERICAN SPRINGFIELD MASSACHUSETTS US M
+HISPANIC PUERTO_RICAN IPSWICH MASSACHUSETTS US F
+HISPANIC PUERTO_RICAN TEMPLETON MASSACHUSETTS US M
+HISPANIC PUERTO_RICAN BAYAMON PUERTO RICO PR F
+HISPANIC PUERTO_RICAN BAYAMON PUERTO RICO PR M
+HISPANIC PUERTO_RICAN CAROLINA PUERTO RICO PR F
+HISPANIC PUERTO_RICAN CAROLINA PUERTO RICO PR M
+HISPANIC PUERTO_RICAN LOWELL MASSACHUSETTS US M
+HISPANIC PUERTO_RICAN PONCE PUERTO RICO PR M
+HISPANIC SOUTH_AMERICAN RIO DE JANERIO RIO DE JANERIO BR M
+HISPANIC MEXICAN LA PAZ BAJA CALIFORNIA MX M
+ASIAN CHINESE QUINCY MASSACHUSETTS US M
+ASIAN CHINESE WENHAM MASSACHUSETTS US F
+ASIAN ASIAN_INDIAN ROCKLAND MASSACHUSETTS US M
+NATIVE AMERICAN_INDIAN NEW BEDFORD MASSACHUSETTS US M
+NATIVE AMERICAN_INDIAN STONEHAM MASSACHUSETTS US F
+BLACK DOMINICAN UXBRIDGE MASSACHUSETTS US M
+BLACK DOMINICAN SALISBURY SAINT JOSEPH PARISH DM F
+BLACK DOMINICAN FALL RIVER MASSACHUSETTS US F
+BLACK DOMINICAN DARTMOUTH MASSACHUSETTS US M
+BLACK DOMINICAN MARIGOT SAINT ANDREW PARISH DM F
+BLACK WEST_INDIAN SANTIAGO DE LOS CABALLEROS SANTIAGO DO M
+BLACK WEST_INDIAN KINGSTON KINGSTON JM M
+
+First we’re going to define a method to get the patient’s age from their birthdate and then use that function to create a list of patient ages.
import csv
+
+from datetime import date, timedelta
+from dateutil import parser
+
+def get_age(birth_date):
+ if isinstance(birth_date, str):
+ birth_date = parser.parse(birth_date).date()
+
+ age = (date.today() - birth_date) // timedelta(days=365)
+ return age
+# calculate a list of patient ages
+patient_ages = []
+with open('../data/csv/patients.csv') as f:
+ reader = csv.DictReader(f)
+ for i, row in enumerate(reader):
+ date_of_birth = row['BIRTHDATE']
+ age = get_age(date_of_birth)
+ patient_ages.append(age)
+print(patient_ages)
+[6, 18, 13, 17, 8, 9, 55, 84, 14, 20, 53, 23, 18, 64, 11, 39, 34, 5, 82, 19, 18, 15, 26, 28, 29, 8, 71, 32, 82, 41, 51, 17, 40, 53, 60, 65, 11, 21, 32, 52, 3, 52, 82, 30, 69, 24, 99, 12, 51, 99, 27, 76, 78, 29, 16, 69, 34, 78, 58, 76, 99, 65, 70, 60, 50, 32, 4, 78, 65, 48, 3, 66, 77, 99, 25, 18, 47, 79, 99, 32, 10, 74, 61, 99, 62, 59, 67, 2, 50, 99, 3, 70, 61, 62, 43, 18, 71, 29, 99, 52, 71, 26, 28, 56, 32, 69, 23, 31, 46, 99, 106, 30, 21, 67, 67, 106, 99, 106, 99, 106, 106, 99, 106, 99, 99, 99, 99, 90, 90]
+# find min and max with a for loop:
+min_age = 0
+max_age = 0
+
+for i, age in enumerate(patient_ages):
+ if i == 0:
+ min_age = age
+ max_age = age
+
+ if age < min_age:
+ min_age = age
+ if age > max_age:
+ max_age = age
+
+print(f'Min Age: {min_age}, Max Age: {max_age}')
+Min Age: 2, Max Age: 106
+# find min and max with a for loop:
+min_age = patient_ages[0]
+max_age = patient_ages[0]
+
+for age in patient_ages:
+ if age < min_age:
+ min_age = age
+ if age > max_age:
+ max_age = age
+
+print(f'Min Age: {min_age}, Max Age: {max_age}')
+Min Age: 2, Max Age: 106
+# find min and max by sorting
+sorted_ages = sorted(patient_ages)
+min_age = sorted_ages[0]
+max_age = sorted_ages[-1]
+print(f'Min Age: {min_age}, Max Age: {max_age}')
+Min Age: 2, Max Age: 106
+# find min and max with reduce
+from functools import reduce
+
+min_age = reduce(lambda a, b : a if a < b else b, patient_ages)
+max_age = reduce(lambda a, b : a if a > b else b, patient_ages)
+print(f'Min Age: {min_age}, Max Age: {max_age}')
+Min Age: 2, Max Age: 106
+# find min and max with min and max
+min_age = min(patient_ages)
+max_age = max(patient_ages)
+print(f'Min Age: {min_age}, Max Age: {max_age}')
+Min Age: 2, Max Age: 106
+# calculate mean with a for loop
+total_age = 0
+age_count = 0
+
+for age in patient_ages:
+ total_age += age
+ age_count += 1
+
+mean_age = total_age / age_count
+print(f'Mean Age: {mean_age}')
+Mean Age: 52.62015503875969
+# calculate mean age with reduce
+import functools
+total_age = functools.reduce(lambda a, b : a + b, patient_ages)
+age_count = len(patient_ages)
+
+mean_age = total_age / age_count
+print(f'Mean Age: {mean_age}')
+Mean Age: 52.62015503875969
+# calculate mean age with sum and len
+mean_age = sum(patient_ages) / len(patient_ages)
+print(f'Mean Age: {mean_age}')
+Mean Age: 52.62015503875969
+# calculate mean with statistics package
+from statistics import mean
+
+mean_age = mean(patient_ages)
+print(f'Mean Age: {mean_age}')
+Mean Age: 52.62015503875969
+# calculate median with a for loop
+from math import floor
+
+sorted_ages = sorted(patient_ages)
+
+midpoint = len(sorted_ages) / 2
+
+if midpoint.is_integer():
+ high_index = floor(midpoint)
+ low_index = high_index - 1
+ median_age = (sorted_ages[high_index] + sorted_ages[low_index]) / 2
+else:
+ median_age = sorted_ages[floor(midpoint)]
+
+print(f'Median Age: {median_age}')
+Median Age: 52
+# calculate median with statistics package
+from statistics import median
+
+median_age = median(patient_ages)
+print(f'Median Age: {median_age}')
+Median Age: 52
+# calculate variance with a for loop
+mean_age = mean(patient_ages)
+squares = []
+
+for age in patient_ages:
+ squares.append((age - mean_age)**2)
+
+age_variance = mean(squares)
+print(f'Variance: {age_variance}')
+Variance: 995.2433147046452
+# calculate variance with a list comprehension
+mean_age = mean(patient_ages)
+squares = [(age - mean_age)**2 for age in patient_ages]
+age_variance = mean(squares)
+print(f'Variance: {age_variance}')
+Variance: 995.2433147046452
+# calculate variance with map and reduce
+from functools import reduce
+
+mean_age = mean(patient_ages)
+
+squares = map(lambda x: (x - mean_age) ** 2, patient_ages)
+age_variance = reduce(lambda a, b: a + b, squares) / len(patient_ages)
+print(f'Variance: {age_variance}')
+Variance: 995.2433147046452
+# calculate variance with statistics package
+from statistics import pvariance
+age_variance = pvariance(patient_ages)
+print(f'Populaiton Variance: {age_variance}')
+Populaiton Variance: 995.2433147046452
+# calculate variance with statistics package
+from statistics import variance
+age_variance = variance(patient_ages)
+print(f'Saple Variance: {age_variance}')
+Saple Variance: 1003.0186531007752
+# calculate variance with statistics package
+import math
+from statistics import pvariance
+
+age_variance = pvariance(patient_ages)
+standard_devation = math.sqrt(age_variance)
+print(f'Standard Deviation: {standard_devation}')
+Standard Deviation: 31.547477152771584
+# calculate variance with statistics package
+from statistics import pstdev
+
+standard_devation = pstdev(patient_ages)
+print(f'Standard Deviation: {standard_devation}')
+Standard Deviation: 31.547477152771584
+# calculate standard deviation with statistics package
+from statistics import pstdev
+
+standard_deviation = pstdev(patient_ages)
+print(f'Standard Deviation: {standard_deviation}')
+Standard Deviation: 31.547477152771584
+
+In this lecture we’re going to learn how to use matplotlib and seaborn by following along with the following example. As always, the source author’s link is listed for reference. This page will evolve over time.
The dataset we’ll use here is the Heart Disease Data Set containing 302 patient data each with 75 attributes. However, this example only uses 14 of them which can be seen below.
The columns used include:
columns = ["age",
+ "sex",
+ "cp",
+ "trestbps",
+ "chol",
+ "fbs",
+ "restecg",
+ "thalach",
+ "exang",
+ "oldpeak",
+ "slope",
+ "ca",
+ "thal",
+ "num"]
+# disable warnings for lecture
+import warnings
+warnings.filterwarnings('ignore')
+import pandas as pd
+
+# import the data and see the basic description
+df = pd.read_csv("https://archive.ics.uci.edu/ml/machine-learning-databases/heart-disease/processed.cleveland.data")
+df.columns = columns
+print("---- Describe ----")
+print(df.describe())
+---- Describe ----
+ age sex cp trestbps chol fbs \
+count 302.000000 302.000000 302.000000 302.000000 302.000000 302.000000
+mean 54.410596 0.678808 3.165563 131.645695 246.738411 0.145695
+std 9.040163 0.467709 0.953612 17.612202 51.856829 0.353386
+min 29.000000 0.000000 1.000000 94.000000 126.000000 0.000000
+25% 48.000000 0.000000 3.000000 120.000000 211.000000 0.000000
+50% 55.500000 1.000000 3.000000 130.000000 241.500000 0.000000
+75% 61.000000 1.000000 4.000000 140.000000 275.000000 0.000000
+max 77.000000 1.000000 4.000000 200.000000 564.000000 1.000000
+
+ restecg thalach exang oldpeak slope num
+count 302.000000 302.000000 302.000000 302.000000 302.000000 302.000000
+mean 0.986755 149.605960 0.327815 1.035430 1.596026 0.940397
+std 0.994916 22.912959 0.470196 1.160723 0.611939 1.229384
+min 0.000000 71.000000 0.000000 0.000000 1.000000 0.000000
+25% 0.000000 133.250000 0.000000 0.000000 1.000000 0.000000
+50% 0.500000 153.000000 0.000000 0.800000 2.000000 0.000000
+75% 2.000000 166.000000 1.000000 1.600000 2.000000 2.000000
+max 2.000000 202.000000 1.000000 6.200000 3.000000 4.000000
+print('---- Info -----')
+print(df.info())
+---- Info -----
+<class 'pandas.core.frame.DataFrame'>
+RangeIndex: 302 entries, 0 to 301
+Data columns (total 14 columns):
+ # Column Non-Null Count Dtype
+--- ------ -------------- -----
+ 0 age 302 non-null float64
+ 1 sex 302 non-null float64
+ 2 cp 302 non-null float64
+ 3 trestbps 302 non-null float64
+ 4 chol 302 non-null float64
+ 5 fbs 302 non-null float64
+ 6 restecg 302 non-null float64
+ 7 thalach 302 non-null float64
+ 8 exang 302 non-null float64
+ 9 oldpeak 302 non-null float64
+ 10 slope 302 non-null float64
+ 11 ca 302 non-null object
+ 12 thal 302 non-null object
+ 13 num 302 non-null int64
+dtypes: float64(11), int64(1), object(2)
+memory usage: 33.2+ KB
+None
+We notice above that the ca and thal data elements are objects which we’ll likely want to remap. Let’s take a look at the data.
df['thal'].unique()
+array(['3.0', '7.0', '6.0', '?'], dtype=object)
+df['ca'].unique()
+array(['3.0', '2.0', '0.0', '1.0', '?'], dtype=object)
+From the codebook above we see these are coded values that we can remap.
# Replace Every Number greater than 0 to 1 to mark heart disease
+df.loc[df['num'] > 0 , 'num'] = 1
+df.ca = pd.to_numeric(df.ca, errors='coerce').fillna(0)
+df.thal = pd.to_numeric(df.thal, errors='coerce').fillna(0)
+df['thal'].unique()
+array([3., 7., 6., 0.])
+df['ca'].unique()
+array([3., 2., 0., 1.])
+Now we can view the datatypes of the remapped data to float64 and int64.
print('---- Dtype ----')
+print(df.dtypes)
+---- Dtype ----
+age float64
+sex float64
+cp float64
+trestbps float64
+chol float64
+fbs float64
+restecg float64
+thalach float64
+exang float64
+oldpeak float64
+slope float64
+ca float64
+thal float64
+num int64
+dtype: object
+Next we’ll want to
print('---- Null Data ----')
+# count how many null values exist
+print(df.isnull().sum())
+---- Null Data ----
+age 0
+sex 0
+cp 0
+trestbps 0
+chol 0
+fbs 0
+restecg 0
+thalach 0
+exang 0
+oldpeak 0
+slope 0
+ca 0
+thal 0
+num 0
+dtype: int64
+# quickly check to see if there are any null values
+print(df.isnull().values.any())
+False
+After doing simple clean up, changing non-numerical value to NaN and replacing NaN with 0 we can safely say our data is somewhat clean.
# print the first 10 and last 10
+print('------ First 10 -------')
+df.head(10)
+------ First 10 -------
+| age | sex | cp | trestbps | chol | fbs | restecg | thalach | exang | oldpeak | slope | ca | thal | num | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 67.0 | 1.0 | 4.0 | 160.0 | 286.0 | 0.0 | 2.0 | 108.0 | 1.0 | 1.5 | 2.0 | 3.0 | 3.0 | 1 |
| 1 | 67.0 | 1.0 | 4.0 | 120.0 | 229.0 | 0.0 | 2.0 | 129.0 | 1.0 | 2.6 | 2.0 | 2.0 | 7.0 | 1 |
| 2 | 37.0 | 1.0 | 3.0 | 130.0 | 250.0 | 0.0 | 0.0 | 187.0 | 0.0 | 3.5 | 3.0 | 0.0 | 3.0 | 0 |
| 3 | 41.0 | 0.0 | 2.0 | 130.0 | 204.0 | 0.0 | 2.0 | 172.0 | 0.0 | 1.4 | 1.0 | 0.0 | 3.0 | 0 |
| 4 | 56.0 | 1.0 | 2.0 | 120.0 | 236.0 | 0.0 | 0.0 | 178.0 | 0.0 | 0.8 | 1.0 | 0.0 | 3.0 | 0 |
| 5 | 62.0 | 0.0 | 4.0 | 140.0 | 268.0 | 0.0 | 2.0 | 160.0 | 0.0 | 3.6 | 3.0 | 2.0 | 3.0 | 1 |
| 6 | 57.0 | 0.0 | 4.0 | 120.0 | 354.0 | 0.0 | 0.0 | 163.0 | 1.0 | 0.6 | 1.0 | 0.0 | 3.0 | 0 |
| 7 | 63.0 | 1.0 | 4.0 | 130.0 | 254.0 | 0.0 | 2.0 | 147.0 | 0.0 | 1.4 | 2.0 | 1.0 | 7.0 | 1 |
| 8 | 53.0 | 1.0 | 4.0 | 140.0 | 203.0 | 1.0 | 2.0 | 155.0 | 1.0 | 3.1 | 3.0 | 0.0 | 7.0 | 1 |
| 9 | 57.0 | 1.0 | 4.0 | 140.0 | 192.0 | 0.0 | 0.0 | 148.0 | 0.0 | 0.4 | 2.0 | 0.0 | 6.0 | 0 |
# Last 10
+print('------ Last 10 -------')
+df.tail(10)
+------ Last 10 -------
+| age | sex | cp | trestbps | chol | fbs | restecg | thalach | exang | oldpeak | slope | ca | thal | num | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 292 | 63.0 | 1.0 | 4.0 | 140.0 | 187.0 | 0.0 | 2.0 | 144.0 | 1.0 | 4.0 | 1.0 | 2.0 | 7.0 | 1 |
| 293 | 63.0 | 0.0 | 4.0 | 124.0 | 197.0 | 0.0 | 0.0 | 136.0 | 1.0 | 0.0 | 2.0 | 0.0 | 3.0 | 1 |
| 294 | 41.0 | 1.0 | 2.0 | 120.0 | 157.0 | 0.0 | 0.0 | 182.0 | 0.0 | 0.0 | 1.0 | 0.0 | 3.0 | 0 |
| 295 | 59.0 | 1.0 | 4.0 | 164.0 | 176.0 | 1.0 | 2.0 | 90.0 | 0.0 | 1.0 | 2.0 | 2.0 | 6.0 | 1 |
| 296 | 57.0 | 0.0 | 4.0 | 140.0 | 241.0 | 0.0 | 0.0 | 123.0 | 1.0 | 0.2 | 2.0 | 0.0 | 7.0 | 1 |
| 297 | 45.0 | 1.0 | 1.0 | 110.0 | 264.0 | 0.0 | 0.0 | 132.0 | 0.0 | 1.2 | 2.0 | 0.0 | 7.0 | 1 |
| 298 | 68.0 | 1.0 | 4.0 | 144.0 | 193.0 | 1.0 | 0.0 | 141.0 | 0.0 | 3.4 | 2.0 | 2.0 | 7.0 | 1 |
| 299 | 57.0 | 1.0 | 4.0 | 130.0 | 131.0 | 0.0 | 0.0 | 115.0 | 1.0 | 1.2 | 2.0 | 1.0 | 7.0 | 1 |
| 300 | 57.0 | 0.0 | 2.0 | 130.0 | 236.0 | 0.0 | 2.0 | 174.0 | 0.0 | 0.0 | 2.0 | 1.0 | 3.0 | 1 |
| 301 | 38.0 | 1.0 | 3.0 | 138.0 | 175.0 | 0.0 | 0.0 | 173.0 | 0.0 | 0.0 | 1.0 | 0.0 | 3.0 | 0 |
After reviewing the data in tabular form we want to visualize all of the data across the variables. We can do this easily with a histogram.
# import matplotlib
+import matplotlib.pyplot as plt
+%matplotlib inline
+# using pandas to generate the plots
+df.hist()
+
+# using matplotlib to render (or show) the plot
+plt.show()
+
# get the histogram of every data points
+fig = plt.figure(figsize = (18, 18))
+ax = fig.gca()
+
+df.hist(ax=ax, bins=30)
+plt.show()
+
With simple histogram of our data, we can easily observe the distribution of different attributes. One thing to note here is the fact that it is extremely easy for us to see which attributes are categorical values and which are not.
We can inspect a little bit more closely and take a look at the distribution of ages and fbs (fasting blood sugar). We can see that the age distribution is closely resembling of Gaussian distribution while fbs is a categorical value.
# import seaborn
+import seaborn as sns
+
+# a closer look at age
+plt.figure(figsize=(8, 8))
+sns.distplot(df.age)
+plt.show()
+plt.close('all')
+
# a closer look at fbs
+plt.figure(figsize=(8, 8))
+sns.distplot(df.fbs)
+plt.show()
+
We can calculate variance-covariance matrices in a number of ways. First we’ll use Numpy and then we’ll use the built-in Dataframe functrion. Once calculated, we can observe that most attributes do not have a strong covariance relationship.
import numpy as np
+from numpy import dot
+
+# calculate the Variance-Covariance Matrix
+sample = df.values
+sample = sample - dot(np.ones((sample.shape[0],sample.shape[0])),sample)/(len(sample)-1)
+covv = dot(sample.T,sample)/(len(sample)-1)
+plt.figure(figsize=(8,8))
+sns.heatmap(covv)
+plt.show()
+
# compare with built in
+plt.figure(figsize=(8,8))
+sns.heatmap(df.cov())
+plt.show()
+
Similarly, the first image is created by manual numpy calculation and the second using the bulit-in method. Ee can observe that among the attributes there are actually strong correlation with one another. (especially heart disease and thal).
# calculate correaltion matrix
+sample = df.values
+certering_mat = np.diag(np.ones((302))) - np.ones((302,302))/302
+std_matrix = np.diag(np.std(sample,0))
+temp = dot(certering_mat,dot(sample, np.linalg.inv(std_matrix) ))
+temp = dot(temp.T,temp)/len(sample)
+
+# plot
+plt.figure(figsize=(13, 13))
+sns.heatmap(np.around(temp,2),annot=True,fmt=".2f",cmap="Blues",annot_kws={"size": 15})
+plt.show()
+
# correaltion matrix
+sns.set(font_scale=2)
+plt.figure(figsize=(13,13))
+
+sns.heatmap(df.corr().round(2),annot=True,fmt=".2f",cmap="Blues",annot_kws={"size": 15})
+plt.show()
+
# plot the people who have heart vs not
+plt.figure(figsize=(13, 13))
+sns.distplot(df.age[df.num==0], label='No Disease', color='blue')
+sns.distplot(df.age[df.num==1], label='Disease', color='Red')
+sns.distplot(df.trestbps[df.num==0],label= 'No Disease', color='Green')
+sns.distplot(df.trestbps[df.num==1], label='Disease', color='violet')
+plt.legend()
+plt.show()
+
%matplotlib inline
+import pygal
+from IPython.display import SVG, HTML
+html_pygal = """
+<!DOCTYPE html>
+<html>
+ <head>
+ <script type="text/javascript" src="http://kozea.github.com/pygal.js/javascripts/svg.jquery.js"></script>
+ <script type="text/javascript" src="http://kozea.github.com/pygal.js/javascripts/pygal-tooltips.js"></script>
+ <!-- ... -->
+ </head>
+ <body>
+ <figure>
+ {pygal_render}
+ </figure>
+ </body>
+</html>
+"""
+
+hist = pygal.Histogram()
+
+count, division = np.histogram(df.age[df.num==0].values,bins=100)
+temp = []
+for c,div in zip(count,division):
+ temp.append((c,div,div+1))
+
+count, division = np.histogram(df.age[df.num==1].values,bins=100)
+temp1 = []
+for c,div in zip(count,division):
+ temp1.append((c,div,div+1))
+
+count, division = np.histogram(df.trestbps[df.num==0].values,bins=100)
+temp2 = []
+for c,div in zip(count,division):
+ temp2.append((c,div,div+1))
+
+count, division = np.histogram(df.trestbps[df.num==1].values,bins=100)
+temp3 = []
+for c,div in zip(count,division):
+ temp3.append((c,div,div+1))
+
+hist.add('No Disease age', temp)
+hist.add('Disease age', temp1)
+hist.add('No Disease ', temp2)
+hist.add('Disease', temp3)
+hist.render()
+HTML(html_pygal.format(pygal_render=hist.render()))
+<!DOCTYPE html>
| \n", + " | age | \n", + "sex | \n", + "cp | \n", + "trestbps | \n", + "chol | \n", + "fbs | \n", + "restecg | \n", + "thalach | \n", + "exang | \n", + "oldpeak | \n", + "slope | \n", + "ca | \n", + "thal | \n", + "num | \n", + "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", + "67.0 | \n", + "1.0 | \n", + "4.0 | \n", + "160.0 | \n", + "286.0 | \n", + "0.0 | \n", + "2.0 | \n", + "108.0 | \n", + "1.0 | \n", + "1.5 | \n", + "2.0 | \n", + "3.0 | \n", + "3.0 | \n", + "1 | \n", + "
| 1 | \n", + "67.0 | \n", + "1.0 | \n", + "4.0 | \n", + "120.0 | \n", + "229.0 | \n", + "0.0 | \n", + "2.0 | \n", + "129.0 | \n", + "1.0 | \n", + "2.6 | \n", + "2.0 | \n", + "2.0 | \n", + "7.0 | \n", + "1 | \n", + "
| 2 | \n", + "37.0 | \n", + "1.0 | \n", + "3.0 | \n", + "130.0 | \n", + "250.0 | \n", + "0.0 | \n", + "0.0 | \n", + "187.0 | \n", + "0.0 | \n", + "3.5 | \n", + "3.0 | \n", + "0.0 | \n", + "3.0 | \n", + "0 | \n", + "
| 3 | \n", + "41.0 | \n", + "0.0 | \n", + "2.0 | \n", + "130.0 | \n", + "204.0 | \n", + "0.0 | \n", + "2.0 | \n", + "172.0 | \n", + "0.0 | \n", + "1.4 | \n", + "1.0 | \n", + "0.0 | \n", + "3.0 | \n", + "0 | \n", + "
| 4 | \n", + "56.0 | \n", + "1.0 | \n", + "2.0 | \n", + "120.0 | \n", + "236.0 | \n", + "0.0 | \n", + "0.0 | \n", + "178.0 | \n", + "0.0 | \n", + "0.8 | \n", + "1.0 | \n", + "0.0 | \n", + "3.0 | \n", + "0 | \n", + "
| 5 | \n", + "62.0 | \n", + "0.0 | \n", + "4.0 | \n", + "140.0 | \n", + "268.0 | \n", + "0.0 | \n", + "2.0 | \n", + "160.0 | \n", + "0.0 | \n", + "3.6 | \n", + "3.0 | \n", + "2.0 | \n", + "3.0 | \n", + "1 | \n", + "
| 6 | \n", + "57.0 | \n", + "0.0 | \n", + "4.0 | \n", + "120.0 | \n", + "354.0 | \n", + "0.0 | \n", + "0.0 | \n", + "163.0 | \n", + "1.0 | \n", + "0.6 | \n", + "1.0 | \n", + "0.0 | \n", + "3.0 | \n", + "0 | \n", + "
| 7 | \n", + "63.0 | \n", + "1.0 | \n", + "4.0 | \n", + "130.0 | \n", + "254.0 | \n", + "0.0 | \n", + "2.0 | \n", + "147.0 | \n", + "0.0 | \n", + "1.4 | \n", + "2.0 | \n", + "1.0 | \n", + "7.0 | \n", + "1 | \n", + "
| 8 | \n", + "53.0 | \n", + "1.0 | \n", + "4.0 | \n", + "140.0 | \n", + "203.0 | \n", + "1.0 | \n", + "2.0 | \n", + "155.0 | \n", + "1.0 | \n", + "3.1 | \n", + "3.0 | \n", + "0.0 | \n", + "7.0 | \n", + "1 | \n", + "
| 9 | \n", + "57.0 | \n", + "1.0 | \n", + "4.0 | \n", + "140.0 | \n", + "192.0 | \n", + "0.0 | \n", + "0.0 | \n", + "148.0 | \n", + "0.0 | \n", + "0.4 | \n", + "2.0 | \n", + "0.0 | \n", + "6.0 | \n", + "0 | \n", + "
| \n", + " | age | \n", + "sex | \n", + "cp | \n", + "trestbps | \n", + "chol | \n", + "fbs | \n", + "restecg | \n", + "thalach | \n", + "exang | \n", + "oldpeak | \n", + "slope | \n", + "ca | \n", + "thal | \n", + "num | \n", + "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 292 | \n", + "63.0 | \n", + "1.0 | \n", + "4.0 | \n", + "140.0 | \n", + "187.0 | \n", + "0.0 | \n", + "2.0 | \n", + "144.0 | \n", + "1.0 | \n", + "4.0 | \n", + "1.0 | \n", + "2.0 | \n", + "7.0 | \n", + "1 | \n", + "
| 293 | \n", + "63.0 | \n", + "0.0 | \n", + "4.0 | \n", + "124.0 | \n", + "197.0 | \n", + "0.0 | \n", + "0.0 | \n", + "136.0 | \n", + "1.0 | \n", + "0.0 | \n", + "2.0 | \n", + "0.0 | \n", + "3.0 | \n", + "1 | \n", + "
| 294 | \n", + "41.0 | \n", + "1.0 | \n", + "2.0 | \n", + "120.0 | \n", + "157.0 | \n", + "0.0 | \n", + "0.0 | \n", + "182.0 | \n", + "0.0 | \n", + "0.0 | \n", + "1.0 | \n", + "0.0 | \n", + "3.0 | \n", + "0 | \n", + "
| 295 | \n", + "59.0 | \n", + "1.0 | \n", + "4.0 | \n", + "164.0 | \n", + "176.0 | \n", + "1.0 | \n", + "2.0 | \n", + "90.0 | \n", + "0.0 | \n", + "1.0 | \n", + "2.0 | \n", + "2.0 | \n", + "6.0 | \n", + "1 | \n", + "
| 296 | \n", + "57.0 | \n", + "0.0 | \n", + "4.0 | \n", + "140.0 | \n", + "241.0 | \n", + "0.0 | \n", + "0.0 | \n", + "123.0 | \n", + "1.0 | \n", + "0.2 | \n", + "2.0 | \n", + "0.0 | \n", + "7.0 | \n", + "1 | \n", + "
| 297 | \n", + "45.0 | \n", + "1.0 | \n", + "1.0 | \n", + "110.0 | \n", + "264.0 | \n", + "0.0 | \n", + "0.0 | \n", + "132.0 | \n", + "0.0 | \n", + "1.2 | \n", + "2.0 | \n", + "0.0 | \n", + "7.0 | \n", + "1 | \n", + "
| 298 | \n", + "68.0 | \n", + "1.0 | \n", + "4.0 | \n", + "144.0 | \n", + "193.0 | \n", + "1.0 | \n", + "0.0 | \n", + "141.0 | \n", + "0.0 | \n", + "3.4 | \n", + "2.0 | \n", + "2.0 | \n", + "7.0 | \n", + "1 | \n", + "
| 299 | \n", + "57.0 | \n", + "1.0 | \n", + "4.0 | \n", + "130.0 | \n", + "131.0 | \n", + "0.0 | \n", + "0.0 | \n", + "115.0 | \n", + "1.0 | \n", + "1.2 | \n", + "2.0 | \n", + "1.0 | \n", + "7.0 | \n", + "1 | \n", + "
| 300 | \n", + "57.0 | \n", + "0.0 | \n", + "2.0 | \n", + "130.0 | \n", + "236.0 | \n", + "0.0 | \n", + "2.0 | \n", + "174.0 | \n", + "0.0 | \n", + "0.0 | \n", + "2.0 | \n", + "1.0 | \n", + "3.0 | \n", + "1 | \n", + "
| 301 | \n", + "38.0 | \n", + "1.0 | \n", + "3.0 | \n", + "138.0 | \n", + "175.0 | \n", + "0.0 | \n", + "0.0 | \n", + "173.0 | \n", + "0.0 | \n", + "0.0 | \n", + "1.0 | \n", + "0.0 | \n", + "3.0 | \n", + "0 | \n", + "
A NumPy tutorial for beginners in which you’ll learn how to create a NumPy array, use broadcasting, access values, manipulate arrays, and much more.
NumPy is, just like SciPy, Scikit-Learn, Pandas, etc. one of the packages that you just can’t miss when you’re learning data science, mainly because this library provides you with an array data structure that holds some benefits over Python lists, such as: being more compact, faster access in reading and writing items, being more convenient and more efficient.
Today we’ll focus precisely on this. This NumPy tutorial will not only show you what NumPy arrays actually are and how you can install Python, but you’ll also learn how to make arrays (even when your data comes from files!), how broadcasting works, how you can ask for help, how to manipulate your arrays and how to visualize them.
You already read in the introduction that NumPy arrays are a bit like Python lists, but still very much different at the same time. For those of you who are new to the topic, let’s clarify what it exactly is and what it’s good for.
As the name gives away, a NumPy array is a central data structure of the numpy library. The library’s name is short for “Numeric Python” or “Numerical Python”.
This already gives an idea of what you’re dealing with, right?
In other words, NumPy is a Python library that is the core library for scientific computing in Python. It contains a collection of tools and techniques that can be used to solve on a computer mathematical models of problems in Science and Engineering. One of these tools is a high-performance multidimensional array object that is a powerful data structure for efficient computation of arrays and matrices. To work with these arrays, there’s a vast amount of high-level mathematical functions operate on these matrices and arrays.
Then, what is an array?
When you look at the print of a couple of arrays, you could see it as a grid that contains values of the same type:
# import the library
+ 03.a - NumPy Introduction - Data Focused Python 03.a - NumPy Introduction | Data Focused Python Python Numpy Tutorial
A NumPy tutorial for beginners in which you’ll learn how to create a NumPy array, use broadcasting, access values, manipulate arrays, and much more.
NumPy is, just like SciPy, Scikit-Learn, Pandas, etc. one of the packages that you just can’t miss when you’re learning data science, mainly because this library provides you with an array data structure that holds some benefits over Python lists, such as: being more compact, faster access in reading and writing items, being more convenient and more efficient.
Today we’ll focus precisely on this. This NumPy tutorial will not only show you what NumPy arrays actually are and how you can install Python, but you’ll also learn how to make arrays (even when your data comes from files!), how broadcasting works, how you can ask for help, how to manipulate your arrays and how to visualize them.
Content
- What Is A Python Numpy Array?
- How To Make NumPy Arrays
- How NumPy Broadcasting Works
- How Do Array Mathematics Work?
- How To Subset, Slice, And Index Arrays
- How To Manipulate Arrays
- How To Visualize NumPy Arrays
- Beyond Data Analysis with NumPy
What Is A Python Numpy Array?
You already read in the introduction that NumPy arrays are a bit like Python lists, but still very much different at the same time. For those of you who are new to the topic, let’s clarify what it exactly is and what it’s good for.
As the name gives away, a NumPy array is a central data structure of the numpy library. The library’s name is short for “Numeric Python” or “Numerical Python”.
This already gives an idea of what you’re dealing with, right?
In other words, NumPy is a Python library that is the core library for scientific computing in Python. It contains a collection of tools and techniques that can be used to solve on a computer mathematical models of problems in Science and Engineering. One of these tools is a high-performance multidimensional array object that is a powerful data structure for efficient computation of arrays and matrices. To work with these arrays, there’s a vast amount of high-level mathematical functions operate on these matrices and arrays.
Then, what is an array?
When you look at the print of a couple of arrays, you could see it as a grid that contains values of the same type:
# import the library
import numpy as np
# create a 1-dimensional array
my_array = np.array([1, 2, 3, 4, 5, 6, 7, 8 ,9, 10, 11, 12])
@@ -29,7 +29,7 @@
# Print out the stride of `my_array`
print('Strides', my_2d_array.strides)
-
Memory Address <memory at 0x1077c1c18>
+
Memory Address <memory at 0x7faf50c48040>
Shape (3, 4)
Data Type int64
Strides (32, 8)
@@ -59,13 +59,13 @@
[0, 0, 0, 0]]], dtype=int16)
# Create an array with random values
np.random.random((2, 2))
-
array([[0.41062006, 0.41917488],
- [0.04339049, 0.77983037]])
+
array([[0.30514961, 0.53408847],
+ [0.5699706 , 0.14205311]])
# Create an empty array
np.empty((3, 2))
-
array([[3.5e-323, 0.0e+000],
- [0.0e+000, 0.0e+000],
- [0.0e+000, 0.0e+000]])
+
array([[1.72723371e-077, 1.72723371e-077],
+ [1.38338381e-322, 0.00000000e+000],
+ [0.00000000e+000, 0.00000000e+000]])
# Create a full array
np.full((2, 2), 6)
array([[6, 6],
@@ -149,17 +149,17 @@
[1. 1. 1. 1.]
[1. 1. 1. 1.]]
(3, 4)
-[[0.10876151 0.31258275 0.65749282 0.71050253]
- [0.90199242 0.08553991 0.91084083 0.53919521]
- [0.39947315 0.37419746 0.32398661 0.31425255]]
+[[0.56519198 0.79797203 0.29302353 0.42378585]
+ [0.20190445 0.37107571 0.50090031 0.36828103]
+ [0.48683869 0.81527133 0.01208389 0.50830169]]
-array([[1.10876151, 1.31258275, 1.65749282, 1.71050253],
- [1.90199242, 1.08553991, 1.91084083, 1.53919521],
- [1.39947315, 1.37419746, 1.32398661, 1.31425255]])
+array([[1.56519198, 1.79797203, 1.29302353, 1.42378585],
+ [1.20190445, 1.37107571, 1.50090031, 1.36828103],
+ [1.48683869, 1.81527133, 1.01208389, 1.50830169]])
- Two dimensions are also compatible when one of them is 1:
# Initialize `x`
x = np.ones((3, 4))
@@ -196,130 +196,130 @@
# Add `x` and `y`
x + y
-
array([[[1.9728683 , 1.96023453, 1.56830623, 1.55632128],
- [1.9728683 , 1.96023453, 1.56830623, 1.55632128],
- [1.9728683 , 1.96023453, 1.56830623, 1.55632128]],
+
array([[[1.83525793, 1.84535921, 1.23908934, 1.51636293],
+ [1.83525793, 1.84535921, 1.23908934, 1.51636293],
+ [1.83525793, 1.84535921, 1.23908934, 1.51636293]],
- [[1.06007097, 1.79532407, 1.89726724, 1.33797925],
- [1.06007097, 1.79532407, 1.89726724, 1.33797925],
- [1.06007097, 1.79532407, 1.89726724, 1.33797925]],
+ [[1.96363415, 1.53040604, 1.28514896, 1.55804427],
+ [1.96363415, 1.53040604, 1.28514896, 1.55804427],
+ [1.96363415, 1.53040604, 1.28514896, 1.55804427]],
- [[1.88084817, 1.66069619, 1.13297031, 1.21667871],
- [1.88084817, 1.66069619, 1.13297031, 1.21667871],
- [1.88084817, 1.66069619, 1.13297031, 1.21667871]],
+ [[1.40773584, 1.34067363, 1.26809999, 1.42783587],
+ [1.40773584, 1.34067363, 1.26809999, 1.42783587],
+ [1.40773584, 1.34067363, 1.26809999, 1.42783587]],
- [[1.4631949 , 1.33212101, 1.82583021, 1.80744998],
- [1.4631949 , 1.33212101, 1.82583021, 1.80744998],
- [1.4631949 , 1.33212101, 1.82583021, 1.80744998]],
+ [[1.24076123, 1.67608402, 1.70843067, 1.96111856],
+ [1.24076123, 1.67608402, 1.70843067, 1.96111856],
+ [1.24076123, 1.67608402, 1.70843067, 1.96111856]],
- [[1.26845678, 1.23624019, 1.85528042, 1.14757416],
- [1.26845678, 1.23624019, 1.85528042, 1.14757416],
- [1.26845678, 1.23624019, 1.85528042, 1.14757416]]])
+ [[1.46894321, 1.37966024, 1.56355702, 1.29747049],
+ [1.46894321, 1.37966024, 1.56355702, 1.29747049],
+ [1.46894321, 1.37966024, 1.56355702, 1.29747049]]])
You see that, even though x and y seem to have somewhat different dimensions, the two can be added together.
That is because they are compatible in all dimensions:
- Array
x has dimensions 3 X 4, - Array
y has dimensions 5 X 1 X 4
Since you have seen above that dimensions are also compatible if one of them is equal to 1, you see that these two arrays are indeed a good candidate for broadcasting!
What you will notice is that in the dimension where y has size 1, and the other array has a size greater than 1 (that is, 3), the first array behaves as if it were copied along that dimension.
Note that the shape of the resulting array will again be the maximum size along each dimension of x and y: the dimension of the result will be (5,3,4)
In short, if you want to make use of broadcasting, you will rely a lot on the shape and dimensions of the arrays with which you’re working.
But what if the dimensions are not compatible?
What if they are not equal or if one of them is not equal to 1?
You’ll have to fix this by manipulating your array! You’ll see how to do this in one of the next sections.
How Do Array Mathematics Work?
You’ve seen that broadcasting is handy when you’re doing arithmetic operations. In this section, you’ll discover some of the functions that you can use to do mathematics with arrays.
As such, it probably won’t surprise you that you can just use +, -, *, / or % to add, subtract, multiply, divide or calculate the remainder of two (or more) arrays. However, a big part of why NumPy is so handy, is because it also has functions to do this. The equivalent functions of the operations that you have seen just now are, respectively, np.add(), np.subtract(), np.multiply(), np.divide() and np.remainder().
You can also easily do exponentiation and taking the square root of your arrays with np.exp() and np.sqrt(), or calculate the sines or cosines of your array with np.sin() and np.cos(). Lastly, its’ also useful to mention that there’s also a way for you to calculate the natural logarithm with np.log() or calculate the dot product by applying the dot() to your array.
# Add `x` and `y`
np.add(x, y)
-
array([[[1.9728683 , 1.96023453, 1.56830623, 1.55632128],
- [1.9728683 , 1.96023453, 1.56830623, 1.55632128],
- [1.9728683 , 1.96023453, 1.56830623, 1.55632128]],
+
array([[[1.83525793, 1.84535921, 1.23908934, 1.51636293],
+ [1.83525793, 1.84535921, 1.23908934, 1.51636293],
+ [1.83525793, 1.84535921, 1.23908934, 1.51636293]],
- [[1.06007097, 1.79532407, 1.89726724, 1.33797925],
- [1.06007097, 1.79532407, 1.89726724, 1.33797925],
- [1.06007097, 1.79532407, 1.89726724, 1.33797925]],
+ [[1.96363415, 1.53040604, 1.28514896, 1.55804427],
+ [1.96363415, 1.53040604, 1.28514896, 1.55804427],
+ [1.96363415, 1.53040604, 1.28514896, 1.55804427]],
- [[1.88084817, 1.66069619, 1.13297031, 1.21667871],
- [1.88084817, 1.66069619, 1.13297031, 1.21667871],
- [1.88084817, 1.66069619, 1.13297031, 1.21667871]],
+ [[1.40773584, 1.34067363, 1.26809999, 1.42783587],
+ [1.40773584, 1.34067363, 1.26809999, 1.42783587],
+ [1.40773584, 1.34067363, 1.26809999, 1.42783587]],
- [[1.4631949 , 1.33212101, 1.82583021, 1.80744998],
- [1.4631949 , 1.33212101, 1.82583021, 1.80744998],
- [1.4631949 , 1.33212101, 1.82583021, 1.80744998]],
+ [[1.24076123, 1.67608402, 1.70843067, 1.96111856],
+ [1.24076123, 1.67608402, 1.70843067, 1.96111856],
+ [1.24076123, 1.67608402, 1.70843067, 1.96111856]],
- [[1.26845678, 1.23624019, 1.85528042, 1.14757416],
- [1.26845678, 1.23624019, 1.85528042, 1.14757416],
- [1.26845678, 1.23624019, 1.85528042, 1.14757416]]])
+ [[1.46894321, 1.37966024, 1.56355702, 1.29747049],
+ [1.46894321, 1.37966024, 1.56355702, 1.29747049],
+ [1.46894321, 1.37966024, 1.56355702, 1.29747049]]])
# Subtract `x` and `y`
np.subtract(x, y)
-
array([[[0.0271317 , 0.03976547, 0.43169377, 0.44367872],
- [0.0271317 , 0.03976547, 0.43169377, 0.44367872],
- [0.0271317 , 0.03976547, 0.43169377, 0.44367872]],
+
array([[[0.16474207, 0.15464079, 0.76091066, 0.48363707],
+ [0.16474207, 0.15464079, 0.76091066, 0.48363707],
+ [0.16474207, 0.15464079, 0.76091066, 0.48363707]],
- [[0.93992903, 0.20467593, 0.10273276, 0.66202075],
- [0.93992903, 0.20467593, 0.10273276, 0.66202075],
- [0.93992903, 0.20467593, 0.10273276, 0.66202075]],
+ [[0.03636585, 0.46959396, 0.71485104, 0.44195573],
+ [0.03636585, 0.46959396, 0.71485104, 0.44195573],
+ [0.03636585, 0.46959396, 0.71485104, 0.44195573]],
- [[0.11915183, 0.33930381, 0.86702969, 0.78332129],
- [0.11915183, 0.33930381, 0.86702969, 0.78332129],
- [0.11915183, 0.33930381, 0.86702969, 0.78332129]],
+ [[0.59226416, 0.65932637, 0.73190001, 0.57216413],
+ [0.59226416, 0.65932637, 0.73190001, 0.57216413],
+ [0.59226416, 0.65932637, 0.73190001, 0.57216413]],
- [[0.5368051 , 0.66787899, 0.17416979, 0.19255002],
- [0.5368051 , 0.66787899, 0.17416979, 0.19255002],
- [0.5368051 , 0.66787899, 0.17416979, 0.19255002]],
+ [[0.75923877, 0.32391598, 0.29156933, 0.03888144],
+ [0.75923877, 0.32391598, 0.29156933, 0.03888144],
+ [0.75923877, 0.32391598, 0.29156933, 0.03888144]],
- [[0.73154322, 0.76375981, 0.14471958, 0.85242584],
- [0.73154322, 0.76375981, 0.14471958, 0.85242584],
- [0.73154322, 0.76375981, 0.14471958, 0.85242584]]])
+ [[0.53105679, 0.62033976, 0.43644298, 0.70252951],
+ [0.53105679, 0.62033976, 0.43644298, 0.70252951],
+ [0.53105679, 0.62033976, 0.43644298, 0.70252951]]])
# Multiply `x` and `y`
np.multiply(x, y)
-
array([[[0.9728683 , 0.96023453, 0.56830623, 0.55632128],
- [0.9728683 , 0.96023453, 0.56830623, 0.55632128],
- [0.9728683 , 0.96023453, 0.56830623, 0.55632128]],
+
array([[[0.83525793, 0.84535921, 0.23908934, 0.51636293],
+ [0.83525793, 0.84535921, 0.23908934, 0.51636293],
+ [0.83525793, 0.84535921, 0.23908934, 0.51636293]],
- [[0.06007097, 0.79532407, 0.89726724, 0.33797925],
- [0.06007097, 0.79532407, 0.89726724, 0.33797925],
- [0.06007097, 0.79532407, 0.89726724, 0.33797925]],
+ [[0.96363415, 0.53040604, 0.28514896, 0.55804427],
+ [0.96363415, 0.53040604, 0.28514896, 0.55804427],
+ [0.96363415, 0.53040604, 0.28514896, 0.55804427]],
- [[0.88084817, 0.66069619, 0.13297031, 0.21667871],
- [0.88084817, 0.66069619, 0.13297031, 0.21667871],
- [0.88084817, 0.66069619, 0.13297031, 0.21667871]],
+ [[0.40773584, 0.34067363, 0.26809999, 0.42783587],
+ [0.40773584, 0.34067363, 0.26809999, 0.42783587],
+ [0.40773584, 0.34067363, 0.26809999, 0.42783587]],
- [[0.4631949 , 0.33212101, 0.82583021, 0.80744998],
- [0.4631949 , 0.33212101, 0.82583021, 0.80744998],
- [0.4631949 , 0.33212101, 0.82583021, 0.80744998]],
+ [[0.24076123, 0.67608402, 0.70843067, 0.96111856],
+ [0.24076123, 0.67608402, 0.70843067, 0.96111856],
+ [0.24076123, 0.67608402, 0.70843067, 0.96111856]],
- [[0.26845678, 0.23624019, 0.85528042, 0.14757416],
- [0.26845678, 0.23624019, 0.85528042, 0.14757416],
- [0.26845678, 0.23624019, 0.85528042, 0.14757416]]])
+ [[0.46894321, 0.37966024, 0.56355702, 0.29747049],
+ [0.46894321, 0.37966024, 0.56355702, 0.29747049],
+ [0.46894321, 0.37966024, 0.56355702, 0.29747049]]])
# Divide `x` and `y`
np.divide(x, y)
-
array([[[ 1.02788836, 1.04141225, 1.75961469, 1.79752246],
- [ 1.02788836, 1.04141225, 1.75961469, 1.79752246],
- [ 1.02788836, 1.04141225, 1.75961469, 1.79752246]],
+
array([[[1.19723497, 1.18292908, 4.18253694, 1.93662236],
+ [1.19723497, 1.18292908, 4.18253694, 1.93662236],
+ [1.19723497, 1.18292908, 4.18253694, 1.93662236]],
- [[16.64697694, 1.2573491 , 1.11449517, 2.95876155],
- [16.64697694, 1.2573491 , 1.11449517, 2.95876155],
- [16.64697694, 1.2573491 , 1.11449517, 2.95876155]],
+ [[1.03773824, 1.88534807, 3.50693892, 1.79197254],
+ [1.03773824, 1.88534807, 3.50693892, 1.79197254],
+ [1.03773824, 1.88534807, 3.50693892, 1.79197254]],
- [[ 1.13526943, 1.51355497, 7.5204758 , 4.61512799],
- [ 1.13526943, 1.51355497, 7.5204758 , 4.61512799],
- [ 1.13526943, 1.51355497, 7.5204758 , 4.61512799]],
+ [[2.45256831, 2.93536074, 3.72995169, 2.33734491],
+ [2.45256831, 2.93536074, 3.72995169, 2.33734491],
+ [2.45256831, 2.93536074, 3.72995169, 2.33734491]],
- [[ 2.15891843, 3.01095073, 1.21090266, 1.23846682],
- [ 2.15891843, 3.01095073, 1.21090266, 1.23846682],
- [ 2.15891843, 3.01095073, 1.21090266, 1.23846682]],
+ [[4.15349266, 1.47910611, 1.41157073, 1.04045436],
+ [4.15349266, 1.47910611, 1.41157073, 1.04045436],
+ [4.15349266, 1.47910611, 1.41157073, 1.04045436]],
- [[ 3.72499439, 4.23298002, 1.16920717, 6.77625391],
- [ 3.72499439, 4.23298002, 1.16920717, 6.77625391],
- [ 3.72499439, 4.23298002, 1.16920717, 6.77625391]]])
+ [[2.13245436, 2.63393394, 1.77444335, 3.36167802],
+ [2.13245436, 2.63393394, 1.77444335, 3.36167802],
+ [2.13245436, 2.63393394, 1.77444335, 3.36167802]]])
# Calculate the remainder of `x` and `y`
np.remainder(x, y)
-
array([[[0.0271317 , 0.03976547, 0.43169377, 0.44367872],
- [0.0271317 , 0.03976547, 0.43169377, 0.44367872],
- [0.0271317 , 0.03976547, 0.43169377, 0.44367872]],
+
array([[[0.16474207, 0.15464079, 0.04364264, 0.48363707],
+ [0.16474207, 0.15464079, 0.04364264, 0.48363707],
+ [0.16474207, 0.15464079, 0.04364264, 0.48363707]],
- [[0.03886453, 0.20467593, 0.10273276, 0.32404151],
- [0.03886453, 0.20467593, 0.10273276, 0.32404151],
- [0.03886453, 0.20467593, 0.10273276, 0.32404151]],
+ [[0.03636585, 0.46959396, 0.14455311, 0.44195573],
+ [0.03636585, 0.46959396, 0.14455311, 0.44195573],
+ [0.03636585, 0.46959396, 0.14455311, 0.44195573]],
- [[0.11915183, 0.33930381, 0.06920783, 0.13328514],
- [0.11915183, 0.33930381, 0.06920783, 0.13328514],
- [0.11915183, 0.33930381, 0.06920783, 0.13328514]],
+ [[0.18452832, 0.31865274, 0.19570004, 0.14432825],
+ [0.18452832, 0.31865274, 0.19570004, 0.14432825],
+ [0.18452832, 0.31865274, 0.19570004, 0.14432825]],
- [[0.0736102 , 0.00363697, 0.17416979, 0.19255002],
- [0.0736102 , 0.00363697, 0.17416979, 0.19255002],
- [0.0736102 , 0.00363697, 0.17416979, 0.19255002]],
+ [[0.03695508, 0.32391598, 0.29156933, 0.03888144],
+ [0.03695508, 0.32391598, 0.29156933, 0.03888144],
+ [0.03695508, 0.32391598, 0.29156933, 0.03888144]],
- [[0.19462966, 0.05503924, 0.14471958, 0.11455502],
- [0.19462966, 0.05503924, 0.14471958, 0.11455502],
- [0.19462966, 0.05503924, 0.14471958, 0.11455502]]])
+ [[0.06211358, 0.24067951, 0.43644298, 0.10758854],
+ [0.06211358, 0.24067951, 0.43644298, 0.10758854],
+ [0.06211358, 0.24067951, 0.43644298, 0.10758854]]])
Remember how broadcasting works? Check out the dimensions and the shapes of both x and y in your IPython shell. Are the rules of broadcasting respected?
But there is more.
Check out this small list of aggregate functions:
Function Description a.sum()Array-wise sum a.min()Array-wise minimum value b.max(axis=0)Maximum value of an array row b.cumsum(axis=1)Cumulative sum of the elements a.mean()Mean b.median()Median a.corrcoef()Correlation coefficient np.std(b)Standard deviation
Besides all of these functions, you might also find it useful to know that there are mechanisms that allow you to compare array elements. For example, if you want to check whether the elements of two arrays are the same, you might use the == operator. To check whether the array elements are smaller or bigger, you use the < or > operators.
This all seems quite straightforward, yes?
However, you can also compare entire arrays with each other! In this case, you use the np.array_equal() function. Just pass in the two arrays that you want to compare with each other, and you’re done.
Note that, besides comparing, you can also perform logical operations on your arrays. You can start with np.logical_or(), np.logical_not() and np.logical_and(). This basically works like your typical OR, NOT and AND logical operations;
In the simplest example, you use OR to see whether your elements are the same (for example, 1), or if one of the two array elements is 1. If both of them are 0, you’ll return FALSE. You would use AND to see whether your second element is also 1 and NOT to see if the second element differs from 1.
# `x` AND `y`
np.logical_and(x, y)
array([[[ True, True, True, True],
@@ -584,8 +584,7 @@
# Show the plot
plt.show()
-
<Figure size 640x480 with 1 Axes>
-
Using np.meshgrid()
Another way to (indirectly) visualize your array is by using np.meshgrid(). The problem that you face with arrays is that you need 2-D arrays of x and y coordinate values. With the above function, you can create a rectangular grid out of an array of x values and an array of y values: the np.meshgrid() function takes two 1D arrays and produces two 2D matrices corresponding to all pairs of (x, y) in the two arrays. Then, you can use these matrices to make all sorts of plots.
np.meshgrid() is particularly useful if you want to evaluate functions on a grid, as the code below demonstrates:
# Import NumPy and Matplotlib
+
Using np.meshgrid()
Another way to (indirectly) visualize your array is by using np.meshgrid(). The problem that you face with arrays is that you need 2-D arrays of x and y coordinate values. With the above function, you can create a rectangular grid out of an array of x values and an array of y values: the np.meshgrid() function takes two 1D arrays and produces two 2D matrices corresponding to all pairs of (x, y) in the two arrays. Then, you can use these matrices to make all sorts of plots.
np.meshgrid() is particularly useful if you want to evaluate functions on a grid, as the code below demonstrates:
# Import NumPy and Matplotlib
import numpy as np
import matplotlib.pyplot as plt
@@ -605,4 +604,4 @@
# Show the plot
plt.show()
-
+
\n",
+ "\n",
+ "The array that you see above is, as its name already suggested, a 2-dimensional array: you have rows and columns. The rows are indicated as the “axis 0”, while the columns are the “axis 1”. The number of the axis goes up accordingly with the number of the dimensions: in 3-D arrays, of which you have also seen an example in the previous code chunk, you’ll have an additional “axis 2”. Note that these axes are only valid for arrays that have at least 2 dimensions, as there is no point in having this for 1-D arrays;\n",
+ "\n",
+ "These axes will come in handy later when you’re manipulating the shape of your NumPy arrays."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## How To Make NumPy Arrays\n",
+ "\n",
+ "To make a numpy array, you can just use the ```np.array()``` function. All you need to do is pass a list to it, and optionally, you can also specify the data type of the data. If you want to know more about the possible data types that you can pick, [go](https://docs.scipy.org/doc/numpy/user/basics.types.html) here or consider taking a brief look at [DataCamp’s NumPy cheat sheet](https://www.datacamp.com/community/blog/python-numpy-cheat-sheet/).\n",
+ "\n",
+ "There’s no need to go and memorize these NumPy data types if you’re a new user; But you do have to know and care what data you’re dealing with. The data types are there when you need more control over how your data is stored in memory and on disk. Especially in cases where you’re working with extensive data, it’s good that you know to control the storage type.\n",
+ "\n",
+ "Don’t forget that, in order to work with the ```np.array()``` function, you need to make sure that the ```numpy``` library is present in your environment. The NumPy library follows an import convention: when you import this library, you have to make sure that you import it as ```np```. By doing this, you’ll make sure that other Pythonistas understand your code more easily.\n",
+ "\n",
+ "In the following example you’ll create the my_array array that you have already played around with above:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "[[1 2 3 4]\n",
+ " [5 6 7 8]] int32\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Import `numpy` as `np`\n",
+ "import numpy as np\n",
+ "\n",
+ "# Make the array `my_array`\n",
+ "my_array = np.array([[1,2,3,4], [5,6,7,8]], dtype=np.int32)\n",
+ "\n",
+ "# Print `my_array`\n",
+ "print(my_array, my_array.dtype)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "However, sometimes you don’t know what data you want to put in your array, or you want to import data into a numpy array from another source. In those cases, you’ll make use of initial placeholders or functions to load data from text into arrays, respectively.\n",
+ "\n",
+ "The following sections will show you how to do this."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### How To Make An “Empty” NumPy Array\n",
+ "\n",
+ "What people often mean when they say that they are creating “empty” arrays is that they want to make use of initial placeholders, which you can fill up afterward. You can initialize arrays with ones or zeros, but you can also create arrays that get filled up with evenly spaced values, constant or random values.\n",
+ "\n",
+ "However, you can still make a totally empty array, too.\n",
+ "\n",
+ "Luckily for us, there are quite a lot of functions to make\n",
+ "\n",
+ "Try it all out below!"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "array([[1., 1., 1., 1.],\n",
+ " [1., 1., 1., 1.],\n",
+ " [1., 1., 1., 1.]])"
+ ]
+ },
+ "execution_count": 7,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "# Create an array of ones\n",
+ "np.ones((3, 4))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "array([[[0, 0, 0, 0],\n",
+ " [0, 0, 0, 0],\n",
+ " [0, 0, 0, 0]],\n",
+ "\n",
+ " [[0, 0, 0, 0],\n",
+ " [0, 0, 0, 0],\n",
+ " [0, 0, 0, 0]]], dtype=int16)"
+ ]
+ },
+ "execution_count": 8,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "# Create an array of zeros\n",
+ "np.zeros((2, 3, 4), dtype=np.int16)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "array([[0.30514961, 0.53408847],\n",
+ " [0.5699706 , 0.14205311]])"
+ ]
+ },
+ "execution_count": 9,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "# Create an array with random values\n",
+ "np.random.random((2, 2))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "array([[1.72723371e-077, 1.72723371e-077],\n",
+ " [1.38338381e-322, 0.00000000e+000],\n",
+ " [0.00000000e+000, 0.00000000e+000]])"
+ ]
+ },
+ "execution_count": 10,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "# Create an empty array\n",
+ "np.empty((3, 2))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 11,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "array([[6, 6],\n",
+ " [6, 6]])"
+ ]
+ },
+ "execution_count": 11,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "# Create a full array\n",
+ "np.full((2, 2), 6)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 12,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "[10 15 20 25 30]\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Create an array of evenly-spaced values\n",
+ "print(np.arange(10, 31, 5))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 13,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "array([0. , 0.25, 0.5 , 0.75, 1. , 1.25, 1.5 , 1.75, 2. ])"
+ ]
+ },
+ "execution_count": 13,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "# Create an array of evenly-spaced values\n",
+ "np.linspace(0, 2, 9)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "* For some, such as ```np.ones()```, ```np.random.random()```, ```np.empty()```, ```np.full()``` or ```np.zeros()``` the only thing that you need to do in order to make arrays with ones or zeros is pass the shape of the array that you want to make. As an option to ```np.ones()``` and ```np.zeros()```, you can also specify the data type. In the case of ```np.full()```, you also have to specify the constant value that you want to insert into the array.\n",
+ "* With ```np.linspace()``` and ```np.arange()``` you can make arrays of evenly spaced values. The difference between these two functions is that the last value of the three that are passed in the code chunk above designates either the step value for np.linspace() or a number of samples for np.arange(). What happens in the first is that you want, for example, an array of 9 values that lie between 0 and 2. For the latter, you specify that you want an array to start at 10 and per steps of 5, generate values for the array that you’re creating.\n",
+ "* Remember that NumPy also allows you to create an identity array or matrix with ```np.eye()``` and ```np.identity()```. An identity matrix is a square matrix of which all elements in the principal diagonal are ones, and all other elements are zeros. When you multiply a matrix with an identity matrix, the given matrix is left unchanged."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### How To Load NumPy Arrays From Text\n",
+ "\n",
+ "Creating arrays with the help of initial placeholders or with some example data is an excellent way of getting started with ```numpy```. But when you want to get started with data analysis, you’ll need to load data from text files.\n",
+ "\n",
+ "With that what you have seen up until now, you won’t really be able to do much. Make use of some specific functions to load data from your files, such as ```loadtxt()``` or ```genfromtxt()```.\n",
+ "\n",
+ "Let’s say you have the following text files with data:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 14,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "[0.2536 0.4839 0.1292 0.1781 0.6253 0.6253]\n",
+ "[0.1008 0.4536 0.6875 0.3049 0.3486 0.3486]\n",
+ "[0.3857 0.3561 0.5929 0.8928 0.8791 0.8791]\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Import your data\n",
+ "x, y, z = np.loadtxt('data.txt',\n",
+ " skiprows=1,\n",
+ " unpack=True)\n",
+ "print(x)\n",
+ "print(y)\n",
+ "print(z)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "In the code above, you use ```loadtxt()``` to load the data in your environment. You see that the first argument that both functions take is the text file ```data.txt```. Next, there are some specific arguments for each: in the first statement, you skip the first row, and you return the columns as separate arrays with ```unpack=True```. This means that the values in column ```Value1``` will be put in ```x```, and so on.\n",
+ "\n",
+ "Note that, in case you have comma-delimited data or if you want to specify the data type, there are also the arguments ```delimiter``` and ```dtype``` that you can add to the ```loadtxt()``` arguments."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 15,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "[[ 4.839e-01 4.536e-01 3.561e-01]\n",
+ " [ 1.292e-01 6.875e-01 -9.990e+02]\n",
+ " [ 1.781e-01 3.049e-01 8.928e-01]\n",
+ " [-9.990e+02 5.801e-01 2.038e-01]\n",
+ " [ 5.993e-01 4.357e-01 7.410e-01]]\n"
+ ]
+ }
+ ],
+ "source": [
+ "my_array2 = np.genfromtxt('data2.txt',\n",
+ " skip_header=1,\n",
+ " filling_values=-999)\n",
+ "print(my_array2)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "You see that here, you resort to ```genfromtxt()``` to load the data. In this case, you have to handle some missing values that are indicated by the ```'MISSING'``` strings. Since the ```genfromtxt()``` function converts character strings in numeric columns to ```nan```, you can convert these values to other ones by specifying the filling_values argument. In this case, you choose to set the value of these missing values to -999.\n",
+ "\n",
+ "If by any chance, you have values that don’t get converted to ```nan``` by ```genfromtxt()```, there’s always the ```missing_values``` argument that allows you to specify what the missing values of your data exactly are.\n",
+ "\n",
+ "But this is not all.\n",
+ "\n",
+ "Tip: check out [this page](https://docs.scipy.org/doc/numpy/reference/generated/numpy.genfromtxt.html#numpy.genfromtxt) to see what other arguments you can add to import your data successfully.\n",
+ "\n",
+ "You now might wonder what the difference between these two functions really is.\n",
+ "\n",
+ "The examples indicated this maybe implicitly, but, in general, ```genfromtxt()``` gives you a little bit more flexibility; It’s more robust than ```loadtxt()```.\n",
+ "\n",
+ "Let’s make this difference a little bit more practical: the latter, loadtxt(), only works when each row in the text file has the same number of values; So when you want to handle missing values easily, you’ll typically find it easier to use genfromtxt().\n",
+ "\n",
+ "But this is definitely not the only reason.\n",
+ "\n",
+ "A brief look on the number of arguments that ```genfromtxt()``` has to offer will teach you that there is really a lot more things that you can specify in your import, such as the maximum number of rows to read or the option to automatically strip white spaces from variables."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### How To Save NumPy Arrays\n",
+ "Once you have done everything that you need to do with your arrays, you can also save them to a file. If you want to save the array to a text file, you can use the ```savetxt()``` function to do this:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 16,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "[0. 1. 2. 3. 4.]\n"
+ ]
+ }
+ ],
+ "source": [
+ "x = np.arange(0.0, 5.0, 1.0)\n",
+ "print(x)\n",
+ "np.savetxt('test.out', x, delimiter=',')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Remember that ```np.arange()``` creates a NumPy array of evenly-spaced values. The third value that you pass to this function is the step value.\n",
+ "\n",
+ "There are, of course, other ways to save your NumPy arrays to text files. Check out the functions in the table below if you want to get your data to binary files or archives:\n",
+ "\n",
+ "| Method | Description |\n",
+ "| --- | --- |\n",
+ "| ```save()``` | Save an array to a binary file in NumPy .npy format |\n",
+ "| ```savez()``` | Save several arrays into an uncompressed .npz archive |\n",
+ "| ```savez_compressed()``` | Save several arrays into a compressed .npz archive |"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### How To Inspect Your NumPy Arrays\n",
+ "\n",
+ "Besides the array attributes that have been mentioned above, namely, data, shape, dtype and strides, there are some more that you can use to easily get to know more about your arrays. The ones that you might find interesting to use when you’re just starting out are the following:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 17,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "2\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Print the number of `my_array`'s dimensions\n",
+ "print(my_array.ndim)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 18,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "8\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Print the number of `my_array`'s elements\n",
+ "print(my_array.size)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 19,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ " C_CONTIGUOUS : True\n",
+ " F_CONTIGUOUS : False\n",
+ " OWNDATA : True\n",
+ " WRITEABLE : True\n",
+ " ALIGNED : True\n",
+ " WRITEBACKIFCOPY : False\n",
+ " UPDATEIFCOPY : False\n",
+ "\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Print information about `my_array`'s memory layout\n",
+ "print(my_array.flags)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 20,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "4\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Print the length of one array element in bytes\n",
+ "print(my_array.itemsize)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 21,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "32\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Print the total consumed bytes by `my_array`'s elements\n",
+ "print(my_array.nbytes)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Also note that, besides the attributes, you also have some other ways of gaining more information on and even tweaking your array slightly:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 22,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "2\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Print the length of `my_array`\n",
+ "print(len(my_array))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 23,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "[[1. 2. 3. 4.]\n",
+ " [5. 6. 7. 8.]]\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Change the data type of `my_array`\n",
+ "my_float_array = my_array.astype(float)\n",
+ "print(my_float_array)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## How NumPy Broadcasting Works\n",
+ "\n",
+ "Before you go deeper into scientific computing, it might be a good idea to first go over what broadcasting exactly is: it’s a mechanism that allows NumPy to work with arrays of different shapes when you’re performing arithmetic operations.\n",
+ "\n",
+ "To put it in a more practical context, you often have an array that’s somewhat larger and another one that’s slightly smaller. Ideally, you want to use the smaller array multiple times to perform an operation (such as a sum, multiplication, etc.) on the larger array.\n",
+ "\n",
+ "To do this, you use the broadcasting mechanism.\n",
+ "\n",
+ "However, there are some rules if you want to use it. And, before you already sigh, you’ll see that these “rules” are very simple and kind of straightforward!\n",
+ "\n",
+ "* First off, to make sure that the broadcasting is successful, the dimensions of your arrays need to be compatible. Two dimensions are compatible when they are equal. Consider the following example:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 24,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "(3, 4)\n",
+ "[[1. 1. 1. 1.]\n",
+ " [1. 1. 1. 1.]\n",
+ " [1. 1. 1. 1.]]\n",
+ "(3, 4)\n",
+ "[[0.56519198 0.79797203 0.29302353 0.42378585]\n",
+ " [0.20190445 0.37107571 0.50090031 0.36828103]\n",
+ " [0.48683869 0.81527133 0.01208389 0.50830169]]\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "array([[1.56519198, 1.79797203, 1.29302353, 1.42378585],\n",
+ " [1.20190445, 1.37107571, 1.50090031, 1.36828103],\n",
+ " [1.48683869, 1.81527133, 1.01208389, 1.50830169]])"
+ ]
+ },
+ "execution_count": 24,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "# Initialize `x`\n",
+ "x = np.ones((3, 4))\n",
+ "\n",
+ "# Check shape of `x`\n",
+ "print(x.shape)\n",
+ "print(x)\n",
+ "\n",
+ "# Initialize `y`\n",
+ "y = np.random.random((3, 4))\n",
+ "\n",
+ "# Check shape of `y`\n",
+ "print(y.shape)\n",
+ "print(y)\n",
+ "\n",
+ "# Add `x` and `y`\n",
+ "x + y"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "* Two dimensions are also compatible when one of them is 1:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 25,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "(3, 4)\n",
+ "[[1. 1. 1. 1.]\n",
+ " [1. 1. 1. 1.]\n",
+ " [1. 1. 1. 1.]]\n",
+ "(4,)\n",
+ "[0 1 2 3]\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "array([[ 1., 0., -1., -2.],\n",
+ " [ 1., 0., -1., -2.],\n",
+ " [ 1., 0., -1., -2.]])"
+ ]
+ },
+ "execution_count": 25,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "# Initialize `x`\n",
+ "x = np.ones((3, 4))\n",
+ "\n",
+ "# Check shape of `x`\n",
+ "print(x.shape)\n",
+ "print(x)\n",
+ "\n",
+ "# Initialize `y`\n",
+ "y = np.arange(4)\n",
+ "\n",
+ "# Check shape of `y`\n",
+ "print(y.shape)\n",
+ "print(y)\n",
+ "\n",
+ "# Subtract `x` and `y`\n",
+ "x - y "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Note that if the dimensions are not compatible, you will get a ```ValueError```.\n",
+ "\n",
+ "*Tip: also test what the size of the resulting array is after you have done the computations! You’ll see that the size is actually the maximum size along each dimension of the input arrays.*\n",
+ "\n",
+ "In other words, you see that the result of ```x-y``` gives an array with shape ```(3,4)```: ```y``` had a shape of ```(4,)``` and ```x``` had a shape of ```(3,4)```. The maximum size along each dimension of x and y is taken to make up the shape of the new, resulting array.\n",
+ "\n",
+ "Lastly, the arrays can only be broadcast together if they are compatible in all dimensions. Consider the following example:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 26,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "array([[[1.83525793, 1.84535921, 1.23908934, 1.51636293],\n",
+ " [1.83525793, 1.84535921, 1.23908934, 1.51636293],\n",
+ " [1.83525793, 1.84535921, 1.23908934, 1.51636293]],\n",
+ "\n",
+ " [[1.96363415, 1.53040604, 1.28514896, 1.55804427],\n",
+ " [1.96363415, 1.53040604, 1.28514896, 1.55804427],\n",
+ " [1.96363415, 1.53040604, 1.28514896, 1.55804427]],\n",
+ "\n",
+ " [[1.40773584, 1.34067363, 1.26809999, 1.42783587],\n",
+ " [1.40773584, 1.34067363, 1.26809999, 1.42783587],\n",
+ " [1.40773584, 1.34067363, 1.26809999, 1.42783587]],\n",
+ "\n",
+ " [[1.24076123, 1.67608402, 1.70843067, 1.96111856],\n",
+ " [1.24076123, 1.67608402, 1.70843067, 1.96111856],\n",
+ " [1.24076123, 1.67608402, 1.70843067, 1.96111856]],\n",
+ "\n",
+ " [[1.46894321, 1.37966024, 1.56355702, 1.29747049],\n",
+ " [1.46894321, 1.37966024, 1.56355702, 1.29747049],\n",
+ " [1.46894321, 1.37966024, 1.56355702, 1.29747049]]])"
+ ]
+ },
+ "execution_count": 26,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "# Initialize `x` and `y`\n",
+ "x = np.ones((3, 4))\n",
+ "y = np.random.random((5,1,4))\n",
+ "\n",
+ "# Add `x` and `y`\n",
+ "x + y"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "You see that, even though ```x``` and ```y``` seem to have somewhat different dimensions, the two can be added together.\n",
+ "\n",
+ "That is because they are compatible in all dimensions:\n",
+ "\n",
+ "* Array ```x``` has dimensions 3 X 4,\n",
+ "* Array ```y``` has dimensions 5 X 1 X 4\n",
+ "\n",
+ "Since you have seen above that dimensions are also compatible if one of them is equal to 1, you see that these two arrays are indeed a good candidate for broadcasting!\n",
+ "\n",
+ "What you will notice is that in the dimension where ```y``` has size 1, and the other array has a size greater than 1 (that is, 3), the first array behaves as if it were copied along that dimension.\n",
+ "\n",
+ "Note that the shape of the resulting array will again be the maximum size along each dimension of ```x``` and ```y```: the dimension of the result will be ```(5,3,4)```\n",
+ "\n",
+ "In short, if you want to make use of broadcasting, you will rely a lot on the shape and dimensions of the arrays with which you’re working.\n",
+ "\n",
+ "But what if the dimensions are not compatible?\n",
+ "\n",
+ "What if they are not equal or if one of them is not equal to 1?\n",
+ "\n",
+ "You’ll have to fix this by manipulating your array! You’ll see how to do this in one of the next sections."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## How Do Array Mathematics Work?\n",
+ "\n",
+ "You’ve seen that broadcasting is handy when you’re doing arithmetic operations. In this section, you’ll discover some of the functions that you can use to do mathematics with arrays.\n",
+ "\n",
+ "As such, it probably won’t surprise you that you can just use ```+```, ```-```, ```*```, ```/``` or ```%``` to add, subtract, multiply, divide or calculate the remainder of two (or more) arrays. However, a big part of why NumPy is so handy, is because it also has functions to do this. The equivalent functions of the operations that you have seen just now are, respectively, ```np.add()```, ```np.subtract()```, ```np.multiply()```, ```np.divide()``` and ```np.remainder()```.\n",
+ "\n",
+ "You can also easily do exponentiation and taking the square root of your arrays with ```np.exp()``` and ```np.sqrt()```, or calculate the sines or cosines of your array with ```np.sin()``` and ```np.cos()```. Lastly, its’ also useful to mention that there’s also a way for you to calculate the natural logarithm with ```np.log()``` or calculate the dot product by applying the ```dot()``` to your array."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 27,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "array([[[1.83525793, 1.84535921, 1.23908934, 1.51636293],\n",
+ " [1.83525793, 1.84535921, 1.23908934, 1.51636293],\n",
+ " [1.83525793, 1.84535921, 1.23908934, 1.51636293]],\n",
+ "\n",
+ " [[1.96363415, 1.53040604, 1.28514896, 1.55804427],\n",
+ " [1.96363415, 1.53040604, 1.28514896, 1.55804427],\n",
+ " [1.96363415, 1.53040604, 1.28514896, 1.55804427]],\n",
+ "\n",
+ " [[1.40773584, 1.34067363, 1.26809999, 1.42783587],\n",
+ " [1.40773584, 1.34067363, 1.26809999, 1.42783587],\n",
+ " [1.40773584, 1.34067363, 1.26809999, 1.42783587]],\n",
+ "\n",
+ " [[1.24076123, 1.67608402, 1.70843067, 1.96111856],\n",
+ " [1.24076123, 1.67608402, 1.70843067, 1.96111856],\n",
+ " [1.24076123, 1.67608402, 1.70843067, 1.96111856]],\n",
+ "\n",
+ " [[1.46894321, 1.37966024, 1.56355702, 1.29747049],\n",
+ " [1.46894321, 1.37966024, 1.56355702, 1.29747049],\n",
+ " [1.46894321, 1.37966024, 1.56355702, 1.29747049]]])"
+ ]
+ },
+ "execution_count": 27,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "# Add `x` and `y`\n",
+ "np.add(x, y)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 28,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "array([[[0.16474207, 0.15464079, 0.76091066, 0.48363707],\n",
+ " [0.16474207, 0.15464079, 0.76091066, 0.48363707],\n",
+ " [0.16474207, 0.15464079, 0.76091066, 0.48363707]],\n",
+ "\n",
+ " [[0.03636585, 0.46959396, 0.71485104, 0.44195573],\n",
+ " [0.03636585, 0.46959396, 0.71485104, 0.44195573],\n",
+ " [0.03636585, 0.46959396, 0.71485104, 0.44195573]],\n",
+ "\n",
+ " [[0.59226416, 0.65932637, 0.73190001, 0.57216413],\n",
+ " [0.59226416, 0.65932637, 0.73190001, 0.57216413],\n",
+ " [0.59226416, 0.65932637, 0.73190001, 0.57216413]],\n",
+ "\n",
+ " [[0.75923877, 0.32391598, 0.29156933, 0.03888144],\n",
+ " [0.75923877, 0.32391598, 0.29156933, 0.03888144],\n",
+ " [0.75923877, 0.32391598, 0.29156933, 0.03888144]],\n",
+ "\n",
+ " [[0.53105679, 0.62033976, 0.43644298, 0.70252951],\n",
+ " [0.53105679, 0.62033976, 0.43644298, 0.70252951],\n",
+ " [0.53105679, 0.62033976, 0.43644298, 0.70252951]]])"
+ ]
+ },
+ "execution_count": 28,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "# Subtract `x` and `y`\n",
+ "np.subtract(x, y)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 29,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "array([[[0.83525793, 0.84535921, 0.23908934, 0.51636293],\n",
+ " [0.83525793, 0.84535921, 0.23908934, 0.51636293],\n",
+ " [0.83525793, 0.84535921, 0.23908934, 0.51636293]],\n",
+ "\n",
+ " [[0.96363415, 0.53040604, 0.28514896, 0.55804427],\n",
+ " [0.96363415, 0.53040604, 0.28514896, 0.55804427],\n",
+ " [0.96363415, 0.53040604, 0.28514896, 0.55804427]],\n",
+ "\n",
+ " [[0.40773584, 0.34067363, 0.26809999, 0.42783587],\n",
+ " [0.40773584, 0.34067363, 0.26809999, 0.42783587],\n",
+ " [0.40773584, 0.34067363, 0.26809999, 0.42783587]],\n",
+ "\n",
+ " [[0.24076123, 0.67608402, 0.70843067, 0.96111856],\n",
+ " [0.24076123, 0.67608402, 0.70843067, 0.96111856],\n",
+ " [0.24076123, 0.67608402, 0.70843067, 0.96111856]],\n",
+ "\n",
+ " [[0.46894321, 0.37966024, 0.56355702, 0.29747049],\n",
+ " [0.46894321, 0.37966024, 0.56355702, 0.29747049],\n",
+ " [0.46894321, 0.37966024, 0.56355702, 0.29747049]]])"
+ ]
+ },
+ "execution_count": 29,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "# Multiply `x` and `y`\n",
+ "np.multiply(x, y)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 30,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "array([[[1.19723497, 1.18292908, 4.18253694, 1.93662236],\n",
+ " [1.19723497, 1.18292908, 4.18253694, 1.93662236],\n",
+ " [1.19723497, 1.18292908, 4.18253694, 1.93662236]],\n",
+ "\n",
+ " [[1.03773824, 1.88534807, 3.50693892, 1.79197254],\n",
+ " [1.03773824, 1.88534807, 3.50693892, 1.79197254],\n",
+ " [1.03773824, 1.88534807, 3.50693892, 1.79197254]],\n",
+ "\n",
+ " [[2.45256831, 2.93536074, 3.72995169, 2.33734491],\n",
+ " [2.45256831, 2.93536074, 3.72995169, 2.33734491],\n",
+ " [2.45256831, 2.93536074, 3.72995169, 2.33734491]],\n",
+ "\n",
+ " [[4.15349266, 1.47910611, 1.41157073, 1.04045436],\n",
+ " [4.15349266, 1.47910611, 1.41157073, 1.04045436],\n",
+ " [4.15349266, 1.47910611, 1.41157073, 1.04045436]],\n",
+ "\n",
+ " [[2.13245436, 2.63393394, 1.77444335, 3.36167802],\n",
+ " [2.13245436, 2.63393394, 1.77444335, 3.36167802],\n",
+ " [2.13245436, 2.63393394, 1.77444335, 3.36167802]]])"
+ ]
+ },
+ "execution_count": 30,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "# Divide `x` and `y`\n",
+ "np.divide(x, y)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 31,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "array([[[0.16474207, 0.15464079, 0.04364264, 0.48363707],\n",
+ " [0.16474207, 0.15464079, 0.04364264, 0.48363707],\n",
+ " [0.16474207, 0.15464079, 0.04364264, 0.48363707]],\n",
+ "\n",
+ " [[0.03636585, 0.46959396, 0.14455311, 0.44195573],\n",
+ " [0.03636585, 0.46959396, 0.14455311, 0.44195573],\n",
+ " [0.03636585, 0.46959396, 0.14455311, 0.44195573]],\n",
+ "\n",
+ " [[0.18452832, 0.31865274, 0.19570004, 0.14432825],\n",
+ " [0.18452832, 0.31865274, 0.19570004, 0.14432825],\n",
+ " [0.18452832, 0.31865274, 0.19570004, 0.14432825]],\n",
+ "\n",
+ " [[0.03695508, 0.32391598, 0.29156933, 0.03888144],\n",
+ " [0.03695508, 0.32391598, 0.29156933, 0.03888144],\n",
+ " [0.03695508, 0.32391598, 0.29156933, 0.03888144]],\n",
+ "\n",
+ " [[0.06211358, 0.24067951, 0.43644298, 0.10758854],\n",
+ " [0.06211358, 0.24067951, 0.43644298, 0.10758854],\n",
+ " [0.06211358, 0.24067951, 0.43644298, 0.10758854]]])"
+ ]
+ },
+ "execution_count": 31,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "# Calculate the remainder of `x` and `y`\n",
+ "np.remainder(x, y)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Remember how broadcasting works? Check out the dimensions and the shapes of both x and y in your IPython shell. Are the rules of broadcasting respected?\n",
+ "\n",
+ "But there is more.\n",
+ "\n",
+ "Check out this small list of aggregate functions:\n",
+ "\n",
+ "| Function | Description |\n",
+ "| --- | --- |\n",
+ "| ```a.sum()``` |\tArray-wise sum |\n",
+ "| ```a.min()``` |\tArray-wise minimum value |\n",
+ "| ```b.max(axis=0)``` |\tMaximum value of an array row |\n",
+ "| ```b.cumsum(axis=1)``` |\tCumulative sum of the elements |\n",
+ "| ```a.mean()``` |\tMean |\n",
+ "| ```b.median()``` |\tMedian |\n",
+ "| ```a.corrcoef()``` |\tCorrelation coefficient |\n",
+ "| ```np.std(b)``` |\tStandard deviation |"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Besides all of these functions, you might also find it useful to know that there are mechanisms that allow you to compare array elements. For example, if you want to check whether the elements of two arrays are the same, you might use the ```==``` operator. To check whether the array elements are smaller or bigger, you use the ```<``` or ```>``` operators.\n",
+ "\n",
+ "This all seems quite straightforward, yes?\n",
+ "\n",
+ "However, you can also compare entire arrays with each other! In this case, you use the ```np.array_equal()``` function. Just pass in the two arrays that you want to compare with each other, and you’re done.\n",
+ "\n",
+ "Note that, besides comparing, you can also perform logical operations on your arrays. You can start with ```np.logical_or()```, ```np.logical_not()``` and ```np.logical_and()```. This basically works like your typical OR, NOT and AND logical operations;\n",
+ "\n",
+ "In the simplest example, you use OR to see whether your elements are the same (for example, 1), or if one of the two array elements is 1. If both of them are 0, you’ll return ```FALSE```. You would use AND to see whether your second element is also 1 and NOT to see if the second element differs from 1."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 32,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "array([[[ True, True, True, True],\n",
+ " [ True, True, True, True],\n",
+ " [ True, True, True, True]],\n",
+ "\n",
+ " [[ True, True, True, True],\n",
+ " [ True, True, True, True],\n",
+ " [ True, True, True, True]],\n",
+ "\n",
+ " [[ True, True, True, True],\n",
+ " [ True, True, True, True],\n",
+ " [ True, True, True, True]],\n",
+ "\n",
+ " [[ True, True, True, True],\n",
+ " [ True, True, True, True],\n",
+ " [ True, True, True, True]],\n",
+ "\n",
+ " [[ True, True, True, True],\n",
+ " [ True, True, True, True],\n",
+ " [ True, True, True, True]]])"
+ ]
+ },
+ "execution_count": 32,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "# `x` AND `y` \n",
+ "np.logical_and(x, y)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 33,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "array([[[ True, True, True, True],\n",
+ " [ True, True, True, True],\n",
+ " [ True, True, True, True]],\n",
+ "\n",
+ " [[ True, True, True, True],\n",
+ " [ True, True, True, True],\n",
+ " [ True, True, True, True]],\n",
+ "\n",
+ " [[ True, True, True, True],\n",
+ " [ True, True, True, True],\n",
+ " [ True, True, True, True]],\n",
+ "\n",
+ " [[ True, True, True, True],\n",
+ " [ True, True, True, True],\n",
+ " [ True, True, True, True]],\n",
+ "\n",
+ " [[ True, True, True, True],\n",
+ " [ True, True, True, True],\n",
+ " [ True, True, True, True]]])"
+ ]
+ },
+ "execution_count": 33,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "# `x` OR `y` \n",
+ "np.logical_or(x, y)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## How To Subset, Slice, And Index Arrays\n",
+ "\n",
+ "Besides mathematical operations, you might also consider taking just a part of the original array (or the resulting array) or just some array elements to use in further analysis or other operations. In such case, you will need to subset, slice and/or index your arrays.\n",
+ "\n",
+ "These operations are very similar to when you perform them on Python lists. If you want to check out the similarities for yourself, or if you want a more elaborate explanation, you might consider checking out DataCamp’s Python list tutorial.\n",
+ "\n",
+ "If you have no clue at all on how these operations work, it suffices for now to know these two basic things:\n",
+ "\n",
+ "* You use square brackets ```[]``` as the index operator, and\n",
+ "* Generally, you pass integers to these square brackets, but you can also put a colon ```:``` or a combination of the colon with integers in it to designate the elements/rows/columns you want to select.\n",
+ "\n",
+ "Besides from these two points, the easiest way to see how this all fits together is by looking at some examples of subsetting:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 34,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "[5 6 7 8]\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Select the element at the 1st index\n",
+ "print(my_array[1])"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 35,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "7\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Select the element at row 1 column 2\n",
+ "print(my_2d_array[1][2])"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 36,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "7\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Select the element at row 1 column 2\n",
+ "print(my_2d_array[1,2])"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 37,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "15\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Select the element at row 1, column 2 and \n",
+ "print(my_3d_array[1,1,2])"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Something a little bit more advanced than subsetting, if you will, is slicing. Here, you consider not just particular values of your arrays, but you go to the level of rows and columns. You’re basically working with “regions” of data instead of pure “locations”."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 38,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "[[1 2 3 4]\n",
+ " [5 6 7 8]]\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Select items at index 0 and 1\n",
+ "print(my_array[0:2])"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 39,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "[2 6]\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Select items at row 0 and 1, column 1\n",
+ "print(my_2d_array[0:2,1])"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 40,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "[[ 9 10 11 12]\n",
+ " [13 14 15 16]]\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Select items at row 1\n",
+ "# This is the same as saying `my_3d_array[1,:,:]\n",
+ "print(my_3d_array[1,...])"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Lastly, there’s also indexing. When it comes to NumPy, there are boolean indexing and advanced or “fancy” indexing.\n",
+ "\n",
+ "First up is boolean indexing. Here, instead of selecting elements, rows or columns based on index number, you select those values from your array that fulfill a certain condition."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 41,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "[[ True False False False]\n",
+ " [False False False False]]\n",
+ "\n",
+ "[1]\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Try out a simple example\n",
+ "mask = my_array < 2\n",
+ "print(mask)\n",
+ "print()\n",
+ "print(my_array[mask])"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 42,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "[1]\n"
+ ]
+ }
+ ],
+ "source": [
+ "print(my_array[my_array < 2])"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 43,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "[[False False False True]\n",
+ " [ True True True True]]\n",
+ "[4 5 6 7 8]\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Try out a simple example\n",
+ "mask = my_array > 3\n",
+ "print(mask)\n",
+ "print(my_array[mask])"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 44,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "[ 3 4 5 6 7 8 9 10 11 12 13 14 15 16]\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Specify a condition\n",
+ "bigger_than_3 = (my_3d_array >= 3)\n",
+ "\n",
+ "# Use the condition to index our 3d array\n",
+ "print(my_3d_array[bigger_than_3])"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 45,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "[3 4 5 6 7 8 9]\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Specify a condition\n",
+ "mask = (my_3d_array >= 3) & (my_3d_array < 10)\n",
+ "\n",
+ "# Use the condition to index our 3d array\n",
+ "print(my_3d_array[mask])"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Note that, to specify a condition, you can also make use of the logical operators ```|``` (OR) and ```&``` (AND). If you would want to rewrite the condition above in such a way (which would be inefficient, but I demonstrate it here for educational purposes :)), you would get ```bigger_than_3 = (my_3d_array > 3) | (my_3d_array == 3)```.\n",
+ "\n",
+ "With the arrays that have been loaded in, there aren’t too many possibilities, but with arrays that contain for example, names or capitals, the possibilities could be endless!\n",
+ "\n",
+ "When it comes to fancy indexing, that what you basically do with it is the following: you pass a list or an array of integers to specify the order of the subset of rows you want to select out of the original array."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 46,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "[5 2 7 1]\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Select elements at (1,0), (0,1), (1,2) and (0,0)\n",
+ "print(my_2d_array[[1, 0, 1, 0],[0, 1, 2, 0]])"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 47,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "[[5 6 7 5]\n",
+ " [1 2 3 1]\n",
+ " [5 6 7 5]\n",
+ " [1 2 3 1]]\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Select a subset of the rows and columns\n",
+ "print(my_2d_array[[1, 0, 1, 0]][:,[0,1,2,0]])"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Now, the second statement might seem to make less sense to you at first sight. This is normal. It might make more sense if you break it down:\n",
+ "\n",
+ "* If you just execute ```my_2d_array[[1,0,1,0]]```, the result is the following:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 48,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "array([[5, 6, 7, 8],\n",
+ " [1, 2, 3, 4],\n",
+ " [5, 6, 7, 8],\n",
+ " [1, 2, 3, 4]])"
+ ]
+ },
+ "execution_count": 48,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "my_2d_array[[1,0,1,0]]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "* What the second part, namely, ```[:,[0,1,2,0]]```, is tell you that you want to keep all the rows of this result, but that you want to change the order of the columns around a bit. You want to display the columns 0, 1, and 2 as they are right now, but you want to repeat column 0 as the last column instead of displaying column number 3. This will give you the following result:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 49,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "array([[ 1, 2, 3, 1],\n",
+ " [ 5, 6, 7, 5],\n",
+ " [ 9, 10, 11, 9]])"
+ ]
+ },
+ "execution_count": 49,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "my_2d_array[:,[0,1,2,0]]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## How To Manipulate Arrays\n",
+ "\n",
+ "Performing mathematical operations on your arrays is one of the things that you’ll be doing, but probably most importantly to make this and the broadcasting work is to know how to manipulate your arrays.\n",
+ "\n",
+ "Below are some of the most common manipulations that you’ll be doing.\n",
+ "\n",
+ "### How To Transpose Your Arrays\n",
+ "\n",
+ "What transposing your arrays actually does is permuting the dimensions of it. Or, in other words, you switch around the shape of the array. Let’s take a small example to show you the effect of transposition:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 50,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "[[ 1 2 3 4]\n",
+ " [ 5 6 7 8]\n",
+ " [ 9 10 11 12]]\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Print `my_2d_array`\n",
+ "print(my_2d_array)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 51,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "[[ 1 5 9]\n",
+ " [ 2 6 10]\n",
+ " [ 3 7 11]\n",
+ " [ 4 8 12]]\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Transpose `my_2d_array`\n",
+ "print(np.transpose(my_2d_array))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 52,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "[[ 1 5 9]\n",
+ " [ 2 6 10]\n",
+ " [ 3 7 11]\n",
+ " [ 4 8 12]]\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Or use `T` to transpose `my_2d_array`\n",
+ "print(my_2d_array.T)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Reshaping Versus Resizing Your Arrays\n",
+ "\n",
+ "You might have read in the broadcasting section that the dimensions of your arrays need to be compatible if you want them to be good candidates for arithmetic operations. But the question of what you should do when that is not the case, was not answered yet.\n",
+ "\n",
+ "Well, this is where you get the answer!\n",
+ "\n",
+ "What you can do if the arrays don’t have the same dimensions, is resize your array. You will then return a new array that has the shape that you passed to the ```np.resize()``` function. If you pass your original array together with the new dimensions, and if that new array is larger than the one that you originally had, the new array will be filled with copies of the original array that are repeated as many times as is needed.\n",
+ "\n",
+ "However, if you just apply ```np.resize()``` to the array and you pass the new shape to it, the new array will be filled with zeros."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 53,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "(3, 4)\n",
+ "[[1. 1. 1. 1.]\n",
+ " [1. 1. 1. 1.]\n",
+ " [1. 1. 1. 1.]]\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Print the shape of `x`\n",
+ "print(x.shape)\n",
+ "print(x)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 54,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "array([[1., 1., 1.],\n",
+ " [1., 1., 1.],\n",
+ " [1., 1., 1.],\n",
+ " [1., 1., 1.]])"
+ ]
+ },
+ "execution_count": 54,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "# Resize `x` to ((6,4))\n",
+ "np.resize(x, (4, 3))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 55,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "array([[1., 1., 1., 1.],\n",
+ " [1., 1., 1., 1.],\n",
+ " [1., 1., 1., 1.]])"
+ ]
+ },
+ "execution_count": 55,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "np.resize(x, (3, 4))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Besides resizing, you can also reshape your array. This means that you give a new shape to an array without changing its data. The key to reshaping is to make sure that the total size of the new array is unchanged. If you take the example of array ```x``` that was used above, which has a size of 3 X 4 or 12, you have to make sure that the new array also has a size of 12.\n",
+ "\n",
+ "If you want to calculate the size of an array with code, make sure to use the size attribute: ```x.size``` or ```x.reshape((2,6)).size```:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 56,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "12\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Print the size of `x` to see what's possible\n",
+ "print(x.size)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 57,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "[1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.]\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Flatten `x`\n",
+ "z = x.ravel()\n",
+ "\n",
+ "# Print `z`\n",
+ "print(z)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "If all else fails, you can also append an array to your original one or insert or delete array elements to make sure that your dimensions fit with the other array that you want to use for your computations.\n",
+ "\n",
+ "Another operation that you might keep handy when you’re changing the shape of arrays is ```ravel()```. This function allows you to flatten your arrays. This means that if you ever have 2D, 3D or n-D arrays, you can just use this function to flatten it all out to a 1-D array."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### How To Append Arrays\n",
+ "When you append arrays to your original array, they are “glued” to the end of that original array. If you want to make sure that what you append does not come at the end of the array, you might consider inserting it. Go to the next section if you want to know more.\n",
+ "\n",
+ "Appending is a pretty easy thing to do thanks to the NumPy library; You can just make use of the ```np.append()```."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 58,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "[[1 2 3 4]\n",
+ " [5 6 7 8]]\n",
+ "[ 1 2 3 4 5 6 7 8 7 8 9 10 11 12]\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Append a 1D array to your `my_array`\n",
+ "new_array = np.append(my_array, [7, 8, 9, 10, 11, 12])\n",
+ "\n",
+ "# Print `new_array`\n",
+ "print(my_array)\n",
+ "print(new_array)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 59,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "[[ 1 2 3 4 7]\n",
+ " [ 5 6 7 8 8]\n",
+ " [ 9 10 11 12 9]]\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Append an extra column to your `my_2d_array`\n",
+ "new_2d_array = np.append(my_2d_array, [[7], [8], [9]], axis=1)\n",
+ "\n",
+ "# Print `new_2d_array`\n",
+ "print(new_2d_array)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Note how, when you append an extra column to my_2d_array, the axis is specified. Remember that axis 1 indicates the columns, while axis 0 indicates the rows in 2-D arrays."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### How To Insert And Delete Array Elements\n",
+ "\n",
+ "Next to appending, you can also insert and delete array elements. As you might have guessed by now, the functions that will allow you to do these operations are ```np.insert()``` and ```np.delete()```:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 60,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "array([1, 5, 2, 3, 4, 5, 6, 7, 8], dtype=int32)"
+ ]
+ },
+ "execution_count": 60,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "# Insert `5` at index 1\n",
+ "np.insert(my_array, 1, 5)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 61,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Delete the value at index 1\n",
+ "# np.delete(my_array,[1])"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### How To Join And Split Arrays\n",
+ "\n",
+ "You can also ‘merge’ or join your arrays. There are a bunch of functions that you can use for that purpose and most of them are listed below.\n",
+ "\n",
+ "Try them out, but also make sure to test out what the shape of the arrays is in the IPython shell. The arrays that have been loaded are ```x```, ```my_array```, ```my_resized_array``` and ```my_2d_array```."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 62,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "[[1. 2. 3. 4.]\n",
+ " [5. 6. 7. 8.]\n",
+ " [1. 1. 1. 1.]\n",
+ " [1. 1. 1. 1.]\n",
+ " [1. 1. 1. 1.]]\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Concatentate `my_array` and `x`\n",
+ "print(np.concatenate((my_array, x)))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 63,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "[[ 1 2 3 4]\n",
+ " [ 5 6 7 8]\n",
+ " [ 1 2 3 4]\n",
+ " [ 5 6 7 8]\n",
+ " [ 9 10 11 12]]\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Stack arrays row-wise\n",
+ "print(np.vstack((my_array, my_2d_array)))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 64,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "[[91 92 93 94]\n",
+ " [91 92 93 94]\n",
+ " [91 92 93 94]]\n"
+ ]
+ }
+ ],
+ "source": [
+ "my_resized_array = np.array([[91, 92, 93, 94], \n",
+ " [91, 92, 93, 94], \n",
+ " [91, 92, 93, 94]])\n",
+ "print(my_resized_array)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 65,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "[[91 92 93 94]\n",
+ " [91 92 93 94]\n",
+ " [91 92 93 94]\n",
+ " [ 1 2 3 4]\n",
+ " [ 5 6 7 8]\n",
+ " [ 9 10 11 12]]\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Stack arrays row-wise\n",
+ "print(np.r_[my_resized_array, my_2d_array])"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 66,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "[[91 92 93 94 1 2 3 4]\n",
+ " [91 92 93 94 5 6 7 8]\n",
+ " [91 92 93 94 9 10 11 12]]\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Stack arrays horizontally\n",
+ "print(np.hstack((my_resized_array, my_2d_array)))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 67,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "[[91 92 93 94 1 2 3 4]\n",
+ " [91 92 93 94 5 6 7 8]\n",
+ " [91 92 93 94 9 10 11 12]]\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Stack arrays column-wise\n",
+ "print(np.column_stack((my_resized_array, my_2d_array)))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 68,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "[[91 92 93 94 1 2 3 4]\n",
+ " [91 92 93 94 5 6 7 8]\n",
+ " [91 92 93 94 9 10 11 12]]\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Stack arrays column-wise\n",
+ "print(np.c_[my_resized_array, my_2d_array])"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "You’ll note a few things as you go through the functions:\n",
+ "\n",
+ "* The number of dimensions needs to be the same if you want to concatenate two arrays with ```np.concatenate()```. As such, if you want to concatenate an array with ```my_array```, which is 1-D, you’ll need to make sure that the second array that you have, is also 1-D.\n",
+ "* With ```np.vstack()```, you effortlessly combine ```my_array``` with ```my_2d_array```. You just have to make sure that, as you’re stacking the arrays row-wise, that the number of columns in both arrays is the same. As such, you could also add an array with shape ```(2,4)``` or ```(3,4)``` to ```my_2d_array```, as long as the number of columns matches. Stated differently, the arrays must have the same shape along all but the first axis. The same holds also for when you want to use ```np.r[]```.\n",
+ "* For ```np.hstack()```, you have to make sure that the number of dimensions is the same and that the number of rows in both arrays is the same. That means that you could stack arrays such as ```(2,3)``` or ```(2,4)``` to ```my_2d_array```, which itself as a shape of (2,4). Anything is possible as long as you make sure that the number of rows matches. This function is still supported by NumPy, but you should prefer ```np.concatenate()``` or ```np.stack()```.\n",
+ "* With ```np.column_stack()```, you have to make sure that the arrays that you input have the same first dimension. In this case, both shapes are the same, but if ```my_resized_array``` were to be ```(2,1)``` or ```(2,)```, the arrays still would have been stacked.\n",
+ "* ```np.c_[]``` is another way to concatenate. Here also, the first dimension of both arrays needs to match.\n",
+ "\n",
+ "When you have joined arrays, you might also want to split them at some point. Just like you can stack them horizontally, you can also do the same but then vertically. You use ```np.hsplit()``` and ```np.vsplit()```, respectively:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 69,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "my_stacked_array = np.r_[my_resized_array, my_2d_array]"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 70,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "[array([[91, 92],\n",
+ " [91, 92],\n",
+ " [91, 92],\n",
+ " [ 1, 2],\n",
+ " [ 5, 6],\n",
+ " [ 9, 10]]), array([[93, 94],\n",
+ " [93, 94],\n",
+ " [93, 94],\n",
+ " [ 3, 4],\n",
+ " [ 7, 8],\n",
+ " [11, 12]])]\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Split `my_stacked_array` horizontally at the 2nd index\n",
+ "print(np.hsplit(my_stacked_array, 2))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 71,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "[array([[91, 92, 93, 94],\n",
+ " [91, 92, 93, 94],\n",
+ " [91, 92, 93, 94]]), array([[ 1, 2, 3, 4],\n",
+ " [ 5, 6, 7, 8],\n",
+ " [ 9, 10, 11, 12]])]\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Split `my_stacked_array` vertically at the 2nd index\n",
+ "print(np.vsplit(my_stacked_array, 2))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "What you need to keep in mind when you’re using both of these split functions is probably the shape of your array. Let’s take the above case as an example: ```my_stacked_array``` has a shape of ```(2,8)```. If you want to select the index at which you want the split to occur, you have to keep the shape in mind."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## How To Visualize NumPy Arrays\n",
+ "Lastly, something that will definitely come in handy is to know how you can plot your arrays. This can especially be handy in data exploration, but also in later stages of the data science workflow, when you want to visualize your arrays.\n",
+ "\n",
+ "### With np.histogram()\n",
+ "Contrary to what the function might suggest, the ```np.histogram()``` function doesn’t draw the histogram but it does compute the occurrences of the array that fall within each bin; This will determine the area that each bar of your histogram takes up.\n",
+ "\n",
+ "What you pass to the ```np.histogram()``` function then is first the input data or the array that you’re working with. The array will be flattened when the histogram is computed."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 72,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Initialize your array\n",
+ "my_3d_array = np.array([[[1,2,3,4], [5,6,7,8]], [[1,2,3,4], [9,10,11,12]]], dtype=np.int64)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 73,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "(array([4, 2, 2, 1, 1, 1, 1, 1, 1, 2]), array([ 1. , 2.1, 3.2, 4.3, 5.4, 6.5, 7.6, 8.7, 9.8, 10.9, 12. ]))\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Pass the array to `np.histogram()`\n",
+ "print(np.histogram(my_3d_array))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 74,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "(array([0, 2, 2, 2, 2, 1, 1, 1, 1, 1, 1, 2]), array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]))\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Specify the number of bins\n",
+ "print(np.histogram(my_3d_array, bins=range(0,13)))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "You’ll see that as a result, the histogram will be computed: the first array lists the frequencies for all the elements of your array, while the second array lists the bins that would be used if you don’t specify any bins.\n",
+ "\n",
+ "If you do specify a number of bins, the result of the computation will be different: the floats will be gone and you’ll see all integers for the bins.\n",
+ "\n",
+ "There are still some other arguments that you can specify that can influence the histogram that is computed. You can find all of them [here](https://docs.scipy.org/doc/numpy/reference/generated/numpy.histogram.html).\n",
+ "\n",
+ "But what is the point of computing such a histogram if you can’t visualize it?\n",
+ "\n",
+ "Visualization is a piece of cake with the help of Matplotlib, but you don’t need ```np.histogram()``` to compute the histogram. ```plt.hist()``` does this for itself when you pass it the (flattened) data and the bins:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 75,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAEICAYAAABRSj9aAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAa9ElEQVR4nO3de5xcZZ3n8c93knBHCKRFyF2NykW5TG+AgdcQL0C4hhmdIVEkODBZHRB0mUUYZ8EF3cXLgBdQzGAWHIHAokBcg5ARWZZLNB0GgQCBTETS4ZKGcA+Kgd/+cZ5mDkVV1+nu6lT6yff9etWr6zzPufxO1elvnTp1qo4iAjMzy9eftLsAMzMbWg56M7PMOejNzDLnoDczy5yD3swscw56M7PMOehtWJD0Xkn3SHpR0qntrmdTIelWSSe1uw4bHAf9RkDSo5JekfRS6bZLu+vayJwB/DIito2Ib9d2pkAKSXvWtF+X2qf1d4GSZkpaLul5SWskXS7pbaX+3uftRUnPSbpT0qclNf2/knSZpPWSdu5vXa0kaZqk12u2vZck7d/OuvqSau5udx3DiYN+43FURGxTuj1e7pQ0sl2FbSQmAsuajPMwcHzvgKQdgf2BngEu8w7ggIjYDngnMBL4cs04R0XEtqm+84EvAD/oa6aStgY+CjwPHNdk3JE1w6ryQtJPj9dse9tExF0tXoa1kYN+I5b2RE+W9AjwSGo7Mh3C6N2D/EBp/L0l3Z32MK+WNF/Sl1PfCZJurzP/d6f7m0v6hqTHJD0l6RJJW6a+aZK6JZ2e9myfkPSp0ny2lPRPkn6X9n5vT20/k/TZmmXeK+kvGqzv0ZKWpXW7VdKuqf0W4IPARWlv8z0NHrIrgGMljUjDs4DrgFfTfN4haV16Aehd5j6SeiSNqp1ZRKyKiKdLTa8B76634Ih4PiIWAMcCsyXt0aBGKEL+OeBcYHbNY/AlSddK+pGkF4AT0mPxFUl3AOuAd0r6lKQH03O9UtJ/Ls3jfklHlYZHSXpa0t591FSJpL9Jy31W0k2SJpb6QtLfSXok1XWepHel7fQFSddI2qw0fl/b8qOS/j5tL8+n7XmL9CJ5I7BL6d3HLpKmSupKy3lK0gWDXdesRIRvbb4BjwIfqdMewCJgB2BLYG9gDbAvMIIiJB4FNgc2A34HfB4YBXwM+CPw5TSvE4Db68z/3en+hcCCtKxtgZ8C/zP1TQPWUwTTKOBwisAZnfovBm4Fxqa6/izV9NfAr0rL2xN4Btiszrq+B3gZODgt4wxgRe+4af4n9fEY3gqcBNwMHJbafk2xR98NTEttC4HPlKa7EPhOH/M9kGLPO1J9h1R43h4rL6NO/y+ArwE7pcf1T0t9X0rP2zEUO2JbpnV7DNid4l3FKOAI4F2AgIPS87FPmscZwNWlec4A7mtQyzSgu9njWprPCmDXVMc/AnfWbE83AG9Ltf4hres7ge2AB4DZadyG23Lpsf01sAvFNvkg8OlGNQN3AZ9M97cB9mv3//XGdGt7Ab69sVG/RLGX9xxwfWoP4EOl8b4HnFcz7fL0j/7nwOOASn13UiHoU1i8DLyr1Lc/8Nt0fxrwCjCy1L8G2C+F0SvAnnXWawvgWWBKGv4G8N0Gj8F/A64pDf8JsJr/COg3AqfB9LdSBP1xwFXA+4CHU1856I8F7kj3RwBPAlMrPEdjKUL4PTXPW72gXwx8scF8JgCvA3ul4ZuAb5X6vwTcVmfdzm1S3/XAaen+LsCLwNvS8LXAGQ2mm5bqea7mtnXt406xJ31izXO0DphY2p4OKPUvBb5QGv4n4JvNtuXSY3tcqe9rwCWlmmuD/jbgvwNjNsT/7HC7+dDNxuOYiNg+3Y4pta8q3Z8InJ7e6j4n6TlgPMU/9i7A6khbffK7isvuALYClpbm+/PU3uuZiFhfGl5Hsec0hiLQ/712phHxe+Bq4Lh0XHkW8C8NatilXG9EvE6x7mMrrkOvnwAfAk5psKwbgN0kTaZ49/B8RPy62UwjYjXFYzK/Qg1jgbUN+j4JPBgR96ThK4CP1xw6WvWWqWraJB0mabGkten5OpziuSCKz3fuAD4qaXvgsLScRh4vbXu9t5frjDcR+FZpG1lLsZNQfo6eKt1/pc7wNqV5NdqWez1Zur+uNG09J1K8K3xI0hJJR/Yx7iZnU/+AbzgoB/cq4CsR8ZXakSQdBIyVpFLYT+A/AvhlijDvHf8dpcmfpvgn3D0FWn88Dfye4jDCb+r0X04RuLcD66Lxh3yPA+8v1SeKf/x+1RMR6yTdCHwm1VTb/3tJ11Ds+b+Pxi889YysN88ySf+JIvhubzDK8cAESb0hNhLYkSKob+gts850b7RJ2hz4cZrXDRHxR0nXU4Rur8sp3uGMBO4awPNaT+/219eLRn/n9ZZtuYK3PD4R8QgwK+1Q/CVwraQdG7xgbXK8Rz+8/DPwaUn7qrC1pCMkbUtxjHI9cGr68O0vgamlaX8D7C5pL0lbUBwiAN7Ye/5n4EJJbweQNFbSoc0KStPOAy5IH4qNkLR/CiNSsL9O8ba9r1C9BjhC0ofT3u3pFMd476z0yLzZP1AcAni0Qf8PKQ5lHd1XTZI+IWlCuj8R+ArFMed6474t7UXOB34UEffVGWd/iheKqcBe6bYHcCWls4Uq2IziM5AeYL2kw4BDasa5HtgHOI1ifVvhEuAsSbsDSNpO0l8NcF59bcvNPAXsKGm73gZJx0nqSNvjc6n59QHWlh0H/TASEV3A3wIXURz7XkERWETEqxR7MidQvKU+luIwRu+0D1N8mPqvFGfw1O5xfiHNb3E62+NfgfdWLO3vgfuAJWnZX+XN29YPKfbWf9THui2n2Mv+DsW7hKMoTl18tWIN5Xk9HhGN9qiJiDsoQuDuiOjr8NZuwJ2SXqY4FLKc4vEv+6mkFyn2UL8IXAB8ivpmU+yB3xcRT/begG8BR0raoeL6vQicSvHi+CzwcYoP0svjvEKx1z+Z0nbQQPkMlt7bR+ss9zqK53Z+2kbupzgs1G99bcsVpn2I4nOYlemwzy7AdGCZpJcoHs+Z6TEw0gd3lidJl1F8aPWPba7jeGBORBzYzjrKVJyyeWVEXNruWoaKpLMpPjzu81x9y5+P0duQkrQV8HfAd9tdS690HH0fitMFs5TeHZxI8eGvbeJ86MaGTDrG30NxTPXKNpcDgKTLKQ5LfS4dAsmOpL+lOJR0Y0Tc1u56rP186MbMLHPeozczy9xGeYx+zJgxMWnSpHaXYWY2bCxduvTpiOio17dRBv2kSZPo6upqdxlmZsOGpIanCvvQjZlZ5hz0ZmaZc9CbmWXOQW9mljkHvZlZ5hz0ZmaZq3K1+vGSfinpARXX8zytzjiS9G1JK9I1Hvcp9c1WcQ3JRyTNrp3WzMyGVpXz6NcDp0fE3em3opdKWhQRD5TGOQyYkm77UlwmbN/0w0rnAJ0UFwtYKmlBRDzb0rUwM7OGmu7RR8QTEXF3uv8ixUV6ay/vNgP4YRQWA9tL2hk4FFgUEWtTuC+i+N1oMzPbQPr1zVhJkyiu3v6rmq6xvPmalt2prVF7vXnPAeYATJgwoT9lbXCTzvxZu0uwOh49/4h2l2AZ2xD/90O1DVf+MFbSNhRXrPlcRLzQ6kIiYm5EdEZEZ0dH3Z9rMDOzAagU9Okanj8GroiIepclW01xIede41Jbo3YzM9tAqpx1I+AHwIMRcUGD0RYAx6ezb/YDno+IJ4CbgEMkjZY0muICxje1qHYzM6ugyjH6AyguR3afpHtS2z8AEwAi4hJgIXA4xQV+15EujhwRayWdR3HRaIBzI2Jty6o3M7OmmgZ9RNwOqMk4AZzcoG8eMG9A1ZmZ2aD5m7FmZplz0JuZZc5Bb2aWOQe9mVnmHPRmZplz0JuZZc5Bb2aWOQe9mVnmHPRmZplz0JuZZc5Bb2aWOQe9mVnmHPRmZplz0JuZZc5Bb2aWOQe9mVnmml54RNI84EhgTUTsUaf/vwKfKM1vV6AjXV3qUeBF4DVgfUR0tqpwMzOrpsoe/WXA9EadEfH1iNgrIvYCzgL+b83lAj+Y+h3yZmZt0DToI+I2oOp1XmcBVw2qIjMza6mWHaOXtBXFnv+PS80B3CxpqaQ5rVqWmZlV1/QYfT8cBdxRc9jmwIhYLentwCJJD6V3CG+RXgjmAEyYMKGFZZmZbdpaedbNTGoO20TE6vR3DXAdMLXRxBExNyI6I6Kzo6OjhWWZmW3aWhL0krYDDgJuKLVtLWnb3vvAIcD9rViemZlVV+X0yquAacAYSd3AOcAogIi4JI32F8DNEfFyadKdgOsk9S7nyoj4eetKNzOzKpoGfUTMqjDOZRSnYZbbVgJ7DrQwMzNrDX8z1swscw56M7PMOejNzDLnoDczy5yD3swscw56M7PMOejNzDLnoDczy5yD3swscw56M7PMOejNzDLnoDczy5yD3swscw56M7PMOejNzDLnoDczy5yD3swsc02DXtI8SWsk1b3eq6Rpkp6XdE+6nV3qmy5puaQVks5sZeFmZlZNlT36y4DpTcb5fxGxV7qdCyBpBHAxcBiwGzBL0m6DKdbMzPqvadBHxG3A2gHMeyqwIiJWRsSrwHxgxgDmY2Zmg9CqY/T7S/qNpBsl7Z7axgKrSuN0p7a6JM2R1CWpq6enp0VlmZlZK4L+bmBiROwJfAe4fiAziYi5EdEZEZ0dHR0tKMvMzKAFQR8RL0TES+n+QmCUpDHAamB8adRxqc3MzDagQQe9pHdIUro/Nc3zGWAJMEXSZEmbATOBBYNdnpmZ9c/IZiNIugqYBoyR1A2cA4wCiIhLgI8Bn5G0HngFmBkRAayXdApwEzACmBcRy4ZkLczMrKGmQR8Rs5r0XwRc1KBvIbBwYKWZmVkr+JuxZmaZc9CbmWXOQW9mljkHvZlZ5hz0ZmaZc9CbmWXOQW9mljkHvZlZ5hz0ZmaZc9CbmWXOQW9mljkHvZlZ5hz0ZmaZc9CbmWXOQW9mljkHvZlZ5hz0ZmaZaxr0kuZJWiPp/gb9n5B0r6T7JN0pac9S36Op/R5JXa0s3MzMqqmyR38ZML2P/t8CB0XE+4HzgLk1/R+MiL0ionNgJZqZ2WBUuWbsbZIm9dF/Z2lwMTCuBXWZmVmLtPoY/YnAjaXhAG6WtFTSnL4mlDRHUpekrp6enhaXZWa26Wq6R1+VpA9SBP2BpeYDI2K1pLcDiyQ9FBG31Zs+IuaSDvt0dnZGq+oyM9vUtWSPXtIHgEuBGRHxTG97RKxOf9cA1wFTW7E8MzOrbtBBL2kC8BPgkxHxcKl9a0nb9t4HDgHqnrljZmZDp+mhG0lXAdOAMZK6gXOAUQARcQlwNrAj8F1JAOvTGTY7AdeltpHAlRHx8yFYBzMz60OVs25mNek/CTipTvtKYM+3TmFmZhuSvxlrZpY5B72ZWeYc9GZmmXPQm5llzkFvZpY5B72ZWeYc9GZmmXPQm5llzkFvZpY5B72ZWeYc9GZmmXPQm5llzkFvZpY5B72ZWeYc9GZmmXPQm5llzkFvZpa5SkEvaZ6kNZLqXvNVhW9LWiHpXkn7lPpmS3ok3Wa3qnAzM6um6h79ZcD0PvoPA6ak2xzgewCSdqC4xuy+wFTgHEmjB1qsmZn1X6Wgj4jbgLV9jDID+GEUFgPbS9oZOBRYFBFrI+JZYBF9v2CYmVmLNb04eEVjgVWl4e7U1qj9LSTNoXg3wIQJE1pUlm1KJp35s3aXYLZR2mg+jI2IuRHRGRGdHR0d7S7HzCwbrQr61cD40vC41Nao3czMNpBWBf0C4Ph09s1+wPMR8QRwE3CIpNHpQ9hDUpuZmW0glY7RS7oKmAaMkdRNcSbNKICIuARYCBwOrADWAZ9KfWslnQcsSbM6NyL6+lDXzMxarFLQR8SsJv0BnNygbx4wr/+lmZlZK2w0H8aamdnQcNCbmWXOQW9mljkHvZlZ5hz0ZmaZc9CbmWXOQW9mljkHvZlZ5hz0ZmaZc9CbmWXOQW9mljkHvZlZ5hz0ZmaZc9CbmWXOQW9mljkHvZlZ5hz0ZmaZqxT0kqZLWi5phaQz6/RfKOmedHtY0nOlvtdKfQtaWLuZmVXQ9FKCkkYAFwMHA93AEkkLIuKB3nEi4vOl8T8L7F2axSsRsVfLKjYzs36pskc/FVgRESsj4lVgPjCjj/FnAVe1ojgzMxu8KkE/FlhVGu5ObW8haSIwGbil1LyFpC5JiyUd02ghkuak8bp6enoqlGVmZlW0+sPYmcC1EfFaqW1iRHQCHwe+Keld9SaMiLkR0RkRnR0dHS0uy8xs01Ul6FcD40vD41JbPTOpOWwTEavT35XArbz5+L2ZmQ2xKkG/BJgiabKkzSjC/C1nz0h6HzAauKvUNlrS5un+GOAA4IHaac3MbOg0PesmItZLOgW4CRgBzIuIZZLOBboiojf0ZwLzIyJKk+8KfF/S6xQvKueXz9YxM7Oh1zToASJiIbCwpu3smuEv1ZnuTuD9g6jPzMwGyd+MNTPLnIPezCxzDnozs8w56M3MMuegNzPLnIPezCxzDnozs8w56M3MMuegNzPLnIPezCxzDnozs8w56M3MMuegNzPLnIPezCxzDnozs8w56M3MMuegNzPLXKWglzRd0nJJKySdWaf/BEk9ku5Jt5NKfbMlPZJus1tZvJmZNdf0UoKSRgAXAwcD3cASSQvqXPv16og4pWbaHYBzgE4ggKVp2mdbUr2ZmTVVZY9+KrAiIlZGxKvAfGBGxfkfCiyKiLUp3BcB0wdWqpmZDUSVoB8LrCoNd6e2Wh+VdK+kayWN7+e0SJojqUtSV09PT4WyzMysilZ9GPtTYFJEfIBir/3y/s4gIuZGRGdEdHZ0dLSoLDMzqxL0q4HxpeFxqe0NEfFMRPwhDV4K/GnVac3MbGhVCfolwBRJkyVtBswEFpRHkLRzafBo4MF0/ybgEEmjJY0GDkltZma2gTQ96yYi1ks6hSKgRwDzImKZpHOBrohYAJwq6WhgPbAWOCFNu1bSeRQvFgDnRsTaIVgPMzNroGnQA0TEQmBhTdvZpftnAWc1mHYeMG8QNZqZ2SD4m7FmZplz0JuZZc5Bb2aWOQe9mVnmHPRmZplz0JuZZc5Bb2aWOQe9mVnmHPRmZplz0JuZZc5Bb2aWOQe9mVnmHPRmZplz0JuZZc5Bb2aWOQe9mVnmHPRmZpmrFPSSpktaLmmFpDPr9P8XSQ9IulfSLyRNLPW9JumedFtQO62ZmQ2tppcSlDQCuBg4GOgGlkhaEBEPlEb7N6AzItZJ+gzwNeDY1PdKROzV2rLNzKyqKnv0U4EVEbEyIl4F5gMzyiNExC8jYl0aXAyMa22ZZmY2UFWCfiywqjTcndoaORG4sTS8haQuSYslHdNoIklz0nhdPT09FcoyM7Mqmh666Q9JxwGdwEGl5okRsVrSO4FbJN0XEf9eO21EzAXmAnR2dkYr6zIz25RV2aNfDYwvDY9LbW8i6SPAF4GjI+IPve0RsTr9XQncCuw9iHrNzKyfqgT9EmCKpMmSNgNmAm86e0bS3sD3KUJ+Tal9tKTN0/0xwAFA+UNcMzMbYk0P3UTEekmnADcBI4B5EbFM0rlAV0QsAL4ObAP8b0kAj0XE0cCuwPclvU7xonJ+zdk6ZmY2xCodo4+IhcDCmrazS/c/0mC6O4H3D6ZAMzMbHH8z1swscw56M7PMOejNzDLnoDczy5yD3swscw56M7PMOejNzDLnoDczy5yD3swscw56M7PMOejNzDLnoDczy5yD3swscw56M7PMOejNzDLnoDczy5yD3swsc5WCXtJ0ScslrZB0Zp3+zSVdnfp/JWlSqe+s1L5c0qEtrN3MzCpoGvSSRgAXA4cBuwGzJO1WM9qJwLMR8W7gQuCradrdKC4mvjswHfhump+ZmW0gVfbopwIrImJlRLwKzAdm1IwzA7g83b8W+LCKq4TPAOZHxB8i4rfAijQ/MzPbQKpcHHwssKo03A3s22iciFgv6Xlgx9S+uGbasfUWImkOMCcNviRpeYXa6hkDPD3AaTc2uaxLLusBXpeNUS7rgb46qHWZ2KijStBvEBExF5g72PlI6oqIzhaU1Ha5rEsu6wFel41RLusBQ7cuVQ7drAbGl4bHpba640gaCWwHPFNxWjMzG0JVgn4JMEXSZEmbUXy4uqBmnAXA7HT/Y8AtERGpfWY6K2cyMAX4dWtKNzOzKpoeuknH3E8BbgJGAPMiYpmkc4GuiFgA/AD4F0krgLUULwak8a4BHgDWAydHxGtDtC69Bn34ZyOSy7rksh7gddkY5bIeMETromLH28zMcuVvxpqZZc5Bb2aWuWyCvtnPNAwXksZL+qWkByQtk3Rau2saLEkjJP2bpP/T7loGQ9L2kq6V9JCkByXt3+6aBkLS59O2db+kqyRt0e6aqpI0T9IaSfeX2naQtEjSI+nv6HbWWFWDdfl62r7ulXSdpO1bsawsgr7izzQMF+uB0yNiN2A/4ORhvC69TgMebHcRLfAt4OcR8T5gT4bhOkkaC5wKdEbEHhQnWMxsb1X9chnFz6mUnQn8IiKmAL9Iw8PBZbx1XRYBe0TEB4CHgbNasaAsgp5qP9MwLETEExFxd7r/IkWY1P028XAgaRxwBHBpu2sZDEnbAX9OcYYZEfFqRDzX1qIGbiSwZfrOy1bA422up7KIuI3izL6y8k+wXA4csyFrGqh66xIRN0fE+jS4mOK7R4OWS9DX+5mGYRuOvdKvgO4N/KrNpQzGN4EzgNfbXMdgTQZ6gP+VDkNdKmnrdhfVXxGxGvgG8BjwBPB8RNzc3qoGbaeIeCLdfxLYqZ3FtNDfADe2Yka5BH12JG0D/Bj4XES80O56BkLSkcCaiFja7lpaYCSwD/C9iNgbeJnhc4jgDen49QyKF65dgK0lHdfeqlonfVFz2J8zLumLFIdxr2jF/HIJ+qx+akHSKIqQvyIiftLuegbhAOBoSY9SHE77kKQftbekAesGuiOi993VtRTBP9x8BPhtRPRExB+BnwB/1uaaBuspSTsDpL9r2lzPoEg6ATgS+ES06ItOuQR9lZ9pGBbSzzv/AHgwIi5odz2DERFnRcS4iJhE8ZzcEhHDcu8xIp4EVkl6b2r6MMU3voebx4D9JG2VtrUPMww/VK5R/gmW2cANbaxlUCRNpzjUeXRErGvVfLMI+vThRe/PNDwIXBMRy9pb1YAdAHySYu/3nnQ7vN1FGQCfBa6QdC+wF/A/2ltO/6V3JNcCdwP3UWTAsPkJAUlXAXcB75XULelE4HzgYEmPULxjOb+dNVbVYF0uArYFFqX//Utasiz/BIKZWd6y2KM3M7PGHPRmZplz0JuZZc5Bb2aWOQe9mVnmHPRmZplz0JuZZe7/A451/NlBdQ28AAAAAElFTkSuQmCC\n",
+ "text/plain": [
+ "NumPy is a commonly used Python data analysis package. By using NumPy, you can speed up your workflow, and interface with other packages in the Python ecosystem, like scikit-learn, that use NumPy under the hood. NumPy was originally developed in the mid 2000s, and arose from an even older package called Numeric. This longevity means that almost every data analysis or machine learning package for Python leverages NumPy in some way.
In this tutorial, we’ll walk through using NumPy to analyze data on wine quality. The data contains information on various attributes of wines, such as pH and fixed acidity, along with a quality score between 0 and 10 for each wine. The quality score is the average of at least 3 human taste testers. As we learn how to work with NumPy, we’ll try to figure out more about the perceived quality of wine.
The wines we’ll be analyzing are from the Minho region of Portugal.
The data was downloaded from the UCI Machine Learning Repository, and is available here. Here are the first few rows of the winequality-red.csv file, which we’ll be using throughout this tutorial:
"fixed acidity";"volatile acidity";"citric acid";"residual sugar";"chlorides";"free sulfur dioxide";"total sulfur dioxide";"density";"pH";"sulphates";"alcohol";"quality"
+ 03.b - NumPy Data analysis - Data Focused Python 03.b - NumPy Data analysis | Data Focused Python NumPy Tutorial: Data analysis with Python
NumPy is a commonly used Python data analysis package. By using NumPy, you can speed up your workflow, and interface with other packages in the Python ecosystem, like scikit-learn, that use NumPy under the hood. NumPy was originally developed in the mid 2000s, and arose from an even older package called Numeric. This longevity means that almost every data analysis or machine learning package for Python leverages NumPy in some way.
In this tutorial, we’ll walk through using NumPy to analyze data on wine quality. The data contains information on various attributes of wines, such as pH and fixed acidity, along with a quality score between 0 and 10 for each wine. The quality score is the average of at least 3 human taste testers. As we learn how to work with NumPy, we’ll try to figure out more about the perceived quality of wine.
The wines we’ll be analyzing are from the Minho region of Portugal.
The data was downloaded from the UCI Machine Learning Repository, and is available here. Here are the first few rows of the winequality-red.csv file, which we’ll be using throughout this tutorial:
"fixed acidity";"volatile acidity";"citric acid";"residual sugar";"chlorides";"free sulfur dioxide";"total sulfur dioxide";"density";"pH";"sulphates";"alcohol";"quality"
7.4;0.7;0;1.9;0.076;11;34;0.9978;3.51;0.56;9.4;5
7.8;0.88;0;2.6;0.098;25;67;0.9968;3.2;0.68;9.8;5
The data is in what I’m going to call ssv (semicolon separated values) format – each record is separated by a semicolon (;), and rows are separated by a new line. There are 1600 rows in the file, including a header row, and 12 columns.
Before we get started, a quick version note – we’ll be using Python 3.5. Our code examples will be done using Jupyter notebook.
If you want to jump right into a specific area, here are the topics:
- Creating an Array
- Reading Text Files
- Array Indexing
- N-Dimensional Arrays
- Data Types
- Array Math
- Array Methods
- Array Comparison and Filtering
- Reshaping and Combining Arrays
Lists Of Lists for CSV Data Before using NumPy, we’ll first try to work with the data using Python and the csv package. We can read in the file using the csv.reader object, which will allow us to read in and split up all the content from the ssv file.
In the below code, we:
- Import the csv library.
- Open the winequality-red.csv file.
- With the file open, create a new csv.reader object.
- Pass in the keyword argument delimiter=”;” to make sure that the records are split up on the semicolon character instead of the default comma character.
- Call the list type to get all the rows from the file.
- Assign the result to wines.
import csv
@@ -35,6 +35,9 @@
# create the numpy array skipping the headers
wines = np.array(wines[1:], dtype=np.float)
+
/var/folders/jd/pq0swyt521jb2424d6fvth840000gn/T/ipykernel_13638/4037387242.py:5: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
+Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
+ wines = np.array(wines[1:], dtype=np.float)
# If we display wines, we'll now get a NumPy array:
print(type(wines), wines)
<class 'numpy.ndarray'> [[ 7.4 0.7 0. ... 0.56 9.4 5. ]
@@ -53,8 +56,8 @@
[0., 0., 0., 0.],
[0., 0., 0., 0.]])
Creating arrays full of random numbers can be useful when you want to quickly test your code with sample arrays. You can also create an array where each element is a random number using numpy.random.rand.
np.random.rand(2, 3)
-
array([[0.45, 0.91, 0.91],
- [0.47, 0.32, 0.32]])
+
array([[0.63, 0.05, 0.44],
+ [0.67, 0.3 , 0.26]])
Using NumPy To Read In Files
It’s possible to use NumPy to directly read csv or other files into arrays. We can do this using the numpy.genfromtxt function. We can use it to read in our initial data on red wines.
In the below code, we:
- Use the
genfromtxt function to read in the winequality-red.csv file. - Specify the keyword argument
delimiter=";" so that the fields are parsed properly. - Specify the keyword argument
skip_header=1 so that the header row is skipped.
wines = np.genfromtxt("winequality-red.csv", delimiter=";", skip_header=1)
wines
array([[ 7.4 , 0.7 , 0. , ..., 0.56, 9.4 , 5. ],
@@ -124,7 +127,7 @@
third_wine[1]
0.7
Most NumPy functions that we’ve worked with, such as numpy.random.rand, can be used with multidimensional arrays. Here’s how we’d use numpy.random.rand to generate a random vector:
np.random.rand(3)
-
array([0.97, 0.95, 0.04])
+
array([0.04, 0.38, 0.08])
Previously, when we called np.random.rand, we passed in a shape for a 2-dimensional array, so the result was a 2-dimensional array. This time, we passed in a shape for a single dimensional array. The shape specifies the number of dimensions, and the size of the array in each dimension.
A shape of (10,10) will be a 2-dimensional array with 10 rows and 10 columns. A shape of (10,) will be a 1-dimensional array with 10 elements.
Where NumPy gets more complex is when we start to deal with arrays that have more than 2 dimensions.
N-Dimensional NumPy Arrays
This doesn’t happen extremely often, but there are cases when you’ll want to deal with arrays that have greater than 3 dimensions. One way to think of this is as a list of lists of lists. Let’s say we want to store the monthly earnings of a store, but we want to be able to quickly lookup the results for a quarter, and for a year. The earnings for one year might look like this:
[500, 505, 490, 810, 450, 678, 234, 897, 430, 560, 1023, 640]
The store earned $500 in January, $505 in February, and so on. We can split up these earnings by quarter into a list of lists:
year_one = [
[500,505,490], # 1st quarter
@@ -380,7 +383,7 @@
ValueError Traceback (most recent call last)
-<ipython-input-82-1671c26db06c> in <module>
+/var/folders/jd/pq0swyt521jb2424d6fvth840000gn/T/ipykernel_13638/2228080465.py in <module>
----> 1 wines * np.array([1,2])
@@ -398,12 +401,12 @@
[7, 9]])
As you can see, array_two has been broadcasted across each row of array_one. Here’s an example with our wines data:
rand_array = np.random.rand(12)
wines + rand_array
-
array([[ 7.78, 1.25, 0.07, ..., 1.32, 20.46, 20.77],
- [ 8.18, 1.43, 0.07, ..., 1.44, 20.46, 20.77],
- [11.58, 0.83, 0.63, ..., 1.34, 20.46, 20.77],
+
array([[ 7.57, 1.1 , 0.66, ..., 0.59, 20.33, 20.78],
+ [ 7.97, 1.28, 0.66, ..., 0.71, 20.33, 20.78],
+ [11.37, 0.68, 1.22, ..., 0.61, 20.33, 20.78],
...,
- [ 6.68, 1.06, 0.2 , ..., 1.51, 20.46, 20.77],
- [ 6.28, 1.19, 0.19, ..., 1.47, 20.46, 20.77],
- [ 6.38, 0.86, 0.54, ..., 1.42, 20.46, 20.77]])
+ [ 6.47, 0.91, 0.79, ..., 0.78, 20.33, 20.78],
+ [ 6.07, 1.04, 0.78, ..., 0.74, 20.33, 20.78],
+ [ 6.17, 0.71, 1.13, ..., 0.69, 20.33, 20.78]])
-
+
+Explore data analysis with Python. Pandas DataFrames make manipulating your data easy, from selecting or replacing columns and indices to reshaping your data. Pandas is a popular Python package for data science, and with good reason: it offers powerful, expressive and flexible data structures that make data manipulation and analysis easy, among many other things. The DataFrame is one of these structures.
This tutorial covers Pandas DataFrames, from basic manipulations to advanced operations, by tackling 11 of the most popular questions so that you understand -and avoid- the doubts of the Pythonistas who have gone before you.
Content
Before you start, let’s have a brief recap of what DataFrames are.
Those who are familiar with R know the data frame as a way to store data in rectangular grids that can easily be overviewed. Each row of these grids corresponds to measurements or values of an instance, while each column is a vector containing data for a specific variable. This means that a data frame’s rows do not need to contain, but can contain, the same type of values: they can be numeric, character, logical, etc.
Now, DataFrames in Python are very similar: they come with the Pandas library, and they are defined as a two-dimensional labeled data structures with columns of potentially different types.
In general, you could say that the Pandas DataFrame consists of three main components: the data, the index, and the columns.
Firstly, the DataFrame can contain data that is:
DataFrameSeries: a one-dimensional labeled array capable of holding any data type with axis labels or index. An example of a Series object is one column from a DataFrame.ndarray, which can be a record or structuredndarrayndarray’s, list’s, dictionarie’s or Series.Note the difference between np.ndarray and np.array(). The former is an actual data type, while the latter is a function to make arrays from other data structures.
Structured arrays allow users to manipulate the data by named fields: in the example below, a structured array of three tuples is created.
The first element of each tuple will be called foo and will be of type int, while the second element will be named bar and will be a float.
import numpy as np
+import pandas as pd
+# A structured array
+my_array = np.ones(3, dtype=([('foo', int), ('bar', float)]))
+
+# Print the structured array
+print(type(my_array['foo']), my_array['foo'])
+<class 'numpy.ndarray'> [1 1 1]
+Record arrays, on the other hand, expand the properties of structured arrays. They allow users to access fields of structured arrays by attribute rather than by index. You see below that the foo values are accessed in the r2 record array.
# A record array
+my_array2 = my_array.view(np.recarray)
+
+# Print the record array
+print(type(my_array2.foo), my_array2.foo)
+<class 'numpy.ndarray'> [1 1 1]
+Besides data, you can also specify the index and column names for your DataFrame. The index, on the one hand, indicates the difference in rows, while the column names indicate the difference in columns. You will see later that these two components of the DataFrame will come in handy when you’re manipulating your data.
Note that in this post, most of the times, the libraries that you need have already been loaded in. The Pandas library is usually imported under the alias pd, while the NumPy library is loaded as np. Remember that when you code in your own data science environment, you shouldn’t forget this import step, which you write just like this:
import numpy as np
+import pandas as pd
+Obviously, making your DataFrames is your first step in almost anything that you want to do when it comes to data munging in Python. Sometimes, you will want to start from scratch, but you can also convert other data structures, such as lists or NumPy arrays, to Pandas DataFrames. In this section, you’ll will only cover the latter.
Among the many things that can serve as input to make a DataFrame, a NumPy ndarray is one of them. To make a data frame from a NumPy array, you can just pass it to the DataFrame() function in the data argument.
columns = ['Col1', 'Col2']
+index = ['Row1', 'Row2']
+data = [[1, 2], [3, 4]]
+
+df = pd.DataFrame(data=data,
+ index=index,
+ columns=columns)
+
+df
+| Col1 | Col2 | |
|---|---|---|
| Row1 | 1 | 2 |
| Row2 | 3 | 4 |
data = np.array([['','Col1','Col2'],
+ ['Row1',1,2],
+ ['Row2',3,4]])
+
+df = pd.DataFrame(data=data[1:,1:],
+ index=data[1:,0],
+ columns=data[0,1:])
+
+df
+| Col1 | Col2 | |
|---|---|---|
| Row1 | 1 | 2 |
| Row2 | 3 | 4 |
Pay attention to how the code chunks above select elements from the NumPy array to construct the DataFrame:
Next, you also see that, in the chunk above, you printed out a small selection of the data. This works the same as subsetting 2D NumPy arrays: you first indicate the row that you want to look in for your data, then the column.
Don’t forget that the indices start at 0! For data in the example above, you go and look in the rows at index 1 to end and you select all elements that come after index 1. As a result, you end up selecting 1, 2, 3 and 4.
This approach to making DataFrames will be the same for all the structures that DataFrame() can take on as input.
# Take a 2D array as input to your DataFrame
+my_2darray = np.array([[1, 2, 3], [4, 5, 6]])
+print(my_2darray)
+[[1 2 3]
+ [4 5 6]]
+# Take a dictionary as input to your DataFrame
+my_dict = {
+ 1: ['1', '3'],
+ 2: ['1', '2'],
+ 3: ['2', '4']}
+my_df = pd.DataFrame(my_dict)
+my_df
+| 1 | 2 | 3 | |
|---|---|---|---|
| 0 | 1 | 1 | 2 |
| 1 | 3 | 2 | 4 |
# Take a DataFrame as input to your DataFrame
+my_df = pd.DataFrame(
+ data=[4,5,6,7],
+ index=range(0,4),
+ columns=['A'])
+my_df
+| A | |
|---|---|
| 0 | 4 |
| 1 | 5 |
| 2 | 6 |
| 3 | 7 |
# Take a Series as input to your DataFrame
+my_series = pd.Series({
+ "Belgium":"Brussels",
+ "India":"New Delhi",
+ "United Kingdom":"London",
+ "United States":"Washington"})
+
+my_series
+Belgium Brussels
+India New Delhi
+United Kingdom London
+United States Washington
+dtype: object
+Note that the index of your Series (and DataFrame) contains the keys of the original dictionary, but that they are sorted: Belgium will be the index at 0, while United States will be the index at 3.
After you have created your DataFrame, you might want to know a little bit more about it. You can use the shape property or the len() function in combination with the .index property:
df = pd.DataFrame(np.array([[1, 2, 3], [4, 5, 6]]))
+
+# Use the `shape` property
+print(df.shape)
+(2, 3)
+# Or use the `len()` function with the `index` property
+print(len(df.index))
+2
+These two options give you slightly different information on your DataFrame:
shape property will give you the dimensions of your DataFrame. That means that you will get to know the width and the height of your DataFrame.len() function, in combination with the index property, will only give you information on the height of your DataFrame.This all is totally not extraordinary, though, as you explicitly give in the index property.
You could also use df[0].count() to get to know more about the height of your DataFrame, but this will exclude the NaN values (if there are any). That is why calling ```.count()`` on your DataFrame is not always the better option.
If you want more information on your DataFrame columns, you can always execute list(df.columns.values).
list(df.columns.values)
+[0, 1, 2]
+Before you start with adding, deleting and renaming the components of your DataFrame, you first need to know how you can select these elements. So, how do you do this?
Even though you might still remember how to do it from the previous section: selecting an index, column or value from your DataFrame isn’t that hard, quite the contrary.
It’s similar to what you see in other languages (or packages!) that are used for data analysis. If you aren’t convinced, consider the following: In R, you use the [,] notation to access the data frame’s values.
Now, let’s say you have a DataFrame like this one
def create_df():
+ vals = [[1, 4, 7], [2, 5, 8], [3, 6, 9]]
+ d = { k:v for (k, v) in zip('ABC', vals)}
+ df = pd.DataFrame(d)
+ return df
+df = create_df()
+df
+| A | B | C | |
|---|---|---|---|
| 0 | 1 | 2 | 3 |
| 1 | 4 | 5 | 6 |
| 2 | 7 | 8 | 9 |
And you want to access the value that is at index 0, in column ‘A’.
There are various options that exist to get your value 1 back:
# Using `iloc[]`
+print(df.iloc[0][0])
+
+# Using `loc[]`
+print(df.loc[0]['A'])
+
+# Using `at[]`
+print(df.at[0,'A'])
+
+# Using `iat[]`
+print(df.iat[0,0])
+1
+1
+1
+1
+The most important ones to remember are, without a doubt, .loc[] and .iloc[]. The subtle differences between these two will be discussed in the next sections.
Enough for now about selecting values from your DataFrame. What about selecting rows and columns? In that case, you would use:
# Use `iloc[]` to select row `0`
+print(df.iloc[0])
+A 1
+B 2
+C 3
+Name: 0, dtype: int64
+# Use `loc[]` to select column `'A'`
+print(df.loc[:,'A'])
+0 1
+1 4
+2 7
+Name: A, dtype: int64
+For now, it’s enough to know that you can either access the values by calling them by their label or by their position in the index or column. If you don’t see this, look again at the slight differences in the commands: one time, you see [0][0], the other time, you see [0,'A'] to retrieve your value 1.
Now that you have learned how to select a value from a DataFrame, it’s time to get to the real work and add an index, row or column to it!
Adding an Index to a DataFrame When you create a DataFrame, you have the option to add input to the index argument to make sure that you have the index that you desire. When you don’t specify this, your DataFrame will have, by default, a numerically valued index that starts with 0 and continues until the last row of your DataFrame.
However, even when your index is specified for you automatically, you still have the power to re-use one of your columns and make it your index. You can easily do this by calling set_index() on your DataFrame.
# Print out your DataFrame `df` to check it out
+df
+| A | B | C | |
|---|---|---|---|
| 0 | 1 | 2 | 3 |
| 1 | 4 | 5 | 6 |
| 2 | 7 | 8 | 9 |
# Set 'C' as the index of your DataFrame
+df.set_index('C')
+| A | B | |
|---|---|---|
| C | ||
| 3 | 1 | 2 |
| 6 | 4 | 5 |
| 9 | 7 | 8 |
Adding Rows to a DataFrame Before you can get to the solution, it’s first a good idea to grasp the concept of loc and how it differs from other indexing attributes such as .iloc[] and .ix[]:
.loc[] works on labels of your index. This means that if you give in loc[2], you look for the values of your DataFrame that have an index labeled 2..iloc[] works on the positions in your index. This means that if you give in iloc[2], you look for the values of your DataFrame that are at index ’2`..ix[] is a more complex case: when the index is integer-based, you pass a label to .ix[] .ix[2] then means that you’re looking in your DataFrame for values that have an index labeled 2. This is just like .loc[]! However, if your index is not solely integer-based, ix will work with positions, just like .iloc[].This all might seem very complicated. Let’s illustrate all of this with a small example:
df = pd.DataFrame(data=np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]]),
+ index=[2, 'A', 4],
+ columns=[48, 49, 50])
+df
+| 48 | 49 | 50 | |
|---|---|---|---|
| 2 | 1 | 2 | 3 |
| A | 4 | 5 | 6 |
| 4 | 7 | 8 | 9 |
# Pass `2` to `loc`
+df.loc[2]
+48 1
+49 2
+50 3
+Name: 2, dtype: int64
+# Pass `2` to `iloc`
+print(df.iloc[2])
+48 7
+49 8
+50 9
+Name: 4, dtype: int64
+# Pass `2` to `ix`
+print(df.ix[2])
+---------------------------------------------------------------------------
+
+AttributeError Traceback (most recent call last)
+
+/var/folders/jd/pq0swyt521jb2424d6fvth840000gn/T/ipykernel_23541/3838644476.py in <module>
+ 1 # Pass `2` to `ix`
+----> 2 print(df.ix[2])
+
+
+~/opt/miniconda3/envs/cmu39/lib/python3.9/site-packages/pandas/core/generic.py in __getattr__(self, name)
+ 5485 ):
+ 5486 return self[name]
+-> 5487 return object.__getattribute__(self, name)
+ 5488
+ 5489 def __setattr__(self, name: str, value) -> None:
+
+
+AttributeError: 'DataFrame' object has no attribute 'ix'
+Note that in this case you used an example of a DataFrame that is not solely integer-based as to make it easier for you to understand the differences. You clearly see that passing 2 to .loc[] or .iloc[]/.ix[] does not give back the same result!
.loc[] will go and look at the values that are at label 2. The result that you get back, will be 48 1
+49 2
+50 3
+.iloc[] will go and look at the positions in the index. When you pass 2, you will get back: 48 7
+49 8
+50 9
+.ix[] will have the same behavior as iloc and look at the positions in the index. You will get back the same result as .iloc[].Now that the difference between .iloc[], .loc[] and .ix[] is clear, you are ready to give adding rows to your DataFrame a go!
Tip: as a consequence of what you have just read, you understand now also that the general recommendation is that you use .loc to insert rows in your DataFrame. That is because if you would use df.ix[], you might try to reference a numerically valued index with the index value and accidentally overwrite an existing row of your DataFrame. You better avoid this!
Check out the difference once more in the DataFrame below:
df = pd.DataFrame(data=np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]]),
+ index= [2.5, 12.6, 4.8],
+ columns=[48, 49, 50])
+# There's no index labeled `2`, so you will change the index at position `2`
+df.ix[2] = [60, 50, 40]
+print(df)
+---------------------------------------------------------------------------
+
+AttributeError Traceback (most recent call last)
+
+/var/folders/jd/pq0swyt521jb2424d6fvth840000gn/T/ipykernel_23541/2080172866.py in <module>
+ 1 # There's no index labeled `2`, so you will change the index at position `2`
+----> 2 df.ix[2] = [60, 50, 40]
+ 3 print(df)
+
+
+~/opt/miniconda3/envs/cmu39/lib/python3.9/site-packages/pandas/core/generic.py in __getattr__(self, name)
+ 5485 ):
+ 5486 return self[name]
+-> 5487 return object.__getattribute__(self, name)
+ 5488
+ 5489 def __setattr__(self, name: str, value) -> None:
+
+
+AttributeError: 'DataFrame' object has no attribute 'ix'
+# This will make an index labeled `2` and add the new values
+df.loc[2] = [11, 12, 13]
+print(df)
+ 48 49 50
+2.5 1 2 3
+12.6 4 5 6
+4.8 7 8 9
+2.0 11 12 13
+In some cases, you want to make your index part of your DataFrame. You can easily do this by taking a column from your DataFrame or by referring to a column that you haven’t made yet and assigning it to the .index property, just like this:
df = pd.DataFrame(data=np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]]),
+ columns=['A', 'B', 'C'])
+df
+| A | B | C | |
|---|---|---|---|
| 0 | 1 | 2 | 3 |
| 1 | 4 | 5 | 6 |
| 2 | 7 | 8 | 9 |
# Use `.index`
+df['D'] = df.index
+
+# Print `df`
+print(df)
+ A B C D
+0 1 2 3 0
+1 4 5 6 1
+2 7 8 9 2
+In other words, you tell your DataFrame that it should take column A as its index.
However, if you want to append columns to your DataFrame, you could also follow the same approach as when you would add an index to your DataFrame: you use .loc[] or .iloc[].
In this case, you add a Series to an existing DataFrame with the help of .loc[]:
# Study the DataFrame `df`
+df
+| A | B | C | D | |
|---|---|---|---|---|
| 0 | 1 | 2 | 3 | 0 |
| 1 | 4 | 5 | 6 | 1 |
| 2 | 7 | 8 | 9 | 2 |
# Append a column to `df`
+df.loc[:, 'E'] = pd.Series(['5', '6', '7'], index=df.index)
+
+# Print out `df` again to see the changes
+df
+| A | B | C | D | E | |
|---|---|---|---|---|---|
| 0 | 1 | 2 | 3 | 0 | 5 |
| 1 | 4 | 5 | 6 | 1 | 6 |
| 2 | 7 | 8 | 9 | 2 | 7 |
Remember a Series object is much like a column of a DataFrame; That explains why you can easily add a Series to an existing DataFrame.
Note also that the observation that was made earlier about .loc[] still stays valid, also when you’re adding columns to your DataFrame!
When your index doesn’t look entirely the way you want it to, you can opt to reset it. You can easily do this with .reset_index(). However, you should still watch out, as you can pass several arguments that can make or break the success of your reset:
# Check out the weird index of your dataframe
+df = df.set_index('C')
+df
+| A | B | D | E | |
|---|---|---|---|---|
| C | ||||
| 3 | 1 | 2 | 0 | 5 |
| 6 | 4 | 5 | 1 | 6 |
| 9 | 7 | 8 | 2 | 7 |
# Use `reset_index()` to reset the values.
+df = df.reset_index(level=0, drop=True)
+df
+| A | B | D | E | |
|---|---|---|---|---|
| 0 | 1 | 2 | 0 | 5 |
| 1 | 4 | 5 | 1 | 6 |
| 2 | 7 | 8 | 2 | 7 |
Now that you have seen how to select and add indices, rows, and columns to your DataFrame, it’s time to consider another use case: removing these three from your data structure.
If you want to remove the index from your DataFrame, you should reconsider because DataFrames and Series always have an index.
However, what you can do is, for example:
del df.index.namedf = pd.DataFrame(data=np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9], [40, 50, 60], [23, 35, 37]]),
+ index= [2.5, 12.6, 4.8, 4.8, 2.5],
+ columns=[48, 49, 50])
+
+print(df)
+ 48 49 50
+2.5 1 2 3
+12.6 4 5 6
+4.8 7 8 9
+4.8 40 50 60
+2.5 23 35 37
+x = df.reset_index().drop_duplicates(subset='index', keep='last').set_index('index')
+print(x)
+ 48 49 50
+index
+12.6 4 5 6
+4.8 40 50 60
+2.5 23 35 37
+To get rid of (a selection of) columns from your DataFrame, you can use the drop() method:
# Check out the DataFrame `df`
+df = create_df()
+print(df)
+ A B C
+0 1 2 3
+1 4 5 6
+2 7 8 9
+# Drop the column with label 'A'
+df.drop('A', axis=1, inplace=True)
+print(df)
+ B C
+0 2 3
+1 5 6
+2 8 9
+# Drop the column at position 1
+df.drop(df.columns[[1]], axis=1)
+print(df)
+ B C
+0 2 3
+1 5 6
+2 8 9
+You might think now: well, this is not so straightforward; There are some extra arguments that are passed to the drop() method!
True to delete the column without having to reassign the DataFrame.You can remove duplicate rows from your DataFrame by executing df.drop_duplicates(). You can also remove rows from your DataFrame, taking into account only the duplicate values that exist in one column.
# Check out your DataFrame `df`
+df = create_df()
+df = df.append({ 'A': 1, 'B': 2, 'C': 3}, ignore_index=True)
+print(df)
+ A B C
+0 1 2 3
+1 4 5 6
+2 7 8 9
+3 1 2 3
+# Drop the duplicates in `df`
+x = df.drop_duplicates(subset='A', keep='last')
+print(x)
+ A B C
+1 4 5 6
+2 7 8 9
+3 1 2 3
+
+| \n", + " | Col1 | \n", + "Col2 | \n", + "
|---|---|---|
| Row1 | \n", + "1 | \n", + "2 | \n", + "
| Row2 | \n", + "3 | \n", + "4 | \n", + "
| \n", + " | Col1 | \n", + "Col2 | \n", + "
|---|---|---|
| Row1 | \n", + "1 | \n", + "2 | \n", + "
| Row2 | \n", + "3 | \n", + "4 | \n", + "
| \n", + " | 1 | \n", + "2 | \n", + "3 | \n", + "
|---|---|---|---|
| 0 | \n", + "1 | \n", + "1 | \n", + "2 | \n", + "
| 1 | \n", + "3 | \n", + "2 | \n", + "4 | \n", + "
| \n", + " | A | \n", + "
|---|---|
| 0 | \n", + "4 | \n", + "
| 1 | \n", + "5 | \n", + "
| 2 | \n", + "6 | \n", + "
| 3 | \n", + "7 | \n", + "
| \n", + " | A | \n", + "B | \n", + "C | \n", + "
|---|---|---|---|
| 0 | \n", + "1 | \n", + "2 | \n", + "3 | \n", + "
| 1 | \n", + "4 | \n", + "5 | \n", + "6 | \n", + "
| 2 | \n", + "7 | \n", + "8 | \n", + "9 | \n", + "
| \n", + " | A | \n", + "B | \n", + "C | \n", + "
|---|---|---|---|
| 0 | \n", + "1 | \n", + "2 | \n", + "3 | \n", + "
| 1 | \n", + "4 | \n", + "5 | \n", + "6 | \n", + "
| 2 | \n", + "7 | \n", + "8 | \n", + "9 | \n", + "
| \n", + " | A | \n", + "B | \n", + "
|---|---|---|
| C | \n", + "\n", + " | \n", + " |
| 3 | \n", + "1 | \n", + "2 | \n", + "
| 6 | \n", + "4 | \n", + "5 | \n", + "
| 9 | \n", + "7 | \n", + "8 | \n", + "
| \n", + " | 48 | \n", + "49 | \n", + "50 | \n", + "
|---|---|---|---|
| 2 | \n", + "1 | \n", + "2 | \n", + "3 | \n", + "
| A | \n", + "4 | \n", + "5 | \n", + "6 | \n", + "
| 4 | \n", + "7 | \n", + "8 | \n", + "9 | \n", + "
| \n", + " | A | \n", + "B | \n", + "C | \n", + "
|---|---|---|---|
| 0 | \n", + "1 | \n", + "2 | \n", + "3 | \n", + "
| 1 | \n", + "4 | \n", + "5 | \n", + "6 | \n", + "
| 2 | \n", + "7 | \n", + "8 | \n", + "9 | \n", + "
| \n", + " | A | \n", + "B | \n", + "C | \n", + "D | \n", + "
|---|---|---|---|---|
| 0 | \n", + "1 | \n", + "2 | \n", + "3 | \n", + "0 | \n", + "
| 1 | \n", + "4 | \n", + "5 | \n", + "6 | \n", + "1 | \n", + "
| 2 | \n", + "7 | \n", + "8 | \n", + "9 | \n", + "2 | \n", + "
| \n", + " | A | \n", + "B | \n", + "C | \n", + "D | \n", + "E | \n", + "
|---|---|---|---|---|---|
| 0 | \n", + "1 | \n", + "2 | \n", + "3 | \n", + "0 | \n", + "5 | \n", + "
| 1 | \n", + "4 | \n", + "5 | \n", + "6 | \n", + "1 | \n", + "6 | \n", + "
| 2 | \n", + "7 | \n", + "8 | \n", + "9 | \n", + "2 | \n", + "7 | \n", + "
| \n", + " | A | \n", + "B | \n", + "D | \n", + "E | \n", + "
|---|---|---|---|---|
| C | \n", + "\n", + " | \n", + " | \n", + " | \n", + " |
| 3 | \n", + "1 | \n", + "2 | \n", + "0 | \n", + "5 | \n", + "
| 6 | \n", + "4 | \n", + "5 | \n", + "1 | \n", + "6 | \n", + "
| 9 | \n", + "7 | \n", + "8 | \n", + "2 | \n", + "7 | \n", + "
| \n", + " | A | \n", + "B | \n", + "D | \n", + "E | \n", + "
|---|---|---|---|---|
| 0 | \n", + "1 | \n", + "2 | \n", + "0 | \n", + "5 | \n", + "
| 1 | \n", + "4 | \n", + "5 | \n", + "1 | \n", + "6 | \n", + "
| 2 | \n", + "7 | \n", + "8 | \n", + "2 | \n", + "7 | \n", + "
Python is a great language for doing data analysis, primarily because of the fantastic ecosystem of data-centric Python packages. Pandas is one of those packages, and makes importing and analyzing data much easier. Pandas builds on packages like NumPy and matplotlib to give you a single, convenient, place to do most of your data analysis and visualization work.
In this introduction, we’ll use Pandas to analyze data on video game reviews from IGN, a popular video game review site. The data was scraped by Eric Grinstein, and can be found here. As we analyze the video game reviews, we’ll learn key Pandas concepts like indexing.
Do games like the Witcher 3 tend to get better reviews on the PS4 than the Xbox One? This dataset can help us find out.
The first step we’ll take is to read the data in. The data is stored as a comma-separated values, or csv, file, where each row is separated by a new line, and each column by a comma (,). Here are the first few rows of the ign.csv file:
,score_phrase,title,url,platform,score,genre,editors_choice,release_year,release_month,release_day
+0,Amazing,LittleBigPlanet PS Vita,/games/littlebigplanet-vita/vita-98907,PlayStation Vita,9.0,Platformer,Y,2012,9,12
+1,Amazing,LittleBigPlanet PS Vita -- Marvel Super Hero Edition,/games/littlebigplanet-ps-vita-marvel-super-hero-edition/vita-20027059,PlayStation Vita,9.0,Platformer,Y,2012,9,12
+2,Great,Splice: Tree of Life,/games/splice/ipad-141070,iPad,8.5,Puzzle,N,2012,9,12
+3,Great,NHL 13,/games/nhl-13/xbox-360-128182,Xbox 360,8.5,Sports,N,2012,9,11
+As you can see above, each row in the data represents a single game that was reviewed by IGN. The columns contain information about that game:
There’s also a leading column that contains row index values. We can safely ignore this column, but we’ll dive into what index values are later on. In order to be able to work with the data in Python, we’ll need to read the csv file into a Pandas DataFrame. A DataFrame is a way to represent and work with tabular data. Tabular data has rows and columns, just like our csv file.
In order to read in the data, we’ll need to use the pandas.read_csv function. This function will take in a csv file and return a DataFrame. The below code will:
# disable warnings for lecture
+import warnings
+warnings.filterwarnings('ignore')
+import pandas as pd
+
+reviews = pd.read_csv("ign.csv")
+Once we read in a DataFrame, Pandas gives us two methods that make it fast to print out the data. These functions are:
pandas.DataFrame.head – prints the first N rows of a DataFrame. By default 5.pandas.DataFrame.tail – prints the last N rows of a DataFrame. By default 5.We’ll use the head method to see what’s in reviews:
reviews.head(3)
+| Unnamed: 0 | score_phrase | title | url | platform | score | genre | editors_choice | release_year | release_month | release_day | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 0 | Amazing | LittleBigPlanet PS Vita | /games/littlebigplanet-vita/vita-98907 | PlayStation Vita | 9.0 | Platformer | Y | 2012 | 9 | 12 |
| 1 | 1 | Amazing | LittleBigPlanet PS Vita -- Marvel Super Hero E... | /games/littlebigplanet-ps-vita-marvel-super-he... | PlayStation Vita | 9.0 | Platformer | Y | 2012 | 9 | 12 |
| 2 | 2 | Great | Splice: Tree of Life | /games/splice/ipad-141070 | iPad | 8.5 | Puzzle | N | 2012 | 9 | 12 |
We can also access the pandas.DataFrame.shape property to see row many rows and columns are in reviews:
reviews.shape
+(18625, 11)
+As you can see, everything has been read in properly – we have 18625 rows and 11 columns.
One of the big advantages of Pandas vs just using NumPy is that Pandas allows you to have columns with different data types. reviews has columns that store float values, like score, string values, like score_phrase, and integers, like release_year.
Now that we’ve read the data in properly, let’s work on indexing reviews to get the rows and columns that we want.
Earlier, we used the head method to print the first 5 rows of reviews. We could accomplish the same thing using the pandas.DataFrame.iloc method. The iloc method allows us to retrieve rows and columns by position. In order to do that, we’ll need to specify the positions of the rows that we want, and the positions of the columns that we want as well.
The below code will replicate reviews.head():
reviews.iloc[0:5,:]
+| Unnamed: 0 | score_phrase | title | url | platform | score | genre | editors_choice | release_year | release_month | release_day | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 0 | Amazing | LittleBigPlanet PS Vita | /games/littlebigplanet-vita/vita-98907 | PlayStation Vita | 9.0 | Platformer | Y | 2012 | 9 | 12 |
| 1 | 1 | Amazing | LittleBigPlanet PS Vita -- Marvel Super Hero E... | /games/littlebigplanet-ps-vita-marvel-super-he... | PlayStation Vita | 9.0 | Platformer | Y | 2012 | 9 | 12 |
| 2 | 2 | Great | Splice: Tree of Life | /games/splice/ipad-141070 | iPad | 8.5 | Puzzle | N | 2012 | 9 | 12 |
| 3 | 3 | Great | NHL 13 | /games/nhl-13/xbox-360-128182 | Xbox 360 | 8.5 | Sports | N | 2012 | 9 | 11 |
| 4 | 4 | Great | NHL 13 | /games/nhl-13/ps3-128181 | PlayStation 3 | 8.5 | Sports | N | 2012 | 9 | 11 |
As you can see above, we specified that we wanted rows 0:5. This means that we wanted the rows from position 0 up to, but not including, position 5. The first row is considered to be in position 0. This gives us the rows at positions 0, 1, 2, 3, and 4.
If we leave off the first position value, like :5, it’s assumed we mean 0. If we leave off the last position value, like 0:, it’s assumed we mean the last row or column in the DataFrame.
We wanted all of the columns, so we specified just a colon (:), without any positions. This gave us the columns from 0 to the last column.
Here are some indexing examples, along with the results:
reviews.iloc[:5,:] — the first 5 rows, and all of the columns for those rows.reviews.iloc[:,:] — the entire DataFrame.reviews.iloc[5:,5:] — rows from position 5 onwards, and columns from position 5 onwards.reviews.iloc[:,0] — the first column, and all of the rows for the column.reviews.iloc[9,:] — the 10th row, and all of the columns for that row.Indexing by position is very similar to NumPy indexing.
Now that we know how to index by position, let’s remove the first column, which doesn’t have any useful information:
reviews = reviews.iloc[:,1:]
+reviews.head()
+| score_phrase | title | url | platform | score | genre | editors_choice | release_year | release_month | release_day | |
|---|---|---|---|---|---|---|---|---|---|---|
| 0 | Amazing | LittleBigPlanet PS Vita | /games/littlebigplanet-vita/vita-98907 | PlayStation Vita | 9.0 | Platformer | Y | 2012 | 9 | 12 |
| 1 | Amazing | LittleBigPlanet PS Vita -- Marvel Super Hero E... | /games/littlebigplanet-ps-vita-marvel-super-he... | PlayStation Vita | 9.0 | Platformer | Y | 2012 | 9 | 12 |
| 2 | Great | Splice: Tree of Life | /games/splice/ipad-141070 | iPad | 8.5 | Puzzle | N | 2012 | 9 | 12 |
| 3 | Great | NHL 13 | /games/nhl-13/xbox-360-128182 | Xbox 360 | 8.5 | Sports | N | 2012 | 9 | 11 |
| 4 | Great | NHL 13 | /games/nhl-13/ps3-128181 | PlayStation 3 | 8.5 | Sports | N | 2012 | 9 | 11 |
reviews.shape
+(18625, 10)
+Now that we know how to retrieve rows and columns by position, it’s worth looking into the other major way to work with DataFrames, which is to retrieve rows and columns by label.
A major advantage of Pandas over NumPy is that each of the columns and rows has a label. Working with column positions is possible, but it can be hard to keep track of which number corresponds to which column.
We can work with labels using the pandas.DataFrame.loc method, which allows us to index using labels instead of positions.
We can display the first five rows of reviews using the loc method like this:
reviews.loc[0:5,:]
+| score_phrase | title | url | platform | score | genre | editors_choice | release_year | release_month | release_day | |
|---|---|---|---|---|---|---|---|---|---|---|
| 0 | Amazing | LittleBigPlanet PS Vita | /games/littlebigplanet-vita/vita-98907 | PlayStation Vita | 9.0 | Platformer | Y | 2012 | 9 | 12 |
| 1 | Amazing | LittleBigPlanet PS Vita -- Marvel Super Hero E... | /games/littlebigplanet-ps-vita-marvel-super-he... | PlayStation Vita | 9.0 | Platformer | Y | 2012 | 9 | 12 |
| 2 | Great | Splice: Tree of Life | /games/splice/ipad-141070 | iPad | 8.5 | Puzzle | N | 2012 | 9 | 12 |
| 3 | Great | NHL 13 | /games/nhl-13/xbox-360-128182 | Xbox 360 | 8.5 | Sports | N | 2012 | 9 | 11 |
| 4 | Great | NHL 13 | /games/nhl-13/ps3-128181 | PlayStation 3 | 8.5 | Sports | N | 2012 | 9 | 11 |
| 5 | Good | Total War Battles: Shogun | /games/total-war-battles-shogun/mac-142565 | Macintosh | 7.0 | Strategy | N | 2012 | 9 | 11 |
The above doesn’t actually look much different from reviews.iloc[0:5,:]. This is because while row labels can take on any values, our row labels match the positions exactly. You can see the row labels on the very left of the table above (they’re in bold). You can also see them by accessing the index property of a DataFrame. We’ll display the row indexes for reviews:
reviews.index
+RangeIndex(start=0, stop=18625, step=1)
+# pull out the 1st twenty indexes
+list(reviews.index)[:20]
+[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
+Indexes don’t always have to match up with positions, though. In the below code cell, we’ll:
Get row 10 to row 20 of reviews, and assign the result to some_reviews. Display the first 5 rows of some_reviews.
some_reviews = reviews.iloc[10:20,]
+some_reviews.head()
+| score_phrase | title | url | platform | score | genre | editors_choice | release_year | release_month | release_day | |
|---|---|---|---|---|---|---|---|---|---|---|
| 10 | Good | Tekken Tag Tournament 2 | /games/tekken-tag-tournament-2/ps3-124584 | PlayStation 3 | 7.5 | Fighting | N | 2012 | 9 | 11 |
| 11 | Good | Tekken Tag Tournament 2 | /games/tekken-tag-tournament-2/xbox-360-124581 | Xbox 360 | 7.5 | Fighting | N | 2012 | 9 | 11 |
| 12 | Good | Wild Blood | /games/wild-blood/iphone-139363 | iPhone | 7.0 | NaN | N | 2012 | 9 | 10 |
| 13 | Amazing | Mark of the Ninja | /games/mark-of-the-ninja-135615/xbox-360-129276 | Xbox 360 | 9.0 | Action, Adventure | Y | 2012 | 9 | 7 |
| 14 | Amazing | Mark of the Ninja | /games/mark-of-the-ninja-135615/pc-143761 | PC | 9.0 | Action, Adventure | Y | 2012 | 9 | 7 |
As we mentioned earlier, column labels can make life much easier when you’re working with data. We can specify column labels in the loc method to retrieve columns by label instead of by position.
reviews.loc[:5, "score"]
+0 9.0
+1 9.0
+2 8.5
+3 8.5
+4 8.5
+5 7.0
+Name: score, dtype: float64
+We can also specify more than one column at a time by passing in a list:
reviews.loc[:5, ["score", "release_year"]]
+| score | release_year | |
|---|---|---|
| 0 | 9.0 | 2012 |
| 1 | 9.0 | 2012 |
| 2 | 8.5 | 2012 |
| 3 | 8.5 | 2012 |
| 4 | 8.5 | 2012 |
| 5 | 7.0 | 2012 |
reviews.loc[[0, 3], ["score", "release_year"]]
+| score | release_year | |
|---|---|---|
| 0 | 9.0 | 2012 |
| 3 | 8.5 | 2012 |
We can retrieve an individual column in Pandas a few different ways. So far, we’ve seen two types of syntax for this:
reviews.iloc[:,1] — will retrieve the second column.reviews.loc[:,"score_phrase"] — will also retrieve the second column.There’s a third, even easier, way to retrieve a whole column. We can just specify the column name in square brackets, like with a dictionary:
reviews["score"].head()
+0 9.0
+1 9.0
+2 8.5
+3 8.5
+4 8.5
+Name: score, dtype: float64
+We can also use lists of columns with this method:
x = reviews[["score", "release_year"]]
+x.head()
+| score | release_year | |
|---|---|---|
| 0 | 9.0 | 2012 |
| 1 | 9.0 | 2012 |
| 2 | 8.5 | 2012 |
| 3 | 8.5 | 2012 |
| 4 | 8.5 | 2012 |
When we retrieve a single column, we’re actually retrieving a Pandas Series object. A DataFrame stores tabular data, but a Series stores a single column or row of data.
We can verify that a single column is a Series:
type(reviews["score"])
+pandas.core.series.Series
+We can create a Series manually to better understand how it works. To create a Series, we pass a list or NumPy array into the Series object when we instantiate it:
s1 = pd.Series([1, 2])
+s1
+0 1
+1 2
+dtype: int64
+A Series can contain any type of data, including mixed types. Here, we create a Series that contains string objects:
s2 = pd.Series(["Boris Yeltsin", "Mikhail Gorbachev"])
+s2
+0 Boris Yeltsin
+1 Mikhail Gorbachev
+dtype: object
+We can create a DataFrame by passing multiple Series into the DataFrame class. Here, we pass in the two Series objects we just created, s1 as the first row, and s2 as the second row:
# create a dataframe from two series
+x = pd.DataFrame()
+x['rank'] = s1
+x['name'] = s2
+x
+| rank | name | |
|---|---|---|
| 0 | 1 | Boris Yeltsin |
| 1 | 2 | Mikhail Gorbachev |
# create a datafrme from a dictionary
+y = pd.DataFrame({
+ 'rank': s1,
+ 'name': s2
+})
+y
+| rank | name | |
|---|---|---|
| 0 | 1 | Boris Yeltsin |
| 1 | 2 | Mikhail Gorbachev |
# create a dataframe from a list
+pd.DataFrame([s1, s2])
+| 0 | 1 | |
|---|---|---|
| 0 | 1 | 2 |
| 1 | Boris Yeltsin | Mikhail Gorbachev |
We can also accomplish the same thing with a list of lists. Each inner list is treated as a row in the resulting DataFrame:
# create a dataframe from a list of lists
+pd.DataFrame(
+ [
+ [1, "Boris Yeltsin"],
+ [2, "Mikhail Gorbachev"]
+ ]
+)
+| 0 | 1 | |
|---|---|---|
| 0 | 1 | Boris Yeltsin |
| 1 | 2 | Mikhail Gorbachev |
We can specify the column labels when we create a DataFrame:
# specify column labels
+pd.DataFrame(
+ [
+ [1, "Boris Yeltsin"],
+ [2, "Mikhail Gorbachev"]
+ ],
+ columns=["rank", "name"]
+)
+| rank | name | |
|---|---|---|
| 0 | 1 | Boris Yeltsin |
| 1 | 2 | Mikhail Gorbachev |
As well as the row labels (the index):
# specify the index labels
+frame = pd.DataFrame(
+ [
+ [1,2],
+ ["Boris Yeltsin", "Mikhail Gorbachev"]
+ ],
+ index=["rank", "name"],
+ columns=["person1", "person2"]
+)
+frame
+| person1 | person2 | |
|---|---|---|
| rank | 1 | 2 |
| name | Boris Yeltsin | Mikhail Gorbachev |
We’re then able index the DataFrame using the labels:
frame.loc["rank":"name", "person1"]
+rank 1
+name Boris Yeltsin
+Name: person1, dtype: object
+We can skip specifying the columns keyword argument if we pass a dictionary into the DataFrame constructor. This will automatically setup column names:
frame = pd.DataFrame(
+ {
+ "person1": [1, "Boris Yeltsin"],
+ "person2": [2, "Mikhail Gorbachev"]
+ }
+)
+frame
+| person1 | person2 | |
|---|---|---|
| 0 | 1 | 2 |
| 1 | Boris Yeltsin | Mikhail Gorbachev |
As we mentioned earlier, each column in a DataFrame is a Series object:
type(reviews["title"])
+pandas.core.series.Series
+We can call most of the same methods on a Series object that we can on a DataFrame, including head:
reviews["title"].head()
+0 LittleBigPlanet PS Vita
+1 LittleBigPlanet PS Vita -- Marvel Super Hero E...
+2 Splice: Tree of Life
+3 NHL 13
+4 NHL 13
+Name: title, dtype: object
+Pandas Series and DataFrame also have other methods that make calculations simpler. For example, we can use the pandas.Series.mean method to find the mean of a Series:
reviews["score"].mean()
+6.950459060402685
+We can also call the similar pandas.DataFrame.mean method, which will find the mean of each numerical column in a DataFrame by default:
x = reviews.mean()
+x
+score 6.950459
+release_year 2006.515329
+release_month 7.138470
+release_day 15.603866
+dtype: float64
+We can modify the axis keyword argument to mean in order to compute the mean of each row or of each column. By default, axis is equal to 0, and will compute the mean of each column. We can also set it to 1 to compute the mean of each row. Note that this will only compute the mean of the numerical values in each row:
reviews.mean(axis=1).head()
+0 510.500
+1 510.500
+2 510.375
+3 510.125
+4 510.125
+dtype: float64
+There are quite a few methods on Series and DataFrame that behave like mean. Here are some handy ones:
pandas.DataFrame.corr — finds the correlation between columns in a DataFrame.pandas.DataFrame.count — counts the number of non-null values in each DataFrame column.pandas.DataFrame.max — finds the highest value in each column.pandas.DataFrame.min — finds the lowest value in each column.pandas.DataFrame.median — finds the median of each column.pandas.DataFrame.std — finds the standard deviation of each column.We can use the corr method to see if any columns correlation with score. For instance, this would tell us if games released more recently have been getting higher reviews (release_year), or if games released towards the end of the year score better (release_month):
reviews.corr()
+| score | release_year | release_month | release_day | |
|---|---|---|---|---|
| score | 1.000000 | 0.062716 | 0.007632 | 0.020079 |
| release_year | 0.062716 | 1.000000 | -0.115515 | 0.016867 |
| release_month | 0.007632 | -0.115515 | 1.000000 | -0.067964 |
| release_day | 0.020079 | 0.016867 | -0.067964 | 1.000000 |
As you can see above, none of our numeric columns correlates with score, meaning that release timing doesn’t linearly relate to review score.
We can also perform math operations on Series or DataFrame objects. For example, we can divide every value in the score column by 2 to switch the scale from 0-10 to 0-5:
reviews["score"] / 2
+0 4.50
+1 4.50
+2 4.25
+3 4.25
+4 4.25
+ ...
+18620 3.80
+18621 4.50
+18622 2.90
+18623 5.00
+18624 5.00
+Name: score, Length: 18625, dtype: float64
+All the common mathematical operators that work in Python, like +, -, *, /, and ^ will work, and will apply to each element in a DataFrame or a Series.
As we saw above, the mean of all the values in the score column of reviews is around 7. What if we wanted to find all the games that got an above average score? We could start by doing a comparison. The comparison compares each value in a Series to a specified value, then generate a Series full of Boolean values indicating the status of the comparison. For example, we can see which of the rows have a score value higher than 7:
score_filter = reviews["score"] > 7
+score_filter
+0 True
+1 True
+2 True
+3 True
+4 True
+ ...
+18620 True
+18621 True
+18622 False
+18623 True
+18624 True
+Name: score, Length: 18625, dtype: bool
+Once we have a Boolean Series, we can use it to select only rows in a DataFrame where the Series contains the value True. So, we could only select rows in reviews where score is greater than 7:
filtered_reviews = reviews[score_filter]
+filtered_reviews.head()
+| score_phrase | title | url | platform | score | genre | editors_choice | release_year | release_month | release_day | |
|---|---|---|---|---|---|---|---|---|---|---|
| 0 | Amazing | LittleBigPlanet PS Vita | /games/littlebigplanet-vita/vita-98907 | PlayStation Vita | 9.0 | Platformer | Y | 2012 | 9 | 12 |
| 1 | Amazing | LittleBigPlanet PS Vita -- Marvel Super Hero E... | /games/littlebigplanet-ps-vita-marvel-super-he... | PlayStation Vita | 9.0 | Platformer | Y | 2012 | 9 | 12 |
| 2 | Great | Splice: Tree of Life | /games/splice/ipad-141070 | iPad | 8.5 | Puzzle | N | 2012 | 9 | 12 |
| 3 | Great | NHL 13 | /games/nhl-13/xbox-360-128182 | Xbox 360 | 8.5 | Sports | N | 2012 | 9 | 11 |
| 4 | Great | NHL 13 | /games/nhl-13/ps3-128181 | PlayStation 3 | 8.5 | Sports | N | 2012 | 9 | 11 |
It’s possible to use multiple conditions for filtering. Let’s say we want to find games released for the Xbox One that have a score of more than 7. In the below code, we:
# setup the boolean filtr
+xbox_one_filter = (reviews["score"] > 7) & (reviews["platform"] == "Xbox One")
+
+# select the data based on the filter
+filtered_reviews = reviews[xbox_one_filter]
+
+# display the first 5 results
+filtered_reviews.head()
+| score_phrase | title | url | platform | score | genre | editors_choice | release_year | release_month | release_day | |
|---|---|---|---|---|---|---|---|---|---|---|
| 17137 | Amazing | Gone Home | /games/gone-home/xbox-one-20014361 | Xbox One | 9.5 | Simulation | Y | 2013 | 8 | 15 |
| 17197 | Amazing | Rayman Legends | /games/rayman-legends/xbox-one-20008449 | Xbox One | 9.5 | Platformer | Y | 2013 | 8 | 26 |
| 17295 | Amazing | LEGO Marvel Super Heroes | /games/lego-marvel-super-heroes/xbox-one-20000826 | Xbox One | 9.0 | Action | Y | 2013 | 10 | 22 |
| 17313 | Great | Dead Rising 3 | /games/dead-rising-3/xbox-one-124306 | Xbox One | 8.3 | Action | N | 2013 | 11 | 18 |
| 17317 | Great | Killer Instinct | /games/killer-instinct-2013/xbox-one-20000538 | Xbox One | 8.4 | Fighting | N | 2013 | 11 | 18 |
When filtering with multiple conditions, it’s important to put each condition in parentheses (), and separate them with a single ampersand &.
Now that we know how to filter, we can create plots to observe the review distribution for the Xbox One vs the review distribution for the PlayStation 4. This will help us figure out which console has better games. We can do this via a histogram, which will plot the frequencies for different score ranges. This will tell us which console has more highly reviewed games.
We can make a histogram for each console using the pandas.DataFrame.plot method. This method utilizes matplotlib, the popular Python plotting library, under the hood to generate good-looking plots.
The plot method defaults to drawing a line graph. We’ll need to pass in the keyword argument kind="hist" to draw a histogram instead.
In the below code, we:
%matplotlib inline to set up plotting inside a Jupyter notebook.reviews["platform"].unique()
+array(['PlayStation Vita', 'iPad', 'Xbox 360', 'PlayStation 3',
+ 'Macintosh', 'PC', 'iPhone', 'Nintendo DS', 'Nintendo 3DS',
+ 'Android', 'Wii', 'PlayStation 4', 'Wii U', 'Linux',
+ 'PlayStation Portable', 'PlayStation', 'Nintendo 64', 'Saturn',
+ 'Lynx', 'Game Boy', 'Game Boy Color', 'NeoGeo Pocket Color',
+ 'Game.Com', 'Dreamcast', 'Dreamcast VMU', 'WonderSwan', 'Arcade',
+ 'Nintendo 64DD', 'PlayStation 2', 'WonderSwan Color',
+ 'Game Boy Advance', 'Xbox', 'GameCube', 'DVD / HD Video Game',
+ 'Wireless', 'Pocket PC', 'N-Gage', 'NES', 'iPod', 'Genesis',
+ 'TurboGrafx-16', 'Super NES', 'NeoGeo', 'Master System',
+ 'Atari 5200', 'TurboGrafx-CD', 'Atari 2600', 'Sega 32X', 'Vectrex',
+ 'Commodore 64/128', 'Sega CD', 'Nintendo DSi', 'Windows Phone',
+ 'Web Games', 'Xbox One', 'Windows Surface', 'Ouya',
+ 'New Nintendo 3DS', 'SteamOS'], dtype=object)
+# enable display of chart inline
+%matplotlib inline
+
+# setup the filter
+platform_filter = reviews["platform"] == "Xbox One"
+
+# select and plot
+reviews[platform_filter]["score"].plot(kind="hist")
+<AxesSubplot:ylabel='Frequency'>
+
We can also do the same for the PS4:
# setup the filter
+mask = reviews["platform"].isin(["Xbox One", "PlayStation 4"])
+
+# select and plot
+reviews[mask]["score"].plot(kind="hist")
+<AxesSubplot:ylabel='Frequency'>
+
It appears from our histogram that the PlayStation 4 has many more highly rated games than the Xbox One.
# plot all
+filtered_reviews["score"].hist()
+<AxesSubplot:>
+
+| \n", + " | Unnamed: 0 | \n", + "score_phrase | \n", + "title | \n", + "url | \n", + "platform | \n", + "score | \n", + "genre | \n", + "editors_choice | \n", + "release_year | \n", + "release_month | \n", + "release_day | \n", + "
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", + "0 | \n", + "Amazing | \n", + "LittleBigPlanet PS Vita | \n", + "/games/littlebigplanet-vita/vita-98907 | \n", + "PlayStation Vita | \n", + "9.0 | \n", + "Platformer | \n", + "Y | \n", + "2012 | \n", + "9 | \n", + "12 | \n", + "
| 1 | \n", + "1 | \n", + "Amazing | \n", + "LittleBigPlanet PS Vita -- Marvel Super Hero E... | \n", + "/games/littlebigplanet-ps-vita-marvel-super-he... | \n", + "PlayStation Vita | \n", + "9.0 | \n", + "Platformer | \n", + "Y | \n", + "2012 | \n", + "9 | \n", + "12 | \n", + "
| 2 | \n", + "2 | \n", + "Great | \n", + "Splice: Tree of Life | \n", + "/games/splice/ipad-141070 | \n", + "iPad | \n", + "8.5 | \n", + "Puzzle | \n", + "N | \n", + "2012 | \n", + "9 | \n", + "12 | \n", + "
| \n", + " | Unnamed: 0 | \n", + "score_phrase | \n", + "title | \n", + "url | \n", + "platform | \n", + "score | \n", + "genre | \n", + "editors_choice | \n", + "release_year | \n", + "release_month | \n", + "release_day | \n", + "
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", + "0 | \n", + "Amazing | \n", + "LittleBigPlanet PS Vita | \n", + "/games/littlebigplanet-vita/vita-98907 | \n", + "PlayStation Vita | \n", + "9.0 | \n", + "Platformer | \n", + "Y | \n", + "2012 | \n", + "9 | \n", + "12 | \n", + "
| 1 | \n", + "1 | \n", + "Amazing | \n", + "LittleBigPlanet PS Vita -- Marvel Super Hero E... | \n", + "/games/littlebigplanet-ps-vita-marvel-super-he... | \n", + "PlayStation Vita | \n", + "9.0 | \n", + "Platformer | \n", + "Y | \n", + "2012 | \n", + "9 | \n", + "12 | \n", + "
| 2 | \n", + "2 | \n", + "Great | \n", + "Splice: Tree of Life | \n", + "/games/splice/ipad-141070 | \n", + "iPad | \n", + "8.5 | \n", + "Puzzle | \n", + "N | \n", + "2012 | \n", + "9 | \n", + "12 | \n", + "
| 3 | \n", + "3 | \n", + "Great | \n", + "NHL 13 | \n", + "/games/nhl-13/xbox-360-128182 | \n", + "Xbox 360 | \n", + "8.5 | \n", + "Sports | \n", + "N | \n", + "2012 | \n", + "9 | \n", + "11 | \n", + "
| 4 | \n", + "4 | \n", + "Great | \n", + "NHL 13 | \n", + "/games/nhl-13/ps3-128181 | \n", + "PlayStation 3 | \n", + "8.5 | \n", + "Sports | \n", + "N | \n", + "2012 | \n", + "9 | \n", + "11 | \n", + "
| \n", + " | score_phrase | \n", + "title | \n", + "url | \n", + "platform | \n", + "score | \n", + "genre | \n", + "editors_choice | \n", + "release_year | \n", + "release_month | \n", + "release_day | \n", + "
|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", + "Amazing | \n", + "LittleBigPlanet PS Vita | \n", + "/games/littlebigplanet-vita/vita-98907 | \n", + "PlayStation Vita | \n", + "9.0 | \n", + "Platformer | \n", + "Y | \n", + "2012 | \n", + "9 | \n", + "12 | \n", + "
| 1 | \n", + "Amazing | \n", + "LittleBigPlanet PS Vita -- Marvel Super Hero E... | \n", + "/games/littlebigplanet-ps-vita-marvel-super-he... | \n", + "PlayStation Vita | \n", + "9.0 | \n", + "Platformer | \n", + "Y | \n", + "2012 | \n", + "9 | \n", + "12 | \n", + "
| 2 | \n", + "Great | \n", + "Splice: Tree of Life | \n", + "/games/splice/ipad-141070 | \n", + "iPad | \n", + "8.5 | \n", + "Puzzle | \n", + "N | \n", + "2012 | \n", + "9 | \n", + "12 | \n", + "
| 3 | \n", + "Great | \n", + "NHL 13 | \n", + "/games/nhl-13/xbox-360-128182 | \n", + "Xbox 360 | \n", + "8.5 | \n", + "Sports | \n", + "N | \n", + "2012 | \n", + "9 | \n", + "11 | \n", + "
| 4 | \n", + "Great | \n", + "NHL 13 | \n", + "/games/nhl-13/ps3-128181 | \n", + "PlayStation 3 | \n", + "8.5 | \n", + "Sports | \n", + "N | \n", + "2012 | \n", + "9 | \n", + "11 | \n", + "
| \n", + " | score_phrase | \n", + "title | \n", + "url | \n", + "platform | \n", + "score | \n", + "genre | \n", + "editors_choice | \n", + "release_year | \n", + "release_month | \n", + "release_day | \n", + "
|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", + "Amazing | \n", + "LittleBigPlanet PS Vita | \n", + "/games/littlebigplanet-vita/vita-98907 | \n", + "PlayStation Vita | \n", + "9.0 | \n", + "Platformer | \n", + "Y | \n", + "2012 | \n", + "9 | \n", + "12 | \n", + "
| 1 | \n", + "Amazing | \n", + "LittleBigPlanet PS Vita -- Marvel Super Hero E... | \n", + "/games/littlebigplanet-ps-vita-marvel-super-he... | \n", + "PlayStation Vita | \n", + "9.0 | \n", + "Platformer | \n", + "Y | \n", + "2012 | \n", + "9 | \n", + "12 | \n", + "
| 2 | \n", + "Great | \n", + "Splice: Tree of Life | \n", + "/games/splice/ipad-141070 | \n", + "iPad | \n", + "8.5 | \n", + "Puzzle | \n", + "N | \n", + "2012 | \n", + "9 | \n", + "12 | \n", + "
| 3 | \n", + "Great | \n", + "NHL 13 | \n", + "/games/nhl-13/xbox-360-128182 | \n", + "Xbox 360 | \n", + "8.5 | \n", + "Sports | \n", + "N | \n", + "2012 | \n", + "9 | \n", + "11 | \n", + "
| 4 | \n", + "Great | \n", + "NHL 13 | \n", + "/games/nhl-13/ps3-128181 | \n", + "PlayStation 3 | \n", + "8.5 | \n", + "Sports | \n", + "N | \n", + "2012 | \n", + "9 | \n", + "11 | \n", + "
| 5 | \n", + "Good | \n", + "Total War Battles: Shogun | \n", + "/games/total-war-battles-shogun/mac-142565 | \n", + "Macintosh | \n", + "7.0 | \n", + "Strategy | \n", + "N | \n", + "2012 | \n", + "9 | \n", + "11 | \n", + "
| \n", + " | score_phrase | \n", + "title | \n", + "url | \n", + "platform | \n", + "score | \n", + "genre | \n", + "editors_choice | \n", + "release_year | \n", + "release_month | \n", + "release_day | \n", + "
|---|---|---|---|---|---|---|---|---|---|---|
| 10 | \n", + "Good | \n", + "Tekken Tag Tournament 2 | \n", + "/games/tekken-tag-tournament-2/ps3-124584 | \n", + "PlayStation 3 | \n", + "7.5 | \n", + "Fighting | \n", + "N | \n", + "2012 | \n", + "9 | \n", + "11 | \n", + "
| 11 | \n", + "Good | \n", + "Tekken Tag Tournament 2 | \n", + "/games/tekken-tag-tournament-2/xbox-360-124581 | \n", + "Xbox 360 | \n", + "7.5 | \n", + "Fighting | \n", + "N | \n", + "2012 | \n", + "9 | \n", + "11 | \n", + "
| 12 | \n", + "Good | \n", + "Wild Blood | \n", + "/games/wild-blood/iphone-139363 | \n", + "iPhone | \n", + "7.0 | \n", + "NaN | \n", + "N | \n", + "2012 | \n", + "9 | \n", + "10 | \n", + "
| 13 | \n", + "Amazing | \n", + "Mark of the Ninja | \n", + "/games/mark-of-the-ninja-135615/xbox-360-129276 | \n", + "Xbox 360 | \n", + "9.0 | \n", + "Action, Adventure | \n", + "Y | \n", + "2012 | \n", + "9 | \n", + "7 | \n", + "
| 14 | \n", + "Amazing | \n", + "Mark of the Ninja | \n", + "/games/mark-of-the-ninja-135615/pc-143761 | \n", + "PC | \n", + "9.0 | \n", + "Action, Adventure | \n", + "Y | \n", + "2012 | \n", + "9 | \n", + "7 | \n", + "
| \n", + " | score | \n", + "release_year | \n", + "
|---|---|---|
| 0 | \n", + "9.0 | \n", + "2012 | \n", + "
| 1 | \n", + "9.0 | \n", + "2012 | \n", + "
| 2 | \n", + "8.5 | \n", + "2012 | \n", + "
| 3 | \n", + "8.5 | \n", + "2012 | \n", + "
| 4 | \n", + "8.5 | \n", + "2012 | \n", + "
| 5 | \n", + "7.0 | \n", + "2012 | \n", + "
| \n", + " | score | \n", + "release_year | \n", + "
|---|---|---|
| 0 | \n", + "9.0 | \n", + "2012 | \n", + "
| 3 | \n", + "8.5 | \n", + "2012 | \n", + "
| \n", + " | score | \n", + "release_year | \n", + "
|---|---|---|
| 0 | \n", + "9.0 | \n", + "2012 | \n", + "
| 1 | \n", + "9.0 | \n", + "2012 | \n", + "
| 2 | \n", + "8.5 | \n", + "2012 | \n", + "
| 3 | \n", + "8.5 | \n", + "2012 | \n", + "
| 4 | \n", + "8.5 | \n", + "2012 | \n", + "
| \n", + " | rank | \n", + "name | \n", + "
|---|---|---|
| 0 | \n", + "1 | \n", + "Boris Yeltsin | \n", + "
| 1 | \n", + "2 | \n", + "Mikhail Gorbachev | \n", + "
| \n", + " | rank | \n", + "name | \n", + "
|---|---|---|
| 0 | \n", + "1 | \n", + "Boris Yeltsin | \n", + "
| 1 | \n", + "2 | \n", + "Mikhail Gorbachev | \n", + "
| \n", + " | 0 | \n", + "1 | \n", + "
|---|---|---|
| 0 | \n", + "1 | \n", + "2 | \n", + "
| 1 | \n", + "Boris Yeltsin | \n", + "Mikhail Gorbachev | \n", + "
| \n", + " | 0 | \n", + "1 | \n", + "
|---|---|---|
| 0 | \n", + "1 | \n", + "Boris Yeltsin | \n", + "
| 1 | \n", + "2 | \n", + "Mikhail Gorbachev | \n", + "
| \n", + " | rank | \n", + "name | \n", + "
|---|---|---|
| 0 | \n", + "1 | \n", + "Boris Yeltsin | \n", + "
| 1 | \n", + "2 | \n", + "Mikhail Gorbachev | \n", + "
| \n", + " | person1 | \n", + "person2 | \n", + "
|---|---|---|
| rank | \n", + "1 | \n", + "2 | \n", + "
| name | \n", + "Boris Yeltsin | \n", + "Mikhail Gorbachev | \n", + "
| \n", + " | person1 | \n", + "person2 | \n", + "
|---|---|---|
| 0 | \n", + "1 | \n", + "2 | \n", + "
| 1 | \n", + "Boris Yeltsin | \n", + "Mikhail Gorbachev | \n", + "
| \n", + " | score | \n", + "release_year | \n", + "release_month | \n", + "release_day | \n", + "
|---|---|---|---|---|
| score | \n", + "1.000000 | \n", + "0.062716 | \n", + "0.007632 | \n", + "0.020079 | \n", + "
| release_year | \n", + "0.062716 | \n", + "1.000000 | \n", + "-0.115515 | \n", + "0.016867 | \n", + "
| release_month | \n", + "0.007632 | \n", + "-0.115515 | \n", + "1.000000 | \n", + "-0.067964 | \n", + "
| release_day | \n", + "0.020079 | \n", + "0.016867 | \n", + "-0.067964 | \n", + "1.000000 | \n", + "
| \n", + " | score_phrase | \n", + "title | \n", + "url | \n", + "platform | \n", + "score | \n", + "genre | \n", + "editors_choice | \n", + "release_year | \n", + "release_month | \n", + "release_day | \n", + "
|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", + "Amazing | \n", + "LittleBigPlanet PS Vita | \n", + "/games/littlebigplanet-vita/vita-98907 | \n", + "PlayStation Vita | \n", + "9.0 | \n", + "Platformer | \n", + "Y | \n", + "2012 | \n", + "9 | \n", + "12 | \n", + "
| 1 | \n", + "Amazing | \n", + "LittleBigPlanet PS Vita -- Marvel Super Hero E... | \n", + "/games/littlebigplanet-ps-vita-marvel-super-he... | \n", + "PlayStation Vita | \n", + "9.0 | \n", + "Platformer | \n", + "Y | \n", + "2012 | \n", + "9 | \n", + "12 | \n", + "
| 2 | \n", + "Great | \n", + "Splice: Tree of Life | \n", + "/games/splice/ipad-141070 | \n", + "iPad | \n", + "8.5 | \n", + "Puzzle | \n", + "N | \n", + "2012 | \n", + "9 | \n", + "12 | \n", + "
| 3 | \n", + "Great | \n", + "NHL 13 | \n", + "/games/nhl-13/xbox-360-128182 | \n", + "Xbox 360 | \n", + "8.5 | \n", + "Sports | \n", + "N | \n", + "2012 | \n", + "9 | \n", + "11 | \n", + "
| 4 | \n", + "Great | \n", + "NHL 13 | \n", + "/games/nhl-13/ps3-128181 | \n", + "PlayStation 3 | \n", + "8.5 | \n", + "Sports | \n", + "N | \n", + "2012 | \n", + "9 | \n", + "11 | \n", + "
| \n", + " | score_phrase | \n", + "title | \n", + "url | \n", + "platform | \n", + "score | \n", + "genre | \n", + "editors_choice | \n", + "release_year | \n", + "release_month | \n", + "release_day | \n", + "
|---|---|---|---|---|---|---|---|---|---|---|
| 17137 | \n", + "Amazing | \n", + "Gone Home | \n", + "/games/gone-home/xbox-one-20014361 | \n", + "Xbox One | \n", + "9.5 | \n", + "Simulation | \n", + "Y | \n", + "2013 | \n", + "8 | \n", + "15 | \n", + "
| 17197 | \n", + "Amazing | \n", + "Rayman Legends | \n", + "/games/rayman-legends/xbox-one-20008449 | \n", + "Xbox One | \n", + "9.5 | \n", + "Platformer | \n", + "Y | \n", + "2013 | \n", + "8 | \n", + "26 | \n", + "
| 17295 | \n", + "Amazing | \n", + "LEGO Marvel Super Heroes | \n", + "/games/lego-marvel-super-heroes/xbox-one-20000826 | \n", + "Xbox One | \n", + "9.0 | \n", + "Action | \n", + "Y | \n", + "2013 | \n", + "10 | \n", + "22 | \n", + "
| 17313 | \n", + "Great | \n", + "Dead Rising 3 | \n", + "/games/dead-rising-3/xbox-one-124306 | \n", + "Xbox One | \n", + "8.3 | \n", + "Action | \n", + "N | \n", + "2013 | \n", + "11 | \n", + "18 | \n", + "
| 17317 | \n", + "Great | \n", + "Killer Instinct | \n", + "/games/killer-instinct-2013/xbox-one-20000538 | \n", + "Xbox One | \n", + "8.4 | \n", + "Fighting | \n", + "N | \n", + "2013 | \n", + "11 | \n", + "18 | \n", + "
In this tutorial, we’ll dive into one of the most powerful aspects of pandas — its grouping and aggregation functionality. With this functionality, it’s dead simple to compute group summary statistics, discover patterns, and slice up your data in various ways.
Since Thanksgiving was just last week, we’ll use a dataset on what Americans typically eat for Thanksgiving dinner as we explore the pandas library. You can download the dataset here.
It contains 1058 online survey responses collected by FiveThirtyEight. Each survey respondent was asked questions about what they typically eat for Thanksgiving, along with some demographic questions, like their gender, income, and location. This dataset will allow us to discover regional and income-based patterns in what Americans eat for Thanksgiving dinner. As we explore the data and try to find patterns, we’ll be heavily using the grouping and aggregation functionality of pandas.
Our first step is to read in the data and do some preliminary exploration. This will help us figure out how we want to approach creating groups and finding patterns.
As you may recall from part one of this tutorial, we can read in the data using the pandas.read_csv function. The data is stored using Latin-1 encoding, so we additionally need to specify the encoding keyword argument. If we don’t, pandas won’t be able to load in the data, and we’ll get an error:
# disable warnings for lecture
+import warnings
+warnings.filterwarnings('ignore')
+import pandas as pd
+
+data = pd.read_csv("thanksgiving-2015-poll-data.csv", encoding="Latin-1")
+data.head()
+| RespondentID | Do you celebrate Thanksgiving? | What is typically the main dish at your Thanksgiving dinner? | What is typically the main dish at your Thanksgiving dinner? - Other (please specify) | How is the main dish typically cooked? | How is the main dish typically cooked? - Other (please specify) | What kind of stuffing/dressing do you typically have? | What kind of stuffing/dressing do you typically have? - Other (please specify) | What type of cranberry saucedo you typically have? | What type of cranberry saucedo you typically have? - Other (please specify) | ... | Have you ever tried to meet up with hometown friends on Thanksgiving night? | Have you ever attended a "Friendsgiving?" | Will you shop any Black Friday sales on Thanksgiving Day? | Do you work in retail? | Will you employer make you work on Black Friday? | How would you describe where you live? | Age | What is your gender? | How much total combined money did all members of your HOUSEHOLD earn last year? | US Region | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 4337954960 | Yes | Turkey | NaN | Baked | NaN | Bread-based | NaN | None | NaN | ... | Yes | No | No | No | NaN | Suburban | 18 - 29 | Male | $75,000 to $99,999 | Middle Atlantic |
| 1 | 4337951949 | Yes | Turkey | NaN | Baked | NaN | Bread-based | NaN | Other (please specify) | Homemade cranberry gelatin ring | ... | No | No | Yes | No | NaN | Rural | 18 - 29 | Female | $50,000 to $74,999 | East South Central |
| 2 | 4337935621 | Yes | Turkey | NaN | Roasted | NaN | Rice-based | NaN | Homemade | NaN | ... | Yes | Yes | Yes | No | NaN | Suburban | 18 - 29 | Male | $0 to $9,999 | Mountain |
| 3 | 4337933040 | Yes | Turkey | NaN | Baked | NaN | Bread-based | NaN | Homemade | NaN | ... | Yes | No | No | No | NaN | Urban | 30 - 44 | Male | $200,000 and up | Pacific |
| 4 | 4337931983 | Yes | Tofurkey | NaN | Baked | NaN | Bread-based | NaN | Canned | NaN | ... | Yes | No | No | No | NaN | Urban | 30 - 44 | Male | $100,000 to $124,999 | Pacific |
5 rows × 65 columns
As you can see above, the data has 65 columns of mostly categorical data. For example, the first column appears to allow for Yes and No responses only. Let’s verify by using the pandas.Series.unique method to see what unique values are in the Do you celebrate Thanksgiving? column of data:
# investigate respondent values
+data["Do you celebrate Thanksgiving?"].unique()
+array(['Yes', 'No'], dtype=object)
+# investigate respondent values
+data["What is typically the main dish at your Thanksgiving dinner?"].unique()
+array(['Turkey', 'Tofurkey', 'Other (please specify)', nan, 'Ham/Pork',
+ 'Turducken', 'Roast beef', 'Chicken', "I don't know"], dtype=object)
+We can also view all the column names to see all of the survey questions. We’ll truncate the output below to save you from having to scroll:
data.columns[50:]
+Index(['Which of these desserts do you typically have at Thanksgiving dinner? Please select all that apply. - Other (please specify).1',
+ 'Do you typically pray before or after the Thanksgiving meal?',
+ 'How far will you travel for Thanksgiving?',
+ 'Will you watch any of the following programs on Thanksgiving? Please select all that apply. - Macy's Parade',
+ 'What's the age cutoff at your "kids' table" at Thanksgiving?',
+ 'Have you ever tried to meet up with hometown friends on Thanksgiving night?',
+ 'Have you ever attended a "Friendsgiving?"',
+ 'Will you shop any Black Friday sales on Thanksgiving Day?',
+ 'Do you work in retail?',
+ 'Will you employer make you work on Black Friday?',
+ 'How would you describe where you live?', 'Age', 'What is your gender?',
+ 'How much total combined money did all members of your HOUSEHOLD earn last year?',
+ 'US Region'],
+ dtype='object')
+Using this Thanksgiving survey data, we can answer quite a few interesting questions, like:
In order to answer these questions and others, we’ll first need to become familiar with applying, grouping and aggregation in Pandas.
There are times when we’re using pandas that we want to apply a function to every row or every column in the data. A good example is getting from the values in our What is your gender? column to numeric values. We’ll assign 0 to Male, and 1 to Female.
Before we dive into transforming the values, let’s confirm that the values in the column are either Male or Female. We can use the pandas.Series.value_counts method to help us with this. We’ll pass the dropna=False keyword argument to also count missing values:
# investigate respondent values
+data["What is your gender?"].unique()
+array(['Male', 'Female', nan], dtype=object)
+# get respondent counts per response
+data["What is your gender?"].value_counts(dropna=False)
+Female 544
+Male 481
+NaN 33
+Name: What is your gender?, dtype: int64
+As you can see, not all of the values are Male or Female. We’ll preserve any missing values in the final output when we transform our column. Here’s a diagram of the input and outputs we need:
Source: https://www.dataquest.io/blog/content/images/2017/12/ditaa_diagram_1-2.png
We’ll need to apply a custom function to each value in the What is your gender? column to get the output we want. Here’s a function that will do the transformation we want:
# print the integer representation of True and False
+print(int(True), int(False))
+1 0
+import math
+
+def gender_code(gender_val):
+ # check to see if the gender iss a float or not specified
+ if isinstance(gender_val, float) and math.isnan(gender_val):
+ # if so return the float or nan value
+ return gender_val
+
+ # otherwise return 1 for female and 0 for male
+ return int(gender_val.lower().strip() == "female")
+In order to apply this function to each item in the What is your gender? column, we could either write a for loop, and loop across each element in the column, or we could use the pandas.Series.apply method.
This method will take a function as input, then return a new pandas Series that contains the results of applying the function to each item in the Series. We can assign the result back to a column in the data DataFrame, then verify the results using value_counts:
# calculate a new gender code column where 1 represents female and 0 represents male
+gender_codes = data["What is your gender?"].apply(gender_code)
+print(type(gender_codes), gender_codes.head())
+<class 'pandas.core.series.Series'> 0 0.0
+1 1.0
+2 0.0
+3 0.0
+4 0.0
+Name: What is your gender?, dtype: float64
+# add a new column containing the series gender_codes
+data["gender"] = gender_codes
+
+# display the value counts
+data["gender"].value_counts(dropna=False)
+1.0 544
+0.0 481
+NaN 33
+Name: gender, dtype: int64
+# check the data type of the new column
+data["gender"].dtype
+dtype('float64')
+# change the datatype to categorical
+data["gender"] = data["gender"].astype('category')
+
+# check the datatype
+data["gender"].dtype
+CategoricalDtype(categories=[0.0, 1.0], ordered=False)
+# print the top 5 values
+data["gender"].head()
+0 0.0
+1 1.0
+2 0.0
+3 0.0
+4 0.0
+Name: gender, dtype: category
+Categories (2, float64): [0.0, 1.0]
+We can use the apply method on DataFrames as well as Series. When we use the pandas.DataFrame.apply method, an entire row or column will be passed into the function we specify. By default, apply will work across each column in the DataFrame. If we pass the axis=1 keyword argument, it will work across each row.
In the below example, we check the data type of each column in data using a lambda function. We also call the head method on the result to avoid having too much output:
# create a method to return the datatype of a row
+def get_type(row):
+ return row.dtype
+
+# apply the method to all rows in the dataframe
+data.apply(get_type).head()
+RespondentID int64
+Do you celebrate Thanksgiving? object
+What is typically the main dish at your Thanksgiving dinner? object
+What is typically the main dish at your Thanksgiving dinner? - Other (please specify) object
+How is the main dish typically cooked? object
+dtype: object
+# return data types with a lambda instead of a function
+data.apply(lambda row: row.dtype).head()
+RespondentID int64
+Do you celebrate Thanksgiving? object
+What is typically the main dish at your Thanksgiving dinner? object
+What is typically the main dish at your Thanksgiving dinner? - Other (please specify) object
+How is the main dish typically cooked? object
+dtype: object
+We can now use what we know about the apply method to clean up the How much total combined money did all members of your HOUSEHOLD earn last year? column.
Cleaning up the income column will allow us to go from string values to numeric values. First, let’s see all the unique values that are in the How much total combined money did all members of your HOUSEHOLD earn last year?
# specify the column by name
+column_name = "How much total combined money did all members of your HOUSEHOLD earn last year?"
+
+# get value counts for the specified column
+data[column_name].value_counts(dropna=False)
+$25,000 to $49,999 180
+Prefer not to answer 136
+$50,000 to $74,999 135
+$75,000 to $99,999 133
+$100,000 to $124,999 111
+$200,000 and up 80
+$10,000 to $24,999 68
+$0 to $9,999 66
+$125,000 to $149,999 49
+$150,000 to $174,999 40
+NaN 33
+$175,000 to $199,999 27
+Name: How much total combined money did all members of your HOUSEHOLD earn last year?, dtype: int64
+Looking at this, there are 4 different patterns for the values in the column:
| Pattern | Example | Notes |
|---|---|---|
X to Y | $25,000 to $49,999 | We can convert this to a numeric value by extracting the numbers and averaging them. |
NaN | We’ll preserve NaN values, and not convert them at all. | |
X and up | $200,000 and up | We can convert this to a numeric value by extracting the number. |
Prefer not to answer | We’ll turn this into an NaN value. |
Here is how we want the transformations to work:
Source: https://www.dataquest.io/blog/content/images/2017/12/ditaa_diagram_2-1.png
We can write a function that covers all of these cases. In the below function, we:
Take a string called value as input.
200000 if so.NaN if so.NaN if so.import numpy as np
+
+def clean_income(value):
+ if value == "$200,000 and up":
+ return 200000
+ elif value == "Prefer not to answer":
+ return np.nan
+ elif isinstance(value, float) and math.isnan(value):
+ return np.nan
+
+ # remove commas and $ with empty spaces
+ value = value.replace(",", "").replace("$", "")
+
+ # split high and low income ranges
+ income_low, income_high = value.split(" to ")
+
+ # return the average between the ranges
+ return (int(income_high) + int(income_low)) / 2
+After creating the function, we can apply it to the How much total combined money did all members of your HOUSEHOLD earn last year? column:
column_name = "How much total combined money did all members of your HOUSEHOLD earn last year?"
+data["income"] = data[column_name].apply(clean_income)
+data["income"].head()
+0 87499.5
+1 62499.5
+2 4999.5
+3 200000.0
+4 112499.5
+Name: income, dtype: float64
+Now that we’ve covered applying functions, we can move on to grouping data using Pandas. When performing data analysis, it’s often useful to explore only a subset of the data.
For example, what if we want to compare income between people who tend to eat homemade cranberry sauce for Thanksgiving vs people who eat canned cranberry sauce?
First, let’s see what the unique values in the column are:
data["What type of cranberry saucedo you typically have?"].value_counts()
+Canned 502
+Homemade 301
+None 146
+Other (please specify) 25
+Name: What type of cranberry saucedo you typically have?, dtype: int64
+We can now filter data to get two DataFrames that only contain rows where the What type of cranberry saucedo you typically have? is Canned or Homemade, respectively:
column_name = "What type of cranberry saucedo you typically have?"
+homemade_mask = data[column_name] == "Homemade"
+print(homemade_mask.head())
+
+homemade = data[homemade_mask]
+0 False
+1 False
+2 True
+3 True
+4 False
+Name: What type of cranberry saucedo you typically have?, dtype: bool
+canned = data[data[column_name] == "Canned"]
+Finally, we can use the pandas.Series.mean method to find the average income in homemade and canned:
print(homemade["income"].mean())
+print(canned["income"].mean())
+94878.1072874494
+83823.40340909091
+We get our answer, but it took more lines of code than it should have. What if we now want to compute the average income for people who didn’t have cranberry sauce?
An easier way to find groupwise summary statistics with pandas is to use the pandas.DataFrame.groupby method. This method will split a DataFrame into groups based on a column or set of columns. We’ll then be able to perform computations on each group.
Here’s how splitting data based on the What type of cranberry saucedo you typically have? column would look:
Source: https://www.dataquest.io/blog/content/images/2017/12/ditaa_diagram_3-1.png
Note how each resulting group only has a single unique value in the What type of cranberry saucedo you typically have? column. One group is created for each unique value in the column we choose to group by.
Let’s create groups from the What type of cranberry saucedo you typically have? column:
column_name = "What type of cranberry saucedo you typically have?"
+grouped = data.groupby(column_name)
+grouped
+<pandas.core.groupby.generic.DataFrameGroupBy object at 0x7faaf8627d90>
+As you can see above, the groupby method returns a DataFrameGroupBy object. We can call the pandas.GroupBy.groups method to see what value for the What type of cranberry saucedo you typically have? column is in each group:
grouped.groups
+{'Canned': [4, 6, 8, 11, 12, 15, 18, 19, 26, 27, 38, 43, 48, 53, 58, 59, 60, 68, 69, 71, 74, 76, 79, 80, 86, 87, 89, 90, 91, 97, 103, 106, 107, 109, 115, 116, 118, 119, 123, 127, 129, 130, 132, 135, 136, 137, 140, 141, 143, 144, 145, 150, 153, 155, 156, 157, 158, 159, 161, 162, 163, 166, 167, 168, 169, 173, 179, 180, 181, 182, 184, 186, 190, 192, 193, 195, 198, 199, 200, 204, 205, 207, 209, 210, 211, 212, 213, 215, 217, 218, 220, 222, 224, 226, 229, 230, 231, 239, 243, 245, ...], 'Homemade': [2, 3, 5, 7, 13, 14, 16, 20, 21, 23, 25, 28, 30, 32, 33, 37, 39, 42, 44, 46, 52, 54, 56, 57, 62, 64, 66, 70, 82, 83, 85, 88, 93, 94, 96, 98, 101, 102, 108, 110, 111, 112, 114, 120, 122, 128, 134, 138, 139, 152, 165, 171, 172, 174, 175, 176, 177, 178, 183, 188, 189, 194, 201, 202, 203, 208, 219, 223, 225, 232, 234, 235, 236, 238, 241, 242, 244, 246, 248, 254, 255, 256, 259, 261, 262, 263, 264, 268, 281, 285, 286, 287, 290, 291, 292, 295, 298, 300, 302, 303, ...], 'None': [0, 17, 24, 29, 34, 36, 40, 47, 49, 51, 55, 61, 67, 72, 73, 77, 78, 81, 92, 99, 100, 104, 105, 117, 121, 124, 126, 131, 133, 142, 146, 148, 149, 160, 164, 185, 187, 191, 197, 227, 228, 237, 240, 274, 275, 319, 321, 329, 337, 362, 370, 377, 391, 395, 406, 409, 414, 417, 421, 437, 439, 466, 480, 491, 492, 495, 505, 514, 526, 529, 532, 537, 540, 553, 560, 564, 571, 573, 580, 584, 591, 594, 598, 602, 605, 606, 609, 610, 618, 626, 631, 639, 647, 658, 672, 673, 684, 700, 701, 716, ...], 'Other (please specify)': [1, 9, 154, 216, 221, 233, 249, 265, 301, 336, 380, 435, 444, 447, 513, 550, 749, 750, 784, 807, 860, 872, 905, 1000, 1007]}
+We can call the pandas.GroupBy.size method to see how many rows are in each group. This is equivalent to the value_counts method on a Series:
grouped.size()
+What type of cranberry saucedo you typically have?
+Canned 502
+Homemade 301
+None 146
+Other (please specify) 25
+dtype: int64
+We can also use a loop to manually iterate through the groups:
for name, group in grouped:
+ print(name)
+ print('\t', group.shape)
+ print('\t', type(group))
+Canned
+ (502, 67)
+ <class 'pandas.core.frame.DataFrame'>
+Homemade
+ (301, 67)
+ <class 'pandas.core.frame.DataFrame'>
+None
+ (146, 67)
+ <class 'pandas.core.frame.DataFrame'>
+Other (please specify)
+ (25, 67)
+ <class 'pandas.core.frame.DataFrame'>
+As you can see above, each group is a DataFrame, and you can use any normal DataFrame methods on it.
We can also extract a single column from a group. This will allow us to perform further computations just on that specific column:
grouped["income"]
+<pandas.core.groupby.generic.SeriesGroupBy object at 0x7faaf864e490>
+As you can see above, this gives us a SeriesGroupBy object. We can then call the normal methods we can call on a DataFrameGroupBy object:
grouped["income"].size()
+What type of cranberry saucedo you typically have?
+Canned 502
+Homemade 301
+None 146
+Other (please specify) 25
+Name: income, dtype: int64
+If all we could do was split a DataFrame into groups, it wouldn’t be of much use. The real power of groups is in the computations we can do after creating groups. We do these computations through the pandas.GroupBy.aggregate method, which we can abbreviate as agg. This method allows us to perform the same computation on every group.
For example, we could find the average income for people who served each type of cranberry sauce for Thanksgiving (Canned, Homemade, None, etc).
In the below code, we:
np.mean as input. grouped["income"].agg(np.mean)
+What type of cranberry saucedo you typically have?
+Canned 83823.403409
+Homemade 94878.107287
+None 78886.084034
+Other (please specify) 86629.978261
+Name: income, dtype: float64
+grouped["income"].agg(np.std)
+What type of cranberry saucedo you typically have?
+Canned 55835.478014
+Homemade 62251.937645
+None 54562.750866
+Other (please specify) 54175.781001
+Name: income, dtype: float64
+grouped["income"].agg(np.median)
+What type of cranberry saucedo you typically have?
+Canned 62499.5
+Homemade 87499.5
+None 62499.5
+Other (please specify) 87499.5
+Name: income, dtype: float64
+grouped["income"].agg([np.mean, np.std, np.median])
+| mean | std | median | |
|---|---|---|---|
| What type of cranberry saucedo you typically have? | |||
| Canned | 83823.403409 | 55835.478014 | 62499.5 |
| Homemade | 94878.107287 | 62251.937645 | 87499.5 |
| None | 78886.084034 | 54562.750866 | 62499.5 |
| Other (please specify) | 86629.978261 | 54175.781001 | 87499.5 |
If we left out only selecting the income column, here’s what we’d get:
grouped.agg(np.mean)
+| RespondentID | income | |
|---|---|---|
| What type of cranberry saucedo you typically have? | ||
| Canned | 4.336699e+09 | 83823.403409 |
| Homemade | 4.336792e+09 | 94878.107287 |
| None | 4.336765e+09 | 78886.084034 |
| Other (please specify) | 4.336763e+09 | 86629.978261 |
The above code will find the mean for each group for every column in data. However, most columns are string columns, not integer or float columns, so Pandas didn’t process them, since calling np.mean on them raised an error.
We can make a plot using the results of our agg method. This will create a bar chart that shows the average income of each category.
%matplotlib inline
+
+sauce = grouped.agg(np.mean)
+sauce["income"].plot(kind="bar")
+<AxesSubplot:xlabel='What type of cranberry saucedo you typically have?'>
+
We can call groupby with multiple columns as input to get more granular groups. If we use the What type of cranberry saucedo you typically have? and What is typically the main dish at your Thanksgiving dinner? columns as input, we’ll be able to find the average income of people who eat Homemade cranberry sauce and Tofurkey, for example:
grouped = data.groupby(
+ ["What type of cranberry saucedo you typically have?",
+ "What is typically the main dish at your Thanksgiving dinner?"])
+grouped.agg(np.mean)
+| RespondentID | income | ||
|---|---|---|---|
| What type of cranberry saucedo you typically have? | What is typically the main dish at your Thanksgiving dinner? | ||
| Canned | Chicken | 4.336354e+09 | 80999.600000 |
| Ham/Pork | 4.336757e+09 | 77499.535714 | |
| I don't know | 4.335987e+09 | 4999.500000 | |
| Other (please specify) | 4.336682e+09 | 53213.785714 | |
| Roast beef | 4.336254e+09 | 25499.500000 | |
| Tofurkey | 4.337157e+09 | 100713.857143 | |
| Turkey | 4.336705e+09 | 85242.682045 | |
| Homemade | Chicken | 4.336540e+09 | 19999.500000 |
| Ham/Pork | 4.337253e+09 | 96874.625000 | |
| I don't know | 4.336084e+09 | NaN | |
| Other (please specify) | 4.336863e+09 | 55356.642857 | |
| Roast beef | 4.336174e+09 | 33749.500000 | |
| Tofurkey | 4.336790e+09 | 57916.166667 | |
| Turducken | 4.337475e+09 | 200000.000000 | |
| Turkey | 4.336791e+09 | 97690.147982 | |
| None | Chicken | 4.336151e+09 | 11249.500000 |
| Ham/Pork | 4.336680e+09 | 61249.500000 | |
| I don't know | 4.336412e+09 | 33749.500000 | |
| Other (please specify) | 4.336688e+09 | 119106.678571 | |
| Roast beef | 4.337424e+09 | 162499.500000 | |
| Tofurkey | 4.336950e+09 | 112499.500000 | |
| Turducken | 4.336739e+09 | NaN | |
| Turkey | 4.336784e+09 | 74606.275281 | |
| Other (please specify) | Ham/Pork | 4.336465e+09 | 87499.500000 |
| Other (please specify) | 4.337335e+09 | 124999.666667 | |
| Tofurkey | 4.336122e+09 | 37499.500000 | |
| Turkey | 4.336724e+09 | 82916.194444 |
As you can see above, we get a nice table that shows us the mean of each column for each group. This enables us to find some interesting patterns, such as:
We can also perform aggregation with multiple functions. This enables us to calculate the mean and standard deviation of a group, for example. In the below code, we find the sum, standard deviation, and mean of each group in the income column:
grouped["income"].agg([np.mean, np.sum, np.std]).head(10)
+| mean | sum | std | ||
|---|---|---|---|---|
| What type of cranberry saucedo you typically have? | What is typically the main dish at your Thanksgiving dinner? | |||
| Canned | Chicken | 80999.600000 | 404998.0 | 75779.481062 |
| Ham/Pork | 77499.535714 | 1084993.5 | 56645.063944 | |
| I don't know | 4999.500000 | 4999.5 | NaN | |
| Other (please specify) | 53213.785714 | 372496.5 | 29780.946290 | |
| Roast beef | 25499.500000 | 127497.5 | 24584.039538 | |
| Tofurkey | 100713.857143 | 704997.0 | 61351.484439 | |
| Turkey | 85242.682045 | 34182315.5 | 55687.436102 | |
| Homemade | Chicken | 19999.500000 | 59998.5 | 16393.596311 |
| Ham/Pork | 96874.625000 | 387498.5 | 77308.452805 | |
| I don't know | NaN | 0.0 | NaN |
One of the limitations of aggregation is that each function has to return a single number. While we can perform computations like finding the mean, we can’t for example, call value_counts to get the exact count of a category. We can do this using the pandas.GroupBy.apply method. This method will apply a function to each group, then combine the results.
In the below code, we’ll apply value_counts to find the number of people who live in each area type (Rural, Suburban, etc) who eat different kinds of main dishes for Thanksgiving:
grouped = data.groupby("How would you describe where you live?")["What is typically the main dish at your Thanksgiving dinner?"]
+grouped.apply(lambda x: x.value_counts())
+How would you describe where you live?
+Rural Turkey 189
+ Other (please specify) 9
+ Ham/Pork 7
+ Tofurkey 3
+ I don't know 3
+ Turducken 2
+ Chicken 2
+ Roast beef 1
+Suburban Turkey 449
+ Ham/Pork 17
+ Other (please specify) 13
+ Tofurkey 9
+ Chicken 3
+ Roast beef 3
+ Turducken 1
+ I don't know 1
+Urban Turkey 198
+ Other (please specify) 13
+ Tofurkey 8
+ Chicken 7
+ Roast beef 6
+ Ham/Pork 4
+Name: What is typically the main dish at your Thanksgiving dinner?, dtype: int64
+The above table shows us that people who live in different types of areas eat different Thanksgiving main dishes at about the same rate.
+| \n", + " | RespondentID | \n", + "Do you celebrate Thanksgiving? | \n", + "What is typically the main dish at your Thanksgiving dinner? | \n", + "What is typically the main dish at your Thanksgiving dinner? - Other (please specify) | \n", + "How is the main dish typically cooked? | \n", + "How is the main dish typically cooked? - Other (please specify) | \n", + "What kind of stuffing/dressing do you typically have? | \n", + "What kind of stuffing/dressing do you typically have? - Other (please specify) | \n", + "What type of cranberry saucedo you typically have? | \n", + "What type of cranberry saucedo you typically have? - Other (please specify) | \n", + "... | \n", + "Have you ever tried to meet up with hometown friends on Thanksgiving night? | \n", + "Have you ever attended a \"Friendsgiving?\" | \n", + "Will you shop any Black Friday sales on Thanksgiving Day? | \n", + "Do you work in retail? | \n", + "Will you employer make you work on Black Friday? | \n", + "How would you describe where you live? | \n", + "Age | \n", + "What is your gender? | \n", + "How much total combined money did all members of your HOUSEHOLD earn last year? | \n", + "US Region | \n", + "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", + "4337954960 | \n", + "Yes | \n", + "Turkey | \n", + "NaN | \n", + "Baked | \n", + "NaN | \n", + "Bread-based | \n", + "NaN | \n", + "None | \n", + "NaN | \n", + "... | \n", + "Yes | \n", + "No | \n", + "No | \n", + "No | \n", + "NaN | \n", + "Suburban | \n", + "18 - 29 | \n", + "Male | \n", + "$75,000 to $99,999 | \n", + "Middle Atlantic | \n", + "
| 1 | \n", + "4337951949 | \n", + "Yes | \n", + "Turkey | \n", + "NaN | \n", + "Baked | \n", + "NaN | \n", + "Bread-based | \n", + "NaN | \n", + "Other (please specify) | \n", + "Homemade cranberry gelatin ring | \n", + "... | \n", + "No | \n", + "No | \n", + "Yes | \n", + "No | \n", + "NaN | \n", + "Rural | \n", + "18 - 29 | \n", + "Female | \n", + "$50,000 to $74,999 | \n", + "East South Central | \n", + "
| 2 | \n", + "4337935621 | \n", + "Yes | \n", + "Turkey | \n", + "NaN | \n", + "Roasted | \n", + "NaN | \n", + "Rice-based | \n", + "NaN | \n", + "Homemade | \n", + "NaN | \n", + "... | \n", + "Yes | \n", + "Yes | \n", + "Yes | \n", + "No | \n", + "NaN | \n", + "Suburban | \n", + "18 - 29 | \n", + "Male | \n", + "$0 to $9,999 | \n", + "Mountain | \n", + "
| 3 | \n", + "4337933040 | \n", + "Yes | \n", + "Turkey | \n", + "NaN | \n", + "Baked | \n", + "NaN | \n", + "Bread-based | \n", + "NaN | \n", + "Homemade | \n", + "NaN | \n", + "... | \n", + "Yes | \n", + "No | \n", + "No | \n", + "No | \n", + "NaN | \n", + "Urban | \n", + "30 - 44 | \n", + "Male | \n", + "$200,000 and up | \n", + "Pacific | \n", + "
| 4 | \n", + "4337931983 | \n", + "Yes | \n", + "Tofurkey | \n", + "NaN | \n", + "Baked | \n", + "NaN | \n", + "Bread-based | \n", + "NaN | \n", + "Canned | \n", + "NaN | \n", + "... | \n", + "Yes | \n", + "No | \n", + "No | \n", + "No | \n", + "NaN | \n", + "Urban | \n", + "30 - 44 | \n", + "Male | \n", + "$100,000 to $124,999 | \n", + "Pacific | \n", + "
5 rows × 65 columns
\n", + "| \n", + " | mean | \n", + "std | \n", + "median | \n", + "
|---|---|---|---|
| What type of cranberry saucedo you typically have? | \n", + "\n", + " | \n", + " | \n", + " |
| Canned | \n", + "83823.403409 | \n", + "55835.478014 | \n", + "62499.5 | \n", + "
| Homemade | \n", + "94878.107287 | \n", + "62251.937645 | \n", + "87499.5 | \n", + "
| None | \n", + "78886.084034 | \n", + "54562.750866 | \n", + "62499.5 | \n", + "
| Other (please specify) | \n", + "86629.978261 | \n", + "54175.781001 | \n", + "87499.5 | \n", + "
| \n", + " | RespondentID | \n", + "income | \n", + "
|---|---|---|
| What type of cranberry saucedo you typically have? | \n", + "\n", + " | \n", + " |
| Canned | \n", + "4.336699e+09 | \n", + "83823.403409 | \n", + "
| Homemade | \n", + "4.336792e+09 | \n", + "94878.107287 | \n", + "
| None | \n", + "4.336765e+09 | \n", + "78886.084034 | \n", + "
| Other (please specify) | \n", + "4.336763e+09 | \n", + "86629.978261 | \n", + "
| \n", + " | \n", + " | RespondentID | \n", + "income | \n", + "
|---|---|---|---|
| What type of cranberry saucedo you typically have? | \n", + "What is typically the main dish at your Thanksgiving dinner? | \n", + "\n", + " | \n", + " |
| Canned | \n", + "Chicken | \n", + "4.336354e+09 | \n", + "80999.600000 | \n", + "
| Ham/Pork | \n", + "4.336757e+09 | \n", + "77499.535714 | \n", + "|
| I don't know | \n", + "4.335987e+09 | \n", + "4999.500000 | \n", + "|
| Other (please specify) | \n", + "4.336682e+09 | \n", + "53213.785714 | \n", + "|
| Roast beef | \n", + "4.336254e+09 | \n", + "25499.500000 | \n", + "|
| Tofurkey | \n", + "4.337157e+09 | \n", + "100713.857143 | \n", + "|
| Turkey | \n", + "4.336705e+09 | \n", + "85242.682045 | \n", + "|
| Homemade | \n", + "Chicken | \n", + "4.336540e+09 | \n", + "19999.500000 | \n", + "
| Ham/Pork | \n", + "4.337253e+09 | \n", + "96874.625000 | \n", + "|
| I don't know | \n", + "4.336084e+09 | \n", + "NaN | \n", + "|
| Other (please specify) | \n", + "4.336863e+09 | \n", + "55356.642857 | \n", + "|
| Roast beef | \n", + "4.336174e+09 | \n", + "33749.500000 | \n", + "|
| Tofurkey | \n", + "4.336790e+09 | \n", + "57916.166667 | \n", + "|
| Turducken | \n", + "4.337475e+09 | \n", + "200000.000000 | \n", + "|
| Turkey | \n", + "4.336791e+09 | \n", + "97690.147982 | \n", + "|
| None | \n", + "Chicken | \n", + "4.336151e+09 | \n", + "11249.500000 | \n", + "
| Ham/Pork | \n", + "4.336680e+09 | \n", + "61249.500000 | \n", + "|
| I don't know | \n", + "4.336412e+09 | \n", + "33749.500000 | \n", + "|
| Other (please specify) | \n", + "4.336688e+09 | \n", + "119106.678571 | \n", + "|
| Roast beef | \n", + "4.337424e+09 | \n", + "162499.500000 | \n", + "|
| Tofurkey | \n", + "4.336950e+09 | \n", + "112499.500000 | \n", + "|
| Turducken | \n", + "4.336739e+09 | \n", + "NaN | \n", + "|
| Turkey | \n", + "4.336784e+09 | \n", + "74606.275281 | \n", + "|
| Other (please specify) | \n", + "Ham/Pork | \n", + "4.336465e+09 | \n", + "87499.500000 | \n", + "
| Other (please specify) | \n", + "4.337335e+09 | \n", + "124999.666667 | \n", + "|
| Tofurkey | \n", + "4.336122e+09 | \n", + "37499.500000 | \n", + "|
| Turkey | \n", + "4.336724e+09 | \n", + "82916.194444 | \n", + "
| \n", + " | \n", + " | mean | \n", + "sum | \n", + "std | \n", + "
|---|---|---|---|---|
| What type of cranberry saucedo you typically have? | \n", + "What is typically the main dish at your Thanksgiving dinner? | \n", + "\n", + " | \n", + " | \n", + " |
| Canned | \n", + "Chicken | \n", + "80999.600000 | \n", + "404998.0 | \n", + "75779.481062 | \n", + "
| Ham/Pork | \n", + "77499.535714 | \n", + "1084993.5 | \n", + "56645.063944 | \n", + "|
| I don't know | \n", + "4999.500000 | \n", + "4999.5 | \n", + "NaN | \n", + "|
| Other (please specify) | \n", + "53213.785714 | \n", + "372496.5 | \n", + "29780.946290 | \n", + "|
| Roast beef | \n", + "25499.500000 | \n", + "127497.5 | \n", + "24584.039538 | \n", + "|
| Tofurkey | \n", + "100713.857143 | \n", + "704997.0 | \n", + "61351.484439 | \n", + "|
| Turkey | \n", + "85242.682045 | \n", + "34182315.5 | \n", + "55687.436102 | \n", + "|
| Homemade | \n", + "Chicken | \n", + "19999.500000 | \n", + "59998.5 | \n", + "16393.596311 | \n", + "
| Ham/Pork | \n", + "96874.625000 | \n", + "387498.5 | \n", + "77308.452805 | \n", + "|
| I don't know | \n", + "NaN | \n", + "0.0 | \n", + "NaN | \n", + "
This Matplotlib tutorial takes you through the basics Python data visualization: the anatomy of a plot, pyplot and pylab, and much more.
Humans are very visual creatures: we understand things better when we see things visualized. However, the step to presenting analyses, results or insights can be a bottleneck: you might not even know where to start or you might have already a right format in mind, but then questions like “Is this the right way to visualize the insights that I want to bring to my audience?” will have definitely come across your mind.
When you’re working with the Python plotting library Matplotlib, the first step to answering the above questions is by building up knowledge on topics like:
At first sight, it will seem that there are quite some components to consider when you start plotting with this Python data visualization library. You’ll probably agree with me that it’s confusing and sometimes even discouraging seeing the amount of code that is necessary for some plots, not knowing where to start yourself and which components you should use.
Luckily, this library is very flexible and has a lot of handy, built-in defaults that will help you out tremendously. As such, you don’t need much to get started: you need to make the necessary imports, prepare some data, and you can start plotting with the help of the plot() function! When you’re ready, don’t forget to show your plot using the show() function.
# Import the necessary packages and modules
+import matplotlib.pyplot as plt
+import numpy as np
+
+# Prepare the data
+x = np.linspace(0, 10, 100)
+
+# Plot the data
+plt.plot(x, x, label='linear')
+
+# Add a legend
+plt.legend()
+
+# Show the plot
+plt.show()
+
Note that you import the pyplot module of the matplotlib library under the alias plt.
Congrats, you have now successfully created your first plot! Now let’s take a look at the resulting plot in a little bit more detail.
What you can’t see on the surface is that you have maybe unconsciously made use of the built-in defaults that take care of the creation of the underlying components, such as the Figure and the Axes. You’ll read more about these defaults in the section that deals with the differences between pylab and pyplot.
For now, you’ll understand that working with matplotlib will already become a lot easier when you understand how the underlying components are instantiated. Or, in other words, what the anatomy of a matplotlib plot looks like:
In essence, there are two big components that you need to take into account:
plot() and scatter() and that can have ticks, labels, etc. associated with it. This explains why Figures can contain multiple Axes.Tip: when you see, for example, plt.xlim, you’ll call ax.set_xlim() behind the covers. All methods of an Axes object exist as a function in the pyplot module and vice versa. Note that mostly, you’ll use the functions of the pyplot module because they’re much cleaner, at least for simple plots!
You’ll see what “clean” means when you take a look at the following pieces of code.
Compare, for example, this piece of code:
fig = plt.figure()
+
+ax = fig.add_subplot(111)
+
+ax.plot([1, 2, 3, 4], [10, 20, 25, 30], color='lightblue', linewidth=3)
+ax.scatter([0.3, 3.8, 1.2, 2.5], [11, 25, 9, 26], color='darkgreen', marker='^')
+ax.set_xlim(0.5, 4.5)
+
+plt.show()
+
With the piece of code below:
plt.plot([1, 2, 3, 4], [10, 20, 25, 30], color='lightblue', linewidth=3)
+plt.scatter([0.3, 3.8, 1.2, 2.5], [11, 25, 9, 26], color='darkgreen', marker='^')
+plt.xlim(0.5, 4.5)
+
+plt.show()
+
The second code chunk is definitely cleaner, isn’it it?
Note: that the above code examples come from the Anatomy of Matplotlib Tutorial by Benjamin Root.
However, if you have multiple axes, it’s still better to make use of the first code chunk because it’s always better to prefer explicit above implicit code! In such cases, you want to make use of the Axes object ax.
Next to these two components, there are a couple more that you can keep in mind:
You see that the right and top spines are set to invisible.
Note that you’ll sometimes also read about Artist objects, which are virtually all objects that the package has to offers to users like yourself. Everything drawn using Matplotlib is part of the Artist module. The containers that you will use to plot your data, such as Axis, Axes and Figure, and other graphical objects such as text, patches, etc. are types of Artists.
For those who have already got some coding experience, it might be good to check out and study the code examples that you find in the Matplotlib gallery.
First off, you’ll already know Matplotlib by now. When you talk about “Matplotlib”, you talk about the whole Python data visualization package. This should not come to you as a big surprise :)
Secondly, pyplot is a module in the matplotlib package. That’s why you often see matplotlib.pyplot in code. The module provides an interface that allows you to implicitly and automatically create figures and axes to achieve the desired plot.
This is especially handy when you want to quickly plot something without instantiating any Figures or Axes, as you saw in the example in the first section of this tutorial. You see, you haven’t explicitly specified these components, yet you manage to output a plot that you have even customized! The defaults are initialized and any customizations that you do, will be done with the current Figure and Axes in mind.
Lastly, pylab is another module, but it gets installed alongside the matplotlib package. It bulk imports pyplot and the numpy library and was generally recommended when you were working with arrays, doing mathematics interactively and wanted access to plotting features.
You might still see this popping up in older tutorials and examples of matplotlib, but its use is no longer recommended, especially not when you’re using the IPython kernel in your Jupyter notebook. You can read more about this here.
As a solution, you can best use %matplotlib magic in combination with the right backend, such as inline, qt, etc. Most of the times, you will want to use inline, as this will make sure that the plots are embedded inside the notebook. Read more about that in DataCamp’s Definitive Guide to Jupyter Notebook.
Note that also when you’re not working in a Jupyter notebook, you’ll still need to choose a different backend, depending on your use case. In other words, if you don’t want to embed plots inside a notebook, but you rather want to embed them into graphical user interfaces, in batch scripts or web application servers, etc., you will need to specify the backend that you want to use. However, this topic is outside the scope of this tutorial; Instead, the tutorial assumes that you will be using Matplotlib to save your images to your local file system.
As you have read in one of the previous sections, Matplotlib is often used to visualize analyses or calcuations. That’s why the first step that you have to take in order to start plotting in Python yourself is to consider revising NumPy, the Python library for scientific computing.
Scientific computing might not really seem of much interest, but when you’re doing data science you’ll find yourself working a lot with data that is stored in arrays. You’ll need to perform operations on them, inspect your arrays and manipulate them so that you’re working with the (subset of the) data that is interesting for your analysis and that is in the right format, etc.
In short, you’ll find NumPy extremely handy when you’re working with this data visualization library. If you’re interested in taking a NumPy tutorial to start well-prepared, go and take DataCamp’s tutorial and make sure to have your copy of their NumPy cheat sheet close!
Of course, arrays are not the only thing that you pass to your plotting functions; There’s also the possibility to, for example, pass Python lists. If you would like to know more about Python lists, consider checking out their Python list tutorial or the free Intro to Python for Data Science course.
Alright, you’re off to create your first plot yourself with Python! As you have read in one of the previous sections, the Figure is the first step and the key to unlocking the power of this package. Next, you see that you initialize the axes of the Figure in the code chunk above with fig.add_axes():
# Initialize a Figure
+fig = plt.figure()
+
+# Add Axes to the Figure
+fig.add_axes([0,0,1,1])
+
+plt.show()
+
You have seen all components of a plot and you have initialized your first figure and Axes, but to make things a bit more complicated, you’ll sometimes see subplots pop up in code.
You use subplots to set up and place your Axes on a regular grid. So that means that in most cases, Axes and subplot are synonymous, they will designate the same thing. When you do call subplot to add Axes to your figure, do so with the add_subplots() function.
There is, however, a difference between the add_axes() and the add_subplots() function, but you’ll learn more about this later on in the tutorial.
Consider the following example:
import numpy as np
+
+# Create a Figure
+fig = plt.figure()
+
+# Set up Axes
+ax = fig.add_subplot(111)
+
+# Scatter the data
+ax.scatter(np.linspace(0, 1, 5), np.linspace(0, 5, 5))
+
+# Show the plot
+plt.show()
+
You see that the add_subplot() function in itsef also poses you with a challenge, because you see add_subplots(111) in the above code chunk.
What does 111 mean?
Well, 111 is equal to 1,1,1, which means that you actually give three arguments to add_subplot(). The three arguments designate the:
So you actually make one subplot.
Note: that you can really go bananas with this function when you are using this function, especially when you’re just starting out with this library and you keep on forgetting for what the three numbers stand.
Consider the following commands and try to envision what the plot will look like and how many Axes your Figure will have: ax = fig.add_subplot(2,2,1).
Your Figure will have four axes in total, arranged in a structure that has two rows and two columns. With the line of code that you have considered, you say that the variable ax is the first of the four axes to which you want to start plotting. The “first” in this case means that it will be the first axes on the left of the 2x2 structure that you have initialized.
The difference between fig.add_axes() and fig.add_subplot() doesn’t lie in the result: they both return an Axes object. However, they do differ in the mechanism that is used to add the axes: you pass a list to add_axes() which is the lower left point, the width and the height. This means that the axes object is positioned in absolute coordinates.
In contrast, the add_subplot() function doesn’t provide the option to put the axes at a certain position: it does, however, allow the axes to be situated according to a subplot grid, as you have seen in the section above.
In most cases, you’ll use add_subplot() to create axes; Only in cases where the positioning matters, you’ll resort to add_axes(). Alternatively, you can also use subplots() if you want to get one or more subplots at the same time. You’ll see an example of how this works in the next section.
Now that you have seen how to initialize a Figure and Axes from scratch, you will also want to know how you can change certain small details that the package sets up for you, such as the figure size.
Let’s say you don’t have the luxury to follow along with the defaults and you want to change this. How do you set the size of your figures manually?
Like everything with this package, it’s pretty easy, but you need to know first what to change.
Add an argument figsize to your plt.figure() function of the pyplot module; You just have to specify a tuple with the width and hight of your figure in inches, just like this plt.figure(figsize=(3,4)), for it to work.
Note that you can also pass figsize to the the plt.subplots() function of the same module; The inner workings are the same as the figure() function that you’ve just seen.
See an example of how this would work here:
# Initialize the plot
+fig = plt.figure(figsize=(20,10))
+ax1 = fig.add_subplot(121)
+ax2 = fig.add_subplot(122)
+
+# Plot the data
+ax1.bar([1,2,3],[3,4,5])
+ax2.barh([0.5,1,2.5],[0,1,2])
+
+# Show the plot
+plt.show()
+
# alternatively
+fig, (ax1, ax2) = plt.subplots(1,2, figsize=(20,10))
+
+# or replace the three lines of code above by the following line:
+#fig, (ax1, ax2) = plt.subplots(1,2, figsize=(20,10))
+
+# Plot the data
+ax1.bar([1,2,3],[3,4,5])
+ax2.barh([0.5,1,2.5],[0,1,2])
+
+# Show the plot
+plt.show()
+
Now that all is set for you to start plotting your data, it’s time to take a closer look at some plotting routines. You’ll often come across functions like plot() and scatter(), which either draw points with lines or markers connecting them, or draw unconnected points, which are scaled or colored.
But, as you have already seen in the example of the first section, you shouldn’t forget to pass the data that you want these functions to use!
These functions are only the bare basics. You will need some other functions to make sure your plots look awesome:
| Method | Description |
|---|---|
| ax.bar() | Vertical rectangles |
| ax.barh() | Horizontal rectangles |
| ax.axhline() | Horizontal line across axes |
| ax.vline() | Vertical line across axes |
| ax.fill() | Filled polygons |
| ax.fill_between() | Fill between y-values and 0 |
| ax.stackplot() | Stack plot |
If you’re curious how you can use these functions to plot your data, consider the following example.
data = [
+ {'x': [1,2,3],
+ 'y':[3,4,5],
+ 'horizontal': False},
+ {'x': [0.5,1,2.5],
+ 'y':[0,1,2],
+ 'horizontal': True}
+]
+
+# alternatively
+fig, axes = plt.subplots(len(data), 1, figsize=(10,20))
+
+# or replace the three lines of code above by the following line:
+#fig, (ax1, ax2) = plt.subplots(1,2, figsize=(20,10))
+for index, ax in enumerate(axes):
+ if data[index]['horizontal']:
+ ax.barh(data[index]['x'], data[index]['y'])
+ else:
+ ax.bar(data[index]['x'], data[index]['y'])
+
+
+# Show the plot
+plt.show()
+
x = np.array([ 0., 0.25, 0.5, 0.75, 1. ], dtype=np.float64)
+y = np.array([ 0., 1.25, 2.5, 3.75, 5. ], dtype=np.float64)
+
+# Initialize the plot
+# fig = plt.figure(figsize=(15,10))
+fig = plt.figure()
+ax1 = fig.add_subplot(131)
+ax2 = fig.add_subplot(132)
+ax3 = fig.add_subplot(133)
+
+# Plot the data
+ax1.bar([1,2,3],[3,4,5])
+ax2.barh([0.5,1,2.5],[0,1,2])
+ax2.axhline(0.45)
+ax1.axvline(0.65)
+ax3.scatter(x,y)
+
+# Show the plot
+plt.show()
+
Most functions speak for themselves because the names are quite clear. But that doesn’t mean that you need to limit yourself: for example, the fill_between() function is perfect for those who want to create area plots, but they can also be used to create a stacked line graph; Just use the plotting function a couple of times to make sure that the areas overlap and give the illusion of being stacked.
Note that, of course, simply passing the data is not enough to create great plots. Make sure to manipulate your data in such a way that the visualization makes sense: don’t be afraid to change your array shape, combine arrays, etc.
When you move on and you start to work with vector fields or data distributions, you might want to check out the following functions:
| Method | Description |
|---|---|
| ax.arrow() | Arrow |
| ax.quiver() | 2D field of arrows |
| ax.streamplot() | 2D vector fields |
| ax.hist() | Histogram |
| ax.boxplot() | Boxplot |
| ax.violinplot() | Violinplot |
Note of course that you probably won’t use all of the functions listed in these tables; It really depends on your data and your use case. If you’re totally new to data science, you might want to check out the statistical plotting routines first!
On the other hand, when you work with 2-D or n-D data, you might also find yourself in need of some more advanced plotting routines, like these ones:
| Method | Description |
|---|---|
| ax.pcolor() | Pseudocolor plot |
| ax.pcolormesh() | Pseudocolor plot |
| ax.contour() | Contour plot |
| ax.contourf() | Filled contour plot |
| ax.clabel() | Labeled contour plot |
Note that contour plots are used to explore the potential relationship between three variables. Just like contour plots, also pseudocolor plots can be used for this purpose, since they are surface plot seen from above. Of course, this are not nearly all the functions that you can use to plot your data.
If you’re working with images or 2D data, for example, you might also want to check out imshow() to show images in your subplots. For a practical example of how to use the imshow() function, go to DataCamp’s scikit-learn tutorial.
The examples in the tutorial also make clear that this data visualization library is really the cherry on the pie in the data science workflow: you have to be quite well-versed in general Python concepts, such as lists and control flow, which can come especially handy if you want to automate the plotting for a great number of subplots. If you feel like revising these concepts, consider taking the free introduction to Python for data science course.
A lot of questions about this package come from the fact that there are a lot of things that you can do to personalize your plots and make sure that they are unique: besides adjusting the colors, you also have the option to change markers, linestyles and linewidths, add text, legend and annotations, and change the limits and layout of your plots.
It’s exactly the fact that there is an endless range of possibilities when it comes to these plots that makes it difficult to set out some things that you need to know when you start working on this topic.
Great tips that you should keep in the back of your mind are not only the gallery, which contains many real-life examples that are already coded for you and which you can use, but also the documentation, which can tell you more about the arguments that you can pass to certain functions to adjust visual features.
Also keep in mind that there are multiple solutions for one problem and that you learn most of this stuff when you’re getting your hands dirty with the package itself and when you run into troubles. You’ll see some of the most common questions and solutions in this section.
If you ever want to remove an axes form your plot, you can use delaxes() to remove and update the current axes.
Note: you can restore a deleted axes by adding fig.add_axes(ax) right after fig.delaxes(ax3).
# Initialize the plot
+fig = plt.figure()
+ax1 = fig.add_subplot(131)
+ax2 = fig.add_subplot(132)
+ax3 = fig.add_subplot(133)
+
+# Plot the data
+ax1.bar([1,2,3],[3,4,5])
+ax2.barh([0.5,1,2.5],[0,1,2])
+ax2.axhline(0.45)
+ax1.axvline(0.65)
+ax3.scatter(np.linspace(0, 1, 5), np.linspace(0, 5, 5))
+
+# Delete `ax3`
+fig.delaxes(ax3)
+
+# Show the plot
+plt.show()
+
There are a number of ways to address this question, but mostly all come back to the arguments that you can provide to legend():
loc or location argument to something like center left or upper right, which ensures that your legend does not fall in the Axes or subplot area.bbox_to_anchor argument to your function and pass a tuple with the coordinates in which you want to put the legend. In this case, the box is put in the upper right corner of the plotting area: ax.legend(bbox_to_anchor=(1.1, 1.05)).To change your plot title and axes labels, you can follow one of the following approaches, depending of which container of which you want to make use:
ax.set(title="A title", xlabel="x", ylabel="y") or ax.set_xlim(), ax.set_ylim() or ax.set_title().fig.suptitle() to add a title to your plot.plt.title(), plt.xlabel(), plt.ylabel().matplotlibrc settings. Read more about this here.A thing to consider when you’re using subplots to build up your plot is the tight_layout function, which will help you to make sure that the plots fit nicely in your figure. You ideally call it after you have plotted your data and customized your plot; So that’s right before you call plt.show() that you should use plt.tight_layout().
Additionally, you might also be interested to use subplots_adjust(), which allows you to manually set the width and height reserved for blank space between subplots, and also fix the left and right sides, and the top and bottom of the subplots.
After you have done all the necessary customizations, you will want to show your plot because, as you will have noticed from working in the terminal, you just get to see that an object is made, but you never see the nice plot every time you make adjustments.
In the first example of this tutorial, this was implicitly done;
Do you remember? It’s this piece of code:
# Prepare the data
+x = np.linspace(0, 10, 100)
+
+# Plot the data
+plt.plot(x, x, label='linear')
+
+# Add a legend
+plt.legend()
+
+# Show the plot
+plt.show()
+
The line plt.show() says indeed that you want to see the plot. If you execute this line, you’ll see a window popping up. And you’ll see if it looks like what you had in your mind!
But this is where your questions start. How can I save this image and if it’s not to your liking, can you clear the image so that you can start anew? The following short sections will cover these questions.
You can easily save a figure to, for example, a png file by making use of plt.savefig(). The only argument you need to pass to this function is the file name, just like in this example:
# Prepare the data
+x = np.linspace(0, 10, 100)
+
+# Plot the data
+plt.plot(x, x, label='linear')
+
+# Add a legend
+plt.legend()
+
+# Save Figure
+plt.savefig("foo.png")
+
# Prepare the data
+x = np.linspace(0, 10, 100)
+
+# Plot the data
+plt.plot(x, x, label='linear')
+
+# Add a legend
+plt.legend()
+
+# Save Transparent Figure
+plt.savefig("foo_transparent.png", transparent=True)
+
If you want to save multiple plots to a pdf file, you want to make use of the pdf backend, which you can easily import:
# Import PdfPages
+from matplotlib.backends.backend_pdf import PdfPages
+
+# Prepare the data
+x = np.linspace(0, 10, 100)
+
+# Plot the data
+plt.plot(x, x, label='linear')
+
+# Add a legend
+plt.legend()
+
+# Initialize the pdf file
+pp = PdfPages('multipage.pdf')
+
+# Save the figure to the file
+pp.savefig()
+
+# Close the file
+pp.close()
+
When you’re finally ready with the inspection of your plot, it’s time to move on with something else (maybe with another plot!). When you’re working with this data visualization library for the first time, it might be weird at start because you can, of course, shut down the GUI window that appears, but that’s usually not the way you want to handle things, because it doesn’t always work as well when you’re working on several things at a time.
You have to explicitly tell Matplotlib to close down the plot that you’ve been working on so that you can move on. There are three functions that will come in handy once you’re at this point:
Use:
plt.cla() to clear an axisplt.clf() to clear the entire figureplt.close() to close a window that has popped up to show you your plotBy now, you’re already familiar with some basic options to customize your plots. But what if the customizations that you want to make situate more on a library level instead of a plot level? In such cases, also, you don’t need to panic: Matplotlib offers you several options to adjust some of the internal workings. This section will just cover two options, namely style sheets and rc settings.
If you want to know more, definitely check out this page.
For the R enthusiasts among you, Matplotlib also offers you the option to set the style of the plots to ggplot. You can easily do this by running the following piece of code:
# Set the style to `ggplot`
+plt.style.use("ggplot")
+“rc” is common for configuration files: they usually end in rc. It comes from the practice of having configs as executables: they are automatically run and configure settings, for example. You can read more about it here. Matplotlib has such an rc file to which you can make adjustments dynamically and statically.
To dynamically change default rc settings, you can use the rcParams variable:
# without rc Settings
+
+# Prepare the data
+x = np.linspace(0, 10, 100)
+
+# Plot the data
+plt.plot(x, x, label='linear')
+
+# Add a legend
+plt.legend()
+
+# Show the plot
+plt.show()
+
# with rc Settings
+import matplotlib as mpl
+
+mpl.rcParams['lines.linewidth'] = 5
+
+# Prepare the data
+x = np.linspace(0, 10, 100)
+
+# Plot the data
+plt.plot(x, x, label='linear')
+
+# Add a legend
+plt.legend()
+
+# Show the plot
+plt.show()
+
You just adjusted the line width in the example above, but you can also change figure size and dpi, line width, color and style, axes, axis and grid properties, text and font properties, …
If you want to work more statically, you should probably also know that you have a matplotlibrc configuration file, which you can use to customize all kinds of properties (just like you did above with the line width parameter). If you want to find this specific file, you can just run the following:
import matplotlib
+
+matplotlib.matplotlib_fname()
+'/Users/bk/opt/miniconda3/envs/cmu39/lib/python3.9/site-packages/matplotlib/mpl-data/matplotlibrc'
+Congratulations! You have gone through today’s Matplotlib tutorial successfully! There is still much to learn, but you’re definitely ready to go out on your own and create your own amazing plots. Don’t miss out on DataCamp’s Matplotlib cheat sheet that can help you to make plots in no time, step by step.
If you’re eager to discover more from Matplotlib, consider checking out DataCamp’s Viewing 3D Volumetric Data With Matplotlib tutorial to learn how to work with matplotlib’s event handler API or this tutorial, in which you’ll learn all about animating your plots.
+In this step-by-step Seaborn tutorial, you’ll learn how to use one of Python’s most convenient libraries for data visualization.
For those who’ve tinkered with Matplotlib before, you may have wondered, “why does it take me 10 lines of code just to make a decent-looking histogram?”
Well, if you’re looking for a simpler way to plot attractive charts, then you’ll love Seaborn. We’ll walk you through everything you need to know to get started, and we’ll use a fun Pokémon dataset (which you can download below).
Seaborn provides a high-level interface to Matplotlib, a powerful but sometimes unwieldy Python visualization library.
On Seaborn’s official website, they state:
If matplotlib “tries to make easy things easy and hard things possible”, seaborn tries to make a well-defined set of hard things easy too.
We’ve found this to be a pretty good summary of Seaborn’s strengths. In practice, the “well-defined set of hard things” includes:
Those last three points are why Seaborn is our tool of choice for Exploratory Analysis. It makes it very easy to “get to know” your data quickly and efficiently.
However, Seaborn is a complement, not a substitute, for Matplotlib. There are some tweaks that still require Matplotlib, and we’ll cover how to do that as well.
While Seaborn simplifies data visualization in Python, it still has many features. Therefore, the best way to learn Seaborn is to learn by doing.
This process will give you intuition about what you can do with Seaborn, leaving documentation to serve as further guidance. This is the fastest way to go from zero to proficient.
A quick tip before we begin:
We tried to make this tutorial as streamlined as possible, which means we won’t go into too much detail for any one topic. It’s helpful to have the Seaborn documentation open beside you, in case you want to learn more about a feature.
Instead of just showing you how to make a bunch of plots, we’re going to walk through the most important paradigms of the Seaborn library. Along the way, we’ll illustrate each concept with examples.
Here are the steps we’ll cover in this tutorial:
We strongly recommend installing the Anaconda Distribution, which comes with all of those packages. Simply follow the instructions on that download page.
Once you have Anaconda installed, simply start Jupyter (either through the command line or the Navigator app) and open a new notebook
Let’s start by importing Pandas, which is a great library for managing relational (i.e. table-format) datasets:
# disable warnings for lecture
+import warnings
+warnings.filterwarnings('ignore')
+# Pandas for managing datasets
+import pandas as pd
+Next, we’ll import Matplotlib, which will help us customize our plots further.
%matplotlib inline to display your plots inside your notebook.# Matplotlib for additional customization
+from matplotlib import pyplot as plt
+%matplotlib inline
+Then, we’ll import the Seaborn library, which is the star of today’s show.
# Seaborn for plotting and styling
+import seaborn as sns
+Now we’re ready to import our dataset.
pd, Matplotlib with plt, and Seaborn with sns.Today, we’ll be using a cool Pokémon dataset (first generation). Here’s the free download:
Dataset for this tutorial.
Once you’ve downloaded the CSV file, you can import it with Pandas.
index_col=0 simply means we’ll treat the first column of the dataset as the ID column.# Read dataset
+df = pd.read_csv('Pokemon.csv', index_col=0)
+# Display first 5 observations
+df.head()
+| Name | Type 1 | Type 2 | Total | HP | Attack | Defense | Sp. Atk | Sp. Def | Speed | Stage | Legendary | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| # | ||||||||||||
| 1 | Bulbasaur | Grass | Poison | 318 | 45 | 49 | 49 | 65 | 65 | 45 | 1 | False |
| 2 | Ivysaur | Grass | Poison | 405 | 60 | 62 | 63 | 80 | 80 | 60 | 2 | False |
| 3 | Venusaur | Grass | Poison | 525 | 80 | 82 | 83 | 100 | 100 | 80 | 3 | False |
| 4 | Charmander | Fire | NaN | 309 | 39 | 52 | 43 | 60 | 50 | 65 | 1 | False |
| 5 | Charmeleon | Fire | NaN | 405 | 58 | 64 | 58 | 80 | 65 | 80 | 2 | False |
One of Seaborn’s greatest strengths is its diversity of plotting functions. For instance, making a scatter plot is just one line of code using the lmplot() function.
There are two ways you can do so.
data= argument, while passing column names to the axes arguments, x= and y=.For example, let’s compare the Attack and Defense stats for our Pokémon:
# Recommended way
+sns.lmplot(x='Attack', y='Defense', data=df)
+<seaborn.axisgrid.FacetGrid at 0x7fd6c805d910>
+
By the way, Seaborn doesn’t have a dedicated scatter plot function, which is why you see a diagonal line. We actually used Seaborn’s function for fitting and plotting a regression line.
Thankfully, each plotting function has several useful options that you can set. Here’s how we can tweak the lmplot():
fit_reg=False to remove the regression line, since we only want a scatter plot.hue='Stage' to color our points by the Pokémon’s evolution stage. This hue argument is very useful because it allows you to express a third dimension of information using color.# Scatterplot arguments
+sns.lmplot(x='Attack', y='Defense', data=df,
+ fit_reg=False, # No regression line
+ hue='Stage') # Color by evolution stage
+<seaborn.axisgrid.FacetGrid at 0x7fd71abc7130>
+
# Scatterplot arguments
+sns.lmplot(x='Attack', y='Defense', data=df,
+ fit_reg=False, # No regression line
+ hue='Type 1') # Color by evolution stage
+<seaborn.axisgrid.FacetGrid at 0x7fd719a289d0>
+
Looking better, but we can improve this scatter plot further. For example, all of our Pokémon have positive Attack and Defense values, yet our axes limits fall below zero.
Let’s see how we can fix that…
Remember, Seaborn is a high-level interface to Matplotlib. From our experience, Seaborn will get you most of the way there, but you’ll sometimes need to bring in Matplotlib.
Setting your axes limits is one of those times, but the process is pretty simple:
ylim() and xlim() functions.Here’s our new scatter plot with sensible axes limits:
# Plot using Seaborn
+sns.lmplot(x='Attack', y='Defense', data=df,
+ fit_reg=False,
+ hue='Stage')
+
+# Tweak using Matplotlib
+plt.ylim(0, None)
+plt.xlim(0, None)
+(0.0, 140.45)
+
Even though this is a Seaborn tutorial, Pandas actually plays a very important role. You see, Seaborn’s plotting functions benefit from a base DataFrame that’s reasonably formatted.
For example, let’s say we wanted to make a box plot for our Pokémon’s combat stats:
plt.figure(figsize = (8, 8))
+
+# Boxplot
+sns.boxplot(data=df)
+<AxesSubplot:>
+
Well, that’s a reasonable start, but there are some columns we’d probably like to remove:
In turns out that this isn’t easy to do within Seaborn alone. Instead, it’s much simpler to pre-format your DataFrame.
Let’s create a new DataFrame called stats_df that only keeps the stats columns:
plt.figure(figsize = (8, 8))
+
+# Pre-format DataFrame
+stats_df = df.drop(['Total', 'Stage', 'Legendary'], axis=1)
+
+# New boxplot using stats_df
+sns.boxplot(data=stats_df)
+<AxesSubplot:>
+
Another advantage of Seaborn is that it comes with decent style themes right out of the box. The default theme is called ‘darkgrid’.
Next, we’ll change the theme to ‘whitegrid’ while making a violin plot.
For example, we can visualize the distribution of Attack by Pokémon’s primary type:
plt.figure(figsize = (12, 12))
+
+# Set theme
+sns.set_style('whitegrid')
+
+# Violin plot
+sns.violinplot(x='Type 1', y='Attack', data=df)
+<AxesSubplot:xlabel='Type 1', ylabel='Attack'>
+
As you can see, Dragon types tend to have higher Attack stats than Ghost types, but they also have greater variance.
Now, Pokémon fans might find something quite jarring about that plot: The colors are nonsensical. Why is the Grass type colored pink or the Water type colored orange? We must fix this!
Fortunately, Seaborn allows us to set custom color palettes. We can simply create an ordered Python list of color hex values.
Let’s use Bulbapedia to help us create a new color palette:
pkmn_type_colors = ['#78C850', # Grass
+ '#F08030', # Fire
+ '#6890F0', # Water
+ '#A8B820', # Bug
+ '#A8A878', # Normal
+ '#A040A0', # Poison
+ '#F8D030', # Electric
+ '#E0C068', # Ground
+ '#EE99AC', # Fairy
+ '#C03028', # Fighting
+ '#F85888', # Psychic
+ '#B8A038', # Rock
+ '#705898', # Ghost
+ '#98D8D8', # Ice
+ '#7038F8', # Dragon
+ ]
+Wonderful. Now we can simply use the palette= argument to recolor our chart.
plt.figure(figsize = (10, 10))
+
+# Violin plot with Pokemon color palette
+sns.violinplot(x='Type 1', y='Attack', data=df,
+ palette=pkmn_type_colors) # Set color palette
+<AxesSubplot:xlabel='Type 1', ylabel='Attack'>
+
Much better!
Violin plots are great for visualizing distributions. However, since we only have 151 Pokémon in our dataset, we may want to simply display each point.
That’s where the swarm plot comes in. This visualization will show each point, while “stacking” those with similar values:
plt.figure(figsize = (10, 10))
+
+# Swarm plot with Pokemon color palette
+sns.swarmplot(x='Type 1', y='Attack', data=df,
+ palette=pkmn_type_colors)
+<AxesSubplot:xlabel='Type 1', ylabel='Attack'>
+
That’s handy, but can’t we combine our swarm plot and the violin plot? After all, they display similar information, right?
The answer is yes.
It’s pretty straightforward to overlay plots using Seaborn, and it works the same way as with Matplotlib. Here’s what we’ll do:
inner=None to remove the bars inside the violins.# Set figure size with matplotlib
+plt.figure(figsize=(10,6))
+
+# Create plot
+sns.violinplot(x='Type 1',
+ y='Attack',
+ data=df,
+ inner=None, # Remove the bars inside the violins
+ palette=pkmn_type_colors)
+
+sns.swarmplot(x='Type 1',
+ y='Attack',
+ data=df,
+ color='k', # Make points black
+ alpha=0.7) # and slightly transparent
+
+# Set title with matplotlib
+plt.title('Attack by Type')
+Text(0.5, 1.0, 'Attack by Type')
+
Awesome, now we have a pretty chart that tells us how Attack values are distributed across different Pokémon types. But what it we want to see all of the other stats as well?
Well, we could certainly repeat that chart for each stat. But we can also combine the information into one chart… we just have to do some data wrangling with Pandas beforehand.
First, here’s a reminder of our data format:
stats_df.head()
+| Name | Type 1 | Type 2 | HP | Attack | Defense | Sp. Atk | Sp. Def | Speed | |
|---|---|---|---|---|---|---|---|---|---|
| # | |||||||||
| 1 | Bulbasaur | Grass | Poison | 45 | 49 | 49 | 65 | 65 | 45 |
| 2 | Ivysaur | Grass | Poison | 60 | 62 | 63 | 80 | 80 | 60 |
| 3 | Venusaur | Grass | Poison | 80 | 82 | 83 | 100 | 100 | 80 |
| 4 | Charmander | Fire | NaN | 39 | 52 | 43 | 60 | 50 | 65 |
| 5 | Charmeleon | Fire | NaN | 58 | 64 | 58 | 80 | 65 | 80 |
As you can see, all of our stats are in separate columns. Instead, we want to “melt” them into one column.
To do so, we’ll use Pandas’s melt() function. It takes 3 arguments:
Here’s the output:
# Melt DataFrame
+melted_df = pd.melt(stats_df,
+ id_vars=["Name", "Type 1", "Type 2"], # Variables to keep
+ var_name="Stat") # Name of melted variable
+melted_df.sort_values('Name').head(10)
+| Name | Type 1 | Type 2 | Stat | value | |
|---|---|---|---|---|---|
| 364 | Abra | Psychic | NaN | Defense | 15 |
| 213 | Abra | Psychic | NaN | Attack | 20 |
| 817 | Abra | Psychic | NaN | Speed | 90 |
| 666 | Abra | Psychic | NaN | Sp. Def | 55 |
| 515 | Abra | Psychic | NaN | Sp. Atk | 105 |
| 62 | Abra | Psychic | NaN | HP | 25 |
| 443 | Aerodactyl | Rock | Flying | Defense | 65 |
| 594 | Aerodactyl | Rock | Flying | Sp. Atk | 60 |
| 745 | Aerodactyl | Rock | Flying | Sp. Def | 75 |
| 292 | Aerodactyl | Rock | Flying | Attack | 105 |
All 6 of the stat columns have been “melted” into one, and the new Stat column indicates the original stat (HP, Attack, Defense, Sp. Attack, Sp. Defense, or Speed). For example, it’s hard to see here, but Bulbasaur now has 6 rows of data.
In fact, if you print the shape of these two DataFrames…
print( stats_df.shape )
+print( melted_df.shape )
+(151, 9)
+(906, 5)
+…you’ll find that melted_df has 6 times the number of rows as stats_df.
Now we can make a swarm plot with melted_df.
x='Stat' and y='value' so our swarms are separated by stat.hue='Type 1' to color our points by the Pokémon type.plt.figure(figsize=(14,10))
+
+# Swarmplot with melted_df
+sns.swarmplot(x='Stat', y='value', data=melted_df,
+ hue='Type 1')
+<AxesSubplot:xlabel='Stat', ylabel='value'>
+
Finally, let’s make a few final tweaks for a more readable chart:
split=True.# 1. Enlarge the plot
+plt.figure(figsize=(14,10))
+
+sns.swarmplot(x='Stat',
+ y='value',
+ data=melted_df,
+ hue='Type 1',
+ split=True, # 2. Separate points by hue
+ palette=pkmn_type_colors) # 3. Use Pokemon palette
+
+# 4. Adjust the y-axis
+plt.ylim(0, 260)
+
+# 5. Place legend to the right
+plt.legend(bbox_to_anchor=(1, 1), loc=2)
+<matplotlib.legend.Legend at 0x7fd6f8a04160>
+
We’re going to conclude this tutorial with a few quick-fire data visualizations, just to give you a sense of what’s possible with Seaborn.
Heatmaps help you visualize matrix-like data.
plt.figure(figsize=(8,6))
+
+# Calculate correlations
+corr = stats_df.corr()
+
+# Heatmap
+sns.heatmap(corr)
+<AxesSubplot:>
+
Histograms allow you to plot the distributions of numeric variables.
plt.figure(figsize=(8,6))
+
+# Distribution Plot (a.k.a. Histogram)
+sns.distplot(df.Attack)
+<AxesSubplot:xlabel='Attack', ylabel='Density'>
+
Bar plots help you visualize the distributions of categorical variables.
plt.figure(figsize=(8,6))
+
+# Count Plot (a.k.a. Bar Plot)
+sns.countplot(x='Type 1', data=df, palette=pkmn_type_colors)
+
+# Rotate x-labels
+plt.xticks(rotation=-45)
+(array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14]),
+ [Text(0, 0, 'Grass'),
+ Text(1, 0, 'Fire'),
+ Text(2, 0, 'Water'),
+ Text(3, 0, 'Bug'),
+ Text(4, 0, 'Normal'),
+ Text(5, 0, 'Poison'),
+ Text(6, 0, 'Electric'),
+ Text(7, 0, 'Ground'),
+ Text(8, 0, 'Fairy'),
+ Text(9, 0, 'Fighting'),
+ Text(10, 0, 'Psychic'),
+ Text(11, 0, 'Rock'),
+ Text(12, 0, 'Ghost'),
+ Text(13, 0, 'Ice'),
+ Text(14, 0, 'Dragon')])
+
Factor plots make it easy to separate plots by categorical classes.
# Factor Plot
+g = sns.factorplot(x='Type 1',
+ y='Attack',
+ data=df,
+ hue='Stage', # Color by stage
+ col='Stage', # Separate by stage
+ kind='swarm') # Swarmplot
+
+# Rotate x-axis labels
+g.set_xticklabels(rotation=-45)
+
+# Doesn't work because only rotates last plot
+# plt.xticks(rotation=-45)
+<seaborn.axisgrid.FacetGrid at 0x7fd729325c40>
+
Density plots display the distribution between two variables.
plt.figure(figsize=(8,6))
+
+# Density Plot
+sns.kdeplot(df.Attack, df.Defense)
+<AxesSubplot:xlabel='Attack', ylabel='Defense'>
+
Joint distribution plots combine information from scatter plots and histograms to give you detailed information for bi-variate distributions.
plt.figure(figsize=(8,6))
+
+# Joint Distribution Plot
+sns.jointplot(x='Attack', y='Defense', data=df)
+<seaborn.axisgrid.JointGrid at 0x7fd708a60f70>
+
+
+
+
+<Figure size 576x432 with 0 Axes>
+
Congratulations… you’ve made it to the end of this Python Seaborn tutorial!
We’ve just concluded a tour of key Seaborn paradigms and showed you many examples along the way. Feel free to use this page along with the official Seaborn gallery as references for your projects going forward.
+| \n", + " | Name | \n", + "Type 1 | \n", + "Type 2 | \n", + "Total | \n", + "HP | \n", + "Attack | \n", + "Defense | \n", + "Sp. Atk | \n", + "Sp. Def | \n", + "Speed | \n", + "Stage | \n", + "Legendary | \n", + "
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| # | \n", + "\n", + " | \n", + " | \n", + " | \n", + " | \n", + " | \n", + " | \n", + " | \n", + " | \n", + " | \n", + " | \n", + " | \n", + " |
| 1 | \n", + "Bulbasaur | \n", + "Grass | \n", + "Poison | \n", + "318 | \n", + "45 | \n", + "49 | \n", + "49 | \n", + "65 | \n", + "65 | \n", + "45 | \n", + "1 | \n", + "False | \n", + "
| 2 | \n", + "Ivysaur | \n", + "Grass | \n", + "Poison | \n", + "405 | \n", + "60 | \n", + "62 | \n", + "63 | \n", + "80 | \n", + "80 | \n", + "60 | \n", + "2 | \n", + "False | \n", + "
| 3 | \n", + "Venusaur | \n", + "Grass | \n", + "Poison | \n", + "525 | \n", + "80 | \n", + "82 | \n", + "83 | \n", + "100 | \n", + "100 | \n", + "80 | \n", + "3 | \n", + "False | \n", + "
| 4 | \n", + "Charmander | \n", + "Fire | \n", + "NaN | \n", + "309 | \n", + "39 | \n", + "52 | \n", + "43 | \n", + "60 | \n", + "50 | \n", + "65 | \n", + "1 | \n", + "False | \n", + "
| 5 | \n", + "Charmeleon | \n", + "Fire | \n", + "NaN | \n", + "405 | \n", + "58 | \n", + "64 | \n", + "58 | \n", + "80 | \n", + "65 | \n", + "80 | \n", + "2 | \n", + "False | \n", + "
| \n", + " | Name | \n", + "Type 1 | \n", + "Type 2 | \n", + "HP | \n", + "Attack | \n", + "Defense | \n", + "Sp. Atk | \n", + "Sp. Def | \n", + "Speed | \n", + "
|---|---|---|---|---|---|---|---|---|---|
| # | \n", + "\n", + " | \n", + " | \n", + " | \n", + " | \n", + " | \n", + " | \n", + " | \n", + " | \n", + " |
| 1 | \n", + "Bulbasaur | \n", + "Grass | \n", + "Poison | \n", + "45 | \n", + "49 | \n", + "49 | \n", + "65 | \n", + "65 | \n", + "45 | \n", + "
| 2 | \n", + "Ivysaur | \n", + "Grass | \n", + "Poison | \n", + "60 | \n", + "62 | \n", + "63 | \n", + "80 | \n", + "80 | \n", + "60 | \n", + "
| 3 | \n", + "Venusaur | \n", + "Grass | \n", + "Poison | \n", + "80 | \n", + "82 | \n", + "83 | \n", + "100 | \n", + "100 | \n", + "80 | \n", + "
| 4 | \n", + "Charmander | \n", + "Fire | \n", + "NaN | \n", + "39 | \n", + "52 | \n", + "43 | \n", + "60 | \n", + "50 | \n", + "65 | \n", + "
| 5 | \n", + "Charmeleon | \n", + "Fire | \n", + "NaN | \n", + "58 | \n", + "64 | \n", + "58 | \n", + "80 | \n", + "65 | \n", + "80 | \n", + "
| \n", + " | Name | \n", + "Type 1 | \n", + "Type 2 | \n", + "Stat | \n", + "value | \n", + "
|---|---|---|---|---|---|
| 364 | \n", + "Abra | \n", + "Psychic | \n", + "NaN | \n", + "Defense | \n", + "15 | \n", + "
| 213 | \n", + "Abra | \n", + "Psychic | \n", + "NaN | \n", + "Attack | \n", + "20 | \n", + "
| 817 | \n", + "Abra | \n", + "Psychic | \n", + "NaN | \n", + "Speed | \n", + "90 | \n", + "
| 666 | \n", + "Abra | \n", + "Psychic | \n", + "NaN | \n", + "Sp. Def | \n", + "55 | \n", + "
| 515 | \n", + "Abra | \n", + "Psychic | \n", + "NaN | \n", + "Sp. Atk | \n", + "105 | \n", + "
| 62 | \n", + "Abra | \n", + "Psychic | \n", + "NaN | \n", + "HP | \n", + "25 | \n", + "
| 443 | \n", + "Aerodactyl | \n", + "Rock | \n", + "Flying | \n", + "Defense | \n", + "65 | \n", + "
| 594 | \n", + "Aerodactyl | \n", + "Rock | \n", + "Flying | \n", + "Sp. Atk | \n", + "60 | \n", + "
| 745 | \n", + "Aerodactyl | \n", + "Rock | \n", + "Flying | \n", + "Sp. Def | \n", + "75 | \n", + "
| 292 | \n", + "Aerodactyl | \n", + "Rock | \n", + "Flying | \n", + "Attack | \n", + "105 | \n", + "
This is the process of extracting information from a webpage by taking advantage of patterns in the web page’s underlying code.
We can use web scraping to gather unstructured data from the internet, process it and store it in a structured format.
In this walkthrough, we’ll be storing our data in a JSON file.
Though web scraping is a useful tool in extracting data from a website, it’s not the only means to achieve this task.
Before starting to web scrape, find out if the page you seek to extract data from provides an API.
Ensure that you check the robots.txt file of a website before making your scrapper. This file tells if the website allows scraping or if they do not.
To check for the file, simply type the base URL followed by “/robots.txt” An example is, “mysite.com/robots.txt”.
For more about robots.txt files click here.
In this tutorial, we’ll be extracting data from books to scrape which you can use to practise your web scraping.
We’ll extract the title, rating, link to more information about the book and the cover image of the book. Code can be found on GitHub.
The python libraries perform the following tasks.
import requests
+import json
+from bs4 import BeautifulSoup
+We’re initializing three variables here.
header = {'User-Agent': 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7'}
+
+base_url = "http://books.toscrape.com/"
+
+r = requests.get(base_url, headers=header)
+To ensure our scraper runs when the http response is ok we’ll use the if statement as a check. The number 200 is the status code for Ok. To get a list of all codes and their meanings check out this resource.
We’ll then parse the response object using the BeautifulSoup method and store the new object to a variable called soup.
if r.status_code == 200:
+ soup = BeautifulSoup(r.text, 'html.parser')
+ books = soup.find_all('li',attrs={"class":"col-xs-6 col-sm-4 col-md-3 col-lg-3"})
+ result=[]
+ for book in books:
+ title=book.find('h3').text
+ link=base_url +book.find('a')['href']
+ stars = str(len(book.find_all('i',attrs= {"class":"icon-star"}))) + " out of 5"
+ price="$"+book.find('p',attrs={'class':'price_color'}).text[2:]
+ picture = base_url + book.find('img')['src']
+ single ={'title':title,'stars':stars,'price':price,'link':link,'picture':picture}
+ result.append(single)
+
+ with open('books.json','w') as f:
+ json.dump(result,f,indent=4)
+else:
+ print(r.status_code)
+import pandas as pd
+
+df = pd.read_json('books.json')
+df.head()
+| title | stars | price | link | picture | |
|---|---|---|---|---|---|
| 0 | A Light in the ... | 5 out of 5 | $51.77 | http://books.toscrape.com/catalogue/a-light-in... | http://books.toscrape.com/media/cache/2c/da/2c... |
| 1 | Tipping the Velvet | 5 out of 5 | $53.74 | http://books.toscrape.com/catalogue/tipping-th... | http://books.toscrape.com/media/cache/26/0c/26... |
| 2 | Soumission | 5 out of 5 | $50.10 | http://books.toscrape.com/catalogue/soumission... | http://books.toscrape.com/media/cache/3e/ef/3e... |
| 3 | Sharp Objects | 5 out of 5 | $47.82 | http://books.toscrape.com/catalogue/sharp-obje... | http://books.toscrape.com/media/cache/32/51/32... |
| 4 | Sapiens: A Brief History ... | 5 out of 5 | $54.23 | http://books.toscrape.com/catalogue/sapiens-a-... | http://books.toscrape.com/media/cache/be/a5/be... |
Let’s take a look at a single record from our webpage to identify the patterns. Once we can see the page, we’ll loop through every record in the page as they contain similar traits.
From the image above, we’ll notice that all books are contained within a list item with the class.
col-xs-6 col-sm-4 col-md-3 col-lg-3
+By using the find_all() method, we can find all references of this HTML tag in the webpage. we pass the tag as the first argument and then using the attrs argument which takes in a python dictionary, we can specify attributes of the HTML tag selected. In this case, it was a class indicated above, but you can even use id as an attribute.
Store the result in a variable, I chose the name books.
title = book.find('h3').text
+link = base_url + book.find('a')['href']
+If we observe keenly, we’ll notice that each of the elements we want to extract is nested within the list item tag are all contained in similar tags, in the example above. The title of the book is between h3 tags.
The find() method returns the first matching tag.
text will simply return any text found within the tags specified.
For the anchor tags, we’ll be extracting the hyper reference link.
As opposed to h3 tag, the href element is within anchor tags in HTML. Like so:
<a href="somelink.com"></a>
+In this case, the returned object will behave like a dictionary where we have a
dictionary_name[key]
+We do this iteratively for all the values we seek to extract because we are taking advantage of the pattern in the underlying code of the webpage. Hence the use of the python for loop.
The extracted elements are then stored in respective variables which we’ll put in a dictionary. With this information, we can then comfortably append the dictionary object to the initialized result list set before our for loop.
single ={'title':title,'stars':stars,'price':price,'link':link,'picture':picture}
+result.append(single)
+with open('books2.json','w') as f:
+ json.dump(result,f,indent=4)
+Finally, store the python list in a JSON file by the name “books.json” with an indent of 4 for readability purposes.
+| \n", + " | title | \n", + "stars | \n", + "price | \n", + "link | \n", + "picture | \n", + "
|---|---|---|---|---|---|
| 0 | \n", + "A Light in the ... | \n", + "5 out of 5 | \n", + "$51.77 | \n", + "http://books.toscrape.com/catalogue/a-light-in... | \n", + "http://books.toscrape.com/media/cache/2c/da/2c... | \n", + "
| 1 | \n", + "Tipping the Velvet | \n", + "5 out of 5 | \n", + "$53.74 | \n", + "http://books.toscrape.com/catalogue/tipping-th... | \n", + "http://books.toscrape.com/media/cache/26/0c/26... | \n", + "
| 2 | \n", + "Soumission | \n", + "5 out of 5 | \n", + "$50.10 | \n", + "http://books.toscrape.com/catalogue/soumission... | \n", + "http://books.toscrape.com/media/cache/3e/ef/3e... | \n", + "
| 3 | \n", + "Sharp Objects | \n", + "5 out of 5 | \n", + "$47.82 | \n", + "http://books.toscrape.com/catalogue/sharp-obje... | \n", + "http://books.toscrape.com/media/cache/32/51/32... | \n", + "
| 4 | \n", + "Sapiens: A Brief History ... | \n", + "5 out of 5 | \n", + "$54.23 | \n", + "http://books.toscrape.com/catalogue/sapiens-a-... | \n", + "http://books.toscrape.com/media/cache/be/a5/be... | \n", + "
Are you new to CSS? This article is for you! Perhaps the biggest key to understanding CSS is understanding selectors. Selectors are what allows you to target specific HTML elements and apply style to them. Let’s not think about style right now though, let’s just focus on the selecting.
In the examples below, the CSS would be in a file called something like style.css that is referenced from an HTML document called something like index.html. They are separate files, which is the great thing about CSS, keeping the design away from the document.
Here’s what that HTML file would be like:
<!DOCTYPE html>
+<html lang="en">
+
+<head>
+ <title>We're learning selectors!</title>
+ <link rel="stylesheet" href="style.css">
+</head>
+
+<body>
+
+ <h1 id="yay">Yay</h1>
+
+<body>
+</html>
+And the CSS file would contain just the selector blocks like you’ll see below.
CSS
#happy-cake {
+
+}
+HTML
<!-- WILL match -->
+<div id="happy-cake"></div>
+
+<!-- WILL match -->
+<aside id="happy-cake"></aside>
+
+<!-- Will NOT match -->
+<div id="sad-cake">Wrong ID!</div>
+
+<!-- Will NOT match -->
+<div class="happy-cake">That's not an ID!</div>
+ID selectors are the most powerful type of selector in terms of CSS specificity. Meaning that they beat out other types of selectors and the styles defined within win. That sounds good, but that’s typically considered bad, because it’s nice to have lower-specificity selectors that are easier to override when needed.
CSS
.module {
+
+}
+HTML*
<!-- WILL match -->
+<div class="module"></div>
+
+<!-- WILL match -->
+<aside class="country module iceland"></aside>
+
+<!-- Will NOT match -->
+<div class=".module">The dot is for CSS, not HTML</div>
+
+<!-- Will NOT match -->
+<div class="bigmodule">Wrong class</div>
+
+Class selectors are your friend. They are probably the most useful and versatile selectors out there. In part because they are well supported in all browsers. In part because you can add multiple classes (just separated by a space) on HTML elements. In part because there are JavaScript things you can do specifically for manipulating classes.
CSS
h2 {
+
+}
+HTML
<!-- WILL match -->
+<h2>Hi, Mom</h2>
+
+<main>
+ <div>
+ <!-- WILL match -->
+ <h2>Anywhere</h2>
+ </div>
+</main>
+
+<!-- Will NOT match -->
+<div class="h2">Wrong tag, can't trick it</div>
+
+<!-- Will NOT match -->
+<h2class="yolo">Make sure that tag has a space after it!</h2>
+Tag selectors are at their most useful when changing properties that are unique to that HTML element. Like setting the list-style on a <ul> or tab-size on a <pre>. Also in reset stylesheets where you are specifically trying to unset styles that browsers apply to certain elements.
Don’t rely on them too much though. It’s typically more useful to have a class define styling that you can use on any HTML element.
CSS
[data-modal="open"] {
+
+}
+HTML
<!-- WILL match -->
+<div data-modal="open"></div>
+
+<!-- WILL match -->
+<aside class='closed' data-modal='open'></aside>
+
+<!-- Will NOT match -->
+<div data-modal="false">Wrong value</div>
+
+<!-- Will NOT match -->
+<div data-modal>No value</div>
+
+<!-- Will NOT match -->
+<div data-modal-open>Wrong attribute</div>
+You might argue that attribute selectors are even more useful than classes because they have the same specificity value, but can be any attribute not just class, plus they can have a value you can select by.
Hardly an issue anymore, but attribute selectors aren’t supported in IE 6.
CSS
:nth-child(2) {
+
+}
+HTML
<ul>
+ <li>nope</li>
+ <!-- WILL match -->
+ <li>yep, I'm #2</li>
+ <li>nope</li>
+</ul>
+There are several different positional selectors beyond :nth-child. Using simple expressions (like 3n = “every third”) you can select elements based on their position in the HTML. You can play with that idea here or check out some useful recipes.
CSS
:empty {
+
+}
+HTML
<!-- WILL match -->
+<div></div>
+
+<!-- WILL match -->
+<aside data-blah><!-- nothin' --></aside>
+
+<!-- Will NOT match -->
+<div> </div>
+
+<!-- Will NOT match -->
+<div>
+</div>
+:empty is one of many pseudo selectors, which you can recognize by the colon (:) in them. They typically represent something that you couldn’t know by just the element and attributes alone.
Note that these are slightly different than pseudo elements, which you can recognize by the double colon (::). They are responsible for adding things to the page by the things they select.
Selectors can be combined together. For instance:
.module.news {
+ /* Selects elements with BOTH of those classes */
+}
+#site-footer::after {
+ /* Adds content after an element with that ID */
+}
+section[data-open] {
+ /* Selects only section elements if they have this attribute */
+}
+
+There are also selector combinators like ~ and + and > that affect selectors, like:
.module > h2 {
+ /* Select h2 elements that are direct children of an element with that class */
+}
+h2 + p {
+ /* Select p elements that are directly following an h2 element */
+}
+li ~ li {
+ /* Select li elements that are siblings (and following) another li element. */
+}
+On CSS-Tricks there is an entire Almanac that covers all the selectors in CSS, as well as properties.
+```. Also in [reset stylesheets](https://css-tricks.com/poll-results-what-css-reset-do-you-use/) where you are specifically trying to unset styles that browsers apply to certain elements.\n",
+ "\n",
+ "Don't rely on them too much though. It's typically more useful to have a class define styling that you can use on any HTML element."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Attribute Selector\n",
+ "\n",
+ "**CSS**\n",
+ "```css\n",
+ "[data-modal=\"open\"] {\n",
+ "\n",
+ "}\n",
+ "```\n",
+ "\n",
+ "**HTML**\n",
+ "```html\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "Wrong value\n",
+ "\n",
+ "\n",
+ "No value\n",
+ "\n",
+ "\n",
+ "Wrong attribute\n",
+ "```\n",
+ "\n",
+ "You might argue that [attribute selectors](https://css-tricks.com/attribute-selectors/) are even more useful than classes because they have the same [specificity value](https://css-tricks.com/specifics-on-css-specificity/), but can be any attribute not just ```class```, plus they can have a value you can select by.\n",
+ "\n",
+ "Hardly an issue anymore, but attribute selectors aren't supported in IE 6."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Positional Selectors\n",
+ "\n",
+ "**CSS**\n",
+ "```css\n",
+ ":nth-child(2) {\n",
+ "\n",
+ "}\n",
+ "```\n",
+ "\n",
+ "**HTML**\n",
+ "```html\n",
+ "Many data analysis, big data, and machine learning projects require scraping websites to gather the data that you’ll be working with. The Python programming language is widely used in the data science community, and therefore has an ecosystem of modules and tools that you can use in your own projects. In this tutorial we will be focusing on the Beautiful Soup module.
Beautiful Soup is a Python library that allows for quick turnaround on web scraping projects. Currently available as Beautiful Soup 4 and compatible with both Python 2.7 and Python 3, Beautiful Soup creates a parse tree from parsed HTML and XML documents (including documents with non-closed tags or tag soup and other malformed markup).
In this tutorial, we will collect and parse a web page in order to grab textual data and write the information we have gathered to a CSV file.
Before working on this tutorial, you should have a local or server-based Python programming environment set up on your machine.
You should have the Requests and Beautiful Soup modules installed, which you can achieve by following the tutorial “How To Work with Web Data Using Requests and Beautiful Soup with Python 3”. It would also be useful to have a working familiarity with these modules.
Additionally, since we will be working with data scraped from the web, you should be comfortable with HTML structure and tagging.
In this tutorial, we’ll be working with data from the official website of the National Gallery of Art in the United States. The National Gallery is an art museum located on the National Mall in Washington, D.C. It holds over 120,000 pieces dated from the Renaissance to the present day done by more than 13,000 artists.
We would like to search the Index of Artists, which, at the time of updating this tutorial, is available via the Internet Archive’s Wayback Machine at the following URL:
https://web.archive.org/web/20170131230332/https://www.nga.gov/collection/an.shtm
Beneath the Internet Archive’s header, you’ll see a page that looks like this:
Since we’ll be doing this project in order to learn about web scraping with Beautiful Soup, we don’t need to pull too much data from the site, so let’s limit the scope of the artist data we are looking to scrape. Let’s therefore choose one letter — in our example we’ll choose the letter Z — and we’ll see a page that looks like this:
In the page above, we see that the first artist listed at the time of writing is Zabaglia, Niccola, which is a good thing to note for when we start pulling data. We’ll start by working with this first page, with the following URL for the letter Z:
https://web.archive.org/web/20121007172955/http://www.nga.gov/collection/anZ1.htm
It is important to note for later how many pages total there are for the letter you are choosing to list, which you can discover by clicking through to the last page of artists. In this case, there are 4 pages total, and the last artist listed at the time of writing is Zykmund, Václav. The last page of Z artists has the following URL:
https://web.archive.org/web/20121010201041/http://www.nga.gov/collection/anZ4.htm
However, you can also access the above page by using the same Internet Archive numeric string of the first page:
https://web.archive.org/web/20121007172955/http://www.nga.gov/collection/anZ4.htm
This is important to note because we’ll be iterating through these pages later in this tutorial.
To begin to familiarize yourself with how this web page is set up, you can take a look at its DOM, which will help you understand how the HTML is structured. In order to inspect the DOM, you can open your browser’s Developer Tools.
We’ll create a new file called nga_z_artists.py in this tutorial.
Within this file, we can begin to import the libraries we’ll be using — Requests and Beautiful Soup.
The Requests library allows you to make use of HTTP within your Python programs in a human readable way, and the Beautiful Soup module is designed to get web scraping done quickly.
We will import both Requests and Beautiful Soup with the import statement. For Beautiful Soup, we’ll be importing it from bs4, the package in which Beautiful Soup 4 is found.
# Import libraries
+import requests
+from bs4 import BeautifulSoup
+The next step we will need to do is collect the URL of the first web page with Requests. We’ll assign the URL for the first page to the variable page by using the method requests.get().
# Collect first page of artists’ list
+page = requests.get('https://web.archive.org/web/20121007172955/https://www.nga.gov/collection/anZ1.htm')
+We’ll now create a BeautifulSoup object, or a parse tree. This object takes as its arguments the page.text document from Requests (the content of the server’s response) and then parses it from Python’s built-in html.parser.
# Create a BeautifulSoup object
+soup = BeautifulSoup(page.text, 'html.parser')
+For this project, we’ll collect artists’ names and the relevant links available on the website. You may want to collect different data, such as the artists’ nationality and dates. Whatever data you would like to collect, you need to find out how it is described by the DOM of the web page.
To do this, in your web browser, right-click — or CTRL + click on macOS — on the first artist’s name, Zabaglia, Niccola. Within the context menu that pops up, you should see a menu item similar to Inspect Element (Firefox) or Inspect (Chrome).
Once you click on the relevant Inspect menu item, the tools for web developers should appear within your browser. We want to look for the class and tags associated with the artists’ names in this list.
We’ll see first that the table of names is within <div> tags where class="BodyText". This is important to note so that we only search for text within this section of the web page. We also notice that the name Zabaglia, Niccola is in a link tag, since the name references a web page that describes the artist. So we will want to reference the <a> tag for links. Each artist’s name is a reference to a link.
To do this, we’ll use Beautiful Soup’s find() and find_all() methods in order to pull the text of the artists’ names from the BodyText <div>.
# Pull all text from the BodyText div
+artist_name_list = soup.find(class_='BodyText')
+# Pull text from all instances of <a> tag within BodyText div
+artist_name_list_items = artist_name_list.find_all('a')
+Next, at the bottom of our program file, we will want to create a for loop in order to iterate over all the artist names that we just put into the artist_name_list_items variable.
We’ll print these names out with the prettify() method in order to turn the Beautiful Soup parse tree into a nicely formatted Unicode string.
# Create for loop to print out the artists' names
+for i, artist_name in enumerate(artist_name_list_items):
+ print(artist_name.prettify())
+ if i > 4:
+ break
+<a href="/web/20121007172955/https://www.nga.gov/cgi-bin/tsearch?artistid=11630">
+ Zabaglia, Niccola
+</a>
+<a href="/web/20121007172955/https://www.nga.gov/cgi-bin/tsearch?artistid=34202">
+ Zaccone, Fabian
+</a>
+<a href="/web/20121007172955/https://www.nga.gov/cgi-bin/tsearch?artistid=3475">
+ Zadkine, Ossip
+</a>
+<a href="/web/20121007172955/https://www.nga.gov/cgi-bin/tsearch?artistid=25135">
+ Zaech, Bernhard
+</a>
+<a href="/web/20121007172955/https://www.nga.gov/cgi-bin/tsearch?artistid=2298">
+ Zagar, Jacob
+</a>
+<a href="/web/20121007172955/https://www.nga.gov/cgi-bin/tsearch?artistid=23988">
+ Zagroba, Idalia
+</a>
+What we see in the output at this point is the full text and tags related to all of the artists’ names within the <a> tags found in the <div class="BodyText"> tag on the first page, as well as some additional link text at the bottom. Since we don’t want this extra information, let’s work on removing this in the next section.
So far, we have been able to collect all the link text data within one <div> section of our web page. However, we don’t want to have the bottom links that don’t reference artists’ names, so let’s work to remove that part.
In order to remove the bottom links of the page, let’s again right-click and Inspect the DOM. We’ll see that the links on the bottom of the <div class="BodyText"> section are contained in an HTML table: <table class="AlphaNav">:
We can therefore use Beautiful Soup to find the AlphaNav class and use the decompose() method to remove a tag from the parse tree and then destroy it along with its contents.
We’ll use the variable last_links to reference these bottom links and add them to the program file:
page = requests.get('https://web.archive.org/web/20121007172955/https://www.nga.gov/collection/anZ1.htm')
+
+soup = BeautifulSoup(page.text, 'html.parser')
+
+# Remove bottom links
+last_links = soup.find(class_='AlphaNav')
+last_links.decompose()
+
+artist_name_list = soup.find(class_='BodyText')
+artist_name_list_items = artist_name_list.find_all('a')
+
+for i, artist_name in enumerate(artist_name_list_items):
+ print(artist_name.prettify())
+ if i > 4:
+ break
+<a href="/web/20121007172955/https://www.nga.gov/cgi-bin/tsearch?artistid=11630">
+ Zabaglia, Niccola
+</a>
+<a href="/web/20121007172955/https://www.nga.gov/cgi-bin/tsearch?artistid=34202">
+ Zaccone, Fabian
+</a>
+<a href="/web/20121007172955/https://www.nga.gov/cgi-bin/tsearch?artistid=3475">
+ Zadkine, Ossip
+</a>
+<a href="/web/20121007172955/https://www.nga.gov/cgi-bin/tsearch?artistid=25135">
+ Zaech, Bernhard
+</a>
+<a href="/web/20121007172955/https://www.nga.gov/cgi-bin/tsearch?artistid=2298">
+ Zagar, Jacob
+</a>
+<a href="/web/20121007172955/https://www.nga.gov/cgi-bin/tsearch?artistid=23988">
+ Zagroba, Idalia
+</a>
+At this point, we see that the output no longer includes the links at the bottom of the web page, and now only displays the links associated with artists’ names.
Until now, we have targeted the links with the artists’ names specifically, but we have the extra tag data that we don’t really want. Let’s remove that in the next section.
In order to access only the actual artists’ names, we’ll want to target the contents of the <a> tags rather than print out the entire link tag.
We can do this with Beautiful Soup’s .contents, which will return the tag’s children as a Python list data type.
Let’s revise the for loop so that instead of printing the entire link and its tag, we’ll print the list of children (i.e. the artists’ full names):
# Use .contents to pull out the <a> tag’s children
+for i, artist_name in enumerate(artist_name_list_items):
+ names = artist_name.contents[0]
+ print(names)
+ if i > 4:
+ break
+Zabaglia, Niccola
+Zaccone, Fabian
+Zadkine, Ossip
+Zaech, Bernhard
+Zagar, Jacob
+Zagroba, Idalia
+Note that we are iterating over the list above by calling on the index number of each item.
We have received back a list of all the artists’ names available on the first page of the letter Z.
However, what if we want to also capture the URLs associated with those artists? We can extract URLs found within a page’s <a> tags by using Beautiful Soup’s get('href') method.
From the output of the links above, we know that the entire URL is not being captured, so we will concatenate the link string with the front of the URL string (in this case https://web.archive.org/).
These lines we’ll also add to the for loop:
for i, artist_name in enumerate(artist_name_list_items):
+ names = artist_name.contents[0]
+ links = 'https://web.archive.org' + artist_name.get('href')
+ print(names)
+ print(links)
+ if i > 4:
+ break
+Zabaglia, Niccola
+https://web.archive.org/web/20121007172955/https://www.nga.gov/cgi-bin/tsearch?artistid=11630
+Zaccone, Fabian
+https://web.archive.org/web/20121007172955/https://www.nga.gov/cgi-bin/tsearch?artistid=34202
+Zadkine, Ossip
+https://web.archive.org/web/20121007172955/https://www.nga.gov/cgi-bin/tsearch?artistid=3475
+Zaech, Bernhard
+https://web.archive.org/web/20121007172955/https://www.nga.gov/cgi-bin/tsearch?artistid=25135
+Zagar, Jacob
+https://web.archive.org/web/20121007172955/https://www.nga.gov/cgi-bin/tsearch?artistid=2298
+Zagroba, Idalia
+https://web.archive.org/web/20121007172955/https://www.nga.gov/cgi-bin/tsearch?artistid=23988
+Although we are now getting information from the website, it is currently just printing to our terminal window. Let’s instead capture this data so that we can use it elsewhere by writing it to a file.
Collecting data that only lives in a terminal window is not very useful. Comma-separated values (CSV) files allow us to store tabular data in plain text, and is a common format for spreadsheets and databases. Before beginning with this section, you should familiarize yourself with how to handle plain text files in Python.
First, we need to import Python’s built-in csv module along with the other modules at the top of the Python programming file:
import csv
+Next, we’ll create and open a file called z-artist-names.csv for us to write to (we’ll use the variable f for file here) by using the 'w' mode. We’ll also write the top row headings: Name and Link which we’ll pass to the writerow() method as a list:
f = csv.writer(open('z-artist-names.csv', 'w'))
+f.writerow(['Name', 'Link'])
+11
+Finally, within our for loop, we’ll write each row with the artists’ names and their associated links:
page = requests.get('https://web.archive.org/web/20121007172955/http://www.nga.gov/collection/anZ1.htm')
+
+soup = BeautifulSoup(page.text, 'html.parser')
+
+last_links = soup.find(class_='AlphaNav')
+last_links.decompose()
+
+# Create a file to write to, add headers row
+f = csv.writer(open('z-artist-names.csv', 'w'))
+f.writerow(['Name', 'Link'])
+
+artist_name_list = soup.find(class_='BodyText')
+artist_name_list_items = artist_name_list.find_all('a')
+
+for artist_name in artist_name_list_items:
+ names = artist_name.contents[0]
+ links = 'https://web.archive.org' + artist_name.get('href')
+
+
+ # Add each artist’s name and associated link to a row
+ f.writerow([names, links])
+We have created a program that will pull data from the first page of the list of artists whose last names start with the letter Z. However, there are 4 pages in total of these artists available on the website.
In order to collect all of these pages, we can perform more iterations with for loops. This will revise most of the code we have written so far, but will employ similar concepts.
To start, we’ll want to initialize a list to hold the pages:
pages = []
+We will populate this initialized list with the following for loop:
for i in range(1, 5):
+ url = 'https://web.archive.org/web/20121007172955/https://www.nga.gov/collection/anZ' + str(i) + '.htm'
+ pages.append(url)
+Earlier in this tutorial, we noted that we should pay attention to the total number of pages there are that contain artists’ names starting with the letter Z (or whatever letter we’re using). Since there are 4 pages for the letter Z, we constructed the for loop above with a range of 1 to 5 so that it will iterate through each of the 4 pages.
For this specific web site, the URLs begin with the string https://web.archive.org/web/20121007172955/https://www.nga.gov/collection/anZ and then are followed with a number for the page (which will be the integer i from the for loop that we convert to a string) and end with .htm. We will concatenate these strings together and then append the result to the pages list.
In addition to this loop, we’ll have a second loop that will go through each of the pages above. The code in this for loop will look similar to the code we have created so far, as it is doing the task we completed for the first page of the letter Z artists for each of the 4 pages total. Note that because we have put the original program into the second for loop, we now have the original loop as a nested for loop contained in it.
The two for loops will look like this:
pages = []
+
+for i in range(1, 5):
+ url = 'https://web.archive.org/web/20121007172955/https://www.nga.gov/collection/anZ' + str(i) + '.htm'
+ pages.append(url)
+
+for item in pages:
+ page = requests.get(item)
+ soup = BeautifulSoup(page.text, 'html.parser')
+
+ last_links = soup.find(class_='AlphaNav')
+ last_links.decompose()
+
+ artist_name_list = soup.find(class_='BodyText')
+ artist_name_list_items = artist_name_list.find_all('a')
+
+ for artist_name in artist_name_list_items:
+ names = artist_name.contents[0]
+ links = 'https://web.archive.org' + artist_name.get('href')
+
+ f.writerow([names, links])
+In the code above, you should see that the first for loop is iterating over the pages and the second for loop is scraping data from each of those pages and then is adding the artists’ names and links line by line through each row of each page.
These two for loops come below the import statements, the CSV file creation and writer (with the line for writing the headers of the file), and the initialization of the pages variable (assigned to a list).
Within the greater context of the programming file, the complete code looks like this:
import requests
+import csv
+from bs4 import BeautifulSoup
+
+
+f = csv.writer(open('z-artist-names.csv', 'w'))
+f.writerow(['Name', 'Link'])
+
+pages = []
+
+for i in range(1, 5):
+ url = 'https://web.archive.org/web/20121007172955/https://www.nga.gov/collection/anZ' + str(i) + '.htm'
+ pages.append(url)
+
+
+for item in pages:
+ page = requests.get(item)
+ soup = BeautifulSoup(page.text, 'html.parser')
+
+ last_links = soup.find(class_='AlphaNav')
+ last_links.decompose()
+
+ artist_name_list = soup.find(class_='BodyText')
+ artist_name_list_items = artist_name_list.find_all('a')
+
+ for artist_name in artist_name_list_items:
+ names = artist_name.contents[0]
+ links = 'https://web.archive.org' + artist_name.get('href')
+
+ f.writerow([names, links])
+Since this program is doing a bit of work, it will take a little while to create the CSV file. Once it is done, the output will be complete, showing the artists’ names and their associated links from Zabaglia, Niccola to Zykmund, Václav.
When scraping web pages, it is important to remain considerate of the servers you are grabbing information from.
Check to see if a site has terms of service or terms of use that pertains to web scraping. Also, check to see if a site has an API that allows you to grab data before scraping it yourself.
Be sure to not continuously hit servers to gather data. Once you have collected what you need from a site, run scripts that will go over the data locally rather than burden someone else’s servers.
Additionally, it is a good idea to scrape with a header that has your name and email so that a website can identify you and follow up if they have any questions. An example of a header you can use with the Python Requests library is as follows:
import requests
+
+headers = {
+ 'User-Agent': 'Your Name, example.com',
+ 'From': 'email@example.com'
+}
+
+url = 'https://example.com'
+
+page = requests.get(url, headers = headers)
+Using headers with identifiable information ensures that the people who go over a server’s logs can reach out to you.
This tutorial went through using Python and Beautiful Soup to scrape data from a website. We stored the text that we gathered within a CSV file.
You can continue working on this project by collecting more data and making your CSV file more robust. For example, you may want to include the nationalities and years of each artist. You can also use what you have learned to scrape data from other websites.
+
![](\"https://images.brickset.com/sets/large/10251-1.jpg?201510121127\" "\\"10251-1:")
