-
Notifications
You must be signed in to change notification settings - Fork 2
Isaac Notes Class 1 3
**February 5 **
Congratulations, you have installed Jupyter Notebook! To run the notebook, run the following command at the Terminal (Mac/Linux) or Command Prompt (Windows): Commad to run :: jupyter notebook Then navigate to this http://localhost:8888/tree/site-packages/notebook http://localhost:8888/tree/site-packages Open Notebooks Dashboard https://jupyter.readthedocs.io/en/latest/running.html#running
We will be doing this course on a T2 micro Everyone uses AWS
Assignment 1) Using Powerpoint or Keynote is cool for creating diagrams
Shell connects your computers operating system. Bash is a born again shell. In the 70’s they called it the Born shell. Commands : pwd # present working directory
Usr - group - everyone Rw r r
Read write execute
Commands Cat / usr
Commands : clear Commands : mkdir #make directory Example : mkdir ssh # directory name ssh Control D ends a thread Commands : cd.. # sends you up a level Note :: the ~ means come Command : cd~ takes you home, users home, not root, home belongs to user, ~ Root or / belongs to computer Command : cd /
Command : CHMOD changes the permissions Command : Touch # makes a new file Touch foo.txt // if the file already exists , if it does exist it just changes latest access Command: less # lets you look at files better than cat command Less foo.txt Q to escape Hit SPACEBAR to go to next page Command cat #lets you look at files and concatenate and print the text to the shell Command: vim foo.txt
Hit i to enter insert mode
Hit escape to enter command moder
Hit :wq to command write quit
Command: rm # removes Command: ls -la # shows hidden file Command: mv foo.txt bob.txt # renames files Command mv foo.txt .foo.txt # Dot files are hidden file and show up when ls-la Note : some people call their config files Commands: x | less # piping x output to less
For example ls -la outputs a lot Pipe that info into less Ls-la | less Look for things starting from the home or ~ Command : ls ~/.ssh # using absolute references instead of relative regerence Command : ssh-keygen # creates rsa key pair Enter passphrase ( dont fill it in unless you want to do it)
Next step set up a new amazon server Ssh connect to new server Step 3 configure jupiter to next server
Step 1 choose oregon ( or any location but be consistent.,, use ec2 ( elastic cloud compute) bread and butter Step 2) Click services → Click EC2 → Launch instance → pick AMI Ubuntu #EVEN numbers are stable ones Step 3) Click Key pairs // setting up key pairs i) import key pairs Give it a name like Jan_2018 Nate_MacPro17 #input in contents Run in terminal Command cat~/.ssh/id_rsa.pub Copy paste into terminal the RSA KEY ii) set up a security group, under network & security Type SSh // ssh always set up to port 22 // 443 https 80 is http These are all protocals, rules for connecting two computers over the internet
Type PORT RANGE sOURCE
Custom TCP PROTOCAL 8888 Anywhere
Custom 2376 // docker hub
Custom tcp 27016 // Mongo db
( not the 27017 which is default mongo with no . security)
Source for all of these is anywhere
You can give it a security group name ( does not matter)
Description : ssh, jupyter, docker, mongo
Then hit create.
# Source where you can connect from
If you click on it
-- Go back to the EC2 dash board and start up an instance
Step 1) (i)choose an amazon machine image // an image is like a cd, frozen in place not moving, static, // these images are like full computer operating systems all ready to go (ii) Select Ubuntu ( usually # 5) // very popular (iii) we will use a t2 micro ( only 2 gigs of memory) (iv) click configure instance details → chose 1 for number of instance (v) add storage 30 gigabits free // 10 cents a gigabyte a month (vi) add security group and pick the one we just made, (vii) click tab 7 the review, get a yellow warning saying we are opening it to the internet (viii) launch, → select an existing key pair. // use the one we just added (ix) click view instances or go to ec2 dashboard and look at running instances → click pencil and give it a name → copy the public IP address IPv4
Go back to terminal // creating a secure shell or ssh Ssh ubuntu@(paste of public ip address) # the user is named ubuntu, # we connect to our computer using born again shell # we connect to amazon using a born again shell. After you hit enter your should see in terminal ubuntu@ip….
//* we are going to use a program called curl. Curl downloads files from a web address. We are going to give it some flags */ Command : curl -sSL https://get.docker.com # good practice Command : curl -sSL https://get.docker.com | sh # bad practice // the |sh lets us pipe in a random script into our shell directly. This one lets docker take over the aws instance. Normally you would download and analyse this.
Next command
Command : Sudo usermod -aG docker ubuntu // if we dont do this, the everytime we run docker we will have to do SUDU or SUDO COMMAND : cntrl D disconnects Command : docker -v (? I think) # this is supposed to recconect us
Command : docker pull jupyter/datascience-notebook Command : docker run -v /home/ubuntu:/home/jovyan -p 8888:8888 -d jupyter/datascience-notebook #Jovyan is the user in the jupyter image #datascience-notebook is the name of the image docker run -v /home/ubuntu:/home/jovyan -p 8888:8888 -d jupyter/datascience-notebook
// jupyter security portion
Command : docker exec 304e
#304e is the first 4 chars he got back in his string. Our string and every individual string will be different, we will get an http::/localhost… jovyan. We copy from http up until the to the ::
Paste this in html
-------
// facebook is preocess intensive, we are data scientists and are memory intensive and are cool with nested abstraction
CLASS 2
Review of HW cmdchallenge
Grep grep "GET" access.log
NOTE on regex rules. Period can mean anything so escape it “.” because “.” will not work. Period means any character
Go to bach and type Vim Tutor into terminal
Go to hackerrank and go to algorithm Other good things include math, sql, python, regex, post classdata structures and functional programming
The way you put code into slack is input 3 backticks: \
In slack Go into preferences and go to advanced and
We have code in our slack canvas to put into our hacker rank
Data = var name Or Data <- var name Both are the same but stack overflow saya <- is better for R best practises
A style is called snake case , for variables using underscorres string_to_vectors A style called Camel Case , for classes , using capitals BobbyClassHere Click box that says : when typing code with ’’’ enter should not send the message
\\\
input_to_data <- function () { con <- file("stdin") data <- (readLines(con, warn = FALSE) ) close(con) return(data) }
string_to_vector <- function (x) { return(as.numeric(unlist(strsplit(x, split=" ")))) }
data <- input_to_data()
n <- strtoi(data[1]) numbers <- string_to_vector(data[2])
cat(sum(numbers), sep="\n")
Set up machine in amazon the T2 2) Look at existing instances Click on descriptions → copy the ip curl -sSL https://get.docker.com | sh # curl lets you download from Ssl lets your secure hmtl link Sudo Usermod -aG docker Ubuntu Logout //* we are going to use a program called curl. Curl downloads files from a web address. We are going to give it some flags */ Command : curl -sSL https://get.docker.com # good practice Command : curl -sSL https://get.docker.com | sh # bad practice but do it here // the |sh lets us pipe in a random script into our shell directly. This one lets docker take over the aws instance. Normally you would download and analyse this.
Next command
sudo usermod -a -G docker ubuntu Command : sudo usermod -aG docker ubuntu // if we dont do this, the everytime we run docker we will have to do SUDU or SUDO COMMAND : cntrl D disconnects Command : docker -v (? I think) # this is supposed to recconect us docker
Command : docker pull jupyter/datascience-notebook Command : docker run -v /home/ubuntu:/home/jovyan -p 8888:8888 -d jupyter/datascience-notebook #Jovyan is the user in the jupyter image #datascience-notebook is the name of the image
Command docker pull jupyter/datascience-notebook Command : docker images # lets you check images // you could do : docker pull mongo
Command : docker tag jupyter/datascience-notebook dsnb # created an alias for dsnb
docker tag jupyter/datascience-notebook dsnb
docker run -v /home/ubuntu:/home/jovyan -p 80:8888 -d dsnb
// for ucla
Image is a class ( defines an object and what it will be ) The object is the container, the object exists in memory
Command : docker images # lets us see classes Command : docker ps # lets us see what is running, what are existing objects // docker defines images in reference to other images
Anything we have to get jupyter to do we ask docker to do for us in jupiter. Docker manages jupyter. We dont have to mange the versions. Other people do via jupyter comittee and mange the docker image for the jupiter project. Then to get the token do Command: docker ps Grab the container ID , which looks like b7ed Command docker exec b7ed jupyter notebook list
Sooooo now what can we do Lets put in the ip address of our amazon server including the port http://IPaddres:8888 http://35.163.88.54:8888
We will run docker ps Then we do docker exec command
If you get disconnected recconect Then to get the token do Command: docker ps Grab the container ID , which looks like b7ed Command docker exec b7ed jupyter notebook list
If you get disconnected recconect Then to get the token do Command: docker ps Grab the container ID , which looks like b7ed Command docker exec b7ed jupyter notebook list
Notes on Linear algebra
Y = mx+b
You can view f(x) → Y As a mapping of X to Y
Note : xx means a vector We can do a vector multiplication, very fast. ( yy <= 0.1*xx + 1)
The way you load a library in R is (repr) Options # lets us format how we want the plotting to look #plot is a command that lets us… well.. plot Plot ( xx, yy, ylim=c(-1,5), xlim=c() // ylim and xlim are just the limits size
#xx <- seq(-1, 2, by=0.1) #Create a sequence from -1 to 2 by 0.1 each time points(xx, yy2, col=”green”)
Vector arithmatic : vector addition and scalar multiplication
Vector Addition
Length nx1
(-1,2) + (3,5) = (2,7)
Scalar multiplication
To make a vector use c A <- c(-1,2) A <- c(3,5) C // if we print we would get 2 7
vecplot(b,a, col=”green”) // lets you plot b, starting at where it intersects with a
Let us do a linear combo ( add tip to tail f <- c(-1,2) h <- c(3,5) G = 2f + 3g
Resources for hacker rank hw in canvas
Notes for Data science class 3
Note Images are like a class and cointainers are like an object Images are like classes and define things The container is running and doing things
If we have a bad container we can do Docker stop b7ed // b7ed is the container name Docker rm b7ed
Or joint Docker stop b53c && rm b53c
Occasionally you may want to do docker prune but you will regain almost no space since you are basically just removing the link from aws to docker
Input : docker run -v /home/ubuntu:/home/jovyan -p 80:8888 -d dsnb // this creates the link between our docker home and our amazon ubuntu home // the -p links it to the ports, 80 is the amazon port, the host system, how we link to the world // we link 88.88 in the docker containter // -d means rune in detached mode and the image is dsnb // if we dont run it in detached mode the jupyter log will print makingh it hard for us to print // this long string is the container Id, which is different from the security token. We dont really need it
Output: C50614f72627a924ff2efd90834258344d42ac2f70fcdfe40fb37d9c6278de5e // this is the name of the container
Input: docker exec C5061 jupyter notebook list // jupyter notebook list shoes us the jupyter list running on that server // docker exec 3 means execute the above command on the container we just ran // but we dont have access to that container. It is running on jupyter. We need to hit up docter Output: Currently running servers: http://localhost:8888/?token=89fdc42367a2b56074612528c3a3987d0b7bfc0a7218da07 :: /home/jovyan
Now type in your ip address from the amazin server name ipv4 So if it were 35.163.88.54 Put this in a browser http://35.163.88.54:80
Then when prompted put in the token listed right after token= So input: 89fdc42367a2b56074612528c3a3987d0b7bfc0a7218da07
Input : git clone include etc
Amazon is a virtual machine You could ssh into your own computer instead of amazon
Virtual private cloud - private ip is only availibe in private cloud. Any machine in that vpc can talk to any other machien in the vpc using the private ip
T series and C compute series M series memory, p series are gpu’s optimized for machine learning There is a python library allded Boto that lets you interface python with aws. There is a bot docker image or you could run a python script and run boto there instead of a full python image
Nvidia makes a tool called invidia for docker and you can run the google tensor flow for running machine learning Kubernides - a tool created by google for managing thousands of docker containers at once. Alternative to EC2 is Linode. Slightly cheaper, digital ocean, google cloud based service
Making a bigger machine is called vertical scaling ( szillard at ucla when asked 1 tb data set just spin up a 1tb instance) Horizontal scaling = have 50 small machines and distribute the work load ( things like spark)
Docker compose lets you create subnetworks in docker aka multi contained applications
You wouldnt use compose for production but it is great for data science adn collaborating with colleages. Just send them compose and you can set up the setup for the containers
Docker hub - github for docker // when we do pull docker pulls from docker hub
Docker is platform agnostic- linux, mac, windows
Docker is much lighter than a virtual machine // it is close to the kernel, a lot fewer processes
Jupyter is not an IDE= integrated development environment It is interactive programming, Browser based and supports 40 languages and saves as JSON Jupyter started as python but they added Julia etc.. E.g. right it, it works, turn into function, change params then put in file. The software development product is interactive You can have insert mode You can have command mode ( if you hit m changes to markdown If you hit y turns into code mode) You can run bash in jupyter all you have to do it is put a bang at the start E.g. ! ls-la (2)Cd or pwd or ls run natively in jupyter too (3) %% bash Eclipse ( others for example) Conventional software is imperative programming Imperative programming goes down list jumps then maybe jumps back (4) Which collumns did we remove and why ( also NA, Null, and multiple unique) data set did we remove from (5) 7 programming topics ( Data types , for loops etc) (6) anything on a past quiz