What do I think when I came back from Hackthon?

 

This Hackathon is held by Cornell,Columbia Data Science institute, sponsored by Accenture, Capital One Lab and balabalala company. If you asked me how do I felt after the hackathon, I would like to say, tired, while it worth.

It is in this hackthon I saw  Hilary Mason, Claudia Perlich and many other interesting people. It is in this hackathon I appreciate some wonderful idea, such as the lyartist (an app help lyric writer), xbox visulization, and the app to help you stay from dangerous area of a city.

Our team analyze 550 indeed job post that are hiring data scientist and we got some interesting result.

Here are the top skills that need for a data scientist. Want to do data science? start to learn python first.

python	270
r	268
sql	258
hadoop	229
java	174
processing	157
excel	138
c	112
matlab	99
c..	88

Another good news for newbie data scientist is that, company are not expected you know all about web developing, algorithm design, SQL and Non-SQL database management, hadoop family and scala. Those full stack data science is few.

Besides, the top 3 algorithm are Clustering, Sampling, Ridge Regression, Ada boosting. In a words, you do not need to know exact detail of complicated algorithm. Understanding and applying those simple algorithm are more important.

While what is the difference between the great and good data scientist?  Claudia Perlich gave me the answer: data intuition. She compare data scientist to detective, great detective always had assumption  then confirm or rejected their assumption by fact. Great data scientist must have sufficient understanding about human behavior and the industry.

Another idea that worth mention is there is no clean data, you had to embrace the randomness of the world. Good data is that truly reflect fact not the clean and well formatted ones.

Also I like those advice given by data scientist panel:

1. When you are young, do the job that you really like. Do not work for money.

2. Always keep writing.

3. Got some sleep after hackathon.

Download all pdfs you need by python.

Yesterday I want to download materials in pdfs from Andrew Moore. while I hate to click the link and save all pdf one by one. Thus I write a script to do that job for me.

Here are the idea: * Get url of all topic * Get url of each pdf * Download it

In []:

import requests
#from pattern import web
import re
import os
import BeautifulSoup
import urllib2
#!~/anaconda/bin/pip install wget
#if you did not install some package, install it!
import wget
#using wget to get pdf

def findlinksnot(url,spec):
    #Find the url of all topics list on the webpage:http://www.autonlab.org/tutorials/list.html
    url=url
    parent_dir=url[:url.rfind("/")]+"/"
    html = requests.get(url).text
    soup = BeautifulSoup.BeautifulSoup(html)
    links=[parent_dir+a['href'] for a in soup.findAll('a') if spec not in a['href']]
    return links

def findlink(url,spec):
    #Find all pdfs in page:http://www.autonlab.org/tutorials/infogain.html
    url=url
    parent_dir=url[:url.rfind("/")]+"/"
    html = requests.get(url).text
    soup = BeautifulSoup.BeautifulSoup(html)
    links=[parent_dir+a['href'] for a in soup.findAll('a') if spec  in a['href']]
    return links

def getpdf(links,newpath=os.getcwd()):
    name=wget.download(links,out=newpath)
    print "%s\n downloaded in %s\n",name, newpath

def main(url,newpath=os.getcwd()):
    #if newpath not set it will creat one
    if not os.path.exists(newpath): os.makedirs(newpath)
    topics=findlinksnot(url,'http')
    print topics
    for topic in topics:
        links=findlink(topic,'.pdf')
        for link in links:
            getpdf(link,newpath=newpath) 
            
#if __name__ == '__main__':
   #main()
main('http://www.autonlab.org/tutorials/list.html')

Following work: * Figure out how to write a script that can work by command line arguement * Learn more about scrapy and do the job more automatically.

In []: