Downloaded 31 times
Uploaded byPeter Skomoroch
O'Reilly Strata: Distilling Data Exhaust
This document discusses extracting insights from data exhaust, or unused data. It provides 10 lessons for doing so: choose a meaningful problem, find relevant data, raw data is better than processed, guide user input, solve easier problems first, create a quick baseline model, test on sample data, use continuous integration, pick the right tools, and prioritize developer productivity. As a case study, it analyzes skills data from attendees of the Strata conference to understand the audience and identify skills clusters. Visualization tools like Gephi are used to analyze similarities between attendees based on their skill vectors.
More Related Content
Similar to O'Reilly Strata: Distilling Data Exhaust
O'Reilly Strata: Distilling Data Exhaust
- 1. Distilling Data Exhaust How toSurface Insights & Build Data Products Feb 2, 2011 Peter Skomoroch LinkedIn @peteskomoroch
- 2. What is DataExhaust?
- 3. What is DataExhaust? My Delicious Tags
- 4. What is DataExhaust? Words I use on Twitter
- 5. What can youdo with it? â˘Data has value â˘Iâll share some lessons Iâve learned about how to extract that value â˘Weâll go through a case study
- 6. Part 1) 10Lessons Learned
- 7. 1) Choose ameaningful problem http://www.flickr.com/photos/aloshbennett/ â˘Find pain points â˘Work on stuff that matters â˘Look for underutilized data
- 8. 2) Find orcollect relevant data â˘DataWrangling â˘InfoChimps â˘Pete Warden â˘Factual, SimpleGeo â˘Mechanical Turk
- 9. 3) Raw isbetter than processed â˘Normalization could be incorrect â˘Data might be lost or corrupted â˘Good approach: public.resource.org http://www.flickr.com/photos/nedraggett/347280918/
- 10. 4) Guide userinput when you can â˘Auto suggest â˘Validate inputs â˘Collect tags, votes â˘Makes data scrubbing easier
- 11. 5) Solve easierproblems first http://where2conf.com/where2010/public/schedule/detail/12400
- 12. 6) Build abaseline model quickly â˘Iterate rapidly after baseline is done â˘Measure accuracy on hold out test set
- 13. 7) Test codeon sample data build logical sample data
- 14. 8) Use ContinuousIntegration
- 15. 8) Use ContinuousIntegration https://github.com/matthayes/azkaban
- 16. 9) Pick theright tool for the job
- 17. 10) Developer productivityis key â˘Fast Iterations: Python, Ruby, Pig â˘Convention over configuration â˘Embrace Github, DevOps, & EC2 â˘Currently using JRuby & Sinatra
- 18.
- 19. Part 2) CaseStudy: Strata
- 20. Conference Insights â˘Iâd liketo understand the audience at Strata â˘What companies do we work for? â˘What are the top skills at Strata? â˘Do attendees cluster together based on skill?
- 21. Round up aData Viz team
- 22. Use the righttools â˘Data Crunching: Hadoop, Pig â˘Statistical Work: Python, NumPy â˘Visualization: Gephi
- 23. Find Some Data:Attendees
- 24.
- 27. Extract Skills fromProfiles What are skills? Extract
- 28. Build Hadoop SkillGraph Discover Core Talent Graph for âHadoopâ Igor Perisic
- 29.
- 40. We can combineskills with the attendee directory to better understand Strata
- 41. What are skills@Strata?
- 42. Extract skills forattendees
- 43.
- 44.
- 45.
- 46.
- 47. Do attendees clustertogether based on skills?
- 48. â˘Compute similarity of attendeesbased on skill vector distance â˘Cluster similarities in Gephi
- 50. More analysis onthe way â˘DJ Patil has a session tomorrow â˘Weâll blog about additional Strata insights soon
- 51.
- 52.