12/19/2015

Python

www.Bigdatainpractice.com

Introduction to Python and Python Streaming

Introduction to Python
Python Data Types
Working with Files
Conditions, Loops etc
Data Structures
Python Streaming on Hadoop

www.Bigdatainpractice.com

12/19/2015

Python Features
1. Open Source, All purpose
Programming Language

10. Versions available for

most of the operating
systems.
9. Extensive standard
libraries and
modules (like sci.py
and num.py)

2. Developed by Guido
Van Rossum (first
released in 1991)
3. Guido wanted to bridge
the gap between C and
Shell

4. Rapid development
(Python is interpreted
Language therefore no
compilation)

8. Very high level dynamic

data types like lists and
dictionaries
7. Supports object oriented as well
as procedural code.
6. A lot of heavy lifting (e.g. working with
hadoop) can be done very easily using
Python

5. Very handy for professionals who dont

have java or c++ or c skills
www.Bigdatainpractice.com

Comparison of Programming languages (popularity)

www.Bigdatainpractice.com

12/19/2015

Working with Python

Working with Python
1. Working with Python Shell:
2. Env Setup add path
3. Python Scripting:
For running python script python hello.py
Blocks.. Blocks blocks no braces just indentation
4. Python Variables:
Created when assigned to
Can hold any type data
Variable name can be of any length and is case-sensitive
Type is given based on assignment (int, float, str)

5. Working with Files

With open

www.Bigdatainpractice.com

Working with Python

Working with Python
6. Decision controls and Loops

For loop (when no of iterations are known)

While loop (loop based on condition)

7. Data Structures: Lists, tuples

Lists are used to store multiple values (of similar type)

Tuples are used to store multiple values (of different type)

Indexing and Slicing is similar for Lists, Tuples and Strings

8. Data Structues: Dictionaries, Sets

9. Working with File System
10. Functions

www.Bigdatainpractice.com

12/19/2015

Python Streaming on Hadoop - mapReduce

******************mapper.py******************
import sys

for line in sys.stdin:

line = line.strip()
memtype, cat, year, month, day, qty, sales = line.split(",")
print '%s\t%s' % (cat, sales)

*********************************************
******************reducer.py******************
Look at script
*********************************************
hadoop jar <streaming.jar> -file /user/cloudera/mapper.py -file
/user/cloudera/reducer.py -mapper /user/cloudera/mapper.py -reducer
/user/cloudera/reducer.py -input /user/cloudera/INPUT1/SalesData.csv output /user/cloudera/OUT_PY

www.Bigdatainpractice.com

Python Streaming on Hadoop - PIG

******************PIG SCRIPT******************
define streampy `/usr/bin/python pigPython.py`
input (stdin using PigStreaming(','))
output (stdout using PigStreaming(','))
ship ('pigPython.py');
salesData = LOAD '/user/cloudera/INPUT1/SalesData.csv' USING
PigStorage(',') AS (member_type:chararray, cat:chararray, year:int,
month:int, day:chararray, quantity:int, sales:float);
salesData2 = FILTER salesData BY cat == 'C1';
salesData3 = STREAM salesData2 THROUGH streampy as
(member_type:chararray, cat:chararray, year:int, month:int, day:chararray,
qty:int, sales:float);
DUMP salesData3;

www.Bigdatainpractice.com

12/19/2015

Python Streaming on Hadoop - PIG

******************pigPython.py******************
#!/usr/bin/python
import sys
name = 'NONE'
salary = 0.0
for line in sys.stdin:
line = line.strip()
member_type, cat, year, month, day, qty, sales = line.split(",")
member_type = 'MEMBER_' + member_type
cat = 'CAT_' + cat
print member_type + "," + cat + "," + year + "," + month + "," + day + "," +
qty + "," + sales

www.Bigdatainpractice.com

Python Streaming on Hadoop - HIVE

******************HIVE SCRIPT*****************
CREATE TABLE employee_python (
empid string,
name string,
assist string,
salary float,
country string,
state string)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '\t
add FILE hiveTran.py;
INSERT OVERWRITE TABLE employee_python
SELECT
TRANSFORM (empid,name,assist,salary,country,state)
USING 'python hiveTran.py'
AS (empid, name, assist, salary, country, state)
FROM employee;
www.Bigdatainpractice.com

12/19/2015

Python Streaming on Hadoop - HIVE

******************hiveTran.py******************
#!/usr/bin/env python
import sys
name = 'NONE'
salary = 0.0
for line in sys.stdin:
line = line.strip()
empid, name, assist, salary, country, state = line.split("\t")
name = str(name)
name = name.upper()
salary = float(salary)
salary = salary/1000
country = str(country)
country = country.upper()
state = str(state)
state = state.upper()
print '\t'.join([empid, name, assist, str(salary),country, state])
www.Bigdatainpractice.com

Thank You

www.Bigdatainpractice.com