Background: one of my friends asked me for a favour and let me write a programme on reading data from text files
& importing them to the specific table in the oracle DB for data analysis.
After a short time consideration, I decided to use pp to accelerate the importing job.
Hardware: My pc has two cores(3.0GHZ) and 2G memory (on XP).
python: py2.6(python 3.0 seems to have long way to walk),cx_oracle,pp and etc.
directory D:\testData
Under this directory, there are 2317 text files,each with thousands of records.
you can see these are stock exchange records.
The record format are as follows:
1999/03/12 3.24 3.33 3.09 3.12 55675300 329260320.000
1999/03/15 3.10 3.20 3.00 3.15 14449600 84195928.000
1999/03/16 3.16 3.61 3.10 3.51 33196100 207144592.000
1999/03/17 3.52 3.63 3.48 3.51 23754800 151684672.000
1999/03/18 3.53 3.61 3.39 3.45 13184000 83092400.000
1999/03/19 3.44 3.79 3.35 3.74 20101300 129390344.000
1999/03/22 3.77 3.84 3.67 3.75 19180800 127824472.000
===========
code area :
===========
# -*- coding: cp936 -*-
'''
Created on 2012-3-1
@Author:aim103@163.com
'''
from __future__ import division
import datetime
import decimal
import os
import cx_Oracle
import sys
import time
def insert(basename):
tnsString = 'username/password@servicename'
try:
connection = cx_Oracle.Connection(tnsString)
cursor = connection.cursor()
dataList = getSetsForEachTXT(basename)
cursor.prepare("INSERT INTO security_day VALUES(:1,:2,:3,:4,:5,:6,:7,:
")
cursor.executemany(None,dataList)
#commit
connection.commit()
print basename + '--has commit'
except cx_Oracle.DatabaseError, exc:
error, = exc.args
print >> sys.stderr, "Oracle-Error-Code:", error.code
print >> sys.stderr, "Oracle-Error-Message:", error.message
finally:
cursor.close()
connection.close()
#example,input :filename = "C:\\testData\\SH600200.TXT"
def getSetsForEachTXT(filename):
rowList = []
f = open(filename,'r')
fileId = os.path.basename(filename)
for one in f.readlines():
lineStr = one[0:-1].split('\t')
lineStr.append(fileId)
i = 0
myList = []
while i < len(lineStr):
if i == 0:
myList.append(datetime.datetime.strptime(str(lineStr
),"%Y/%m/%d"))
elif i in range(1,7):
myList.append(decimal.Decimal(lineStr))
else:
myList.append(fileId[0:-4])
i = i + 1
lineTuple = tuple(myList)
rowList.append(lineTuple)
f.close()
return rowList
#input "C:\\testData"
#output: C:\testData\SH600010.TXT
# C:\testData\SH600178.TXT
def getAllTxtByDir(dirname):
textList = os.listdir(dirname)
absList = []
for one in textList:
absList.append(dirname + os.sep + one)
del textList
return absList
def parallelImport(dirname):
print 'parallel is beginning'
startTime = time.time()
inputs = getAllTxtByDir(dirname)
print 'The number of data file is %s' % len(inputs)
#launch the parallel
import pp
jobsServer = pp.Server()
print 'Launch the PP'
print 'The number of PP workers is %s' % jobsServer.get_ncpus()
#pp job submit
jobs = [jobsServer.submit(insert,args=(input,),depfuncs=(getSetsForEachTXT,),
modules=('cx_Oracle','datetime','time','sys','decimal'))for input in inputs]
for job in jobs:
job()
job = ''
jobsServer.print_stats()
jobsServer.destroy()
print 'Parallel consumed %s seconds' % (time.time() - startTime)
print 'The number of data file is %s' % len(inputs)
if __name__ == "__main__":
dirname = "D:\\testData"
parallelImport(dirname)
================================
conventional coding:
================================
def normalImport(dirname):
print 'Import is beginning'
startTime = time.time()
txtList = getAllTxtByDir(dirname)
txtNumber = len(txtList)
i = 1
for one in txtList:
insert(one)
print 'progess: %s' % (str(i/txtNumber*100)+ '%')
i = i + 1
print 'time consumed %s seconds' % (time.time() - startTime)
print 'totally %s txt files' % txtNumber
----------------------
compare and contrast
----------------------
1.in parallel：
Job execution statistics:
job count | % of all jobs | job time sum | time per job | job server
2317 | 100.00 | 1166.4560 | 0.503434 | local
Time elapsed since server creation 583.673000097
Parallel consumed 584.125999928 seconds
The number of data file is 2317
2.conventional ：
time consumed 1001.68599987 seconds
totally 2317 txt files
======>584.125999928 seconds VS 1001.68599987 seconds
Conclusion: which one will you prefer?
=====================================================================
self-introduction:
I am jj (nickname),a Chinese boy.
I work in an insurance company as a software test engineer.
In addition to python,I like maven,hudson,selenium webdriver and etc.
Welcome to contact me, aim103@163.com.
To Vitalii , thank you for the contribution to parallel python!
I love pp!
Forums 
