Establishing why an object can't be pickled

I'm receiving an object, t, from an api of type Object. I am unable to pickle it, getting the error:

  File "p.py", line 55, in <module>
    pickle.dump(t, open('data.pkl', 'wb'))
  File "/usr/lib/python2.6/pickle.py", line 1362, in dump
    Pickler(file, protocol).dump(obj)
  File "/usr/lib/python2.6/pickle.py", line 224, in dump
    self.save(obj)
  File "/usr/lib/python2.6/pickle.py", line 313, in save
    (t.__name__, obj))
pickle.PicklingError: Can't pickle 'Object' object: <Object object at 0xb77b11a0>

When I do the following:

for i in dir(t): print(type(i))

I get only string objects:

<type 'str'>
<type 'str'>
<type 'str'>
...
<type 'str'>
<type 'str'>
<type 'str'>

How can I print the contents of my Object object in order to understand why it cant be pickled?

Its also possible that the object contains C pointers to QT objects, in which case it wouldn't make sense for me to pickle the object. But again I would like to see the internal structure of the object in order to establish this.

Difference between counting and totaling? [on hold]

For e.g we need to calculate the total marks of students in a class and take an average: so will I need to need to use a counter first to store the marks?

Finding count of distinct elements in DataFrame in each column

I am trying to find the count of distinct values in each column using Pandas. This is what I did.

import pandas as pd

df = pd.read_csv('train.csv')
# print(df)

a = pd.unique(df.values.ravel())
print(a)

It counts unique elements in the DataFrame irrespective of rows/columns, but I need to count for each column with output formatted as below.

policyID              0
statecode             0
county                0
eq_site_limit         0
hu_site_limit         454
fl_site_limit         647
fr_site_limit         0
tiv_2011              0
tiv_2012              0
eq_site_deductible    0
hu_site_deductible    0
fl_site_deductible    0
fr_site_deductible    0
point_latitude        0
point_longitude       0
line                  0
construction          0
point_granularity     0

What would be the most efficient way to do this, as this method will be applied to files which have size greater than 1.5GB?

---

Based upon the answers, df.apply(lambda x: len(x.unique())) is the fastest.

In[23]: %timeit df.apply(pd.Series.nunique)
1 loops, best of 3: 1.45 s per loop
In[24]: %timeit df.apply(lambda x: len(x.unique()))
1 loops, best of 3: 335 ms per loop
In[25]: %timeit df.T.apply(lambda x: x.nunique(), axis=1)
1 loops, best of 3: 1.45 s per loop

Algorithm equalivence from Matlab to Python

I've plotted a 3-d mesh in Matlab by below little m-file:

[x,n] = meshgrid(0:0.1:20, 1:1:100);

mu = 0;
sigma = sqrt(2)./n;

f = normcdf(x,mu,sigma);

mesh(x,n,f);

I am going to acquire the same result by utilization of Python and its corresponding modules, by below code snippet:

import numpy as np
from scipy.integrate import quad
import matplotlib.pyplot as plt

sigma = 1

def integrand(x, n):
    return (n/(2*sigma*np.sqrt(np.pi)))*np.exp(-(n**2*x**2)/(4*sigma**2))

tt = np.linspace(0, 20, 2000)
nn = np.linspace(1, 100, 100)  

T = np.zeros([len(tt), len(nn)])

for i,t in enumerate(tt):
    for j,n in enumerate(nn):
        T[i, j], _ = quad(integrand, -np.inf, t, args=(n,))

x, y = np.mgrid[0:20:0.01, 1:101:1]

plt.pcolormesh(x, y, T)

plt.show()

But the output of the Python is is considerably different with the Matlab one, and as a matter of fact is unacceptable. I am afraid of wrong utilization of the functions just like linespace, enumerate or mgrid...

Does somebody have any idea about?!...

PS. Unfortunately, I couldn't insert the output plots within this thread...!

Best

..............................

Edit: I changed the linespace and mgrid intervals and replaced plot_surface method... The output is 3d now with the suitable accuracy and smoothness...

Add values to a class in Python

Let's say I have the following in Python:

class Test():
    self.value1 = 1
    self.value2 = 2

def setvalue1(self, value):
    self.value1 = value

So one can set up value1 by doing:

Test.setvalue1('Hola')

or

Test.Value1 = 'Hola'

So far so good. My problem is I would like to set the values by reading them somewhere else so for instance I could have the following:

A = [['Value1','Hola'],['Value2','Adios']]

I would like to be able to run something that will do (in pseudo code):

for each in A:
    Test.each[0] = A[1]

Is this possible? Thanks so much!

Sorting todo.txt lines by its properties

I'm after a simple code to organize my todo.txt that has Gina Trapani's syntax, that is contexts are preceded by @, projects by +, priorities are marked by (A), (B) etc.. A task can have multiple contexts and projects.

What I would like to achieve is to first sort the lines by context and in the block of a contexts lines should be ordered by projects and lines with priorities comes first in the project.

My code until now:

import os
import sys
import re

# Configuration
todo_path = notepad.getCurrentFilename()

def ordered_set(inlist):
    out_list = []
    for val in inlist:
        if not val in out_list:
            out_list.append(val)
    return out_list

class Todo:
    def __init__(self, priority, context, project, due, task, cdate):
        self.__priority = priority
        self.__context = context
        self.__project = project
        self.__due = due
        self.__task = task
        self.__cdate = cdate

    def __len__(self):
     return len(str(re.sub(' +',' ',str(self.__priority) +' '+' '.join(self.__context) + ' ' + ' '.join(self.__project) + ' ' + str(self.__due) + ' ' + str(self.__task) + ' ' + str(self.__cdate) + '\n')))

    def priority(self):
        return self.__priority

    def context(self):
        return self.__context

    def project(self):
        return self.__project

    def due(self):
        return self.__due

    def task(self):
        return self.__task

    def cdate(self):
        return self.__cdate

def BuildTodos():
 global todos
 todo_file = open(todo_path, 'r')
 raw_todos = todo_file.readlines()
 todo_file.close()
 todos = []

 for item in raw_todos:
  item = item.strip("\n")
  todos.append(item)
 console.write("Loaded Todos\n")
 for idx, item in enumerate(todos):  
  words = item.split(' ')  
  priority = [word for word in words if re.match('^\([A-Z]\)',word)]
  context = [word for word in words if word.startswith('@')]  
  project = [word for word in words if word.startswith('+')]  
  due = [word for word in words if word.startswith('due:')]  
  task = [word for word in words if not re.match('^\([A-Z]\)',word) and not word.startswith('@') and not word.startswith('+') and not word.startswith('due:') and not re.match('[0-9]{4}-[0-9]{2}-[0-9]{2}',word)]  
  cdate = [word for word in words if re.match('[0-9]{4}-[0-9]{2}-[0-9]{2}',word)]
  todos[idx] = Todo(priority, context, project, due, task, cdate)
 console.write("Built Todos\n")
 todos.sort(key=lambda t: t.context())
# ----------------
# HELP NEEDED HERE
# sort the lines by context and within the block of contexts lines  should be
# ordered by projects and lines with priorities comes first in the project.
# ---------------- 

def OutTodos():
 for t in todos:
    console.write(re.sub(' +',' ',' '.join(t.priority()) + ' ' + ' '.join(t.context()) + ' ' + ' '.join(t.project()) + ' ' + ' '.join(t.due()) + ' ' + ' '.join(t.task()) + ' ' + ' '.join(t.cdate()) + '\n'))

console.clear()
BuildTodos()
OutTodos()

Example todo.txt file, contains utf-8 characters (!):

(A) @personal +study +python organize todo.txt áőúíéá
(A) Schedule annual checkup +Health áőúíéá
(B) Outline chapter 5 +Novel @Computer áőúíéá
(C) Add cover sheets @Office +TPSReports áőúíéá
Plan backyard herb garden @Home áőúíéá
Pick up milk @GroceryStore áőúíéá
Research self-publishing services +Novel @Computer áőúíéá
Download Todo.txt mobile app @Phone áőúíéá

I'm squeezing my mind on how to construct this sorting so to not end up with a monster. My guess would be to iterate on the todos list and have cascading ifs but not having any experience in sorting/list manipulations in python I'm out for advices.

Having issues with a corrupt encryption error

Im trying to encrypt every line of the file test.txt the function encrypt() will output basically the text all jumbled up. For an end of year project in my computer science class. But my problem is when i try to run the code "backwards" with decrypt_all it doesnt work and the file is still corrupted

from Crypto.Cipher import XOR
import base64
import os

def encrypt(key=None, plaintext=None):
    if key == None:
        key = "This_is_my_hidden_key"
    cipher = XOR.new(key)
    return base64.b64encode(cipher.encrypt(plaintext))
def decrypt(key=None, ciphertext=None):
    if key == None:
            key = "This_is_my_hidden_key"
    cipher = XOR.new(key)
    return cipher.decrypt(base64.b64decode(ciphertext))


#####Run below to encrypt all files in folder and each sub folder#######

def encrypt_all(UselessVariable = None):
    root = os.getcwd()
    path = os.path.join(root, "targetdirectory")
    x=0
    for path, subdirs, files in os.walk(root):
        for name in files:
            openfile = os.path.join(path, name)
            print openfile
            try:
                with open(openfile, 'r+') as a:
                    encrypted_text = []
                    for line in a:
                        encrypted_text.append(decrypt(None, line))
                    open(openfile,"w").close()
                    for text in range(len(encrypted_text)):
                        a.write((str(encrypted_text[text]))+ '\n')
            except IOError as e:
                print 'Operation failed: %s' % e.strerror
            x+=1
        print ""
    print x



#####Run below to decrypt all files in folder and each sub folder#######

def decrypt_all(UselessVariable = None):
    root = os.getcwd()
    path = os.path.join(root, "targetdirectory")
    x=0
    for path, subdirs, files in os.walk(root):
        for name in files:
            openfile = os.path.join(path, name)
            print openfile
            try:
                with open(openfile, 'r+') as a:
                    encrypted_text = []
                    for line in a:
                        encrypted_text.append(decrypt(None, line))
                    open(openfile,"w").close()
                    for text in range(len(encrypted_text)):
                        a.write((str(encrypted_text[text])))
            except IOError as e:
                print 'Operation failed: %s' % e.strerror
            x+=1
        print ""
    print x

Active Shape Models: matching model points to target points

I have a question regarding Active Shape Models. I am using the paper of T. Coots (which can be found here.)

I have done all of the initial steps (Procrustes Analysis to calculate mean shape, PCA to reduce dimensions) but am stuck on fitting.

This is the situation I am in now: I have calculated the mean shape with points X and have also calculated a new set of points Y that X should move to, to better fit my image.

I am using the following algorithm, which can be found on page 23 of the paper previously linked:

---

---

To clarify: is the mean shape calculated with Procrustes Analysis, and the is the matrix containing the eigenvectors calculated with PCA.

Everything goes well up to step 4. I can calculate the pose parameters and invert the transformation onto the points Y.

However, in stap 5, something strange happens. Whatever the pose parameters are calculated in stap 3 and applied in stap 4, stap 5 always results in almost exactly the same vector y' with very low values (one of them being 1.17747114e-05 for example). (So whether i calculated a scale of 1/10 or 1/1000, y' barely changes).

This results in the algorithm always converging to the same value of b, and thus in the same output shape x, no matter what the input set of target points Y are that I want the model points X to match with.

This sure is not the goal of the algorithm... Could anyone explain this strange behaviour? Somehow, projecting my calculated vector y in step 4 into the "tangent plane" does not take into account any of the changes made in step 4.

---

Edit: I have some more reasoning, though no explanation or solution. If, in step 5, i manually set y' to consist only of zeros, then in step 6, b is equal to the matrix of eigenvectors multiplicated by our meanshape. And this results in the same b I always get (since y' is always a vector with very low values).

But these eigenvectors are calculated from the meanshape using PCA... So what's expected, is that no change should take place, I think?

---

Can re.findall() return only the part of the regex in parens?

Looping through some data, I want to capture string of numbers that appear as page IDs (with more than one per line.) However, I only want to match number strings as part of a particular URL, but I DON'T want to record the URL, just the number.

I am currently using re.findall to identify the right URLs, and then re.sub to extract the number strings.

views = re.findall(r"/view/\d*?.htm", line)
for view in views:
    view = re.sub(r"/view/(\d+).htm", r"\1", view)
    pagelist.append(view)

Is there a way to do something like

views = re.findall(r"/view/(\d*?).htm", r"\1", line)   #I know this doesn't work

where the original findall() only returns the part of the match in parens?

how to install Lasagne package with python con windows

I'm new on python and I'm running some script on python 3.4. I'm getting the following error: ImportError: No module named 'lasagne'. Does someone know how to install this package on Python please?

Using postgres thru ODBC in python 2.7

• I have installed Postgres.app and started it.
• I have pip installed pypyodbc
• I have copied the hello world lines from the Pypyodbc docs, and received the error below. any ideas what the issue might be?

Here is my code

  from __future__ import print_function
  import pypyodbc
  import datetime
  conn = pypyodbc.connect("DRIVER={psqlOBDC};SERVER=localhost") 

And I receive this error:

File "/ob/pkg/python/dan27/lib/python2.7/site-packages/pypyodbc.py", line 975, in ctrl_err
  err_list.append((from_buffer_u(state), from_buffer_u(Message), NativeError.value))
File "/ob/pkg/python/dan27/lib/python2.7/site-packages/pypyodbc.py", line 482, in UCS_dec
  uchar = buffer.raw[i:i + ucs_length].decode(odbc_decoding)
File "/ob/pkg/python/dan27/lib/python2.7/encodings/utf_32.py", line 11, in decode
  return codecs.utf_32_decode(input, errors, True)
UnicodeDecodeError: 'utf32' codec can't decode bytes in position 0-1:   truncated data

what am I doing wrong?

Do I need to somehow initialize the DB / tables first? it is a weird error if that is the issue.

Different behavior of same regular expression in python and java

Firstly, my apologies as I don't know regular expressions that well.

I am using a regular expression to match a string. I tested it on python command line interface but when I ran it in Java, it produced a different result.

Python execution:

re.search("[0-9]*[\\.[0-9]+]?[^0-9]*D\\([M|W]\\)\\s*US", "9.5 D(M) US");

gives the result as:

<_sre.SRE_Match object; span=(0, 11), match='9.5 D(M) US'>

But the Java code

import java.util.Set;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

class RegexTest {
    private static final Pattern FALLBACK_MEN_SIZE_PATTERN = Pattern.compile("[0-9]*[\\.[0-9]+]?[^0-9]*D\\([M|W]\\)\\s*US");

    public static void main(String[] args) {
    String strTest = "9.5 D(M) US";
    Matcher matcher = FALLBACK_MEN_SIZE_PATTERN.matcher(strTest);
        if (matcher.find()) {
            System.out.println(matcher.group(0));
        }
    }
}

gives the output as:

5 D(M) US

I don't understand why it is behaving the different way.

Python time comparison at midnight

I have to save the time in AM PM format.

But i am having trouble in deciding how to enter midnight time.

Suppose the time is 9PM to 6AM next morning. I have to divide it into day to day basis . Like this

t1 = datetime.datetime.strptime('09:00PM', '%I:%M%p').time()

t2 = datetime.datetime.strptime('12:00AM', '%I:%M%p').time()

t3 = datetime.datetime.strptime('06:00AM', '%I:%M%p').time()

Now i want to know whether the t2 should be

12:00 AM or 11.59PM

If i do 12:00AM then i can't compare if 9pm > 12am but 11.59 looks odd or may be it is right way

PHP's array_slice vs Python's splitting arrays

Some background

I was having a go at the common "MaxProfit" programming challenge. It basically goes like this:

Given a zero-indexed array A consisting of N integers containing daily prices of a stock share for a period of N consecutive days, returns the maximum possible profit from one transaction during this period.

I was quite pleased with this PHP algorithm I came up, having avoided the naive brute-force attempt:

public function maxProfit($prices)
{
    $maxProfit = 0;
    $key = 0;
    $n = count($prices);

    while ($key < $n - 1) {
        $buyPrice = $prices[$key];
        $maxFuturePrice = max( array_slice($prices, $key+1) );          
        $profit = $maxFuturePrice - $buyPrice;

        if ($profit > $maxProfit) $maxProfit = $profit;
        $key++;
    }
    return $maxProfit;
}

However, having tested my solution it seems to perform badly performance-wise, perhaps even in O(n²) time.

I did a bit of reading around the subject and discovered a very similar python solution. Python has some quite handy array abilities which allow splitting an array with a a[s : e] syntax, unlike in PHP where I used the array_slice function. I decided this must be the bottleneck so I did some tests:

Tests

PHP array_slice()

$n = 10000;    
$a = range(0,$n);

$start = microtime(1);
foreach ($a as $key => $elem) {
    $subArray = array_slice($a, $key);
}
$end = microtime(1);

echo sprintf("Time taken: %sms", round(1000 * ($end - $start), 4)) . PHP_EOL;

Results:

$ php phpSlice.php
Time taken: 4473.9199ms
Time taken: 4474.633ms
Time taken: 4499.434ms

Python a[s : e]

import time

n = 10000
a = range(0, n)

start = time.time()
for key, elem in enumerate(a):
    subArray = a[key : ]
end = time.time()

print "Time taken: {0}ms".format(round(1000 * (end - start), 4))

Results:

$ python pySlice.py 
Time taken: 213.202ms
Time taken: 212.198ms
Time taken: 215.7381ms
Time taken: 213.8121ms

Question

1. Why is PHP's array_slice() around 20x less efficient than Python?
2. Is there an equivalently efficient method in PHP that achieves the above and thus hopefully makes my maxProfit algorithm run in O(N) time?

Showing sprite wihout a group

I have two sprites in a class for esier control (specifickly: tank turret and suspension). If I try to launch program it works wihout any errors, but it dosen't show anything. I also tried to put both of spirtes in group in class, bu it throved errorTypeError: draw() missing 1 required positional argument: 'surface'. How I should do the displaying of my tank wihout disambling of my group?

Reading mulitple data from a text file

I am trying to read two pieces of data from a single text file. Here is how the file looks:

PaxHeader/data-science000755 777777 777777 00000000262 12525446741 015207 xustar00armourp000000 000000 18 gid=1050026054
17 uid=488147323
20 ctime=1431779590
20 atime=1431779720
38 LIBARCHIVE.creationtime=1431719347
23 SCHILY.dev=16777218
24 SCHILY.ino=110226037
18 SCHILY.nlink=4
data-science/000755 Äâ{Ä>ñ F00000000000 12525446741 013547 5ustar00armourp000000 000000 data-science/PaxHeader/merged-sensor-files.csv000644 777777 777777 00000000214 12525446724 021646 xustar00armourp000000 000000 18 gid=1050026054
17 uid=488147323
20 ctime=1431779590
20 atime=1431779720
23 SCHILY.dev=16777218
24 SCHILY.ino=110226038
18 SCHILY.nlink=1
data-science/merged-sensor-files.csv000644 Äâ{Ä>ñ F00016452751 12525446724 020164 0ustar00armourp000000 000000 MTU, Time, Power, Cost, Voltage
MTU1,05/11/2015 19:59:06,4.102,0.62,122.4
MTU1,05/11/2015 19:59:05,4.089,0.62,122.3
MTU1,05/11/2015 19:59:04,4.089,0.62,122.3
MTU1,05/11/2015 19:59:06,4.089,0.62,122.3
MTU1,05/11/2015 19:59:04,4.097,0.62,122.4
MTU1,05/11/2015 19:59:03,4.097,0.62,122.4
MTU1,05/11/2015 19:59:02,4.111,0.62,122.5
MTU1,05/11/2015 19:59:03,4.111,0.62,122.5
MTU1,05/11/2015 19:59:02,4.104,0.62,122.5
MTU1,05/11/2015 19:59:01,4.090,0.62,122.4
MTU1,05/11/2015 19:59:00,4.093,0.62,122.4
MTU1,05/11/2015 19:58:59,4.112,0.62,122.5
data-science/PaxHeader/weather.json000644 777777 777777 00000000214 12525446741 017610 xustar00armourp000000 000000 18 gid=1050026054
17 uid=488147323
20 ctime=1431779590
20 atime=1431779720
23 SCHILY.dev=16777218
24 SCHILY.ino=110226039
18 SCHILY.nlink=1
data-science/weather.json000644 Äâ{Ä>ñ F00000000766 12525446741 016112 0ustar00armourp000000 000000 {"1431388800":"75.4","1431392400":"73.2","1431396000":"72.1","1431399600":"71.0", "1431403200":"70.7","1431406800":"69.6","1431410400":"69.0","1431414000":"68.8","1431417600":"69.2","1431421200":"67.9","1431424800":"68.6","1431428400":"68.7","1431432000":"72.1","1431435600":"76.2","1431439200":"80.1","1431442800":"80.7","1431446400":"80.9","1431450000":"83.3","1431453600":"84.5","1431457200":"85.1","1431460800":"87.0","1431464400":"84.2","1431468000":"84.4","1431471600":"83.0","1431475200":"81.1"}

So basically I want to get the values like below

MTU, Time, Power, Cost, Voltage
    MTU1,05/11/2015 19:59:06,4.102,0.62,122.4

as separate pandas frame and then another frame for the below dictionary.

{"1431388800":"75.4","1431392400":"73.2","1431396000":"72.1","1431399600":"71.0", "1431403200":"70.7","1431406800":"69.6","1431410400":"69.0","1431414000":"68.8","1431417600":"69.2","1431421200":"67.9","1431424800":"68.6","1431428400":"68.7","1431432000":"72.1","1431435600":"76.2","1431439200":"80.1","1431442800":"80.7","1431446400":"80.9","1431450000":"83.3","1431453600":"84.5","1431457200":"85.1","1431460800":"87.0","1431464400":"84.2","1431468000":"84.4","1431471600":"83.0","1431475200":"81.1"}

I can manually cut and copy paste these two portions in separate files and read in, but I want to automate it using regex. I think I know how we can regex it, but while reading the whole file as a text, I am seeing the following values.

So I did this:

f=open("file",'r').read()
print(f)

'PaxHeader/data-science\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00000755 \x00777777 \x00777777 \x0000000000262 12

These are the first few lines of file. Not sure why I see \x00 a lot. Is it becauuse of some space or some non -recognised character?

Any idea how to get the desired result?

Thanks

Find duplicate element in a list of lists [on hold]

I am looking for Python ideas for the following problem.

Given a list of lists...

[[20, 21, 22], [17, 18, 19, 20], [10, 11, 12, 13]]

If there is a duplicate element that is common between any or all the lists, return True. If all of the elements are unique, return False.

In the example above, 20 is common and would return True. The example below would return False because all the numbers are unique between the lists.

[[20, 21, 22], [17, 18, 19], [10, 11, 12, 13]]

Lastly, testing for duplicates in an individual list is not needed because the numbers are always sequential.

FYI - this problem will be used to optimize an airline crew members monthly schedule. Each list represents a 3, 4, or 5 day airline trips that can't overlap.

BTW - this problem is not an assignment but a personal quest to work less and get paid more :) Sorry it was unclear. I tried a brute force method which works but was hoping for a more elegant Pythonic method. I appreciate all the responses as they are leading me into new areas of Python programming.

How to explicitly control task schedule of Spark exactly?

I tried to achieve a parallelized image processing technique using Spark. Different from conventional Spark work with millions of tasks. I only want to separate the image into the number of worker (machine) I have and let one worker process one image patch. So one image patch is one task, if I have 12 image patches, I have 12 tasks. The question is how to explicitly control the schedule of task to each worker. The current situation happens that if I parallelize the image patches, they often send several patches to one or two worker and leave the others not working. I tried to set the system property of spark to control the spark.cores.max and spark.default.parallelism. But it seems not helpful. The only way to make the task send to different workers as separate as possible is to enlarge the second parameter of SparkContext.parallelize - numSlices. Here is the code:

img = misc.imread('test_.bmp')
height, width = img.shape
divisions, patch_width, patch_height = partitionParameters(width, height, 2, 2, border=100)

spark = SparkContext(appName="Miner")
# spark.setSystemProperty('spark.cores.max','1')
spark.setSystemProperty('spark.default.parallelism','24')

broadcast_img = spark.broadcast(img)

start = datetime.now()
print "--------------------", divisions
# run spark
run = spark.parallelize(divisions, 24).cache()
print "--------------- RDD size: ", run._jrdd.splits().size()
result = run.map(lambda (x, y): crop_sub_img(broadcast_img.value, x, y, patch_width, patch_height, width, height)) \
                .map(lambda ((x, y), subimg): fastSeg.process(subimg, x, y)) \
                .collect()

img = cat_sub_img(result, width, height)
end = datetime.now()

print "time cost:", (end-start) 

As you can see, I only have four patches set in divisions. divisions is a list of tuple with x and y-axis of the image patch. Only I set the numSlices to a high value 24 which far exceeds the actual tasks I have in divisions, most of workers are used now. But it seems not reasonable. If I set to 4, it will sent all tasks to only one worker! There must be someway to control how many task one worker accept. I am not familiar with the core of Spark. Can anyone help me, Thanks?

One thought it happens is that the image size is too small for one worker. So spark will assume one worker could handle that and send all to one.

Override a package method

If I import a package, let's say networkx.

How to override one method inside it so that it's the one called by every other function inside the package ?

Example :

import networkx

def _draw_networkx_nodes(G, pos,
                        nodelist=None,
                        node_size=300,
                        node_color='r',
                        node_shape='o',
                        alpha=1.0,
                        cmap=None,
                        vmin=None,
                        vmax=None,
                        ax=None,
                        linewidths=None,
                        label=None,
                        **kwds):
     print 'OK'

nx.draw_networkx_nodes = _draw_networkx_nodes

nx.draw(G, pos)

I want the method draw to call other methods that will call my overriden function

Python - Pyramid and matplotlib - Cannot Have More Than One View Output A SVG?

I am developing a Python Pyramid application where I am intending to create more than one SVG image to chart statistics using pie charts. In my testing I find that one SVG view works correctly and as soon as I add a second SVG output view, and a second SVG image is loaded (order of SVG image load doesn't matter), whether directly through its view, or through another view that references this view, the SVG images "are combined" in any other further calls to load a SVG file. This appears to be a bug somewhere in the Python stack as it appears memory is not cleared properly (primarily in the case of more than one SVG file, see further details below). Also note below that after enough image/page loads a TclError is encountered.

Since I was using SVG in a more detailed application with many more views, I am reproducing this in a minimized/reduced application to show it isn't something extra I'm doing and this code is generated right from the Pyramid alchemy template and database calls are not involved. The database is actively utilized in my more details application. This application only has 3 views, where the first view is part of the original template. I am also adding DEBUG logging to make it clear that there is no indication that there is any internal calling of the other SVG view.

Some of the view code is based on Matplotlib svg as string and not a file primarily for the use of StringIO. Note that as a pie chart is needed, that is the primary reason why my code differs from the code in referenced question. I find the issue is essentially the same whether I use StringIO or cStringIO. In my code I am using cStringIO.

The full application code is available at: http://ift.tt/1HAeOYJ

Code From First SVG View:

@view_config(route_name='view_test_svg')
def test_svg_view(request):
    # Full module import is not allowed by Pyramid
    #from pylab import *
    # Do individual required imports instead
    from pylab import figure, axes, pie, title, savefig
    log.debug('In test_svg_view')
    figure(1, figsize=(6,6))
    ax = axes([0.1, 0.1, 0.8, 0.8])
    labels = ['Frogs', 'Hogs', 'Dogs', 'Logs']
    fracs = [15, 30, 45, 10]
    explode=(0, 0.05, 0, 0)
    pie(fracs, explode=explode, labels=labels,
                                autopct='%1.1f%%', shadow=True, startangle=90)
    title('Raining Hogs and Dogs', bbox={'facecolor':'0.8', 'pad':5})
    imgdata = cStringIO.StringIO()
    savefig(imgdata, format='svg')
    imgdata.seek(0)
    svg_dta = imgdata.getvalue()
    # Close the StringIO buffer
    imgdata.close()
    return Response(svg_dta, content_type='image/svg+xml')

Python Version: Python 2.7.5

Python Package Configuration (Primary Packages Only)

• pyramid-1.6a1-py2.7
• matplotlib-1.4.3-py2.7-win32

Steps Taken To Reproduce:

1. pserve pyramidapp.

Command: pserve development.ini --reload

Starting server in PID 4912.
serving on http://0.0.0.0:6543

2. Load http://localhost:6543/test.svg

Note this works properly

DEBUG [pyramidapp.views:22][Dummy-2] In test_svg_view

3. Load http://localhost:6543/test2.svg

Note this "combines" both SVG files together

DEBUG [pyramidapp.views:45][Dummy-3] In test2_svg_view

4. Load http://localhost:6543/test.svg

Note this works exactly like test2.svg, with the correct title, since they are also of similar length, and now images are combined in this view as well

DEBUG [pyramidapp.views:22][Dummy-4] In test_svg_view

5. Rehost application and only load http://localhost:6543/test2.svg

Note this works properly for first load as this view was loaded before test.svg this time

DEBUG [pyramidapp.views:45][Dummy-2] In test2_svg_view

Tracelog when using Control+C to terminate the pserve process

Error in sys.exitfunc:
Traceback (most recent call last):
  File "--python_path--\lib\atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "--python_path--\lib\site-packages\matplotlib-1.4.3-py2.7-win32.egg\ma
tplotlib\_pylab_helpers.py", line 89, in destroy_all
    manager.destroy()
  File "--python_path--\lib\site-packages\matplotlib-1.4.3-py2.7-win32.egg\ma
tplotlib\backends\backend_tkagg.py", line 588, in destroy
    self.window.destroy()
  File "--python_path--\lib\lib-tk\Tkinter.py", line 1789, in destroy
    for c in self.children.values(): c.destroy()
  File "--python_path--\lib\lib-tk\Tkinter.py", line 2042, in destroy
    self.tk.call('destroy', self._w)
_tkinter.TclError: out of stack space (infinite loop?)
^C caught in monitor process

Important: After enough SVG image loads the following is encountered:

The only way to fix this currently is to restart pserve. Also note that views, such as the my_view load properly as long as SVG images are not referenced, or utilized, by such views.

Another important note, as long as only one SVG file, i.e. http://localhost:6543/test.svg, is loaded the entire time of pserve it seems that image can be reloaded/refreshed (potentially) infinite times without any apparent issue, or encountering of the following:

_tkinter.TclError
TclError: out of stack space (infinite loop?)
Traceback (most recent call last)
File "--python_path--\lib\site-packages\pyramid_debugtoolbar-2.0.2-py2.7.egg\pyramid_debugtoolbar\panels

\performance.py", line 69, in noresource_timer_handler
Display the sourcecode for this frameOpen an interactive python shell in this frameresult = handler(request)
File "--python_path--\lib\site-packages\pyramid-1.6a1-py2.7.egg\pyramid\tweens.py", line 20, in excview_tween
Display the sourcecode for this frameOpen an interactive python shell in this frameresponse = handler(request)
File "--python_path--\lib\site-packages\pyramid_tm-0.11-py2.7.egg\pyramid_tm\__init__.py", line 94, in tm_tween
Display the sourcecode for this frameOpen an interactive python shell in this framereraise(*exc_info)
File "--python_path--\lib\site-packages\pyramid_tm-0.11-py2.7.egg\pyramid_tm\__init__.py", line 75, in tm_tween
Display the sourcecode for this frameOpen an interactive python shell in this frameresponse = handler(request)
File "--python_path--\lib\site-packages\pyramid-1.6a1-py2.7.egg\pyramid\router.py", line 145, in handle_request
Display the sourcecode for this frameOpen an interactive python shell in this frameview_name
File "--python_path--\lib\site-packages\pyramid-1.6a1-py2.7.egg\pyramid\view.py", line 527, in _call_view
Display the sourcecode for this frameOpen an interactive python shell in this frameresponse = view_callable

(context, request)
File "--python_path--\lib\site-packages\pyramid-1.6a1-py2.7.egg\pyramid\config\views.py", line 384, in 

viewresult_to_response
Display the sourcecode for this frameOpen an interactive python shell in this frameresult = view(context, 

request)
File "--python_path--\lib\site-packages\pyramid-1.6a1-py2.7.egg\pyramid\config\views.py", line 506, in 

_requestonly_view
Display the sourcecode for this frameOpen an interactive python shell in this frameresponse = view(request)
File "c:\projects\python\pyramid\pyramidapp\pyramidapp\views.py", line 55, in test2_svg_view
Display the sourcecode for this frameOpen an interactive python shell in this framesavefig(imgdata, 

format='svg')
File "--python_path--\lib\site-packages\matplotlib-1.4.3-py2.7-win32.egg\matplotlib\pyplot.py", line 578, in 

savefig
Display the sourcecode for this frameOpen an interactive python shell in this framedraw()   # need this if 

'transparent=True' to reset colors
File "--python_path--\lib\site-packages\matplotlib-1.4.3-py2.7-win32.egg\matplotlib\pyplot.py", line 571, in 

draw
Display the sourcecode for this frameOpen an interactive python shell in this frameget_current_fig_manager

().canvas.draw()
File "--python_path--\lib\site-packages\matplotlib-1.4.3-py2.7-win32.egg\matplotlib\backends\backend_tkagg.py", 

line 350, in draw
Display the sourcecode for this frameOpen an interactive python shell in this frametkagg.blit(self._tkphoto, 

self.renderer._renderer, colormode=2)
File "--python_path--\lib\site-packages\matplotlib-1.4.3-py2.7-win32.egg\matplotlib\backends\tkagg.py", line 

24, in blit
Display the sourcecode for this frameOpen an interactive python shell in this frametk.call("PyAggImagePhoto", 

photoimage, id(aggimage), colormode, id(bbox_array))
TclError: out of stack space (infinite loop?)