Sunday, April 3, 2011

How can I check the memory usage of objects in ipython?

Duplicate

http://stackoverflow.com/questions/33978/find-out-how-much-memory-is-being-used-by-an-object-in-python

I am using ipython to run my code. I wonder if there is any module or command which allow me to check the memory usage of an object. For instance:

1> a = range(10000)
2> %memusage a
1MB

Something like %memusage and return the memory used by the object.

Thank you very much.

From stackoverflow
  • UPDATE: Here is another, maybe more thorough recipe for estimating the size of a python object.

    Here is a thread addressing a similar question

    The solution proposed is to write your own... using some estimates of the known size of primitives, python's object overhead, and the sizes of built in container types.

    Since the code is not that long, here is a direct copy of it:

    def sizeof(obj):
        """APPROXIMATE memory taken by some Python objects in 
        the current 32-bit CPython implementation.
    
        Excludes the space used by items in containers; does not
        take into account overhead of memory allocation from the
        operating system, or over-allocation by lists and dicts.
        """
        T = type(obj)
        if T is int:
            kind = "fixed"
            container = False
            size = 4
        elif T is list or T is tuple:
            kind = "variable"
            container = True
            size = 4*len(obj)
        elif T is dict:
            kind = "variable"
            container = True
            size = 144
            if len(obj) > 8:
                size += 12*(len(obj)-8)
        elif T is str:
            kind = "variable"
            container = False
            size = len(obj) + 1
        else:
            raise TypeError("don't know about this kind of object")
        if kind == "fixed":
            overhead = 8
        else: # "variable"
            overhead = 12
        if container:
            garbage_collector = 8
        else:
            garbage_collector = 0
        malloc = 8 # in most cases
        size = size + overhead + garbage_collector + malloc
        # Round to nearest multiple of 8 bytes
        x = size % 8
        if x != 0:
            size += 8-x
            size = (size + 8)
        return size
    
  • Unfortunately this is not possible, but there are a number of ways of approximating the answer:

    1. for very simple objects (e.g. ints, strings, floats, doubles) which are represented more or less as simple C-language types you can simply calculate the number of bytes as with John Mulder's solution above.

    2. For more complex objects a good approximation is to serialize the object to a string using cPickle.dumps. The length of the string is a good approximation of the amount of memory required to store an object.

    There is one big snag with solution 2, which is that objects usually contain references to other objects. For example a dict contains string-keys and other objects as values. Those other objects might be shared. Since pickle always tries to do a complete serialization of the object it will always over-estimate the amount of memory required to store an object.

    Constantin : But if you pickle a list containing all root objects you're interested in, there will be no overestimation.
    i-freaker : Thank you very much. But I wonder if pickle will do any compression or not.
    Salim Fadhley : No, pickle does not compress. It simply eliminates redundancy.

0 comments:

Post a Comment