Tuesday, August 09, 2005

Pythonic file I/O?

A coworker of mine has recently been using Python on and off for basic administrative tasks and ran into a problem this week with modifying the contents of a file without rewriting the entire file. He wasn't trying to modify the file size, just change a number of bytes that were somewhere in the file. I had forgotten about mmap, but is that the best Python module for file modification?

It has two main limitations: 1) No way to insert into or delete from a random location in the file. 2) A mmap object maps a file in part or in total, but the mapping must start at the beginning of the file. This is a problem when modifying large files which can't be loaded entirely into memory, a solution would be to allow the mmap to begin a file mapping at an arbitrary position.

I understand there are some technical details to make this work. Completely rewriting a file out may still be needed in some cases; but, all of this should be implemented in a slick module and hidden from the developer. It's just not very pythonic.


Post a Comment

<< Home