| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
| |
Reading a file into memory resulted in depletion of memory. Now files
are read and written lazily, by chunks.
|
| | |
|
| | |
|
| | |
|
| | |
|
| |\
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
* cwebber/254_delete_queue_directories:
Removing docstring bit about delete_file possibly deleting directories in the future
Convert media processing backends to delete the queue directory (#254)
Implement delete_dir in the FileStorage
Conflicts:
mediagoblin/media_types/image/processing.py
mediagoblin/media_types/video/processing.py
Conflicts, because those media_types already use the newer
proc_state.delete_queue_file() method (which needs
updating.
|
| | |
| |
| |
| |
| |
| |
| |
| | |
plus options for deleting only empty directories and deleting
them recursively.
Not sure how cloudfile storage is or should be handled here. Are
things such as a "directory" even a concept there?
|
| |/
|
|
|
|
|
|
|
|
|
|
|
| |
The copy_locally and copy_local_to_storage (very inconsistent terms BTW)
were simply slurping in everything in RAM and writing it out at once.
(the copy_locally was actually memory efficient if the remote system was local)
Use shutil.copyfileobj which does chunked reads/writes on file objects.
The default buffer size is 16kb, and as each chunk means a separate HTTP
request for e.g. cloudfiles, we use a chunksize of 4MB here (which has
just been arbitrarily set by me without tests).
This should help with the failure to upload large files issue #419.
|
| | |
|
| |
|
|
|
|
|
| |
versions
This utility should allow for easy copying from a local filesystem to
the storage instance.
|
|
|
* Removed storage.py
* Created submodules for filestorage, cloudfiles, mountstorage
* Changed test_storage to reflect the changes made in the storage
module structure
* Added mediagoblin.storage.filestorage.BasicFileStorage as a
default for both publicstore and queuestore's `storage_class`
|
