aboutsummaryrefslogtreecommitdiffstats
path: root/mediagoblin/processing
Commit message (Collapse)AuthorAgeFilesLines
* Oh god how did I leave a pdb.set_trace() in hereChristopher Allan Webber2013-11-101-2/+0
| | | | | | Well, cleaning that up :) This commit sponsored by Enrico Zini. Thanks!
* Support python 2.6 again! Thanks to julianoliver for catching this.Christopher Allan Webber2013-09-091-2/+11
| | | | This commit sponsored by Sam Clegg. Thank you!
* should use public_store.file_exists to check if the file copied correctlyRodney Ewing2013-09-021-2/+1
|
* Merge remote-tracking branch 'refs/remotes/rodney757/reprocessing'Christopher Allan Webber2013-08-212-57/+306
|\ | | | | | | | | | | Conflicts: mediagoblin/processing/task.py mediagoblin/submit/lib.py
| * refactor get_orig_filename to return an acceptable filename to the processor.Rodney Ewing2013-08-161-14/+27
| | | | | | | | If there is an original video file and we skip transcoding, delete the webm_640 file
| * only try and delete queue file if it existsRodney Ewing2013-08-161-3/+4
| |
| * catch processing exceptions and if entry_orig_state is processed, then ↵Rodney Ewing2013-08-161-1/+12
| | | | | | | | ignore the exception
| * raise an error if the file failed to copy to public storageRodney Ewing2013-08-161-1/+19
| | | | | | | | catch copy_local_to_storage errors and raise PublicStoreFail, saving the keyname
| * run initial processing on all failed entriesRodney Ewing2013-08-161-0/+5
| |
| * add bulk_run, thumbs, and initial sub_commandsRodney Ewing2013-08-161-2/+13
| |
| * remove ProcessingStateRodney Ewing2013-08-161-82/+0
| |
| * delete existing file in store_publicRodney Ewing2013-08-161-1/+4
| |
| * Add image resizer and some cleanup of old codeRodney Ewing2013-08-161-2/+2
| |
| * Fixing normal submission of media (well for images anyway)Christopher Allan Webber2013-08-161-0/+2
| |
| * Fixing the MediaProcessor context manager so it actually, you know, works :)Christopher Allan Webber2013-08-161-0/+1
| | | | | | | | This commit sponsored by Mikiya Okuno. Thank you!
| * Theoretically the last steps to get reprocessing working for initial & imagesChristopher Allan Webber2013-08-161-10/+34
| | | | | | | | | | | | Haven't tested it yet though :) This commit sponsored by Samuel Bächler. Thank you!
| * Factored the get_orig_filename from processing state and put it to use.Christopher Allan Webber2013-08-161-0/+34
| | | | | | | | This commit sponsored by Vincent Demeester. Thank you!
| * Record the original state of the media entry in the processorChristopher Allan Webber2013-08-162-4/+7
| | | | | | | | | | | | | | This allows our processor to make some informed decisions based on the state by still having access to the original state. This commit sponsored by William Rico. Thank you!
| * Renaming the processing manager stuff to be less ambiguous.Christopher Allan Webber2013-08-162-5/+12
| | | | | | | | | | | | BONUS COMMIT to Ben Finney and the Free Software Melbourne crew. :) IRONY: Initially I committed this as "media manager".
| * Updating to the point where we can allllmost run with the new reprocessing codeChristopher Allan Webber2013-08-162-20/+46
| | | | | | | | This commit sponsored by Odin Hørthe Omdal. Thank you!
| * Should be enough to get to the point where you can actually initialize a ↵Christopher Allan Webber2013-08-161-2/+14
| | | | | | | | | | | | | | | | processing command now. However, it doesn't celery task-ify it... This commit sponsored by Catalin Cosovanu. Thank you!
| * Steps toward working "run" reprocessing command.Christopher Allan Webber2013-08-161-1/+25
| | | | | | | | This commit sponsored by Philippe Casteleyn. Thank you!
| * `gmg reprocess available --action-help` now tells you processor arguments!Christopher Allan Webber2013-08-161-3/+3
| | | | | | | | | | Every reprocessing action possible can inform you of its command line argument stuff! Is that awesome or what?
| * "initial" reprocessing subcommand now works!Christopher Allan Webber2013-08-161-2/+6
| | | | | | | | | | | | | | We are on our way now to a working reprocessing system under this redesign! This commit sponsored by Bjarni Rúnar Einarsson. Thank you!
| * More steps towards a working reprocessing system.Christopher Allan Webber2013-08-161-9/+19
| | | | | | | | | | | | | | | | | | Fleshing out the base classes and setting up some docstrings. Not everything is totally clear yet, but I think it's on a good track, and getting clearer. This commit sponsored by Ben Finney, on behalf of Free Software Melbourne. Thank you all!
| * Documentation for the MediaProcessorChristopher Allan Webber2013-08-161-4/+54
| |
| * started coding basics of new processing codeChristopher Allan Webber2013-08-161-0/+52
| |
| * fix storage paramater in get_orig_filename(), fix __init__ for ↵Rodney Ewing2013-08-161-1/+3
| | | | | | | | ProceessImage, better description for --size flag
| * change get_queued_filename to get_orig_filename and modified functionRodney Ewing2013-08-161-27/+21
| |
| * added comments and did a little refactoring. not sure if it is actually any ↵Rodney Ewing2013-08-161-2/+10
| | | | | | | | clearer though
| * very rough working version of image reprocessingRodney Ewing2013-08-161-3/+3
| |
| * added image reprocessingRodney Ewing2013-08-162-2/+21
| |
* | - need self.metadata with BaseProcessingFailRodney Ewing2013-08-191-1/+2
| | | | | | | | - pass feed_url into ProcessMedia run()
* | -update to latest masterRodney Ewing2013-08-192-11/+12
| | | | | | | | | | | | - have mg generate task_id remove
* | Tweak Celery TaskSebastian Spaeth2013-08-192-15/+14
|/ | | | | | | | | | | | - Make sure Exceptions are pickleable (not sure if this was not the case but this is the pattern as documented in the celery docs. - Don't create a task_id in the GMG code, but save the one implicitely created by celery. - Don't create a task-id directory per upload. Just store queued uploads in a single directory (this is the most controversial change and might need discussion!!!) Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
* Revert "Leave slug empty until we are sure media processing was successful."Rodney Ewing2013-08-081-2/+0
| | | | | | | This reverts commit f67611fb485b5a84cedc62b73beb1e551e8cb934. For some reason, generating a slug here throws an integrity error during a query when there is a duplicate slug.
* Leave slug empty until we are sure media processing was successful.Rodney Ewing2013-08-071-0/+2
| | | | Patch submitted by LotusEcho
* Doc string for ProcessingState.Elrond2013-04-221-0/+8
|
* MediaManager: Use .foo instead of ['foo'].Elrond2013-04-171-1/+1
| | | | To make .media_fetch_order work, create a property.
* Implement queue dir delete for proc_state.delete_queue_fileElrond2013-04-081-1/+6
| | | | | Implement queue dir deleting in the proc_state.delete_queue_file helper function.
* Create ProcessingState.store_public.Elrond2013-04-031-2/+10
| | | | | | The ideas is by Alon Levy. Use it in ProcessingState.copy_original for now.
* Kill monkeypatching of ProcessingState.Elrond2013-02-082-14/+1
| | | | | | | And change the process_foo() API to accept a processingstate now. image and video are tested, the others are UNTESTED.
* Also refactor "copy original into public storage".Elrond2013-02-081-0/+6
| | | | | | | This makes the processing code easier to read/write and alos will help the reprocessing once we get to it. Thanks to Joar Wandborg for testing!
* ProcessingState: Document monkey patching.Elrond2013-02-081-0/+10
|
* Implement ProcessingState class and use for imagesElrond2013-02-082-3/+37
| | | | | | | | | | The idea is to have a class that has the knowledge of the currently being processed media and also has tools for that. The long term idea is to make reprocessing easier by for example hiding the way the original comes into the processing code.
* Don't pass request into run_process_mediaSebastian Spaeth2013-01-151-1/+2
| | | | | | | | | People(tm) want to start run_process_media from the CLI and might not have a request object handy. So pass in the feed_url into run_process_media rather than the request object and allow the feed url to be empty (resulting in no PuSH notification at all then). Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
* Make PuSHing the Pubhubsubbub server an async task (#436, #585)Sebastian Spaeth2013-01-151-3/+45
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Notifying the PuSH servers had 3 problems.  1) it was done immediately after sending of the processing task to celery. So if celery was run in a separate process we would notify the PuSH servers before the new media was processed/ visible. (#436) 2) Notification code was called in submit/views.py, so submitting via the API never resulted in notifications. (#585) 3) If Notifying the PuSH server failed, we would never retry. The solution was to make the PuSH notification an asynchronous subtask. This way: 1) it will only be called once async processing has finished, 2) it is in the main processing code path, so even API calls will result in notifications, and 3) We retry 3 times in case of failure before giving up. If the server is in a separate process, we will wait 3x 2 minutes before retrying the notification. The only downside is that the celery server needs to have access to the internet to ping the PuSH server. If that is a problem, we need to make the task belong to a special group of celery servers that has access to the internet. As a side effect, I believe I removed the limitation that prevented us from upgrading celery. Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
* Move db.sql.models* to db.models*Sebastian Spaeth2013-01-071-1/+1
|
* Remove ObjectId from the treeSebastian Spaeth2012-12-251-3/+2
| | | | | | | This was one of the last remaining Mongo holdouts and has been removed from the tree herewith. Good bye, ObjectId. Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
* Move DBModel._id -> DBModel.idSebastian Spaeth2012-12-212-7/+7
| | | | | | | | | | | | | | | | | | | | | | | We were refering to model._id in most of the code base as this is what Mongo uses. However, each use of _id required a) fixup of queries: e.g. what we did in our find() and find_one() functions moving all '_id' to 'id'. It also required using AliasFields to make the ._id attribute available. This all means lots of superfluous fixing and transitioning in a SQL world. It will also not work in the long run. Much newer code already refers to the objects by model.id (e.g. in the oauth plugin), which will break with Mongo. So let's be honest, rip out the _id mongoism and live with .id as the one canonical way to address objects. This commit modifies all users and providers of model._id to use model.id instead. This patch works with or without Mongo removed first, but will break Mongo usage (even more than before) I have not bothered to fixup db.mongo.* and db.sql.convert (which converts from Mongo to SQL) Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
generated by cgit v1.2.3 (git 2.25.1) at 2025-09-13 14:34:31 +0000