diff options
Diffstat (limited to 'docs/source/devel')
| -rw-r--r-- | docs/source/devel/codebase.rst | 148 | ||||
| -rw-r--r-- | docs/source/devel/migrations.rst | 62 | ||||
| -rw-r--r-- | docs/source/devel/storage.rst | 94 |
3 files changed, 226 insertions, 78 deletions
diff --git a/docs/source/devel/codebase.rst b/docs/source/devel/codebase.rst index 5e8cbcc6..122a3297 100644 --- a/docs/source/devel/codebase.rst +++ b/docs/source/devel/codebase.rst @@ -34,78 +34,11 @@ various recipes for getting things done. for where we hang out. For more information on how to get started hacking on GNU MediaGoblin, -see `the wiki <http://wiki.mediagoblin.org/>`_. - - -Software Stack -============== - -* Project infrastructure - - * `Python <http://python.org/>`_: the language we're using to write - this - - * `Nose <http://somethingaboutorange.com/mrl/projects/nose/>`_: - for unit tests - - * `virtualenv <http://www.virtualenv.org/>`_: for setting up an - isolated environment to keep mediagoblin and related packages - (potentially not required if MediaGoblin is packaged for your - distro) - -* Data storage - - * `SQLAlchemy <http://sqlalchemy.org/>`_: SQL ORM and database - interaction library for Python. Currently we support sqlite and - postgress as backends. - -* Web application - - * `Paste Deploy <http://pythonpaste.org/deploy/>`_ and - `Paste Script <http://pythonpaste.org/script/>`_: we'll use this for - configuring and launching the application - - * `werkzeug <http://werkzeug.pocoo.org/>`_: nice abstraction layer - from HTTP requests, responses and WSGI bits - - * `Beaker <http://beaker.groovie.org/>`_: for handling sessions and - caching - - * `Jinja2 <http://jinja.pocoo.org/docs/>`_: the templating engine - - * `WTForms <http://wtforms.simplecodes.com/>`_: for handling, - validation, and abstraction from HTML forms - - * `Celery <http://celeryproject.org/>`_: for task queuing (resizing - images, encoding video, ...) - - * `Babel <http://babel.edgewall.org>`_: Used to extract and compile - translations. - - * `Markdown (for python) <http://pypi.python.org/pypi/Markdown>`_: - implementation of `Markdown <http://daringfireball.net/projects/markdown/>`_ - text-to-html tool to make it easy for people to write richtext - comments, descriptions, and etc. - - * `lxml <http://lxml.de/>`_: nice xml and html processing for - python. - -* Media processing libraries - - * `Python Imaging Library <http://www.pythonware.com/products/pil/>`_: - used to resize and otherwise convert images for display. - - * `GStreamer <http://gstreamer.freedesktop.org/>`_: (Optional, for - video hosting sites only) Used to transcode video, and in the - future, probably audio too. - - * `chardet <http://pypi.python.org/pypi/chardet>`_: (Optional, for - ascii art hosting sites only) Used to make ascii art thumbnails. - -* Front end - - * `JQuery <http://jquery.com/>`_: for groovy JavaScript things - +see `the wiki <http://wiki.mediagoblin.org/>`_, and specifically, go +through the +`Hacking HOWTO <http://wiki.mediagoblin.org/HackingHowto>`_ +which explains generally how to get going with running an instance for +development. What's where @@ -177,3 +110,74 @@ including the following: :migrations.py: When creating a new migration (a change to the database structure), we put it here + +Software Stack +============== + +* Project infrastructure + + * `Python <http://python.org/>`_: the language we're using to write + this + + * `Py.Test <http://pytest.org/>`_: + for unit tests + + * `virtualenv <http://www.virtualenv.org/>`_: for setting up an + isolated environment to keep mediagoblin and related packages + (potentially not required if MediaGoblin is packaged for your + distro) + +* Data storage + + * `SQLAlchemy <http://sqlalchemy.org/>`_: SQL ORM and database + interaction library for Python. Currently we support sqlite and + postgress as backends. + +* Web application + + * `Paste Deploy <http://pythonpaste.org/deploy/>`_ and + `Paste Script <http://pythonpaste.org/script/>`_: we'll use this for + configuring and launching the application + + * `werkzeug <http://werkzeug.pocoo.org/>`_: nice abstraction layer + from HTTP requests, responses and WSGI bits + + * `itsdangerous <http://pythonhosted.org/itsdangerous/>`_: + for handling sessions + + * `Jinja2 <http://jinja.pocoo.org/docs/>`_: the templating engine + + * `WTForms <http://wtforms.simplecodes.com/>`_: for handling, + validation, and abstraction from HTML forms + + * `Celery <http://celeryproject.org/>`_: for task queuing (resizing + images, encoding video, ...) + + * `Babel <http://babel.edgewall.org>`_: Used to extract and compile + translations. + + * `Markdown (for python) <http://pypi.python.org/pypi/Markdown>`_: + implementation of `Markdown <http://daringfireball.net/projects/markdown/>`_ + text-to-html tool to make it easy for people to write richtext + comments, descriptions, and etc. + + * `lxml <http://lxml.de/>`_: nice xml and html processing for + python. + +* Media processing libraries + + * `Python Imaging Library <http://www.pythonware.com/products/pil/>`_: + used to resize and otherwise convert images for display. + + * `GStreamer <http://gstreamer.freedesktop.org/>`_: (Optional, for + video hosting sites only) Used to transcode video, and in the + future, probably audio too. + + * `chardet <http://pypi.python.org/pypi/chardet>`_: (Optional, for + ascii art hosting sites only) Used to make ascii art thumbnails. + +* Front end + + * `JQuery <http://jquery.com/>`_: for groovy JavaScript things + + diff --git a/docs/source/devel/migrations.rst b/docs/source/devel/migrations.rst new file mode 100644 index 00000000..16c02b04 --- /dev/null +++ b/docs/source/devel/migrations.rst @@ -0,0 +1,62 @@ +.. MediaGoblin Documentation + + Written in 2011, 2012 by MediaGoblin contributors + + To the extent possible under law, the author(s) have dedicated all + copyright and related and neighboring rights to this software to + the public domain worldwide. This software is distributed without + any warranty. + + You should have received a copy of the CC0 Public Domain + Dedication along with this software. If not, see + <http://creativecommons.org/publicdomain/zero/1.0/>. + +========== +Migrations +========== + +So, about migrations. Every time we change the way the database +structure works, we need to add a migration so that people running +older codebases can have their databases updated to the new structure +when they run `./bin/gmg dbupdate`. + +The first time `./bin/gmg dbupdate` is run by a user, it creates the +tables at the current state that they're defined in models.py and sets +the migration number to the current migration... after all, migrations +only exist to get things to the current state of the db. After that, +every migration is run with dbupdate. + +There's a few things you need to know: + +- We use `sqlalchemy-migrate + <http://code.google.com/p/sqlalchemy-migrate/>`_. + See `their docs <https://sqlalchemy-migrate.readthedocs.org/>`_. +- `Alembic <https://bitbucket.org/zzzeek/alembic>`_ might be a better + choice than sqlalchemy-migrate now or in the future, but we + originally decided not to use it because it didn't have sqlite + support. It's not clear if that's changed. +- SQLAlchemy has two parts to it, the ORM and the "core" interface. + We DO NOT use the ORM when running migrations. Think about it: the + ORM is set up with an expectation that the models already reflect a + certain pattern. But if a person is moving from their old patern + and are running tools to *get to* the current pattern, of course + their current database structure doesn't match the state of the ORM! +- How to write migrations? Maybe there will be a tutorial here in the + future... in the meanwhile, look at existing migrations in + `mediagoblin/db/migrations.py` and look in + `mediagoblin/tests/test_sql_migrations.py` for examples. +- Common pattern: use `inspect_table` to get the current state + of the table before we run alterations on it. +- Make sure you set the RegisterMigration to be the next migration in + order. +- What happens if you're adding a *totally new* table? In this case, + you should copy the table in entirety as it exists into + migrations.py then create the tables based off of that... see + add_collection_tables. This is easier than reproducing the SQL by + hand. +- If you're writing a feature branch, you don't need to keep adding + migrations every time you change things around if your database + structure is in flux. Just alter your migrations so that they're + correct for the merge into master. + +That's it for now! Good luck! diff --git a/docs/source/devel/storage.rst b/docs/source/devel/storage.rst index 52406c4e..215f9579 100644 --- a/docs/source/devel/storage.rst +++ b/docs/source/devel/storage.rst @@ -2,18 +2,28 @@ Storage ========= - -See for now: http://wiki.mediagoblin.org/Storage - -Things get moved here. - - The storage systems attached to your app ---------------------------------------- Dynamic content: queue_store and public_store ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Two instances of the StorageInterface come attached to your app. These +are: + ++ **queue_store:** When a user submits a fresh piece of media for + their gallery, before the Processing stage, that piece of media sits + here in the queue_store. (It's possible that we'll rename this to + "private_store" and start storing more non-publicly-stored stuff in + the future...). This is a StorageInterface implementation + instance. Visitors to your site probably cannot see it... it isn't + designed to be seen, anyway. + ++ **public_store:** After your media goes through processing it gets + moved to the public store. This is also a StorageInterface + implelementation, and is for stuff that's intended to be seen by + site visitors. + The workbench ~~~~~~~~~~~~~ @@ -32,6 +42,28 @@ See the workbench module documentation for more. Static assets / staticdirect ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +On top of all that, there is some static media that comes bundled with your +application. This stuff is kept in: + + mediagoblin/static/ + +These files are for mediagoblin base assets. Things like the CSS files, +logos, etc. You can mount these at whatever location is appropriate to you +(see the direct_remote_path option in the config file) so if your users +are keeping their static assets at http://static.mgoblin.example.org/ but +their actual site is at http://mgoblin.example.org/, you need to be able +to get your static files in a where-it's-mounted agnostic way. There's a +"staticdirector" attached to the request object. It's pretty easy to use; +just look at this bit taken from the +mediagoblin/templates/mediagoblin/base.html main template: + + <link rel="stylesheet" type="text/css" + href="Template:Request.staticdirect('/css/extlib/text.css')"/> + +see? Not too hard. As expected, if you configured direct_remote_path to be +http://static.mgoblin.example.org/ you'll get back +http://static.mgoblin.example.org/css/extlib/text.css just as you'd +probably expect. StorageInterface and implementations ------------------------------------ @@ -39,5 +71,55 @@ StorageInterface and implementations The guts of StorageInterface and friends ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +So, the StorageInterface! + +So, the public and queue stores both use StorageInterface implementations +... but what does that mean? It's not too hard. + +Open up: + + mediagoblin/storage.py + +In here you'll see a couple of things. First of all, there's the +StorageInterface class. What you'll see is that this is just a very simple +python class. A few of the methods actually implement things, but for the +most part, they don't. What really matters about this class is the +docstrings. Each expected method is documented as to how it should be +constructed. Want to make a new StorageInterface? Simply subclass it. Want +to know how to use the methods of your storage system? Read these docs, +they span all implementations. + +There are a couple of implementations of these classes bundled in +storage.py as well. The most simple of these is BasicFileStorage, which is +also the default storage system used. As expected, this stores files +locally on your machine. + +There's also a CloudFileStorage system. This provides a mapping to +[OpenStack's swift http://swift.openstack.org/] storage system (used by +RackSpace Cloud files and etc). + +Between these two examples you should be able to get a pretty good idea of +how to write your own storage systems, for storing data across your +beowulf cluster of radioactive monkey brains, whatever. + Writing code to store stuff ~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +So what does coding for StorageInterface implementations actually look +like? It's pretty simple, really. For one thing, the design is fairly +inspired by [Django's file storage API +https://docs.djangoproject.com/en/dev/ref/files/storage/]... with some +differences. + +Basically, you access files on "file paths", which aren't exactly like +unix file paths, but are close. If you wanted to store a file on a path +like dir1/dir2/filename.jpg you'd actually write that file path like: + +['dir1', 'dir2', 'filename.jpg'] + +This way we can be *sure* that each component is actually a component of +the path that's expected... we do some filename cleaning on each component. + +Your StorageInterface should pass in and out "file like objects". In other +words, they should provide .read() and .write() at minimum, and probably +also .seek() and .close(). |