diff options
Diffstat (limited to 'docs/source')
| -rw-r--r-- | docs/source/conf.py | 3 | ||||
| -rw-r--r-- | docs/source/devel/codebase.rst (renamed from docs/source/siteadmin/codebase.rst) | 135 | ||||
| -rw-r--r-- | docs/source/devel/originaldesigndecisions.rst | 336 | ||||
| -rw-r--r-- | docs/source/devel/storage.rst | 125 | ||||
| -rw-r--r-- | docs/source/index.rst | 18 | ||||
| -rw-r--r-- | docs/source/plugindocs/raven.rst | 2 | ||||
| -rw-r--r-- | docs/source/plugindocs/trim_whitespace.rst | 1 | ||||
| -rw-r--r-- | docs/source/pluginwriter/api.rst | 127 | ||||
| -rw-r--r-- | docs/source/pluginwriter/database.rst | 111 | ||||
| -rw-r--r-- | docs/source/siteadmin/deploying.rst | 70 | ||||
| -rw-r--r-- | docs/source/siteadmin/media-types.rst | 112 | ||||
| -rw-r--r-- | docs/source/siteadmin/plugins.rst | 22 | ||||
| -rw-r--r-- | docs/source/siteadmin/production-deployments.rst | 13 | ||||
| -rw-r--r-- | docs/source/siteadmin/relnotes.rst | 160 |
14 files changed, 1155 insertions, 80 deletions
diff --git a/docs/source/conf.py b/docs/source/conf.py index 4209acc8..0b2bccac 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -26,7 +26,8 @@ sys.path.insert(0, os.path.abspath(os.path.join('..', '..'))) # Add any Sphinx extension module names here, as strings. They can be extensions # coming with Sphinx (named 'sphinx.ext.*') or your custom ones. -extensions = [] +extensions = ['sphinx.ext.autodoc', 'sphinx.ext.intersphinx'] +intersphinx_mapping = {'python': ('http://docs.python.org/2.7', None)} # Add any paths that contain templates here, relative to this directory. templates_path = ['source/_templates'] diff --git a/docs/source/siteadmin/codebase.rst b/docs/source/devel/codebase.rst index 3ef91290..122a3297 100644 --- a/docs/source/siteadmin/codebase.rst +++ b/docs/source/devel/codebase.rst @@ -34,7 +34,81 @@ various recipes for getting things done. for where we hang out. For more information on how to get started hacking on GNU MediaGoblin, -see `the wiki <http://wiki.mediagoblin.org/>`_. +see `the wiki <http://wiki.mediagoblin.org/>`_, and specifically, go +through the +`Hacking HOWTO <http://wiki.mediagoblin.org/HackingHowto>`_ +which explains generally how to get going with running an instance for +development. + + +What's where +============ + +After you've run checked out mediagoblin and followed the virtualenv +instantiation instructions, you're faced with the following directory +tree:: + + mediagoblin/ + |- mediagoblin/ # source code + | |- db/ # database setup + | |- tools/ # various utilities + | |- init/ # "initialization" tools (arguably should be in tools/) + | |- tests/ # unit tests + | |- templates/ # templates for this application + | |- media_types/ # code for processing, displaying different media + | |- storage/ # different storage backends + | |- gmg_commands/ # command line tools (./bin/gmg) + | |- themes/ # pre-bundled themes + | | + | | # ... some submodules here as well for different sections + | | # of the application... here's just a few + | |- auth/ # authentication (login/registration) code + | |- user_dev/ # user pages (under /u/), including media pages + | \- submit/ # submitting media for processing + | + |- docs/ # documentation + |- devtools/ # some scripts for developer convenience + | + |- user_dev/ # local instance sessions, media, etc + | + | # the below directories are installed into your virtualenv checkout + | + |- bin/ # scripts + |- develop-eggs/ + |- lib/ # python libraries installed into your virtualenv + |- include/ + |- mediagoblin.egg-info/ + \- parts/ + + +As you can see, all the code for GNU MediaGoblin is in the +``mediagoblin`` directory. + +Here are some interesting files and what they do: + +:routing.py: maps url paths to views +:views.py: views handle http requests +:forms.py: wtforms stuff for this submodule + +You'll notice that there are several sub-directories: tests, +templates, auth, submit, ... + +``tests`` holds the unit test code. + +``templates`` holds all the templates for the output. + +``auth`` and ``submit`` are modules that enacpsulate authentication +and media item submission. If you look in these directories, you'll +see they have their own ``routing.py``, ``view.py``, and forms.py in +addition to some other code. + +You'll also notice that mediagoblin/db/ contains quite a few things, +including the following: + +:models.py: This is where the database is set up +:mixin.py: Certain functions appended to models from here +:migrations.py: When creating a new migration (a change to the + database structure), we put it here Software Stack @@ -45,7 +119,7 @@ Software Stack * `Python <http://python.org/>`_: the language we're using to write this - * `Nose <http://somethingaboutorange.com/mrl/projects/nose/>`_: + * `Py.Test <http://pytest.org/>`_: for unit tests * `virtualenv <http://www.virtualenv.org/>`_: for setting up an @@ -65,13 +139,11 @@ Software Stack `Paste Script <http://pythonpaste.org/script/>`_: we'll use this for configuring and launching the application - * `WebOb <http://pythonpaste.org/webob/>`_: nice abstraction layer + * `werkzeug <http://werkzeug.pocoo.org/>`_: nice abstraction layer from HTTP requests, responses and WSGI bits - * `Routes <http://routes.groovie.org/>`_: for URL routing - - * `Beaker <http://beaker.groovie.org/>`_: for handling sessions and - caching + * `itsdangerous <http://pythonhosted.org/itsdangerous/>`_: + for handling sessions * `Jinja2 <http://jinja.pocoo.org/docs/>`_: the templating engine @@ -109,52 +181,3 @@ Software Stack * `JQuery <http://jquery.com/>`_: for groovy JavaScript things - -What's where -============ - -After you've run checked out mediagoblin and followed the virtualenv -instantiation instructions, you're faced with the following directory -tree:: - - mediagoblin/ - |- mediagoblin/ # source code - | |- tests/ - | |- templates/ - | |- auth/ - | \- submit/ - |- docs/ # documentation - |- devtools/ # some scripts for developer convenience - | - | # the below directories are installed into your virtualenv checkout - | - |- bin/ # scripts - |- develop-eggs/ - |- lib/ # python libraries installed into your virtualenv - |- include/ - |- mediagoblin.egg-info/ - |- parts/ - |- user_dev/ # sessions, etc - - -As you can see, all the code for GNU MediaGoblin is in the -``mediagoblin`` directory. - -Here are some interesting files and what they do: - -:routing.py: maps url paths to views -:views.py: views handle http requests -:models.py: holds the sqlalchemy schemas---these are the data structures - we're working with - -You'll notice that there are several sub-directories: tests, -templates, auth, submit, ... - -``tests`` holds the unit test code. - -``templates`` holds all the templates for the output. - -``auth`` and ``submit`` are modules that enacpsulate authentication -and media item submission. If you look in these directories, you'll -see they have their own ``routing.py``, ``view.py``, and -``models.py`` in addition to some other code. diff --git a/docs/source/devel/originaldesigndecisions.rst b/docs/source/devel/originaldesigndecisions.rst new file mode 100644 index 00000000..2843870c --- /dev/null +++ b/docs/source/devel/originaldesigndecisions.rst @@ -0,0 +1,336 @@ +.. _original-design-decisions-chapter: + +=========================== + Original Design Decisions +=========================== + +.. contents:: Sections + :local: + + +This chapter talks a bit about design decisions. + +Note: This is an outdated document. It's more or less the historical +reasons for a lot of things. That doesn't mean these decisions have +stayed the same or we haven't changed our minds on some things! + + +Why GNU MediaGoblin? +==================== + +Chris and Will on "Why GNU MediaGoblin": + + Chris came up with the name MediaGoblin. The name is pretty fun. + It merges the idea that this is a Media hosting project with + Goblin which sort of sounds like gobbling. Here's a piece of + software that gobbles up your media for all to see. + + `According to Wikipedia <http://en.wikipedia.org/wiki/Goblin>`_, a + goblin is: + + a legendary evil or mischievous illiterate creature, described + as grotesquely evil or evil-like phantom + + So are we evil? No. Are we mischievous or illiterate? Not + really. So what kind of goblin are we thinking about? We're + thinking about these goblins: + + .. figure:: ../_static/goblin.png + :alt: Cute goblin with a beret. + + *Figure 1: Cute goblin with a beret. llustrated by Chris + Webber* + + .. figure:: ../_static/snugglygoblin.png + :scale: 50% + :alt: Snuggly goblin with a beret. + + *Figure 2: Snuggly goblin. Illustrated by Karen Rustad* + + Those are pretty cute goblins. Those are the kinds of goblins + we're thinking about. + + Chris started doing work on the project after thinking about it + for a year. Then, after talking with Matt and Rob, it became an + official GNU project. Thus we now call it GNU MediaGoblin. + + That's a lot of letters, though, so in the interest of brevity and + facilitating easier casual conversation and balancing that with + what's important to us, we have the following rules: + + 1. "GNU MediaGoblin" is the name we're going to use in all official + capacities: web site, documentation, press releases, ... + + 2. In casual conversation, it's ok to use more casual names. + + 3. If you're writing about the project, we ask that you call it GNU + MediaGoblin. + + 4. If you don't like the name, we kindly ask you to take a deep + breath, think a happy thought about cute little goblins playing + on a playground and taking cute pictures of themselves, and let + it go. (Will added this one.) + + +Why Python +========== + +Chris Webber on "Why Python": + + Because I know Python, love Python, am capable of actually making + this thing happen in Python (I've worked on a lot of large free + software web applications before in Python, including `Miro + Community`_, the `Miro Guide`_, a large portion of `Creative + Commons`_, and a whole bunch of things while working at `Imaginary + Landscape`_). Me starting a project like this makes sense if it's + done in Python. + + You might say that PHP is way more deployable, that Rails has way + more cool developers riding around on fixie bikes---and all of + those things are true. But I know Python, like Python, and think + that Python is pretty great. I do think that deployment in Python + is not as good as with PHP, but I think the days of shared hosting + are (thankfully) coming to an end, and will probably be replaced + by cheap virtual machines spun up on the fly for people who want + that sort of stuff, and Python will be a huge part of that future, + maybe even more than PHP will. The deployment tools are getting + better. Maybe we can use something like Silver Lining. Maybe we + can just distribute as ``.debs`` or ``.rpms``. We'll figure it + out when we get there. + + Regardless, if I'm starting this project, which I am, it's gonna + be in Python. + +.. _Miro Community: http://mirocommunity.org/ +.. _Miro Guide: http://miroguide.org/ +.. _Creative Commons: http://creativecommons.org/ +.. _Imaginary Landscape: http://www.imagescape.com/ + + +Why WSGI Minimalism +=================== + +Chris Webber on "Why WSGI Minimalism": + + If you notice in the technology list I list a lot of components + that are very "django-like", but not actually `Django`_ + components. What can I say, I really like a lot of the ideas in + Django! Which leads to the question: why not just use Django? + + While I really like Django's ideas and a lot of its components, I + also feel that most of the best ideas in Django I want have been + implemented as good or even better outside of Django. I could + just use Django and replace the templating system with Jinja2, and + the form system with wtforms, and the database with MongoDB and + MongoKit, but at that point, how much of Django is really left? + + I also am sometimes saddened and irritated by how coupled all of + Django's components are. Loosely coupled yes, but still coupled. + WSGI has done a good job of providing a base layer for running + applications on and if you know how to do it yourself [1]_, it's + not hard or many lines of code at all to bind them together + without any framework at all (not even say `Pylons`_, `Pyramid`_ + or `Flask`_ which I think are still great projects, especially for + people who want this sort of thing but have no idea how to get + started). And even at this already really early stage of writing + MediaGoblin, that glue work is mostly done. + + Not to say I don't think Django isn't great for a lot of things. + For a lot of stuff, it's still the best, but not for MediaGoblin, + I think. + + One thing that Django does super well though is documentation. It + still has some faults, but even with those considered I can hardly + think of any other project in Python that has as nice of + documentation as Django. It may be worth learning some lessons on + documentation from Django [2]_, on that note. + + I'd really like to have a good, thorough hacking-howto and + deployment-howto, especially in the former making some notes on + how to make it easier for Django hackers to get started. + +.. _Django: http://www.djangoproject.com/ +.. _Pylons: http://pylonshq.com/ +.. _Pyramid: http://docs.pylonsproject.org/projects/pyramid/dev/ +.. _Flask: http://flask.pocoo.org/ + +.. [1] http://pythonpaste.org/webob/do-it-yourself.html +.. [2] http://pycon.blip.tv/file/4881071/ + + +Why MongoDB +=========== + +(Note: We don't use MongoDB anymore. This is the original rationale, +however.) + +Chris Webber on "Why MongoDB": + + In case you were wondering, I am not a NOSQL fanboy, I do not go + around telling people that MongoDB is web scale. Actually my + choice for MongoDB isn't scalability, though scaling up really + nicely is a pretty good feature and sets us up well in case large + volume sites eventually do use MediaGoblin. But there's another + side of scalability, and that's scaling down, which is important + for federation, maybe even more important than scaling up in an + ideal universe where everyone ran servers out of their own + housing. As a memory-mapped database, MongoDB is pretty hungry, + so actually I spent a lot of time debating whether the inability + to scale down as nicely as something like SQL has with sqlite + meant that it was out. + + But I decided in the end that I really want MongoDB, not for + scalability, but for flexibility. Schema evolution pains in SQL + are almost enough reason for me to want MongoDB, but not quite. + The real reason is because I want the ability to eventually handle + multiple media types through MediaGoblin, and also allow for + plugins, without the rigidity of tables making that difficult. In + other words, something like:: + + {"title": "Me talking until you are bored", + "description": "blah blah blah", + "media_type": "audio", + "media_data": { + "length": "2:30", + "codec": "OGG Vorbis"}, + "plugin_data": { + "licensing": { + "license": "http://creativecommons.org/licenses/by-sa/3.0/"}}} + + + Being able to just dump media-specific information in a media_data + hashtable is pretty great, and even better is having a plugin + system where you can just let plugins have their own entire + key-value space cleanly inside the document that doesn't interfere + with anyone else's stuff. If we were to let plugins to deposit + their own information inside the database, either we'd let plugins + create their own tables which makes SQL migrations even harder + than they already are, or we'd probably end up creating a table + with a column for key, a column for value, and a column for type + in one huge table called "plugin_data" or something similar. (Yo + dawg, I heard you liked plugins, so I put a database in your + database so you can query while you query.) Gross. + + I also don't want things to be too loose so that we forget or lose + the structure of things, and that's one reason why I want to use + MongoKit, because we can cleanly define a much structure as we + want and verify that documents match that structure generally + without adding too much bloat or overhead (MongoKit is a pretty + lightweight wrapper and doesn't inject extra MongoKit-specific + stuff into the database, which is nice and nicer than many other + ORMs in that way). + + +Why Sphinx for documentation +============================ + +Will Kahn-Greene on "Why Sphinx": + + `Sphinx`_ is a fantastic tool for organizing documentation for a + Python-based project that makes it pretty easy to write docs that + are readable in source form and can be "compiled" into HTML, LaTeX + and other formats. + + There are other doc systems out there, but given that GNU + MediaGoblin is being written in Python and I've done a ton of + documentation using Sphinx, it makes sense to use Sphinx for now. + +.. _Sphinx: http://sphinx.pocoo.org/ + + +Why AGPLv3 and CC0? +=================== + +Chris, Brett, Will, Rob, Matt, et al curated into a story where +everyone is the hero by Will on "Why AGPLv3 and CC0": + + The `AGPL v3`_ preserves the freedoms guaranteed by the GPL v3 in + the context of software as a service. Using this license ensures + that users of the service have the ability to examine the source, + deploy their own instance, and implement their own version. This + is really important to us and a core mission component of this + project. Thus we decided that the software parts should be under + this license. + + However, the project is made up of more than just software: + there's CSS, images, and other output-related things. We wanted + the templates/images/css side of the project all permissive and + permissive in the same absolutely permissive way. We're waiving + our copyrights to non-software things under the CC0 waiver. + + That brings us to the templates where there's some code and some + output. The template engine we're using is called Jinja2. It + mixes HTML markup with Python code to render the output of the + software. We decided the templates are part of the output of the + software and not the software itself. We wanted the output of the + software to be licensed in a hassle-free way so that when someone + deploys their own GNU MediaGoblin instance with their own + templates, they don't have to deal with the copyleft aspects of + the AGPLv3 and we'd be fine with that because the changes they're + making are identity-related. So at first we decided to waive our + copyrights to the templates with a CC0 waiver and then add an + exception to the AGPLv3 for the software such that the templates + can make calls into the software and yet be a separately licensed + work. However, Brett brought up the question of whether this + allows some unscrupulous person to make changes to the software + through the templates in such a way that they're not bound by the + AGPLv3: i.e. a loophole. We thought about this loophole and + between this and the extra legalese involved in the exception to + the AGPLv3, we decided that it's just way simpler if the templates + were also licensed under the AGPLv3. + + Then we have the licensing for the documentation. Given that the + documentation is tied to the software content-wise, we don't feel + like we have to worry about ensuring freedom of the documentation + or worry about attribution concerns. Thus we're waiving our + copyrights to the documentation under CC0 as well. + + Lastly, we have branding. This covers logos and other things that + are distinctive to GNU MediaGoblin that we feel represents this + project. Since we don't currently have any branding, this is an + open issue, but we're thinking we'll go with a CC BY-SA license. + + By licensing in this way, we make sure that users of the software + receive the freedoms that the AGPLv3 ensures regardless of what + fate befalls this project. + + So to summarize: + + * software (Python, JavaScript, HTML templates): licensed + under AGPLv3 + * non-software things (CSS, images, video): copyrights waived + under CC0 because this is output of the software + * documentation: copyrights waived under CC0 because it's not part + of the software + * branding assets: we're kicking this can down the road, but + probably CC BY-SA + + This is all codified in the ``COPYING`` file. + +.. _AGPL v3: http://www.gnu.org/licenses/agpl.html +.. _CC0 v1: http://creativecommons.org/publicdomain/zero/1.0/ + + +Why (non-mandatory) copyright assignment? +========================================= + +Chris Webber on "Why copyright assignment?": + + GNU MediaGoblin is a GNU project with non-mandatory but heavily + encouraged copyright assignment to the FSF. Most, if not all, of + the core contributors to GNU MediaGoblin will have done a + copyright assignment, but unlike some other GNU projects, it isn't + required here. We think this is the best choice for GNU + MediaGoblin: it ensures that the Free Software Foundation may + protect the software by enforcing the AGPL if the FSF sees fit, + but it also means that we can immediately merge in changes from a + new contributor. It also means that some significant non-FSF + contributors might also be able to enforce the AGPL if seen fit. + + Again, assignment is not mandatory, but it is heavily encouraged, + even incentivized: significant contributors who do a copyright + assignment to the FSF are eligible to have a unique goblin drawing + produced for them by the project's main founder, Christopher Allan + Webber. See `the wiki <http://wiki.mediagoblin.org/>`_ for details. + + diff --git a/docs/source/devel/storage.rst b/docs/source/devel/storage.rst new file mode 100644 index 00000000..215f9579 --- /dev/null +++ b/docs/source/devel/storage.rst @@ -0,0 +1,125 @@ +========= + Storage +========= + +The storage systems attached to your app +---------------------------------------- + +Dynamic content: queue_store and public_store +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Two instances of the StorageInterface come attached to your app. These +are: + ++ **queue_store:** When a user submits a fresh piece of media for + their gallery, before the Processing stage, that piece of media sits + here in the queue_store. (It's possible that we'll rename this to + "private_store" and start storing more non-publicly-stored stuff in + the future...). This is a StorageInterface implementation + instance. Visitors to your site probably cannot see it... it isn't + designed to be seen, anyway. + ++ **public_store:** After your media goes through processing it gets + moved to the public store. This is also a StorageInterface + implelementation, and is for stuff that's intended to be seen by + site visitors. + +The workbench +~~~~~~~~~~~~~ + +In addition, there's a "workbench" used during +processing... it's just for temporary files during +processing, and also for making local copies of stuff that +might be on remote storage interfaces while transitionally +moving/converting from the queue_store to the public store. +See the workbench module documentation for more. + +.. automodule:: mediagoblin.tools.workbench + :members: + :show-inheritance: + + +Static assets / staticdirect +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +On top of all that, there is some static media that comes bundled with your +application. This stuff is kept in: + + mediagoblin/static/ + +These files are for mediagoblin base assets. Things like the CSS files, +logos, etc. You can mount these at whatever location is appropriate to you +(see the direct_remote_path option in the config file) so if your users +are keeping their static assets at http://static.mgoblin.example.org/ but +their actual site is at http://mgoblin.example.org/, you need to be able +to get your static files in a where-it's-mounted agnostic way. There's a +"staticdirector" attached to the request object. It's pretty easy to use; +just look at this bit taken from the +mediagoblin/templates/mediagoblin/base.html main template: + + <link rel="stylesheet" type="text/css" + href="Template:Request.staticdirect('/css/extlib/text.css')"/> + +see? Not too hard. As expected, if you configured direct_remote_path to be +http://static.mgoblin.example.org/ you'll get back +http://static.mgoblin.example.org/css/extlib/text.css just as you'd +probably expect. + +StorageInterface and implementations +------------------------------------ + +The guts of StorageInterface and friends +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +So, the StorageInterface! + +So, the public and queue stores both use StorageInterface implementations +... but what does that mean? It's not too hard. + +Open up: + + mediagoblin/storage.py + +In here you'll see a couple of things. First of all, there's the +StorageInterface class. What you'll see is that this is just a very simple +python class. A few of the methods actually implement things, but for the +most part, they don't. What really matters about this class is the +docstrings. Each expected method is documented as to how it should be +constructed. Want to make a new StorageInterface? Simply subclass it. Want +to know how to use the methods of your storage system? Read these docs, +they span all implementations. + +There are a couple of implementations of these classes bundled in +storage.py as well. The most simple of these is BasicFileStorage, which is +also the default storage system used. As expected, this stores files +locally on your machine. + +There's also a CloudFileStorage system. This provides a mapping to +[OpenStack's swift http://swift.openstack.org/] storage system (used by +RackSpace Cloud files and etc). + +Between these two examples you should be able to get a pretty good idea of +how to write your own storage systems, for storing data across your +beowulf cluster of radioactive monkey brains, whatever. + +Writing code to store stuff +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +So what does coding for StorageInterface implementations actually look +like? It's pretty simple, really. For one thing, the design is fairly +inspired by [Django's file storage API +https://docs.djangoproject.com/en/dev/ref/files/storage/]... with some +differences. + +Basically, you access files on "file paths", which aren't exactly like +unix file paths, but are close. If you wanted to store a file on a path +like dir1/dir2/filename.jpg you'd actually write that file path like: + +['dir1', 'dir2', 'filename.jpg'] + +This way we can be *sure* that each component is actually a component of +the path that's expected... we do some filename cleaning on each component. + +Your StorageInterface should pass in and out "file like objects". In other +words, they should provide .read() and .write() at minimum, and probably +also .seek() and .close(). diff --git a/docs/source/index.rst b/docs/source/index.rst index ac8bd110..7f692d57 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -44,7 +44,6 @@ MediaGoblin website. It is written for site administrators. siteadmin/relnotes siteadmin/theming siteadmin/plugins - siteadmin/codebase .. _core-plugin-section: @@ -58,6 +57,8 @@ Part 2: Core plugin documentation plugindocs/flatpagesfile plugindocs/sampleplugin plugindocs/oauth + plugindocs/trim_whitespace + plugindocs/raven Part 3: Plugin Writer's Guide @@ -70,6 +71,21 @@ This guide covers writing new GNU MediaGoblin plugins. pluginwriter/foreward pluginwriter/quickstart + pluginwriter/database + pluginwriter/api + + +Part 4: Developer's Zone +======================== + +This chapter contains various information for developers. + +.. toctree:: + :maxdepth: 1 + + devel/codebase + devel/storage + devel/originaldesigndecisions Indices and tables diff --git a/docs/source/plugindocs/raven.rst b/docs/source/plugindocs/raven.rst new file mode 100644 index 00000000..71e284d0 --- /dev/null +++ b/docs/source/plugindocs/raven.rst @@ -0,0 +1,2 @@ +.. _raven-setup: Set up the raven plugin +.. include:: ../../../mediagoblin/plugins/raven/README.rst diff --git a/docs/source/plugindocs/trim_whitespace.rst b/docs/source/plugindocs/trim_whitespace.rst new file mode 100644 index 00000000..eb38e0e5 --- /dev/null +++ b/docs/source/plugindocs/trim_whitespace.rst @@ -0,0 +1 @@ +.. include:: ../../../mediagoblin/plugins/trim_whitespace/README.rst diff --git a/docs/source/pluginwriter/api.rst b/docs/source/pluginwriter/api.rst new file mode 100644 index 00000000..5e0568fd --- /dev/null +++ b/docs/source/pluginwriter/api.rst @@ -0,0 +1,127 @@ +.. MediaGoblin Documentation + + Written in 2013 by MediaGoblin contributors + + To the extent possible under law, the author(s) have dedicated all + copyright and related and neighboring rights to this software to + the public domain worldwide. This software is distributed without + any warranty. + + You should have received a copy of the CC0 Public Domain + Dedication along with this software. If not, see + <http://creativecommons.org/publicdomain/zero/1.0/>. + + +========== +Plugin API +========== + +This documents the general plugin API. + +Please note, at this point OUR PLUGIN HOOKS MAY AND WILL CHANGE. +Authors are encouraged to develop plugins and work with the +MediaGoblin community to keep them up to date, but this API will be a +moving target for a few releases. + +Please check the release notes for updates! + +:mod:`pluginapi` Module +----------------------- + +.. automodule:: mediagoblin.tools.pluginapi + :members: get_config, register_routes, register_template_path, + register_template_hooks, get_hook_templates, + hook_handle, hook_runall, hook_transform + +Configuration +------------- + +Your plugin may define its own configuration defaults. + +Simply add to the directory of your plugin a config_spec.ini file. An +example might look like:: + + [plugin_spec] + some_string = string(default="blork") + some_int = integer(default=50) + +This means that when people enable your plugin in their config you'll +be able to provide defaults as well as type validation. + + +Context Hooks +------------- + +View specific hooks ++++++++++++++++++++ + +You can hook up to almost any template called by any specific view +fairly easily. As long as the view directly or indirectly uses the +method ``render_to_response`` you can access the context via a hook +that has a key in the format of the tuple:: + + (view_symbolic_name, view_template_path) + +Where the "view symbolic name" is the same parameter used in +``request.urlgen()`` to look up the view. So say we're wanting to add +something to the context of the user's homepage. We look in +mediagoblin/user_pages/routing.py and see:: + + add_route('mediagoblin.user_pages.user_home', + '/u/<string:user>/', + 'mediagoblin.user_pages.views:user_home') + +Aha! That means that the name is ``mediagoblin.user_pages.user_home``. +Okay, so then we look at the view at the +``mediagoblin.user_pages.user_home`` method:: + + @uses_pagination + def user_home(request, page): + # [...] whole bunch of stuff here + return render_to_response( + request, + 'mediagoblin/user_pages/user.html', + {'user': user, + 'user_gallery_url': user_gallery_url, + 'media_entries': media_entries, + 'pagination': pagination}) + +Nice! So the template appears to be +``mediagoblin/user_pages/user.html``. Cool, that means that the key +is:: + + ("mediagoblin.user_pages.user_home", + "mediagoblin/user_pages/user.html") + +The context hook uses ``hook_transform()`` so that means that if we're +hooking into it, our hook will both accept one argument, ``context``, +and should return that modified object, like so:: + + def add_to_user_home_context(context): + context['foo'] = 'bar' + return context + + hooks = { + ("mediagoblin.user_pages.user_home", + "mediagoblin/user_pages/user.html"): add_to_user_home_context} + + +Global context hooks +++++++++++++++++++++ + +If you need to add something to the context of *every* view, it is not +hard; there are two hooks hook that also uses hook_transform (like the +above) but make available what you are providing to *every* view. + +Note that there is a slight, but critical, difference between the two. + +The most general one is the ``'template_global_context'`` hook. This +one is run only once, and is read into the global context... all views +will get access to what are in this dict. + +The slightly more expensive but more powerful one is +``'template_context_prerender'``. This one is not added to the global +context... it is added to the actual context of each individual +template render right before it is run! Because of this you also can +do some powerful and crazy things, such as checking the request object +or other parts of the context before passing them on. diff --git a/docs/source/pluginwriter/database.rst b/docs/source/pluginwriter/database.rst new file mode 100644 index 00000000..58edf3a0 --- /dev/null +++ b/docs/source/pluginwriter/database.rst @@ -0,0 +1,111 @@ +.. MediaGoblin Documentation + + Written in 2013 by MediaGoblin contributors + + To the extent possible under law, the author(s) have dedicated all + copyright and related and neighboring rights to this software to + the public domain worldwide. This software is distributed without + any warranty. + + You should have received a copy of the CC0 Public Domain + Dedication along with this software. If not, see + <http://creativecommons.org/publicdomain/zero/1.0/>. + + +======== +Database +======== + + +Accessing Existing Data +======================= + +If your plugin wants to access existing data, this is quite +straight forward. Just import the appropiate models and use +the full power of SQLAlchemy. Take a look at the (upcoming) +database section in the Developer's Chapter. + + +Creating new Tables +=================== + +If your plugin needs some new space to store data, you +should create a new table. Please do not modify core +tables. Not doing so might seem inefficient and possibly +is. It will help keep things sane and easier to upgrade +versions later. + +So if you create a new plugin and need new tables, create a +file named ``models.py`` in your plugin directory. You +might take a look at the core's db.models for some ideas. +Here's a simple one: + +.. code-block:: python + + from mediagoblin.db.base import Base + from sqlalchemy import Column, Integer, Unicode, ForeignKey + + class MediaSecurity(Base): + __tablename__ = "yourplugin__media_security" + + # The primary key *and* reference to the main media_entry + media_entry = Column(Integer, ForeignKey('core__media_entries.id'), + primary_key=True) + get_media_entry = relationship("MediaEntry", + backref=backref("security_rating", cascade="all, delete-orphan")) + + rating = Column(Unicode) + + MODELS = [MediaSecurity] + +That's it. + +Some notes: + +* Make sure all your ``__tablename__`` start with your + plugin's name so the tables of various plugins can't + conflict in the database. (Conflicts in python naming are + much easier to fix later). +* Try to get your database design as good as possible in + the first attempt. Changing the database design later, + when people already have data using the old design, is + possible (see next chapter), but it's not easy. + + +Changing the Database Schema Later +================================== + +If your plugin is in use and instances use it to store some +data, changing the database design is a tricky thing. + +1. Make up your mind how the new schema should look like. +2. Change ``models.py`` to contain the new schema. Keep a + copy of the old version around for your personal + reference later. +3. Now make up your mind (possibly using your old and new + ``models.py``) what steps in SQL are needed to convert + the old schema to the new one. + This is called a "migration". +4. Create a file ``migrations.py`` that will contain all + your migrations and add your new migration. + +Take a look at the core's ``db/migrations.py`` for some +good examples on what you might be able to do. Here's a +simple one to add one column: + +.. code-block:: python + + from mediagoblin.db.migration_tools import RegisterMigration, inspect_table + from sqlalchemy import MetaData, Column, Integer + + MIGRATIONS = {} + + @RegisterMigration(1, MIGRATIONS) + def add_license_preference(db): + metadata = MetaData(bind=db.bind) + + security_table = inspect_table(metadata, 'yourplugin__media_security') + + col = Column('security_level', Integer) + col.create(security_table) + db.commit() diff --git a/docs/source/siteadmin/deploying.rst b/docs/source/siteadmin/deploying.rst index a270c723..9bcc0637 100644 --- a/docs/source/siteadmin/deploying.rst +++ b/docs/source/siteadmin/deploying.rst @@ -32,6 +32,11 @@ GNU/Linux distro. install. If instead you want to join in as a contributor, see our `Hacking HOWTO <http://wiki.mediagoblin.org/HackingHowto>`_ instead. + There are also many ways to install servers... for the sake of + simplicity, our instructions below describe installing with nginx. + For more recipes, including Apache, see + `our wiki <http://wiki.mediagoblin.org/Deployment>`_. + Prepare System -------------- @@ -165,7 +170,7 @@ And set up the in-package virtualenv:: If you have problems here, consider trying to install virtualenv with the ``--distribute`` or ``--no-site-packages`` options. If - your system's default Python is in the 3.x series you man need to + your system's default Python is in the 3.x series you may need to run ``virtualenv`` with the ``--python=python2.7`` or ``--python=python2.6`` options. @@ -173,22 +178,50 @@ The above provides an in-package install of ``virtualenv``. While this is counter to the conventional ``virtualenv`` configuration, it is more reliable and considerably easier to configure and illustrate. If you're familiar with Python packaging you may consider deploying with -your preferred the method. +your preferred method. Assuming you are going to deploy with FastCGI, you should also install flup:: ./bin/easy_install flup +(Sometimes this breaks because flup's site is flakey. If it does for +you, try):: + + ./bin/easy_install https://pypi.python.org/pypi/flup/1.0.3.dev-20110405 + This concludes the initial configuration of the development environment. In the future, when you update your codebase, you should also run:: ./bin/python setup.py develop --upgrade && ./bin/gmg dbupdate +Note: If you are running an active site, depending on your server +configuration, you may need to stop it first or the dbupdate command +may hang (and it's certainly a good idea to restart it after the +update) + + Deploy MediaGoblin Services --------------------------- +Edit site configuration +~~~~~~~~~~~~~~~~~~~~~~~ + +A few basic properties must be set before MediaGoblin will work. First +make a copy of ``mediagoblin.ini`` for editing so the original config +file isn't lost:: + + cp mediagoblin.ini mediagoblin_local.ini + +Then: + - Set ``email_sender_address`` to the address you wish to be used as + the sender for system-generated emails + - Edit ``direct_remote_path``, ``base_dir``, and ``base_url`` if + your mediagoblin directory is not the root directory of your + vhost. + + Configure MediaGoblin to use the PostgreSQL database ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -224,11 +257,11 @@ browser to confirm that the service is operable. .. _webserver-config: -Connect the Webserver to MediaGoblin with FastCGI -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -This section describes how to configure MediaGoblin to work via -FastCGI. Our configuration example will use nginx, however, you may +FastCGI and nginx +~~~~~~~~~~~~~~~~~ + +This configuration example will use nginx, however, you may use any webserver of your choice as long as it supports the FastCGI protocol. If you do not already have a web server, consider nginx, as the configuration files may be more clear than the @@ -271,6 +304,10 @@ this ``nginx.conf`` file should be modeled on the following:: # Change this to update the upload size limit for your users client_max_body_size 8m; + # prevent attacks (someone uploading a .txt file that the browser + # interprets as an HTML file, etc.) + add_header X-Content-Type-Options nosniff; + server_name mediagoblin.example.org www.mediagoblin.example.org; access_log /var/log/nginx/mediagoblin.example.access.log; error_log /var/log/nginx/mediagoblin.example.error.log; @@ -325,3 +362,24 @@ Visit the site you've set up in your browser by visiting smaller deployments. However, for larger production deployments with larger processing requirements, see the ":doc:`production-deployments`" documentation. + + +Apache +~~~~~~ + +Instructions and scripts for running MediaGoblin on an Apache server +can be found on the `MediaGoblin wiki <http://wiki.mediagoblin.org/Deployment>`_. + + +Security Considerations +~~~~~~~~~~~~~~~~~~~~~~~ + +.. warning:: + + The directory ``user_dev/crypto/`` contains some very + sensitive files. + Especially the ``itsdangeroussecret.bin`` is very important + for session security. Make sure not to leak its contents anywhere. + If the contents gets leaked nevertheless, delete your file + and restart the server, so that it creates a new secret key. + All previous sessions will be invalifated then. diff --git a/docs/source/siteadmin/media-types.rst b/docs/source/siteadmin/media-types.rst index 5653217f..210094b9 100644 --- a/docs/source/siteadmin/media-types.rst +++ b/docs/source/siteadmin/media-types.rst @@ -43,6 +43,15 @@ video media types, then the list would look like this:: media_types = mediagoblin.media_types.image, mediagoblin.media_types.video +Note that after enabling new media types, you must run dbupdate like so:: + + ./bin/gmg dbupdate + +If you are running an active site, depending on your server +configuration, you may need to stop it first (and it's certainly a +good idea to restart it after the update). + + How does MediaGoblin decide which media type to use for a file? =============================================================== @@ -62,12 +71,27 @@ Video To enable video, first install gstreamer and the python-gstreamer bindings (as well as whatever gstremaer extensions you want, -good/bad/ugly). On Debianoid systems:: +good/bad/ugly). On Debianoid systems - sudo apt-get install python-gst0.10 gstreamer0.10-plugins-{base,bad,good,ugly} \ +.. code-block:: bash + + sudo apt-get install python-gst0.10 \ + gstreamer0.10-plugins-base \ + gstreamer0.10-plugins-bad \ + gstreamer0.10-plugins-good \ + gstreamer0.10-plugins-ugly \ gstreamer0.10-ffmpeg +Add ``mediagoblin.media_types.video`` to the ``media_types`` list in your +``mediagoblin_local.ini`` and restart MediaGoblin. + +Run + +.. code-block:: bash + + ./bin/gmg dbupdate + Now you should be able to submit videos, and mediagoblin should transcode them. @@ -92,7 +116,9 @@ To install these on Debianoid systems, run:: The ``scikits.audiolab`` package you will install in the next step depends on the ``libsndfile1-dev`` package, so we should install it. -On Debianoid systems, run:: +On Debianoid systems, run + +.. code-block:: bash sudo apt-get install libsndfile1-dev @@ -108,8 +134,15 @@ Then install ``scikits.audiolab`` for the spectrograms:: ./bin/pip install scikits.audiolab Add ``mediagoblin.media_types.audio`` to the ``media_types`` list in your -``mediagoblin_local.ini`` and restart MediaGoblin. You should now be able to -upload and listen to audio files! +``mediagoblin_local.ini`` and restart MediaGoblin. + +Run + +.. code-block:: bash + + ./bin/gmg dbupdate + +You should now be able to upload and listen to audio files! Ascii art @@ -117,7 +150,9 @@ Ascii art To enable ascii art support, first install the `chardet <http://pypi.python.org/pypi/chardet>`_ -library, which is necessary for creating thumbnails of ascii art:: +library, which is necessary for creating thumbnails of ascii art + +.. code-block:: bash ./bin/easy_install chardet @@ -131,4 +166,69 @@ the list would look like this:: media_types = mediagoblin.media_types.image, mediagoblin.media_types.ascii +Run + +.. code-block:: bash + + ./bin/gmg dbupdate + Now any .txt file you uploaded will be processed as ascii art! + + +STL / 3d model support +====================== + +To enable the "STL" 3d model support plugin, first make sure you have +a recentish `Blender <http://blender.org>`_ installed and available on +your execution path. This feature has been tested with Blender 2.63. +It may work on some earlier versions, but that is not guaranteed (and +is surely not to work prior to Blender 2.5X). + +Add ``mediagoblin.media_types.stl`` to the ``media_types`` list in your +``mediagoblin_local.ini`` and restart MediaGoblin. + +Run + +.. code-block:: bash + + ./bin/gmg dbupdate + +You should now be able to upload .obj and .stl files and MediaGoblin +will be able to present them to your wide audience of admirers! + +PDF and Document +================ + +To enable the "PDF and Document" support plugin, you need pdftocairo, pdfinfo, +unoconv with headless support. All executables must be on your execution path. + +To install this on Fedora: + +.. code-block:: bash + + sudo yum install -y poppler-utils unoconv libreoffice-headless + +pdf.js relies on git submodules, so be sure you have fetched them: + +.. code-block:: bash + + git submodule init + git submodule update + +This feature has been tested on Fedora with: + poppler-utils-0.20.2-9.fc18.x86_64 + unoconv-0.5-2.fc18.noarch + libreoffice-headless-3.6.5.2-8.fc18.x86_64 + +It may work on some earlier versions, but that is not guaranteed. + +Add ``mediagoblin.media_types.pdf`` to the ``media_types`` list in your +``mediagoblin_local.ini`` and restart MediaGoblin. + +Run + +.. code-block:: bash + + ./bin/gmg dbupdate + + diff --git a/docs/source/siteadmin/plugins.rst b/docs/source/siteadmin/plugins.rst index 75562d4b..baca381d 100644 --- a/docs/source/siteadmin/plugins.rst +++ b/docs/source/siteadmin/plugins.rst @@ -44,29 +44,33 @@ If the plugin is available on the `Python Package Index pip install <plugin-name> For example, if we wanted to install the plugin named -"mediagoblin-restrictfive", we would do:: +"mediagoblin-licenses" (which allows you to customize the licenses you +offer for your media), we would do:: - pip install mediagoblin-restrictfive + pip install mediagoblin-licenses .. Note:: If you're using a virtual environment, make sure to activate the - virtual environment before installing with pip. Otherwise the - plugin may get installed in a different environment than the one - MediaGoblin is installed in. + virtual environment before installing with pip. Otherwise the plugin + may get installed in a different environment than the one MediaGoblin + is installed in. Also make sure, you use e.g. pip-2.7 if your default + python (and thus pip) is python 3 (e.g. in Ubuntu). Once you've installed the plugin software, you need to tell MediaGoblin that this is a plugin you want MediaGoblin to use. To do that, you edit the ``mediagoblin.ini`` file and add the plugin as a subsection of the plugin section. -For example, say the "mediagoblin-restrictfive" plugin had the Python -package path ``restrictfive``, then you would add ``restrictfive`` to +For example, say the "mediagoblin-licenses" plugin has the Python +package path ``mediagoblin_licenses``, then you would add ``mediagoblin_licenses`` to the ``plugins`` section as a subsection:: [plugins] - [[restrictfive]] + [[mediagoblin_licenses]] + license_01=abbrev1, name1, http://url1 + license_02=abbrev2, name1, http://url2 Configuring plugins @@ -112,7 +116,7 @@ Removing plugins To remove a plugin, use ``pip uninstall``. For example:: - pip uninstall mediagoblin-restrictfive + pip uninstall mediagoblin-licenses .. Note:: diff --git a/docs/source/siteadmin/production-deployments.rst b/docs/source/siteadmin/production-deployments.rst index 356fab7f..1a32d95e 100644 --- a/docs/source/siteadmin/production-deployments.rst +++ b/docs/source/siteadmin/production-deployments.rst @@ -52,7 +52,7 @@ as the basis for your script: :: Separate Celery --------------- -While the ``./lazyserer.sh`` configuration provides an efficient way to +While the ``./lazyserver.sh`` configuration provides an efficient way to start using a MediaGoblin instance, it is not suitable for production deployments for several reasons: @@ -77,6 +77,17 @@ Modify your existing MediaGoblin and application init scripts, if necessary, to prevent them from starting their own ``celeryd`` processes. +.. _sentry: + +Set up sentry to monitor exceptions +----------------------------------- + +We have a plugin for `raven`_ integration, see the ":doc:`/plugindocs/raven`" +documentation. + +.. _`raven`: http://raven.readthedocs.org + + .. _init-script: Use an Init Script diff --git a/docs/source/siteadmin/relnotes.rst b/docs/source/siteadmin/relnotes.rst index 1b70b31c..04863ec6 100644 --- a/docs/source/siteadmin/relnotes.rst +++ b/docs/source/siteadmin/relnotes.rst @@ -19,6 +19,166 @@ This chapter has important information for releases in it. If you're upgrading from a previous release, please read it carefully, or at least skim over it. +0.3.3 +===== + +**Do this to upgrade** + +1. Make sure to run ``bin/gmg dbupdate`` after upgrading. +2. OpenStreetMap is now a plugin, so if you want to use it, add the + following to your config file: + + .. code-block:: ini + + [plugins] + [[mediagoblin.plugins.geolocation]] + +If you have your own theme, you may need to make some adjustments to +it as some theme related things may have changed in this release. If +you run into problems, don't hesitate to +`contact us <http://mediagoblin.org/pages/join.html>`_ +(IRC is often best). + +**New features** + +* New dropdown menu for accessing various features. + +* Significantly improved URL generation. Now mediagoblin won't give + up on making a slug if it looks like there will be a duplicate; + it'll try extra hard to generate a meaningful one instead. + + Similarly, linking to an id no longer can possibly conflict with + linking to a slug; /u/username/m/id:35/ is the kind of reference we + now use to linking to entries with ids. However, old links with + entries that linked to ids should work just fine with our migration. + The only urls that might break in this release are ones using colons + or equal signs. + +* New template hooks for plugin authoring. + +* As a demonstration of new template hooks for plugin authoring, + openstreetmap support now moved to a plugin! + +* Method to add media to collections switched from icon of paperclip + to button with "add to collection" text. + +* Bug where videos often failed to produce a proper thumbnail fixed! + +* Copying around files in MediaGoblin now much more efficient, doesn't + waste gobs of memory. + +* Video transcoding now optional for videos that meet certain + criteria. By default, MediaGoblin will not transcode webm videos + that are smaller in resolution than the MediaGoblin defaults, and + MediaGoblin can also be configured to allow theora files to not be + transcoded as well. + +* Per-user license preference option; always want your uploads to be + BY-SA and tired of changing that field? You can now set your + license preference in your user settings. + +* Video player now responsive; better for mobile! + +* You can now delete your account from the user preferences page if + you so wish. + +**Other changes** + +* Plugin writers: Internal restructuring led to mediagoblin.db.sql* be + mediagoblin.db.* starting from 0.3.3 + +* Dependency list has been reduced not requiring the "webob" package anymore. + +* And many small fixes/improvements, too numerous to list! + + +0.3.2 +===== + +This will be the last release that is capable of converting from an earlier +MongoDB-based MediaGoblin instance to the newer SQL-based system. + +**Do this to upgrade** + + # directory of your mediagoblin install + cd /srv/mediagoblin.example.org + + # copy source for this release + git fetch + git checkout tags/v0.3.2 + + # perform any needed database updates + bin/gmg dbupdate + + # restart your servers however you do that, e.g., + sudo service mediagoblin-paster restart + sudo service mediagoblin-celeryd restart + + +**New features** + +* **3d model support!** + + You can now upload STL and OBJ files and display them in + MediaGoblin. Requires a recent-ish Blender; for details see: + :ref:`deploying-chapter` + +* **trim_whitespace** + + We bundle the optional plugin trim_whitespace which reduces the size + of the delivered html output by reducing redundant whitespace. + + See :ref:`core-plugin-section` for plugin documentation + +* **A new API!** + + It isn't well documented yet but we do have an API. There is an + `android application in progress <https://gitorious.org/mediagoblin/mediagoblin-android>`_ + which makes use of it, and there are some demo applications between + `automgtic <https://github.com/jwandborg/automgtic>`_, an + automatic media uploader for your desktop + and `OMGMG <https://github.com/jwandborg/omgmg>`_, an example of + a web application hooking up to the API. + + This is a plugin, so you have to enable it in your mediagoblin + config file by adding a section under [plugins] like:: + + [plugins] + [[mediagoblin.plugins.api]] + + Note that the API works but is not nailed down... the way it is + called may change in future releases. + +* **OAuth login support** + + For applications that use OAuth to connect to the API. + + This is a plugin, so you have to enable it in your mediagoblin + config file by adding a section under [plugins] like:: + + [plugins] + [[mediagoblin.plugins.oauth]] + +* **Collections** + + We now have user-curated collections support. These are arbitrary + galleries that are customizable by users. You can add media to + these by clicking on the paperclip icon when logged in and looking + at a media entry. + +* **OpenStreetMap licensing display improvements** + + More accurate display of OSM licensing, and less disruptive: you + click to "expand" the display of said licensing. + + Geolocation is also now on by default. + +* **Miscelaneous visual improvements** + + We've made a number of small visual improvements including newer and + nicer looking thumbnails and improved checkbox placement. + + 0.3.1 ===== |