Merge branch 'master' into merge-python3-port

Has some issues, will iteratively fix!

Conflicts:
	mediagoblin/gmg_commands/__init__.py
	mediagoblin/gmg_commands/deletemedia.py
	mediagoblin/gmg_commands/users.py
	mediagoblin/oauth/views.py
	mediagoblin/plugins/api/views.py
	mediagoblin/tests/test_api.py
	mediagoblin/tests/test_edit.py
	mediagoblin/tests/test_oauth1.py
	mediagoblin/tests/test_util.py
	mediagoblin/tools/mail.py
	mediagoblin/webfinger/views.py
	setup.py

8 files changed, 468 insertions, 18 deletions

diff --git a/docs/source/api/media.rst b/docs/source/api/media.rst
new file mode 100644
index 00000000..bafe43d3
--- /dev/null
+++ b/docs/source/api/media.rst
@@ -0,0 +1,155 @@
+.. MediaGoblin Documentation
+
+   Written in 2011, 2012 by MediaGoblin contributors
+
+   To the extent possible under law, the author(s) have dedicated all
+   copyright and related and neighboring rights to this software to
+   the public domain worldwide. This software is distributed without
+   any warranty.
+
+   You should have received a copy of the CC0 Public Domain
+   Dedication along with this software. If not, see
+   <http://creativecommons.org/publicdomain/zero/1.0/>.
+
+.. info:: Currently only image uploading is supported.
+
+===============
+Uploading Media
+===============
+
+To use any the APIs mentioned in this document you will required :doc:`oauth`
+
+Uploading and posting an media requiest you to make two to three requests:
+
+1) Uploads the data to the server
+2) Post media to feed
+3) Update media to have title, description, license, etc. (optional)
+
+These steps could be condenced in the future however currently this is how the
+pump.io API works. There is currently an issue open, if you would like to change
+how this works please contribute upstream: https://github.com/e14n/pump.io/issues/657
+
+----------------------
+Upload Media to Server
+----------------------
+
+To upload media you should use the URI `/api/user/<username>/uploads`.
+
+A POST request should be made to the media upload URI submitting at least two header:
+
+* `Content-Type` - This being a valid mimetype for the media.
+* `Content-Length` - size in bytes of the media.
+
+The media data should be submitted as POST data to the image upload URI.
+You will get back a JSON encoded response which will look similiar to::
+
+    {
+        "updated": "2014-01-11T09:45:48Z",
+        "links": {
+            "self": {
+                "href": "https://<server>/image/4wiBUV1HT8GRqseyvX8m-w"
+            }
+        },
+        "fullImage": {
+            "url": "https://<server>//uploads/<username>/2014/1/11/V3cBMw.jpg",
+            "width": 505,
+            "height": 600
+        },
+        "replies": {
+            "url": "https://<server>//api/image/4wiBUV1HT8GRqseyvX8m-w/replies"
+        },
+        "image": {
+            "url": "https://<server>/uploads/<username>/2014/1/11/V3cBMw_thumb.jpg",
+            "width": 269,
+            "height": 320
+        },
+        "author": {
+            "preferredUsername": "<username>",
+            "displayName": "<username>",
+            "links": {
+                "activity-outbox": {
+                    "href": "https://<server>/api/user/<username>/feed"
+                },
+                "self": {
+                    "href": "https://<server>/api/user/<username>/profile"
+                },
+                "activity-inbox": {
+                    "href": "https://<server>/api/user/<username>/inbox"
+                }
+            },
+            "url": "https://<server>/<username>",
+            "updated": "2013-08-14T10:01:21Z",
+            "id": "acct:<username>@<server>",
+            "objectType": "person"
+        },
+        "url": "https://<server>/<username>/image/4wiBUV1HT8GRqseyvX8m-w",
+        "published": "2014-01-11T09:45:48Z",
+        "id": "https://<server>/api/image/4wiBUV1HT8GRqseyvX8m-w",
+        "objectType": "image"
+    }
+
+The main things in this response is `fullImage` which contains `url` (the URL
+of the original image - i.e. fullsize) and `image` which contains `url` (the URL
+of a thumbnail version).
+
+.. warning:: Media which have been uploaded but not submitted to a feed will
+             periodically be deleted.
+
+--------------
+Submit to feed
+--------------
+
+This is submitting the media to appear on the website. This will create an
+object in your feed which will then appear on the GNU MediaGoblin website so the
+user and others can view and interact with the media.
+
+The URL you need to POST to is `/api/user/<username>/feed`
+
+You first should do a post to the feed URI with some of the information you got
+back from the above request (which uploaded the media). The request should look
+something like::
+
+    {
+        "verb": "post",
+        "object": {
+            "id": "https://<server>/api/image/6_K9m-2NQFi37je845c83w",
+            "objectType": "image"
+        }
+    }
+
+.. warning:: Any other data submitted **will** be ignored
+
+-------------------
+Submitting Metadata
+-------------------
+
+Finally if you wish to set a title, description and license you will need to do
+and update request to the endpoint, the following attributes can be submitted:
+
++--------------+---------------------------------------+-------------------+
+| Name         | Description                           | Required/Optional |
++==============+=======================================+===================+
+| displayName  | This is the title for the media       | Optional          |
++--------------+---------------------------------------+-------------------+
+| content      | This is the description for the media | Optional          |
++--------------+---------------------------------------+-------------------+
+| license      | This is the license to be used        | Optional          |
++--------------+---------------------------------------+-------------------+
+
+.. note:: license attribute is mediagoblin specific, pump.io does not support this attribute
+
+
+The update request should look something similiar to::
+
+    {
+        "verb": "update",
+        "object": {
+            "displayName": "My super awesome image!",
+            "content": "The awesome image I took while backpacking to modor",
+            "license": "creativecommons.org/licenses/by-sa/3.0/",
+            "id": "https://<server>/api/image/6_K9m-2NQFi37je845c83w",
+            "objectType": "image"
+        }
+    }
+
+.. warning:: Any other data submitted **will** be ignored.
diff --git a/docs/source/api/media_interaction.rst b/docs/source/api/media_interaction.rst
new file mode 100644
index 00000000..41114a71
--- /dev/null
+++ b/docs/source/api/media_interaction.rst
@@ -0,0 +1,65 @@
+.. MediaGoblin Documentation
+
+   Written in 2011, 2012 by MediaGoblin contributors
+
+   To the extent possible under law, the author(s) have dedicated all
+   copyright and related and neighboring rights to this software to
+   the public domain worldwide. This software is distributed without
+   any warranty.
+
+   You should have received a copy of the CC0 Public Domain
+   Dedication along with this software. If not, see
+   <http://creativecommons.org/publicdomain/zero/1.0/>.
+
+Pump.io supports a number of different interactions that can happen against
+media. Theser are commenting, liking/favoriting and (re-)sharing. Currently
+MediaGoblin supports just commenting although other interactions will come at
+a later date.
+
+--------------
+How to comment
+--------------
+
+.. warning:: Commenting on a comment currently is NOT supported.
+
+Commenting is done by posting a comment activity to the users feed. The
+activity should look similiar to::
+
+    {
+        "verb": "post",
+        "object": {
+            "objectType": "comment",
+            "inReplyTo": <media>
+        }
+    }
+
+This is where `<media>` is the media object you have got with from the server.
+
+----------------
+Getting comments
+----------------
+
+The media object you get back should have a `replies` section. This should
+be an object which contains the number of replies and if there are any (i.e.
+number of replies > 0) then `items` will include an array of every item::
+
+    {
+        "totalItems": 2,
+        "items: [
+            {
+                "id": 1,
+                "objectType": "comment",
+                "content": "I'm a comment ^_^",
+                "author": <author user object>
+            },
+            {
+                "id": 4,
+                "objectType": "comment",
+                "content": "Another comment! Blimey!",
+                "author": <author user object>
+            }
+        ],
+        "url": "http://some.server/api/images/1/comments/"
+    }
+
+
diff --git a/docs/source/index.rst b/docs/source/index.rst
index 3ead6136..8e49d1d1 100644
--- a/docs/source/index.rst
+++ b/docs/source/index.rst
@@ -78,6 +78,7 @@ This guide covers writing new GNU MediaGoblin plugins.
    pluginwriter/database
    pluginwriter/api
    pluginwriter/tests
+   pluginwriter/hooks
    pluginwriter/media_type_hooks
    pluginwriter/authhooks
 
@@ -96,6 +97,26 @@ This chapter contains various information for developers.
    devel/migrations
 
 
+Part 5: Pump API
+================
+
+This chapter covers MediaGoblin's `Pump API
+<https://github.com/e14n/pump.io/blob/master/API.md>`_ support.  (A
+work in progress; full federation is not supported at the moment, but
+media uploading works!  You can use something like
+`PyPump <http://pypump.org>`_
+to write MediaGoblin uploadable applications.)
+
+.. toctree::
+   :maxdepth: 1
+
+   api/client_register
+   api/oauth
+   api/media
+   api/media_interaction
+
+
+
 Indices and tables
 ==================
 
diff --git a/docs/source/pluginwriter/hooks.rst b/docs/source/pluginwriter/hooks.rst
new file mode 100644
index 00000000..4aa062e8
--- /dev/null
+++ b/docs/source/pluginwriter/hooks.rst
@@ -0,0 +1,35 @@
+.. MediaGoblin Documentation
+
+   Written in 2014 by MediaGoblin contributors
+
+   To the extent possible under law, the author(s) have dedicated all
+   copyright and related and neighboring rights to this software to
+   the public domain worldwide. This software is distributed without
+   any warranty.
+
+   You should have received a copy of the CC0 Public Domain
+   Dedication along with this software. If not, see
+   <http://creativecommons.org/publicdomain/zero/1.0/>.
+
+
+===============================
+Documentation on Built-in Hooks
+===============================
+
+This section explains built-in hooks to MediaGoblin.
+
+
+What hooks are available?
+=========================
+
+'collection_add_media'
+----------------------
+
+This hook is used by ``add_media_to_collection``
+in ``mediagoblin.user_pages.lib``.
+It gets a ``CollectionItem`` as its argument.
+It's the newly created item just before getting commited.
+So the item can be modified by the hook, if needed.
+Changing the session regarding this item is currently
+undefined behaviour, as the SQL Session might contain other
+things.
diff --git a/docs/source/siteadmin/commandline-upload.rst b/docs/source/siteadmin/commandline-upload.rst
index be19df58..69098312 100644
--- a/docs/source/siteadmin/commandline-upload.rst
+++ b/docs/source/siteadmin/commandline-upload.rst
@@ -15,7 +15,13 @@
 Command-line uploading
 ======================
 
-Want to submit media via the command line?  It's fairly easy to do::
+If you're a site administrator and have access to the server then you
+can use the 'addmedia' task. If you're just a user and want to upload
+media by the command line you can. This can be done with the pump.io
+API. There is `p <https://github.com/xray7224/p/>`_, which will allow you
+to easily upload media from the command line, follow p's docs to do that.
+
+To use the addmedia command::
 
   ./bin/gmg addmedia username your_media.jpg
 
@@ -39,3 +45,70 @@ You can also pass in the `--celery` option if you would prefer that
 your media be passed over to celery to be processed rather than be
 processed immediately.
 
+============================
+Command-line batch uploading
+============================
+
+There's another way to submit media, and it can be much more powerful, although
+it is a bit more complex.
+
+  ./bin/gmg batchaddmedia admin /path/to/your/metadata.csv
+
+This is an example of what a script may look like. The important part here is
+that you have to create the 'metadata.csv' file.::
+
+  media:location,dcterms:title,dcterms:creator,dcterms:type
+  "http://www.example.net/path/to/nap.png","Goblin taking a nap",,"Image"
+  "http://www.example.net/path/to/snore.ogg","Goblin Snoring","Me","Audio"
+
+The above is an example of a very simple metadata.csv file. The batchaddmedia
+script would read this and attempt to upload only two pieces of media, and would
+be able to automatically name them appropriately.
+
+The csv file
+============
+The location column
+-------------------
+The location column is the one column that is absolutely necessary for
+uploading your media. This gives a path to each piece of media you upload. This
+can either a path to a local file or a direct link to remote media (with the
+link in http format). As you can see in the example above the (fake) media was
+stored remotely on "www.example.net".
+
+Other internal nodes
+--------------------
+There are other columns which can be used by the script to provide information.
+These are not stored as part of the media's metadata. You can use these columns to
+provide default information for your media entry, but as you'll see below, it's
+just as easy to provide this information through the correct metadata columns.
+
+- **id** is used to identify the media entry to the user in case of an error in the batchaddmedia script.
+- **license** is used to set a license for your piece a media for mediagoblin's use. This must be a URI.
+- **title** will set the title displayed to mediagoblin users.
+- **description** will set a description of your media.
+
+Metadata columns
+----------------
+Other columns can be used to provide detailed metadata about each media entry.
+Our metadata system accepts any information provided for in the
+`RDFa Core Initial Context`_, and the batchupload script recognizes all of the
+resources provided within it.
+
+.. _RDFa Core Initial Context: http://www.w3.org/2011/rdfa-context/rdfa-1.1
+
+The uploader may include the metadata for each piece of media, or
+leave them blank if they want to. A few columns from `Dublin Core`_ are
+notable because the batchaddmedia script also uses them to set the default
+information of uploaded media entries.
+
+.. _Dublin Core: http://wiki.dublincore.org/index.php/User_Guide
+
+- **dc:title** sets a title for your media entry.
+- **dc:description** sets a description of your media entry.
+
+If both a metadata column and an internal node for the title are provided, mediagoblin
+will use the internal node as the media entry's display name. This makes it so
+that if you want to display a piece of media with a different title
+than the one provided in its metadata, you can just provide different data for
+the 'dc:title' and 'title' columns. The same is true of the 'description' and
+'dc:description'.
diff --git a/docs/source/siteadmin/deploying.rst b/docs/source/siteadmin/deploying.rst
index 0dde3b6a..ad68c897 100644
--- a/docs/source/siteadmin/deploying.rst
+++ b/docs/source/siteadmin/deploying.rst
@@ -165,11 +165,11 @@ to the unpriviledged system account.
 To do this, enter either of the following commands, changing the defaults
 to suit your particular requirements::
 
-  sudo mkdir -p /srv/mediagoblin.example.org && sudo chown -hR mediagoblin:mediagoblin /srv/mediagoblin.example.org
+  sudo mkdir -p /srv/mediagoblin.example.org && sudo chown -hR mediagoblin: /srv/mediagoblin.example.org
 
 or (as the root user)::
 
-  mkdir -p /srv/mediagoblin.example.org && chown -hR mediagoblin:mediagoblin /srv/mediagoblin.example.org
+  mkdir -p /srv/mediagoblin.example.org && chown -hR mediagoblin: /srv/mediagoblin.example.org
 
 
 Install MediaGoblin and Virtualenv
@@ -200,7 +200,7 @@ Clone the MediaGoblin repository and set up the git submodules::
 
 And set up the in-package virtualenv::
 
-    (virtualenv --system-site-packages . || virtualenv .) && ./bin/python setup.py develop
+    (virtualenv --python=python2 --system-site-packages . || virtualenv --python=python2 .) && ./bin/python setup.py develop
 
 .. note::
 
@@ -214,16 +214,6 @@ And set up the in-package virtualenv::
 
    Note: this is liable to break.  Use this method with caution.
 
-.. ::
-
-   (NOTE: Is this still relevant?)
-
-   If you have problems here, consider trying to install virtualenv
-   with the ``--distribute`` or ``--no-site-packages`` options. If
-   your system's default Python is in the 3.x series you may need to
-   run ``virtualenv`` with the  ``--python=python2.7`` or
-   ``--python=python2.6`` options.
-
 The above provides an in-package install of ``virtualenv``. While this
 is counter to the conventional ``virtualenv`` configuration, it is
 more reliable and considerably easier to configure and illustrate. If
@@ -244,7 +234,7 @@ This concludes the initial configuration of the development
 environment. In the future, when you update your
 codebase, you should also run::
 
-    ./bin/python setup.py develop --upgrade && ./bin/gmg dbupdate && git submodule fetch
+    git submodule update && ./bin/python setup.py develop --upgrade && ./bin/gmg dbupdate
 
 Note: If you are running an active site, depending on your server
 configuration, you may need to stop it first or the dbupdate command
diff --git a/docs/source/siteadmin/media-types.rst b/docs/source/siteadmin/media-types.rst
index 3e8a94e9..f8030081 100644
--- a/docs/source/siteadmin/media-types.rst
+++ b/docs/source/siteadmin/media-types.rst
@@ -1,6 +1,6 @@
 .. MediaGoblin Documentation
 
-   Written in 2011, 2012 by MediaGoblin contributors
+   Written in 2011, 2012, 2014 by MediaGoblin contributors
 
    To the extent possible under law, the author(s) have dedicated all
    copyright and related and neighboring rights to this software to
@@ -18,8 +18,8 @@ Media Types
 ====================
 
 In the future, there will be all sorts of media types you can enable,
-but in the meanwhile there are five additional media types: video, audio,
-ascii art, STL/3d models, PDF and Document.
+but in the meanwhile there are six additional media types: video, audio,
+raw image, ascii art, STL/3d models, PDF and Document.
 
 First, you should probably read ":doc:`configuration`" to make sure
 you know how to modify the mediagoblin config file.
@@ -149,6 +149,28 @@ Run
 You should now be able to upload and listen to audio files!
 
 
+Raw image
+=========
+
+To enable raw image you need to install pyexiv2.  On Debianoid systems
+
+.. code-block:: bash
+
+    sudo apt-get install python-pyexiv2
+
+Add ``[[mediagoblin.media_types.raw_image]]`` under the ``[plugins]``
+section in your ``mediagoblin_local.ini`` and restart MediaGoblin.
+
+Run
+
+.. code-block:: bash
+
+    ./bin/gmg dbupdate
+
+Now you should be able to submit raw images, and mediagoblin should
+extract the JPEG preview from them.
+
+
 Ascii art
 =========
 
@@ -242,3 +264,13 @@ Run
     ./bin/gmg dbupdate
 
 
+Blog (HIGHLY EXPERIMENTAL)
+==========================
+
+MediaGoblin has a blog media type, which you might notice by looking
+through the docs!  However, it is *highly experimental*.  We have not
+security reviewed this, and it acts in a way that is not like normal
+blogs (the blogposts are themselves media types!).
+
+So you can play with this, but it is not necessarily recommended yet
+for production use! :)
diff --git a/docs/source/siteadmin/relnotes.rst b/docs/source/siteadmin/relnotes.rst
index 3542bdcb..ff701b1f 100644
--- a/docs/source/siteadmin/relnotes.rst
+++ b/docs/source/siteadmin/relnotes.rst
@@ -21,6 +21,85 @@ This chapter has important information for releases in it.
 If you're upgrading from a previous release, please read it
 carefully, or at least skim over it.
 
+0.7.0
+====
+
+**Do this to upgrade**
+
+1. Update to the latest release.  If checked out from git, run:
+   ``git fetch && git checkout -q v0.7.0 && git submodule init && git submodule update``
+2. Make sure to run
+   ``./bin/python setup.py develop --upgrade && ./bin/gmg dbupdate``
+
+(NOTE: earlier versions of the 0.7.0 release instructions left out the
+``git submodule init`` step!  If you did an upgrade earlier based on
+these instructions and your theme looks weirdly aligned, try running
+the following:)
+
+  ``git submodule init && git submodule update``
+
+That's it, probably!  If you run into problems, don't hesitate to
+`contact us <http://mediagoblin.org/pages/join.html>`_
+(IRC is often best).
+
+**New features:**
+
+- New mobile upload API making use of the
+  `Pump API <https://github.com/e14n/pump.io/blob/master/API.md>`_
+  (which will be the foundation for MediaGoblin's federation)
+- New theme: Sandy 70s Speedboat!
+
+- Metadata features!  We also now have a json-ld context. 
+
+- Many improvements for archival institutions, including metadata
+  support and featuring items on the homepage.  With the (new!)
+  archivalook plugin enabled, featuring media is possible.
+  Additionally, metadata about the particular media item will show up
+  in the sidebar.
+
+  In the future these plugins may be separated, but for now they have
+  come together as part of the same plugin.
+
+- There is a new gmg subcommand called batchaddmedia that allows for
+  uploading many files at once.  This is aimed to be useful for
+  archival institutions and groups where there is an already existing
+  and large set of available media that needs to be included.
+- Speaking of, the call to postgres in the makefile is fixed.
+- We have a new, generic media-page context hook that allows for
+  adding context depending on the type of media.
+- Tired of video thumbnails breaking during processing all the time?
+  Good news, everyone!  Video thumbnail generation should not fail
+  frequently anymore.  (We think...)
+- You can now set default permissions for new users in the config.
+
+- bootstrap.sh / gnu configuration stuff still exists, but moves to be
+  experimental-bootstrap.sh so as to not confuse newcomers.  There are
+  some problems currently with the autoconf stuff that we need to work
+  out... we still have interest in supporting it, though help is
+  welcome.
+
+- MediaGoblin now checks whether or not the database is up to date
+  when starting.
+- Switched to `Skeleton <http://www.getskeleton.com/>`_ as a system for
+  graphic design.
+- New gmg subcommands for administrators:
+  - A "deletemedia" command
+  - A "deleteuser" command
+- We now have a blogging media type... it's very experimental,
+  however.  Use with caution!
+- We have switched to exifread as an external library for reading EXIF
+  data.  It's basically the same thing as before, but packaged
+  separately from MediaGoblin.
+- Many improvements to internationalization.  Also (still rudimentary,
+  but existant!) RTL language support!
+
+**Known issues:**
+ - The host-meta is now json by default; in the spec it should be xml by
+   default.  We have done this because of compatibility with the pump
+   API.  We are checking with upstream to see if there is a way to
+   resolve this discrepancy.
+
+
 0.6.1
 =====