Interfacing DocMoto with a third party application using Python
You can download the key scripts from here .
The article demonstrates how DocMoto can be accessed using standard WebDAV calls.
For the purposes of a demonstration we are going to create a scripts that hooks into the popular "Dropbox" web application and copy any modifications to a DocMoto server.
The Dropbox API
Dropbox has an API which is specifically designed to allow developers to interface with the service.
Python is a very popular programming language. It is well supported on a Mac, and most cloud service providers who publish APIs have Python based code libraries or examples.
Since Dropbox's Python support is particulary good we decided to use this to create our sample application.
Essentially we want to create a system whereby changes made to a Dropbox account are reflected in a DocMoto folder.
Fortunately the Dropbox API has the very useful delta function which provides a list of all changes made to an account since the last query to the function.
DocMoto will automatically version documents if a connection is made via port 4983 (non-secure) or 4984 (secure).
When hooked up to Dropbox this means a new version will be added to a file's version history within DocMoto whenever Dropbox detects the file has changed!
DocMoto has an API, which is based around the fact that DocMoto is, at heart, a WebDAV server.
So the first task is to create a simple library of DocMoto function calls using Python.
LwpMotoUtils.py is a small library of utilities that allow you to connect to a DocMoto server and perform simple operations.
The library uses Python's popular urllib2 networking package which is installed by default on a Mac.
To overcome some issues with moving large files LwpMotoUtils uses a custom urllib2 handler CHLHTTPDigestAuthHandler which is an extension of urllib2's standard HTTPDigestAuthHandler .
Included in the Dropbox SDK download is an example script search_cache.py . This demonstrates how to use the delta function to maintain an accurate copy (cache) of the Dropbox folder and file structure.
The example does something very similar to our objective already, all we really need to do is replace the cache with a DocMoto folder. DropBoxDocMotoUtils.py is the result.
The work is done by the update function. This connects to the Dropbox account, calls the delta function and decides whether or not any processing is required.
def update(count=None): page_limit = None # Load state state = load_state() access_token = state['access_token'] cursor = state.get('cursor') # Connect to Dropbox try: sess = dropbox.session.DropboxSession(APP_KEY, APP_SECRET, ACCESS_TYPE) sess.set_token(*access_token) c = dropbox.client.DropboxClient(sess) except dropbox.rest.ErrorResponse, e: print "Update process error: " + str(e.status) + " " + str(e.reason) + "n" return except: print "Unexpected update process errorn" return page = 0 changed = False while (page_limit is None) or (page < page_limit): # Get /delta results from Dropbox try: result = c.delta(cursor) except dropbox.rest.ErrorResponse, e: print "Dropbox delta return error: " + str(e.status) + " " + str(e.reason) + "n" return except: print "Unexpected update process Dropbox delta return errorn" return page += 1 if result['reset'] == True: sys.stdout.write('resetn') changed = True cursor = result['cursor'] # Apply the entries one by one to DocMoto. for delta_entry in result['entries']: changed = True apply_delta(delta_entry, c) cursor = result['cursor'] if not result['has_more']: break if not changed: sys.stdout.write("No updates " + str(count) + "n") else: # Save state state['cursor'] = cursor save_state(state)
The replication of the Dropbox structure is performed by the apply_delta function.
apply_delta makes use of several LwpMotoUtils functions to create folders, delete folders and files. Copying a file from Dropbox to DocMoto is a three stage process:
- get the file from Dropbox and write it to a temporary folder, using writeFile
- send the file to DocMoto using sendFile
- remove the file from the temporary folder using removeFile
def apply_delta(entry, c): path, metadata = entry branch, leaf = split_path(path) root = DOCMOTO_SERVER_URL + "/" + DOCMOTO_BASE_FOLDER # Securely connect to DocMoto myRet = LwpMotoUtils.secureinitialize(DOCMOTO_USERNAME,DOCMOTO_PWD,DOCMOTO_SERVER_URL) if not myRet['returnOK']: print "DocMoto connection error " + str(myRet['responseString']) + "n" return if metadata is not None: sys.stdout.write('+ %sn' % path) # Traverse down the tree until we find the parent folder of the entry # we want to add. Create any missing folders along the way. myURL = root #Use str() to convert from unicode otherwise get problems with WebDAV myDropBoxPath = ""#DROPBOX_URL for part in branch: myURL = myURL + "/" + urllib.quote(str(part)) myDropBoxPath = myDropBoxPath + "/" + part myRet = LwpMotoUtils.checkExists(myURL) if(not myRet['exists']): LwpMotoUtils.createCollection(myURL) # Create the file/folder. if metadata['is_dir']: # Only create an empty folder if there isn't one there already. myURL = myURL + "/" + urllib.quote(str(leaf)) myRet = LwpMotoUtils.checkExists(myURL) if(not myRet['exists']): LwpMotoUtils.createCollection(myURL) else: #Go get the file, then put it into DocMoto myURL = myURL + "/" + urllib.quote(str(leaf)) myDropBoxPath = myDropBoxPath + "/" + leaf writeFile(myDropBoxPath,leaf, c) sendFile(myURL,leaf) removeFile(leaf) else: sys.stdout.write('- %sn' % path) # Traverse down the tree until we find the parent of the entry we # want to delete. myURL = root for part in branch: myURL = myURL + "/" + urllib.quote(str(part)) myRet = LwpMotoUtils.checkExists(myURL) if(not myRet['exists'] or not myRet['isCollection']): break else: # If we made it all the way, delete the file/folder (if it exists). # If this is a folder we need to add a final /. myURL = myURL + "/" + urllib.quote(str(leaf)) myRet = LwpMotoUtils.checkExists(myURL) if (myRet['isCollection']): myURL = myURL + "/" #Delete it LwpMotoUtils.createDelete(myURL)
Of the three control functions sendFile deserves a special mention since this function calls the methods that write to DocMoto.
The method calls "put", "proppatch" and "version-control" which correspond to the equivalent WebDAV commands.
The proppatch in particular is interesting since we can use this to set property data. In this example we are setting a value for the "comments" property only.
def sendFile(myURL,fileName): try: f = open(TEMP_FOLDER + fileName,'rb') buff = f.read() f.close() except: print "Unexpected sendFile return errorn" return LwpMotoUtils.createPut(myURL,buff) # Add any properties. Correspond to DocMoto tags request = """<!--?xml version='1.0' encoding='utf-8'?--> <propertyupdate xmlns="DAV:" xmlns:dm="http://www.docmoto.com/2008/03/uiconfig"> <set> <prop> <comment>Uploaded by script</comment> </prop> </set> </propertyupdate>""" # Confirm LwpMotoUtils.createPropPatch(myURL, request) LwpMotoUtils.createVersionControl(myURL)
Putting it all Together
Now we have enough code to make a simple system to replicate a Dropbox account in a DocMoto repository.
But to make it work we need to do the following:
- Download and install Dropbox's Python SDK
- Create an "app" in a Dropbox account
- Add the Dropbox "App key" and "App Secret" to any scripts
- Link to a user's account , use the DropBoxDocMotoUtils Link function
- Create a script to call the update function periodically
DropboxMonitor is a simple script that runs permanently and periodically calls the DropBoxDocMotoUtils update function to synchronize a DocMoto folder with a Dropbox account.
The "count" isn't strictly necessary, it's just handy when running the script from the command line as it makes it clear a scan is actually happening. In a production environment it wouldn't be used.
import DropBoxDocMotoUtils import time #The DocMoto folder eg. "Contents/dropboxtest" DropBoxDocMotoUtils.DOCMOTO_BASE_FOLDER = "" #Not to end in. / #The DocMoto server URL. https for secure. Port 4984 for autoversioning. #eg. https://<mydocmotoserver>:4984 DropBoxDocMotoUtils.DOCMOTO_SERVER_URL = "" #Not to end in / #DocMoto user name and password. DropBoxDocMotoUtils.DOCMOTO_USERNAME = "" DropBoxDocMotoUtils.DOCMOTO_PWD = "" #Dropbox app key and secret DropBoxDocMotoUtils.APP_KEY = '' DropBoxDocMotoUtils.APP_SECRET = '' #POSIX path to the temp folder eg "/Users/Fred/temp" DropBoxDocMotoUtils.TEMP_FOLDER = "" #Ends in / #Dropbox application level. Two options, app_folder or dropbox #If set to dropbox the app will synchronize the whole account. DropBoxDocMotoUtils.ACCESS_TYPE = 'app_folder' #The name of the file used to store persistent data. An example #is included in the project download DropBoxDocMotoUtils.STATE_FILE = 'docmoto_dropbox_cache.json' def main(): var = 1 count = 1 while var == 1: DropBoxDocMotoUtils.update(count) count = count + 1 time.sleep(1) if __name__ == '__main__': main() </mydocmotoserver>
Dropbox access type controls the scope of an application. If set to app_folder the scope is limited to the contents of the application's folder within apps .
If set to dropbox the scope covers the whole Dropbox account.
All the code in this example has the access type set to app_folder
This article shows how to link a Dropbox account and DocMoto folder.
The scripts included are not intended to be to a production level but to act as a good basis for a production project.
Below are some of the more obvious improvements that could be made in a real life scenario.
- The DropboxMonitor could be called from a plist at startup.
- Proper logging could be developed to record any errors.
- The system described only looks for changes. It doesn't initialize by copying the contents of a Dropbox account. For DocMoto this isn't a big issue as the contents could be simply dragged in, and the monitor switched on thereafter to update the changes.
- Extend the access type to dropbox but add a filter to exclude certain folders from the scan.