Bootstrapping "opengoupware.us"

Thanks to the addition of the "pages" feature in Blogspot I've finally created a site at opengroupware.us, The OpenGroupware [Legacy] project website (I won't even bother to link to it) has been worthless for some time, and the information about the constellation of projects beyond legacy is very scattered. opengroupware.us is an attempt to at least create an index of that information as well as resurrect some of the good content from the abyss that is the docs plone.  If you want to submit content to opengroupware.us or want to help edit / maintain content just let me know any I'll add your Blogspot account to the site permissions.

XSLT Transform to TXT, with LXML

Maybe this should be obvious, but it wasn't to me. I've got an XML document and an XSLT stylesheet. But that stylesheet just produces text, not XML; it is essentially a template for an e-mail [with some iterations, so something more complex than OIE's very convenient rowTemplateAction can handle]. So I was extending the transformAction for performing XSLT transforms that produce other than XML... but the documentation is a bit thin and every example is XML results. The trick is pretty simple, just

unicode(result)

and make sure [of course] that you have

<xsl:output method="txt" encoding="utf-8" omit-xml-declaration="yes"/>

declared in the template. So it should be as simple as:

def do_action(self):
        source = etree.parse(self._rfile)
        xslt = etree.fromstring(self._xslt)
        transform = etree.XSLT(xslt)
        result = transform(source)
        self._wfile.write(unicode(result))

in order to transform to just about anything.

Coils: Features merged into "default"

This morning the following features were merged into OpenGroupware Coils "default" branch, and will be available in the next release.

•  The scheduler service no longer depends on the ticktock heartbeat.  It checks the run queue in its work method.
• Format(s) now support logging of rejected records to a buffer.  This feature is available via a new parameter on the readAction: "rejectionsLabel".  This is documented in the readAction's wiki page.
•  The above motivated me to abstract how ActionCommand goes about creating [output] messages.  It is now simple for an action to create additional messages [beyond it's 'default' output message] by just calling the store_in_message(label, wfile, mimetype='application/octet-stream') method.
• New CLI tools: "coils-list-schedule" and "coils-unschedule-process" that let the administrator view and modify the workflow scheduler from the command line.  Yes, we still need a "coils-schedule-process". 
• getopt isn't the greatest command line option parser,  would be great if someone would look into swapping that out with something better.
• The above motivated me to add an initialize_tool(name, argv, arguments=['',[]]) function to coils.core.utility.  This provides a simple way for a tool [vs. a service] to bootstrap itself onto the OpenGroupware Coils service bus so that it can receive callbacks, etc... Return value is an AdministrativeContext (with a registered Broker for IPC) and dictionary of unconsumed command line parameters. initialize_tool automatically consumes the --store=, --add-bundle=, and --ban-bundle parameters in order to initialize the Coils environment.

In the last week or so there has been significant work on tidying things up, and the deployment instructions on the wiki have been updated as well.  Getting a working OpenGroupware Coils instance should now be very straight-forward.   Big work ahead is the move to WSGI which is underway in the NetComm branch (which needs to be remerged with default, btw).  And with the impending release of openSUSE 11.3, and its inclusion of Evolution 1.30.x,  the GroupDAV address book support can be completely tested and polished.  Evolution 1.30.x contains several enhancements to make it's WebDAV address book support fully compatible with GroupDAV.

Keeping an OIE Route After Completion

Once an OpenGroupware Integration Engine route has successfully completed it is automatically garbage collected.  This prevents the system from filling up with messages and version information.  But if an external application, like ogo-curses-putter, wants to retrieve the output of the route... it's gone!  To facilitate this use-case the garbage collection can be suppressed per-route by setting the {http://www.opengroupware.us/oie}preserveAfterCompletion property of the route entity with a value of "YES".  If this property value exists the processes derived from that route will not be garbage collected.  The simplest way to set the property is to retrieve the route via "route::get" and use your context's property manager:

from coils.core import *
ctx = AdministrativeContext()
r = ctx.run_command('route::get', name='MTAStockOrder_TEST')
ctx.property_manager.set_property(r, 'http://www.opengroupware.us/oie', 'preserveAfterCompletion', 'YES')
ctx.commit() 

The above will set the value of {http://www.opengroupware.us/oie}preserveAfterCompletion to "YES" for the route named "MTAStockOrder_TEST". The property manager will update the value of the property if such a property already exists, or it will create a new property with the specified value.

Performing PROPFIND from Python

Surprising, when I went looking for examples of performing PROPFIND requests from Python I found very little information. For testing OpenGroupware Coil's WebDAV presentation I needed a simple way to perform the PROPFIND required to list the entities in a collection and then request each entity. With a bit of scratching through the thin documentation I was able to come up with the following:

1. import httplib, urllib2, base64, sys
2. from lxml import etree
4. PROPFIND = u'''<?xml version="1.0" encoding="utf-8"?>
5. <propfind xmlns="DAV:">
6. <prop>
7. <getetag/>
8. </prop>
9. </propfind>'''
11. SERVER_HOST = '127.0.0.1'
12. SERVER_PORT = 8080
13. SERVER_PATH = '/dav/Contacts'
14. CLIENT_AGENT = 'Whitemice rules/so very true'
16. auth_string = 'Basic {0}'.format(base64.encodestring('adam:fred123')[:-1])
18. urllib2.install_opener(urllib2.build_opener(urllib2.HTTPHandler()))
20. connection = httplib.HTTPConnection(SERVER_HOST, SERVER_PORT)
21. connection.putrequest('PROPFIND', SERVER_PATH)
22. connection.putheader('Authorization', auth_string)
23. connection.putheader('User-Agent', CLIENT_AGENT)
24. connection.putheader('Depth', 1)
25. connection.putheader('Content-Length', str(len(PROPFIND)))
26. connection.endheaders()
27. connection.send(PROPFIND)
28. response = connection.getresponse()
29. if response.status == 207:
30.   data = response.read()
31.   connection.close()
32.   response = None
33.   namespace_prefix_map = { 'D' : 'DAV:' }
34.   document = etree.fromstring(data)
35.   for path in document.xpath('/D:multistatus/D:response/D:href/text()',
36.       namespaces=namespace_prefix_map):
37.     if (path != SERVER_PATH):
38.       connection = httplib.HTTPConnection('127.0.0.1', 8080)
39.       connection.putrequest('GET', path)
40.       connection.putheader('Authorization', auth_string)
41.       connection.putheader('User-Agent', CLIENT_AGENT)
42.       connection.endheaders()
43.       response = connection.getresponse()
44.       if response.status != 200:
45.         print 'Error retrieving {0}'.format(path)
46.         sys.exit(1)

This will make the PROPFIND for all the items in the collection, requesting the getetag property, and then from the response perform an HTTP GET for every item. Even better would be if it requested isCollection and knew better than to perform GETs on collections.  I hope this simple example will be of use to someone.

OIE Progress, and lots of it.

The workflow engine [OIE] in the OpenGroupware Coils project has reached a point a general usefulness.  OIE is accessible via WebDAV for creating, browsing, and starting flows.  The engine compiles BPML 1.0 markup to an internal format (adding additional markup-notations, such as XPDL,  shouldn't be much harder than writing a SaX parser to produce the internal format).  The foreach flow control structure is implemented;  switch, until, and while are simply stubs at this point.  switch is actually compiled but can't be executed yet.  Message scope is handled correctly.  Even with just linear execution and foreach a great many business processes can be modelled, especially ETL type flows.

The actions implemented so far are:

• read - Translate a message using a format; fixed record length and XLS formats have been implemented.
• select - SELECT from a defined RDBMS.
• write - Write a StandardXML message out using a specified format.
• assign - XPath select a value or assign a static value to a message.
• getEntity - Retrieve an entity from the groupware database.
• xpath - Create a new message from an XPath query.
• ldapSearch - Search an LDAP DSA, results are in DSMLv1.0.
• regularExpressionFind - Perform regexp matches.
• readJSON - Translate JSON data into XML.
• sendMail - Send a message via an SMTP server, including support for attachments.

All commands support label substitution in appropriate parameters.

As of yesterday there is also an insert action for inserting data into a RDMS table;  this action hasn't yet been as heavily tested as the others,  but it completes the tool set required for simple ETL flows.

Disabling GNOME Automount

When you connect a mass storage device to a computer running the GNOME desktop environment it automatically mounts the device in /media and places a short-cut to the device on your desktop.  This is an excellent default behavior;  but if you are working with various devices sometimes it can get in the way.  To correctly disable this feature simply execute:

gconftool-2 --type bool --set /apps/nautilus/preferences/media_automount false

The devices will still appear in computer:/// (in nautilus) where you can right click on them to select the mount action if and when you want them to be mounted.  If and when you want to re-enable to auto-mounting feature execute:

gconftool-2 --type bool --set /apps/nautilus/preferences/media_automount true

Passing a column set to an SQLalchemy query

In implementing the List method of the EntityAccessManager provided by the Contacts bundle in OpenGroupware Coils it seemed like it would be very efficient to allow the consumer to request what set of attributes it needed;  for instance, if a WebDAV client's PROPFIND request didn't ask for a given property, why request the corresponding attribute in the query?  Especially since the result set for PROPFIND queries are frequently very large [on the order of 20,000 records or so].  But how to pass a set of attributes as a parameter?

def List(ctx, attributes):
  ...
  db = query(attributes).filter(....)
  return db.all

does not work;  where attributes is a list, or a set of more than one attribute. Some fiddling around and I discover that the following works:

def List(ctx, *attributes):
  ...
  db = query(*attributes).filter(...)
  return db.all()

...
ctx = AssumedContext(10100)
a = BundleManager.get_access_manager('Contact', ctx)
a.List(ctx, (Contact.object_id, Contact.version))

That single asterisk is the trick.

Configuring OpenLDAP's dynlist in cn=config

Step#1: Make sure the dynlist module is loaded on the server.  What modules are loaded are typically controlled by the "cn=modules{0}, cn=config" object.  If your server doesn't have a modules object it probably isn't loading any dynamic modules.  Create an LDIF file and import it.  Our "cn=modules{0}, cn=config" looks like:

dn: cn=modules{0}, cn=config
olcModuleLoad: {0}accesslog.la
olcModuleLoad: {1}auditlog.la
olcModuleLoad: {2}constraint.la
olcModuleLoad: {3}dynlist.la
olcModuleLoad: {4}memberof.la
olcModuleLoad: {5}ppolicy.la
olcModuleLoad: {6}refint.la
olcModuleLoad: {7}seqmod.la
olcModuleLoad: {8}syncprov.la
olcModuleLoad: {9}sssvlv.la
olcModuleLoad: {10}translucent.la
olcModuleLoad: {11}unique.la
olcModuleLoad: {12}back_monitor.la
olcModulePath: /usr/lib/openldap2.4
objectClass: olcModuleList
cn: modules{0}

NOTE: Double check that the "olcModulePath" is the absolute path to the directory containing the OpenLDAP modules.

Step#2: Check that your schema has the groupOfURLs objectclass defined.  If you attempt to configure the dynlist module without that schema available you still crash slapd.  To define the dynamicGroup schema (if it is missing) you can import the following LDIF into your cn=config:

dn: cn=dynamicGroup, cn=schema, cn=config
olcObjectClasses: {0}( 2.16.840.1.113730.2.33         NAME 'groupOfURLs'   
     SUP top STRUCTURAL         MUST cn         MAY ( memberURL $ businessCat
 egory $ description $ o $ ou $                 owner $ seeAlso ) )
olcAttributeTypes: {0}( 2.16.840.1.113730.1.198         NAME 'memberURL'   
     DESC 'Identifies an URL associated with each member of a group. Any type
  of labeled URL can be used.'         SUP labeledURI )
objectClass: olcSchemaConfig
cn: dynamicGroup

Step#3: Create an olcOverlayConfig object in the scope of your Dit database.  For example, our Dit is the 1st HDB database on the server so the appropriate object is:

dn: olcOverlay=dynlist,olcDatabase={1}hdb,cn=config
objectClass: olcOverlayConfig
objectClass: olcDynamicList
olcOverlay: dynlist

Step#4: Now you are ready to actually use the dynlist module.  A common use-case is to create dynamic mail alias objects;  with dynlist you don't need to maintain mail aliases, they will automatically contain everyone who matches the relevent criteria.  Provided you use the traditional nisMailAlias objectclass in order to define mail aliases adding the attribute -

olcDlAttrSet: {0}nisMailAlias labeledURI rfc822mailmember:mail

 - to "olcOverlay=dynlist,olcDatabase={1}hdb,cn=config" will enable dynamic mail aliases.  Specifically any nisMailAlias containing a labeledURI attribute will be expanded by the query specified in that attribute.  The results of that query will be rewritten by the optional rfc822mailmember:mail clause which will rename the mail attributes resulting from the query into the rfc822mailmember attribute required by consumers of the nisMailAlias objects.  So rather than populating the mail alias object with rfc822mailmember attributes manually, you extend the object with the auxilliary labeledURIObject objectclass and define the query in the labeledURI attribute.

dn: cn=gr_parts, ou=ListAliases, ou=Aliases, ou=Mail, ou=SubSystems, o=Morrison Industries,c=US
mail: gr_parts@morrison-ind.com
labeledURI: ldap:///ou=People,ou=Entities,ou=SAM,o=Morrison Industries,c=US?mail?one?(&(morrisonactiveuser=Y)(objectclass=morrisonuser)(departmentNumber=*P*)(morrisonbranch=GRD))
objectClass: nisMailAlias
objectClass: top
objectClass: labeledURIObject
cn: gr_parts

The object will immediately populate with rfc822mailmember attributes derived from the mail attribute of those objects matching the specified filter: "(&(morrisonactiveuser=Y)(objectclass=morrisonuser)(departmentNumber=*P*)(morrisonbranch=GRD))".  The third parameter, "one", of the URI is the scope of the query so only objects immediately subordinate to "ou=People,ou=Entities,ou=SAM,o=Morrison Industries,c=US" are candidates for the filter.