A friend was asking "what is Demisauce", of which i have been horrible at articulating, so here goes an attempt:

Demisauce is a set of Services, and a Services framework

• A set of services to use within your app (as service oriented/easy to integrate javascript or xml): "feedback/help/ideas", activity tracking (per person google analytics), comment system, poll's, group management. All very boring EXCEPT: these are all services able to be plugged in to your app instead of having to build on top of wordpress/joomla/sharepoint to get access to them.
• A plugin services framework: think service oriented Wordpress plugin's, again available to any app for plugin. (cloud service plugin framework). The goal was to create an plugin based open source tool, that cater's to developer's adding in their own plugin's. However, since it was service oriented, a services framework was needed. Instead of having each developer integrate with 50 api's (see below) this provides a common place to exchange services similar to Opensocial gadgets, or Google gadget's but not based on javascript client side integration (slow, slow, slow).
• Integration as a service: As a side product of moving to a cross-application, cross-data-center services framework you can now provide integration as a service. Think of an "add user" web service that you call, you have previously configured through web based admin what you want to happen: "when new user gets added, add them to: (wordpress, google apps, salesforce.com, basecamp, etc,etc)". Think of having one api for add user, behind which is integrations' with services so instead of each developer/company doing integration with google apps, Salesforce.com, Wordpress, instead you call this one and it has pre done integration. The idea, as more and more cloud services pop-up you are going to need to register your user's on those systems. Why have each company do 50 "add user" integration services? Instead of integrate with one, then it integrates with the 50?

I have been quiet but still making lots of updates to Demisauce. Most recently i have updated the Help and Feedback processing system. It is somewhat of a light scale embeddable app that can collect feedback and suggestions, ideas, and trouble from your user's.

It shows off several of the new features i have been working on. The biggest of which is the tagging system. I am working on extracting the tagging as a fully embeddable service (json, javascript, xml, html api's).

So, this help system is definitely not a full blown ticketing system. It is geared towards several things: More towards feedback, and easy to embed. If you are looking for full blown open source trouble ticket or help systems, here are a few: http://www.eticketsupport.com/, Simpleticket.net, OTRS.org, TroubleTicketExpress.com, http://www.osticket.com/, http://www.hulihanapplications.com/projects_mystic.php.

• Easy To Embed in Your App: Using Javascript, or xml api's easy to put collection of user's input. Since it is service oriented, you can serve many app's with the same implementation (hosted on Demisauce.com soon) or install on your servers.
• Feedback Processing: It is geared more towards feedback and suggestions by end user's. So, you can put a link inside your footer or in your website to ask for feedback. It pop's up a form that is hosted on Demisauce, allowing zero install implementation's other than javascript. But, more XML api's are available if needed. Currently tagging works great, working on ability to assign item's to other user's as task's that send notifications.
• Event Driven Architecture: Currently being worked on is a set of Event callback api's. So, you can recieve call back via web service api's or simple Web Hooks API.

We have been working on building some Google Gadget's at work as a way to do distributed application development. We are looking to use Shindig, the Open source Gadget server. To interface with it we were looking into Thrift, a cross-language services framework. However, to get it working it has quite a few non-trivial dependencies, and not the best documentation. So, anyway, I followed This article on installing Thrift, however i received the following error:

--->  Building boost with target all
Error: Target org.macports.build returned: shell command " cd "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_devel_boost/work/boost_1_35_0" && gmake all " returned error 2
Command output: Makefile:2: *** missing separator.  Stop.

Error: Status 1 encountered during processing.

checking for libevent >= 1.0... yes
checking for zlib >= 1.2.3... yes
./configure: line 20726: syntax error near unexpected token `MONO,'
./configure: line 20726: `  PKG_CHECK_MODULES(MONO, mono >= 1.2.6, have_mono=yes, have_mono=no)'
mac:thrift aaron$ sudo make install
Password:
make: *** No rule to make target `install'.  Stop.
mac:thrift aaron$ sudo port install mono

cp /opt/local/share/aclocal/pkg.m4 aclocal

sh bootstrap.sh

While playing with Thrift, i also found ThruDB which sounds like a very interesting service as well. I started to follow this article on installing ThruDB. I am tackling that now. Have to start to find the Python API's.

The newest version of Demisauce in svn includes a new activity tracker (I will blog in more detail in the future). This activity tracker utilizes the Google Visualization API on a new Dashboard (screenshot below) to present visual graphs of user activity. Big deal, its not as nice as Google Analytics you say? Well, true, but it does allow you to track information about users so that you know the exact usage for a particular user, instead of anonymous statistics for all users.

However, while playing with the Visualization API, which I found to be a great product and a great abstraction, I really was wanting some different graph formats. So, I went in search of some different visualization API's in Flash which were hopefully opensource, here are some i found. I am maybe going to dig in and make a treemap version of a Google Visualization API? We will see....

A lot of work on the Demisauce Help system, using facebox jquery plugin for the popup page, or at least a heavily modifed version of Facebox. I had to implement quite a few changes to allow for content to be gathered from the Demisauce server, which would be a different domain. Therefore, the traditional ajax get in the Facebox wouldn't work, and i had to modify it to do a Document.write instead.

I added this method to Facebox to route requests via an inserted script.

// in facebox.js 
function fillFaceboxFromDocumentWrite(href, klass) {
    if ($('#facebox .content').attr('src') == href){
        $.facebox.reveal(null, klass,href);
    }else {
        var ds_html_script = document.createElement('script');
        ds_html_script.type = 'text/javascript';
        ds_html_script.src = $.facebox.settings.base_url + href;
        document.getElementById('facebox').appendChild(ds_html_script)
        $.facebox.reveal(null, klass,href);
    }
}

The script that gets called is dynamically generated to produce javascript, which will get inserted into the Dom of the remote server.

//  python code to generate ds_help_output
document.getElementById('facebox_content_holder').innerHTML = ds_help_output;

The end result is this help system. You can edit content at the Demisauce Hosted Server or can host a version of demisauce, then include a line of javascript in your apps to get built in help system. Here is a screenshot.

Also, modified Demisauce to allow for Comment notification's.

I made a number of improvements to this blog engine over the last couple of weeks. Most note-able was performance improvements to pre-render and cache HTML. There has been a lot of discussion and movement in the web architecture community towards not-traditional Database designs, not normalizing data, and storing caclulated info, and even moving to non-relational database designs. All because Internet scale data-bases built on transactional systems just seem fraught with problems and forces you to build around the db instead of around the needs of the web apps. The Google App Engine article on High Scalability gives a lot of great insight into App Engine which i had been discovering separately.

In order to incorporate a lot of the improvements possible out of Google App Engine Datastore, you have to start to do things different, here are what i did to help build a better google app engine based app:

• OnChange Handler: Since we are often pre-rendering, and caching html, we need to get hooked into the events which signify when we need to make a change. Digging into the Google Datastore API, i did some reverse engineering to try to figure out how I can figure out when something is changing. What i came up with is to provide a Base model which i inherit from which allows me to write OnChange methods per attribute.

  from google.appengine.ext.db import Model as DBModel
  
  class BaseModel(db.Model):
      def __init__(self, parent=None, key_name=None, _app=None, **kwds):
          self.__isdirty = False
          DBModel.__init__(self, parent=None, key_name=None, _app=None, **kwds)
      
      def __setattr__(self,attrname,value):
          """
          DataStore api stores all prop values say "email" is stored in "_email" so
          we intercept the set attribute, see if it has changed, then check for an
          onchanged method for that property to call
          """
          if (attrname.find('_') != 0):
              if hasattr(self,'_' + attrname):
                  curval = getattr(self,'_' + attrname)
                  if curval != value:
                      self.__isdirty = True
                      if hasattr(self,attrname + '_onchange'):
                          getattr(self,attrname + '_onchange')(curval,value)
          
          DBModel.__setattr__(self,attrname,value)
    
  

  Then, on my Entry class (blog entry), i implemented a method whenever an entry changes from published to draft and back, and updated the archive.

  class Entry(BaseModel):
      author = db.UserProperty()
      blog = db.ReferenceProperty(Blog)
      published = db.BooleanProperty(default=False)
      content = db.TextProperty(default='')
      # more cut out
      
      def published_onchange(self,curval,newval):
          """
          Gets called every time published status changes
          """
          if self.entrytype == 'post':
              my = self.date.strftime('%b-%Y') # May-2008
              archive = Archive.all().filter('monthyear',my).fetch(10)
              if curval == False and newval == True:
                  # add to archive
                  if archive == []: # new month
                      archive = Archive(blog=self.blog,monthyear=my)
                  else: 
                      archive = archive[0]
                  archive.entrycount += 1
                  archive.put()
                  self.blog.entrycount += 1
              else:
                  # remove from archive
                  if archive and archive[0]:
                      archive = archive[0]
                      archive.entrycount -= 1
                      if archive.entrycount == 0:
                          archive.delete()
                      else:
                          archive.put()
                  self.blog.entrycount -= 1
              
              self.blog.save()
    
  

• Pre-Render HTML: On the previous entry, we are storing some tables (Archive) which are purely used for convenience. They are never called at runtime, and in fact the Archive table is then rendered into html and stored again. Here is an admin entry "POST" update that then calls to update the cache. The Cache is actually a Blog entity which is used on every single request. It is half way between a Cache, configuration and context in normal web apps.

  class AdminEntry(BaseController):
      def post(self,entrytype='post',key=None):
          #update entry from form post
          entry.save()
          rebuild_cache(self.blog)
  
  def rebuild_cache(blog):
      """
      Pre-Render's and cache's html in blog object.  Everything
      in here doesn't change very often, so we can update it at point of change
      """
      pages = Entry.all().filter("entrytype =", "page").filter("published =", True).fetch(20)
      archives = Archive.all().order('-date').fetch(10)
      recententries = Entry.all().filter('entrytype =','post').filter("published", True).order('-date').fetch(10)
      links = Link.all().filter('linktype =','blogroll')
      template_vals = {'recententries':recententries,'pages':pages,
              'links':links,'archives':archives}
  
      path = os.path.join(os.path.dirname(__file__), 'views/sidebar.html')
      blog.sidebar = template.render(path, template_vals)
      path = os.path.join(os.path.dirname(__file__), 'views/topmenu.html')
      blog.topmenu = template.render(path, template_vals) 
      blog.save()
      
  

These tricks helped make writing App engine even easier, moving more to an event driven model that updates the cache. You can find all the code from this app at the Github code repository.

I just created a Feature and Bug tracking project at Lighthouse a new online request/feature/bug system. Mostly used by Ruby on Rails developers but seems like a nice system, a bit nicer than the ones on Google Code (which i use for other projects) so will try it out.

Wow! A whirlwind of hacking has produced the first version of the comment system that i am going to use on this Potlatch blog, a Google App engine powered blog. The comment system is built as a proof of concept for Demisauce, the open source project i have been working on. The core idea of Demisauce is an open source set of web services that allow you to add functionality to your app. This service is mostly a javascript service that you embed on your app, allowing comments on anything. The other Demisauce services so far are XML services and html. This service (comments) is not better than the other comment systems out there (Disqus.com is very nice, and obviously Wordpress.com has comments for all blogs at Wordpress), but it is able to be installed and run behind a corporate firewall. I am thinking of using it for some applications at work to comment on just about anything, not just blog posts. Think of comments on images (flickr), or commenting on a purchase order, or requistion, or request to hire someone that are in corporate systems.

The features

• Google Authentication for Users (optional)
• Gravatar Icons
• Native authentication for users (if Not using Google)
• Configuration tools to setup javascript for your app

The application is built on the Python, Pylons base of Demisauce, but the core of the presentation is Javascript/Jquery. You can get the code at Demisauce on google code or get this blog code at Potlatch Blog on Github.

Although Comment systems are not the services I wanted to build first for Demisauce, I decided it is a great example for a service that combines Javascript, Data/Persistence, Widget/Embedding. I am planning on enhancing this a bit more, possibly doing a Django Book style pop-up comment system. Also, been wanting to enhance the help system some more, which uses the content management system which although not fancy is usable. If anyone else has some ideas on what they would like to see first I would love to hear them.

• Added archive capability, traditional blog archive by month
• Added support for "migrations" in the blog. see it here. Not very scalable going forward (can time out). But, seems to address the need of changes on app engine for now.
• Cleaned up slugs a bit
• Added Draft mode/Publish capability

And started working on the Demisauce comment system, hopefully this week...