Subscribe to my RSS feed RSS

Posted: August 27th, 2008

Demisauce Summary, what is it?

A friend was asking "what is Demisauce", of which i have been horrible at articulating, so here goes an attempt:

Demisauce is a set of Services, and a Services framework

  • A set of services to use within your app (as service oriented/easy to integrate javascript or xml): "feedback/help/ideas", activity tracking (per person google analytics), comment system, poll's, group management. All very boring EXCEPT: these are all services able to be plugged in to your app instead of having to build on top of wordpress/joomla/sharepoint to get access to them.
  • A plugin services framework: think service oriented Wordpress plugin's, again available to any app for plugin. (cloud service plugin framework). The goal was to create an plugin based open source tool, that cater's to developer's adding in their own plugin's. However, since it was service oriented, a services framework was needed. Instead of having each developer integrate with 50 api's (see below) this provides a common place to exchange services similar to Opensocial gadgets, or Google gadget's but not based on javascript client side integration (slow, slow, slow).
  • Integration as a service: As a side product of moving to a cross-application, cross-data-center services framework you can now provide integration as a service. Think of an "add user" web service that you call, you have previously configured through web based admin what you want to happen: "when new user gets added, add them to: (wordpress, google apps, salesforce.com, basecamp, etc,etc)". Think of having one api for add user, behind which is integrations' with services so instead of each developer/company doing integration with google apps, Salesforce.com, Wordpress, instead you call this one and it has pre done integration. The idea, as more and more cloud services pop-up you are going to need to register your user's on those systems. Why have each company do 50 "add user" integration services? Instead of integrate with one, then it integrates with the 50?

Posted: August 16th, 2008

Help and Feedback system on Demisauce

I have been quiet but still making lots of updates to Demisauce. Most recently i have updated the Help and Feedback processing system. It is somewhat of a light scale embeddable app that can collect feedback and suggestions, ideas, and trouble from your user's.
demisauce help system with tags
It shows off several of the new features i have been working on. The biggest of which is the tagging system. I am working on extracting the tagging as a fully embeddable service (json, javascript, xml, html api's).

So, this help system is definitely not a full blown ticketing system. It is geared towards several things: More towards feedback, and easy to embed. If you are looking for full blown open source trouble ticket or help systems, here are a few: http://www.eticketsupport.com/, Simpleticket.net, OTRS.org, TroubleTicketExpress.com, http://www.osticket.com/, http://www.hulihanapplications.com/projects_mystic.php.

  • Easy To Embed in Your App: Using Javascript, or xml api's easy to put collection of user's input. Since it is service oriented, you can serve many app's with the same implementation (hosted on Demisauce.com soon) or install on your servers.
  • Feedback Processing: It is geared more towards feedback and suggestions by end user's. So, you can put a link inside your footer or in your website to ask for feedback. It pop's up a form that is hosted on Demisauce, allowing zero install implementation's other than javascript. But, more XML api's are available if needed. Currently tagging works great, working on ability to assign item's to other user's as task's that send notifications.
  • Event Driven Architecture: Currently being worked on is a set of Event callback api's. So, you can recieve call back via web service api's or simple Web Hooks API.

tags: demisauce
Posted: July 6th, 2008

Installing Thrift on Mac

We have been working on building some Google Gadget's at work as a way to do distributed application development. We are looking to use Shindig, the Open source Gadget server. To interface with it we were looking into Thrift, a cross-language services framework. However, to get it working it has quite a few non-trivial dependencies, and not the best documentation. So, anyway, I followed This article on installing Thrift, however i received the following error:

--->  Building boost with target all
Error: Target org.macports.build returned: shell command " cd "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_devel_boost/work/boost_1_35_0" && gmake all " returned error 2
Command output: Makefile:2: *** missing separator.  Stop.

Error: Status 1 encountered during processing.

Some more googling led me to This MacPorts trac ticket which has a mention from someone on getting it working (with the Clean, then reinstall steps).

That Nearly got me there, but i ran into this error.
checking for libevent >= 1.0... yes
checking for zlib >= 1.2.3... yes
./configure: line 20726: syntax error near unexpected token `MONO,'
./configure: line 20726: `  PKG_CHECK_MODULES(MONO, mono >= 1.2.6, have_mono=yes, have_mono=no)'
mac:thrift aaron$ sudo make install
Password:
make: *** No rule to make target `install'.  Stop.
mac:thrift aaron$ sudo port install mono


Some more googling led me to this http://wiki.apache.org/thrift/ThriftInstallationWin32, the section about the missing .m4 file. My .m4 file was actually in /opt/local/share/acloal so i modifed it for the following:
cp /opt/local/share/aclocal/pkg.m4 aclocal


I was still getting the same missing mono, error, some more googling led me to http://blog.jakbot.com/?p=6 I did couple things this page, but then re-ran
sh bootstrap.sh


I think that is what did it, although if not I also followed the steps in the above article about exporting mono path, and also the additional installs (autconf,automake, libtools). Then, i was able to re-run the Thrift .configure command and it was good.



While playing with Thrift, i also found ThruDB which sounds like a very interesting service as well. I started to follow this article on installing ThruDB. I am tackling that now. Have to start to find the Python API's.

tags: thrift, shindig, mac
Posted: June 10th, 2008

New Demisauce Dashboard and Google Visualization API

The newest version of Demisauce in svn includes a new activity tracker (I will blog in more detail in the future). This activity tracker utilizes the Google Visualization API on a new Dashboard (screenshot below) to present visual graphs of user activity. Big deal, its not as nice as Google Analytics you say? Well, true, but it does allow you to track information about users so that you know the exact usage for a particular user, instead of anonymous statistics for all users.

demisauce dashboard with visualization api

However, while playing with the Visualization API, which I found to be a great product and a great abstraction, I really was wanting some different graph formats. So, I went in search of some different visualization API's in Flash which were hopefully opensource, here are some i found. I am maybe going to dig in and make a treemap version of a Google Visualization API? We will see....

Posted: May 26th, 2008

Demisauce Comment notifications and Help System

A lot of work on the Demisauce Help system, using facebox jquery plugin for the popup page, or at least a heavily modifed version of Facebox. I had to implement quite a few changes to allow for content to be gathered from the Demisauce server, which would be a different domain. Therefore, the traditional ajax get in the Facebox wouldn't work, and i had to modify it to do a Document.write instead.

I added this method to Facebox to route requests via an inserted script.

// in facebox.js 
function fillFaceboxFromDocumentWrite(href, klass) {
    if ($('#facebox .content').attr('src') == href){
        $.facebox.reveal(null, klass,href);
    }else {
        var ds_html_script = document.createElement('script');
        ds_html_script.type = 'text/javascript';
        ds_html_script.src = $.facebox.settings.base_url + href;
        document.getElementById('facebox').appendChild(ds_html_script)
        $.facebox.reveal(null, klass,href);
    }
}

The script that gets called is dynamically generated to produce javascript, which will get inserted into the Dom of the remote server.

//  python code to generate ds_help_output
document.getElementById('facebox_content_holder').innerHTML = ds_help_output;

The end result is this help system. You can edit content at the Demisauce Hosted Server or can host a version of demisauce, then include a line of javascript in your apps to get built in help system. Here is a screenshot. Help System

Also, modified Demisauce to allow for Comment notification's. demisaucecommentsnotification

Posted: May 25th, 2008

Blog updates for App Engine, Caching data and non-normalized database work.

I made a number of improvements to this blog engine over the last couple of weeks. Most note-able was performance improvements to pre-render and cache HTML. There has been a lot of discussion and movement in the web architecture community towards not-traditional Database designs, not normalizing data, and storing caclulated info, and even moving to non-relational database designs. All because Internet scale data-bases built on transactional systems just seem fraught with problems and forces you to build around the db instead of around the needs of the web apps. The Google App Engine article on High Scalability gives a lot of great insight into App Engine which i had been discovering separately.

In order to incorporate a lot of the improvements possible out of Google App Engine Datastore, you have to start to do things different, here are what i did to help build a better google app engine based app:

  • OnChange Handler: Since we are often pre-rendering, and caching html, we need to get hooked into the events which signify when we need to make a change. Digging into the Google Datastore API, i did some reverse engineering to try to figure out how I can figure out when something is changing. What i came up with is to provide a Base model which i inherit from which allows me to write OnChange methods per attribute.
    from google.appengine.ext.db import Model as DBModel
    
    class BaseModel(db.Model):
        def __init__(self, parent=None, key_name=None, _app=None, **kwds):
            self.__isdirty = False
            DBModel.__init__(self, parent=None, key_name=None, _app=None, **kwds)
        
        def __setattr__(self,attrname,value):
            """
            DataStore api stores all prop values say "email" is stored in "_email" so
            we intercept the set attribute, see if it has changed, then check for an
            onchanged method for that property to call
            """
            if (attrname.find('_') != 0):
                if hasattr(self,'_' + attrname):
                    curval = getattr(self,'_' + attrname)
                    if curval != value:
                        self.__isdirty = True
                        if hasattr(self,attrname + '_onchange'):
                            getattr(self,attrname + '_onchange')(curval,value)
            
            DBModel.__setattr__(self,attrname,value)
      
    
    Then, on my Entry class (blog entry), i implemented a method whenever an entry changes from published to draft and back, and updated the archive.
    class Entry(BaseModel):
        author = db.UserProperty()
        blog = db.ReferenceProperty(Blog)
        published = db.BooleanProperty(default=False)
        content = db.TextProperty(default='')
        # more cut out
        
        def published_onchange(self,curval,newval):
            """
            Gets called every time published status changes
            """
            if self.entrytype == 'post':
                my = self.date.strftime('%b-%Y') # May-2008
                archive = Archive.all().filter('monthyear',my).fetch(10)
                if curval == False and newval == True:
                    # add to archive
                    if archive == []: # new month
                        archive = Archive(blog=self.blog,monthyear=my)
                    else: 
                        archive = archive[0]
                    archive.entrycount += 1
                    archive.put()
                    self.blog.entrycount += 1
                else:
                    # remove from archive
                    if archive and archive[0]:
                        archive = archive[0]
                        archive.entrycount -= 1
                        if archive.entrycount == 0:
                            archive.delete()
                        else:
                            archive.put()
                    self.blog.entrycount -= 1
                
                self.blog.save()
      
    
  • Pre-Render HTML: On the previous entry, we are storing some tables (Archive) which are purely used for convenience. They are never called at runtime, and in fact the Archive table is then rendered into html and stored again. Here is an admin entry "POST" update that then calls to update the cache. The Cache is actually a Blog entity which is used on every single request. It is half way between a Cache, configuration and context in normal web apps.
    class AdminEntry(BaseController):
        def post(self,entrytype='post',key=None):
            #update entry from form post
            entry.save()
            rebuild_cache(self.blog)
    
    def rebuild_cache(blog):
        """
        Pre-Render's and cache's html in blog object.  Everything
        in here doesn't change very often, so we can update it at point of change
        """
        pages = Entry.all().filter("entrytype =", "page").filter("published =", True).fetch(20)
        archives = Archive.all().order('-date').fetch(10)
        recententries = Entry.all().filter('entrytype =','post').filter("published", True).order('-date').fetch(10)
        links = Link.all().filter('linktype =','blogroll')
        template_vals = {'recententries':recententries,'pages':pages,
                'links':links,'archives':archives}
    
        path = os.path.join(os.path.dirname(__file__), 'views/sidebar.html')
        blog.sidebar = template.render(path, template_vals)
        path = os.path.join(os.path.dirname(__file__), 'views/topmenu.html')
        blog.topmenu = template.render(path, template_vals) 
        blog.save()
        
    

These tricks helped make writing App engine even easier, moving more to an event driven model that updates the cache. You can find all the code from this app at the Github code repository.

Posted: May 25th, 2008

Demisauce Upgrades and Help System

Another release of Demisauce including bug fixes, and the start of a help system. Here is a screenshot of the Help System which is an popup over the rest of the page. There are a couple of different styles. The content comes from Demisauce, and both the help (javascript) and its content come from Demisauce server and are easily embeddable in an app. ds04help
Posted: May 19th, 2008

Added new Lighthouse Project Tracking for Potlatch Blog

I just created a Feature and Bug tracking project at Lighthouse a new online request/feature/bug system. Mostly used by Ruby on Rails developers but seems like a nice system, a bit nicer than the ones on Google Code (which i use for other projects) so will try it out.

Posted: May 18th, 2008

First Version of the Demisauce Comment System!

Wow! A whirlwind of hacking has produced the first version of the comment system that i am going to use on this Potlatch blog, a Google App engine powered blog. The comment system is built as a proof of concept for Demisauce, the open source project i have been working on. The core idea of Demisauce is an open source set of web services that allow you to add functionality to your app. This service is mostly a javascript service that you embed on your app, allowing comments on anything. The other Demisauce services so far are XML services and html. This service (comments) is not better than the other comment systems out there (Disqus.com is very nice, and obviously Wordpress.com has comments for all blogs at Wordpress), but it is able to be installed and run behind a corporate firewall. I am thinking of using it for some applications at work to comment on just about anything, not just blog posts. Think of comments on images (flickr), or commenting on a purchase order, or requistion, or request to hire someone that are in corporate systems.

The configuration Page with the javascript you would need to copy, and embed in your pages. Also shows comments by this user.
Comment configuration
Here is an actual view of the comments in a blog, showing comments, comment box, login etc.
Comment box for Authenticated user
Here is a view by a user not logged in (uses google Authentication or you can create an account).
Comment form for Anonymous User
This shows a user that has clicked on "Logon" but is already authenticated.
Logon Page for logged on user
These are the Logon Comments
login comments


The features

  • Google Authentication for Users (optional)
  • Gravatar Icons
  • Native authentication for users (if Not using Google)
  • Configuration tools to setup javascript for your app

The application is built on the Python, Pylons base of Demisauce, but the core of the presentation is Javascript/Jquery. You can get the code at Demisauce on google code or get this blog code at Potlatch Blog on Github.

Although Comment systems are not the services I wanted to build first for Demisauce, I decided it is a great example for a service that combines Javascript, Data/Persistence, Widget/Embedding. I am planning on enhancing this a bit more, possibly doing a Django Book style pop-up comment system. Also, been wanting to enhance the help system some more, which uses the content management system which although not fancy is usable. If anyone else has some ideas on what they would like to see first I would love to hear them.

Posted: May 14th, 2008

A few more updates to Blog!

A few more updates today:
  • Added archive capability, traditional blog archive by month
  • Added support for "migrations" in the blog. see it here. Not very scalable going forward (can time out). But, seems to address the need of changes on app engine for now.
  • Cleaned up slugs a bit
  • Added Draft mode/Publish capability

And started working on the Demisauce comment system, hopefully this week...