Wednesday, October 2, 2013

Getting CRUDy with Meteor

So, over the past couple of months I have told a number of people on the Google meteor-talk forum and on irc that I would whip up an article about how I have implemented forms handling and basic crud in my Meteor application.

This article comes with a few caveats.
  1. I started my application before Meteorite.  So, I don't use 3rd party packages and I haven't converted these application design concepts into packages.
  2. This isn't a cut & paste recipe.  This is a high level overview of my application design.  It should be sufficient for someone to understand and potentially implement.
  3. I am still on Meteor 6.4.1.  No idea if it will work with 6.5.1
  4. I am simplifying what I did to make it easier to follow and leaving A LOT out.
  5. I wrote this fast.  It will be littered with misspellings and grammatical errors.

I built a real-time retail point of sale system called EtherPOS in Meteor.  Yeah, the website sucks.  As of this article, it is just a place-holder while I focus on the application.  

Last month, I wrote an article called Meteor Is Not Just A Toy that provides some insight into the eight month process of writing this application.  I plan on writing a more in depth white paper/business case analysis that provides more details on the cost savings and overall benefits.

If you can imagine, a retail point of sale system has a lot data models and objects.  For example, beyond just users there are data models like a default settings, chart of accounts,  tax rates, cash paid outs, tender types, departments, sales groups, commission groups, manufacturers, vendors, items, discounts, locations, registers, customers, gift certificates, gift cards, purchase orders, time clock records, payroll records, commission and sales records, transfer orders, inventory counts, invoices, invoice items, invoice item discounts, invoice payments and much more...

As you know these are collections in Meteor and generally most examples show writing template event maps bound to an event trigger to perform some kind of insert, update or remove.  As I started to build my application, with all those collections, it was a bit maddening and cumbersome to have to write template event maps for inserts, updates and removes for all of these collections.

When I first started developing my application I was very frustrated because I was approaching it from the point of view of other frameworks and I struggled with basic CRUD.  I wanted a simple reusable pattern similar to the router-controller-action model of other frameworks.

Basically, what I really wanted was a generic abstracted way to handle create, update and delete based on the router controller path that allowed me to implement inserts, updates and removes.  I also wanted to do this only with HTML/Handlebars and without having to write event handlers for every insert, update or remove.  With as many collections as my application has it was important to find a solution to this early, otherwise I would have spent a ridiculous amount of time writing what I consider boilerplate.  Unfortunately, nothing like what I wanted existed in Meteor.  Not only did it not exist but a router controller concept didn't exist.  

Thankfully, the madewith.meteor.com resource was available and provides a lot of  excellent application examples many of which have source code.  There was an early application called Arae (source) that had a decent backbone based router.  I studied this project and then rolled my own router that suited my needs.  Like I said, my project goes way back to before there were really any router packages.

I am not going to belabor the router details because I think most readers are probably already using a package based router or rolled their own.

The important details are as follows.
  1. Provides a :route/:action/:id type of mapping.
  2. Implements a Meteor.request with
    1. setController sets this.controller which is Meteor.request.controller
    2. setAction sets this.controller which is Meteor.request.action
    3. setQuery sets this.controller which is Meteor.request.query
  3. Implements the typical helper based content rendering other routers use via Template[Meteor.get_template()]()
  4. The Meteor.get_template() looks something like this.
Meteor.get_template = function() {
  if(Meteor.request.action){
    Meteor.view = Meteor.request.controller + '/' + Meteor.request.action;
  } else {
    Meteor.view = Meteor.request.controller;
  }
  try {
      if ( !_.has(Template, Meteor.view) )
        throw new Meteor.Error(404, "Not found", "The page does not exist.");
  } catch (e) {
    Template.error.informations = function () {
      return e;
    }
    return 'error';
  }
  return Meteor.view;
}

In the end, I didn't even really use the action or id url model and I rarely use the Meteor.request.action and never use the Meteor.request.query. 

Once I started working with Meteor it became obvious that it was unnecessary to follow the typical CRUD and REST type of paradigm.

Though I did realize that the Meteor.request.controller was very valuable for a simple and efficient way to handle CRUD routing.

First, in line with other frameworks I decided that urls and the value of Meteor.request.controller are the name of the collection for which a CRUD action is going to take place.  Therefore, /discounts must render a generic interface to read, insert, update and remove records from Discounts and for that matter any other /collection respectively.

Second, to implement the READ for my CRUD I created a client handlebars helper.  This is a highly simplified version of what I did.  Mine is more complex and you can pass a name, query, sort field and various other options.  This provides you a basic idea and shows the value of the Meteor.request.controller and how it is used to access the model.  To continue with our Discounts example, this helper calls Discounts.getRecords().

Handlebars.registerHelper('get_records', function(){
  return window[Meteor.request.controller.capitalise()]['getRecords']();
});

Third, instead of implementing a get_record on each collection model I extended Meteor.collection.  Again, this is highly simplified just to give you an idea.  You would probably pass options such as a query, sorting information and whatever else you deem necessary.    Continuing with our Discounts example this returns Discounts.find().

Meteor.Collection.prototype.getRecords = function(){
  return this.find();
};

Fourth, now I was able to implement the READ part of any CRUD interface with nothing more than handlebars code.  For example, in my application if you are on /discounts or /tendertypes the page merely has the following clause with whatever custom elements you want to display.  Again, continuing with our Discounts example because we are on /discounts the value of Meteor.request.controller is the word discounts and therefore the Handlebars helper get_records calls Discounts.getRecords.  The same holds true for ANY url.   I never have to write a template helper again to get records for rendering.

{{#each get_records}} ... {{/each}}

Fifth, for inserts and updates of records with multiple fields I like to use the form tag.  Remember, I wanted to only write HTML and never have to write a Meteor event map handler again for most cases.  I do this with a single form.   Using the example of Discounts, both the insert form and update form use the same template.

I use Twitter bootstrap tabs so I have to track the tab name with Sessions.  If you are using Twitter bootstrap tabs you already understand why that is important.

I used to use the Meteor.request.action and if this makes any sense to you I think you can see how this would work and I will provide examples of both

Both my insert and update tabs just include the form.

{{> discounts_form}}

The form is rendered with the following tag.

<form class="form-horizontal well" id="{{active_tab}}Form"> 
The Handlebars helper looks like this.

Handlebars.registerHelper('active_tab', function(){
  return Session.get('active_tab');
});

I set the Session variable when the tab show event triggers.

$('body').on('show','a[data-toggle="tab"]', tabs_show());

tabs_show: function(){
  Session.set('active_tab', event.target.hash.replace("#","") );
}

Now, you could accomplish the same thing by using the Meteor.request.action.  I know my use of the Twitter bootstrap tabs probably complicates the understanding of this strategy.

So, if you were using the Meteor.request.action method you would do something like this.  You would have links for insert and update.  Such as /discounts/insert and /discounts/update/_id which each render a template that just includes the form in the same fashion describe above via the Twitter bootstrap tab method.

<form class="form-horizontal well" id="{{action}}Form"> 

With a corresponding Handlebars helper.

Handlebars.registerHelper('action', function(){
  return Meteor.request.action;
});

Now, only the update action requires a little special sauce.  Basically, you need an event that sets a Session variable place holder.  You can use a button or a link but however you do it just wire up a generic handler in your template event map.  Yes, you still have to use event maps to a degree.  You could probably abstract this a bit more but this provides you with some flexibility to control the variables passed to the handler.

'click .btnEdit': function(event, template){buttonEdit(event, template, this, Meteor.request.controller.capitalise())}

buttonEdit: function(event, template, that, name){
  event.stopPropagation();
  event.preventDefault();
  Session.set('editing', that._id);
}

My buttonEdit handler has more involved processing logic like checking ACL, the above is just a simplified example.  An example might be adding action validation to buttonEdit.

window[name].validateAction(that._id, 'update');

If you guessed correctly, yes this calls a validateAction method on the Collection by the Meteor.request.controller received from the  /url.  I used validateAction to validate that the user can perform that action against that collection via an ACL subsystem.  Use your imagination though, because you can call any method on any collection by collection name.

You can now render your input elements on the form updates quite easily and use the same form for inserts.

Your insert template merely includes.

{{> discounts_form}}

Your update template merely includes the following.  Obviously, you can see it is using a get_record and get_editing helper.

{{#with get_record get_editing}}  
   {{> discounts_form}}
{{/with}}

The corresponding helpers are.

Handlebars.registerHelper('get_editing', function () {
  return Session.get('editing');
});

Handlebars.registerHelper('get_record', function(id){
  if(id){
      return window[Meteor.request.controller.capitalise()]['getRecord'](id);
  }
});

The corresponding Meteor.collection.getRecord.

Meteor.Collection.prototype.getRecord = function(id){
  if(id)
  return this.findOne(id);
};

Input elements are dead simple.

<input rel="tooltip" title="The name must be unique." type="text" id="name" name="name" {{#if name}}value="{{name}}"{{/if}}>

While that seems like a lot, now all I have to do is write HTML/Handlebars forms.  There is no other work involved.  No models to write, except for custom validators.  No template helpers.  No template event maps.

Sixth, now it is data processing time!  Happy! Happy! Joy! Joy!  This gets even easier.  So, I wire up a generic event handler to a generic submit button.

For example, in my case using Twitter bootstrap tabs.

<button id="{{active_tab}}" type="button">Submit</button>

Or just using the action model.

<button id="{{action}}" type="button">Submit</button>

Which can be wired up with JQuery like this.

$(document).on("click", '#insert', Meteor.eventhandler.btnClickHandler());
$(document).on("click", '#update, Meteor.eventhandler.btnClickHandler());

So, if the Meteor.request.action or in my case the active_tab has a value of insert or update when the button is clicked on the form the event handler is fired.

Seventh, one EventHandler to rule them all!  Again, like all of the above, I have kept this as simple as possible.  This is a trimmed down version of my event handler.  So, what is happening here.

  1. The btnClickHandler is like an action router and calls a method for the corresponding target.
  2. Grab all the form params with a simple Jquery plugin that puts them in a JSON doc that can be inserted directly into the database.
  3. Leverage the Meteor.request.controller to call a validateParams and/or validateAction method on your model.  In this example, that code would call Discounts.validateParams and/or Discounts.validateAction.  Put any validation methods on your /lib/Discounts.js collections model (or however you manage models) so that you can use it server and client side to do things like validate the params or validate that the user can execute this method based on an ACL.

var EventHandler = Base.extend({
  btnClickHandler: function(){
    return function (event) {
      event.preventDefault();
      Meteor.eventhandler[event.currentTarget.id](event);
    }
  },
 insert: function(event){
    event.preventDefault();
    var params =  $('#insert-form').toJSON(); 
    try{
      window[Meteor.request.controller.capitalise()]['validateParams'](params);
      window[Meteor.request.controller.capitalise()]['insert'](params, function(error, _id){
        if(error){
          // do something clever
        } else {
          // do something even more clever
          $("#insertForm").reset();
        }
      });
    } catch(error) {
      // this better be something clever 
    }
  },
  update: function(event){
    event.preventDefault();
    try{
      var params =  $('#updateForm').toJSON();
      //console.log('_eventhandler.js update params: ' + EJSON.stringify(params) );
      window[Meteor.request.controller.capitalise()]['validateAction'](Session.get('editing'), 'update');
      window[Meteor.request.controller.capitalise()]['validateParams'](params, Session.get('editing'));
      window[Meteor.request.controller.capitalise()]['update']({_id: Session.get('editing')}, { $set : params }, function(error){
        if(error){
          // do something clever
        } else {
          // do something even more clever
        }
      });
    } catch(error) {
      // this better be something clever 
    }
  },
});

Meteor.eventhandler = new EventHandler;

So, you never have to write an event map handler for an insert or an update.  But wait, we are not done, we still have to handle removes.

Eighth, handling the big D of CRUD which is actually remove in Mongo.  This should be very simple for you to understand.  You need a button or link on the CRUD page that renders the records.

Here is a simple example.  For remove, I use a custom element instead of leveraging the Meteor Sessions.  It just seemed easier.  It is after all just a remove.

<button id="remove" rel="tooltip" title="remove" data-id="{{_id}}">

The JQuery binding.

$(document).on("click", '#remove', Meteor.eventhandler.btnClickHandler());

The method for the event handler.

remove: function(event){
    event.preventDefault();
    try{  
      var data_id = event.currentTarget.getAttribute('data-id');
      window[Meteor.request.controller.capitalise()]['validateAction'](data_id, 'remove');
      window[Meteor.request.controller.capitalise()]['remove']({ _id: data_id }, function(error){
        if(error){
          // do something clever
        } else {
          // do something even more clever
        }
      });      
    } catch(error) {
      // this better be something clever 
    }
  },

Now, where you see the comments about being clever, clearly be clever.  I do things like message back to the UI through a flash object that has methods for updating the flash divs.  You can do things like implement bootbox to ask if they are sure they want to remove/delete something.  Use your imagination.  I stripped out all of my magic to make the concept easier to understand.

So, personally I think that is quite simple.  I wrote the code once and now I can quickly, efficiently and effectively create templates for.

  1. CREATE: inserting records.
  2. READ: displaying multiple records.
  3. UPDATE: updating records.
  4. DELETE: removing records.
All without writing any real javascript.  In fact, once I built this I was able to (and I am still able to) roll out really fast iterations when a new data model is added.

You can even get more advanced.  I have abstracted a generic event handler for toggle records as active or published, inserting and updating subfields and toggling parent and child relationships between records.  


Ultimately, some interfaces will require more pizzazz.  In my case, I don't use this type of design in the actual cash register interface and some of the more complex interfaces like purchase and transfer orders or inventory counts.  I also don't use this on interfaces that are more spreadsheet oriented, though I have abstracted that to some easy to use boiler plate as well.   Though, this strategy is very efficient and effective for managing all of the underlying record sets of the system.

Now, I suspect you are wondering about security.
  • How do you prevent users from seeing all records or all collections?  
  • Can users could just submit random parameters and they will be inserted?
  • How do you prevent users from accessing a form?
  • How do prevent users from accessing sections of the application they shouldn't?
  • How do you prevent users from seeing parts of the application they shouldn't?
Simple.
  1. Implement or use a resource based security system.  Namely something with groups, which have one or more permissions or roles.  Users are assigned to one ore more groups and inherit those permissions.
  2. Publish on a need to know basis based on your resource based security system.
  3. Implement model validators and accessors.  For example, I like to use Model.validateParams() and Model.validateAction().
  4. Use _.extend and _.pick to strip out params that don't belong.
  5. Lock down your app.  Use allow and deny.
  6. Use your resource based security system to limit what the user can see and do.
I will explore these briefly.

First, implement or use a resource based security system.  I believe there are packages out there.  When I started developing my application there weren't.  So, I built one and have been using it every since.  Basically, I have a Groups and a Sections collection.  Sections are areas of the application, which as illustrated above with the Meteor.request.controller usually correspond to a collection.

Sections, my Sections generally have a document structure like this.  As you can see, there is a controller, name, description and various other elements.  Notice, the acl_group_fetch and other acl fields.  Those are the groups that have access to that section of the application and subsequently that collection and can execute those rights.  You could call this anything.  It could be called Resources and maybe should rightfully be called that.

{
"active" : true,
"parent" : "AeKqvL4ot2Rt7f5La",
"controller" : "discounts",
"name" : "Discounts",
"description" : "Discount Management System",
"created_td" : ISODate("2013-03-30T18:26:03.714Z"),
"updated_td" : ISODate("2013-03-30T18:26:03.714Z"),
"client_updated_td" : ISODate("2013-03-30T18:26:03.714Z"),
"show_on_menu" : true,
"menu_order" : 12,
"acl_group_fetch" : [
"SQmFtWTwJhkTRtMFB",
"PP7iXtM7bHEfzPgjT"
],
"acl_group_insert" : [
"SQmFtWTwJhkTRtMFB",
"PP7iXtM7bHEfzPgjT"
],
"acl_group_update" : [
"SQmFtWTwJhkTRtMFB",
"PP7iXtM7bHEfzPgjT"
],
"acl_group_remove" : [
"SQmFtWTwJhkTRtMFB",
"PP7iXtM7bHEfzPgjT"
],
"_id" : "HQAMEHKqKdWAGbPm5"
}

Groups look something like this.  As you can see there are Users assigned to the group.  The group has one or more sets of rights on one or more Sections.

{
"_id" : "PP7iXtM7bHEfzPgjT",
"active" : true,
"client_updated_td" : ISODate("2013-04-11T18:17:57.156Z"),
"created_td" : ISODate("2013-03-30T18:26:03.714Z"),
"name" : "Company Managers",
"updated_td" : ISODate("2013-04-11T18:17:57.804Z"),
"users" : [
"w3RFM3bQMbEfT8kbB"
]
}

I also store what Groups the User belongs to on their document record.  The example is trimmed to just show the groups and basic user data.

{
  "_id" : "w3RFM3bQMbEfT8kbB",
  "groups" : [
    "GyNZM3E6tGxMNYNkf",
    "PP7iXtM7bHEfzPgjT",
    "kmygimm9F8SPJFs7X",
    "k8K5RuHQdoqaJo9Cq",
    "6r8nPzsaNCpewxnGE"
  ],
  "updated_td" : ISODate("2013-05-20T17:34:15.857Z"),
  "username" : "krusty.the.clown"
}

To implement the resource based security system it is as simple as the following.

Resource based READ via a publish using our Discounts example again.  We are publishing on a need to know basis according to our resource based ACL.

Meteor.publish("discounts", function() {
  if(this.userId){
    if( Sections.find({
      controller: 'discounts',  
      acl_group_fetch: {
          $in: Meteor.users.findOne(this.userId).groups
        }
      }).count() > 0 ){
      return Discounts.find();
    } else {
      return null;
    }
  } else {
    return null;
  }
});

Here is an example of resource based CREATE aka INSERT using our Discounts example again.  

See the reuse of the validator Discounts.validateParams?  

You would have put something in there to validate params and potentially prevent submission of validate or invalidate params.  If you remember, this is executed on the client in the event handler, but if the client hacks and still sends the insert to the server it is executed on the server again.  The advantage is that on the client the params were a JSON document and on the server they are also a JSON document.  The same code can easily execute on both the client and server.  

Throw errors from your validators so you can catch them on the client or the server.  The nice thing is that if you throw an error from validateParams in the below example the insert method doesn't return true.  The error is caught on the client side in the event handler and can be nicely handled in the UI.  Use the _.extend and _.pick to pluck out only the variables you want inserted.

Discounts.allow({
  insert: function (userId, doc) { 
  try{
    if( Sections.find({
      controller: 'discounts',  
      acl_group_insert: {
        $in: Users.getGroups(userId)
      }
    }).count() > 0 ){
      Discounts.validateParams(doc);
      // do something clever here like use _.extend(_.pick(doc, 'this', 'that'))
      return true;
    }else {
        return false;
      } 
    }catch(error){
      throw new Meteor.Error(600, 'Server error: ' + error);
      return false;
    }
  },
});

Here is a quick example of the Discounts.validateParams validator showing if the Discount.value isNAN throw an error.  This will be caught client side, but if the client tampers with the code, it is also called server side in the allow insert and because it throws an error it is caught and rethrown to the client and of course caught in the event handler insert method.

Discounts.validateParams = function(params, id) {
  if(params.value){
    if( isNaN(params.value) ){
      throw 'Discount value must be a decimal number.';
    }
  }
};

I think from here you can figure out update and remove on your own.  

So, that answers #1 through #5 but not #6.  For a little icing on the cake, here is what I do to prevent them from seeing or accessing parts of the application they should not.

Basically, my menu system renders based on their subscription from Sections.

Meteor.publish("sections", function(groups) {
  if(this.userId){
    return Sections.find(
      {
        active: true,
        acl_group_fetch: {
          $in: Meteor.users.findOne(this.userId).groups
        }
      });
  } else{
    return null;
  }
});

So, the UI renders the Sections.name based on the above publication.  So, they only see menu or navigational elements based on their subscription which is controlled by the underlying resource based security system.

Yes, a user could find the templates because all templates are served to all clients, but the average user does not know that.  A crafty user could get to a template, but because all other collections are published on a need to know basis according to the resource based ACL nothing would render and they would not be able to insert, update or remove records.  So, it becomes a bit of a moot point really.

Though, just for a little extra protection and to make it painful for a crafty user I wired up a few Handlebars helpers that check if the user has access to that Sections.controller and it's corresponding acl fields.  Of course, if the Section record document is not in their collection they are subscribed to, which is already a need to know ACL controlled publication, nothing renders. 

I boiled it down to the simplest recipe possible.  Personally, I found it really quite easy to implement.  This is really the first thing I implemented alongside the Users/Groups/Sections resource based ACL system because my application design demanded it.  

It took a couple of weeks and then after that developing my application was fast tracked because I was able to get routine, boiler plate, CRUDy activities resolved to simple HTML/Handlebars code.  In fact, most of the time I just cut & paste and change a couple of things when adding new collections to my application.  I could probably just write generators or factories that take a dictionary of names to dynamically render the HTML read, insert and update elements rather than having the actual HTML/Handlebars.  Though, designed this way I could bring on someone who doesn't know Javascript that well and can still develop well crafted data management interfaces.

If you have any questions, please feel free to contact me.

2 comments:

  1. That was a very clever approach on build any meteor app. I'm going to try to replicate this on my own app. Thanks! Steeve.

    ReplyDelete