Templatized customization a.k.a. never give a gun to a monkey
My good friends at Southworks told me about a saying in Argentina going like this: "mas peligroso que mono con navaja" which loosely translates to "never give a gun to a monkey. " This is a very good approach when thinking about the customization features of your SaaS solution.
No, I am not saying that your customers are monkeys :) The point I am trying to make is that when you allow customization of your application in a shared environment (such as a single instance multitenant saas solution) you need to make sure you are not putting too much power in the hands of the "customizers" who could misuse it, either maliciously or most probably mistakenly.
In our LitwareHR sample application, we allow tenants to customize their UI, business processes, data model and roles, but we do not allow "free form" customization. All the customizations are governed through very specific designers offering the appropriate level of "expressiveness". Allowing the right level of expressiveness is the key element here. If you restrict it too much, the customization looses of its usefulness, if the expressiveness is too large, bad behavior could be introduced.
So from an architecture perspective, we introduced the notion of Templatized Customization. The picture below summarizes the concept, but let me give you some details.
First, you need to define a set of policies that define appropriate behavior in your share environment. Good behavior will depend on many factors, but will primarly be dicated by making sure that customizations of one tenant will not impact the SLAs that you offer to all the others. Real life examples of policies would be "you can bring only up to 3 guest to the communal swimming pool or take your trash out only on Tuesdays" not that doing differently would be wrong for that tenant (inviting 4 people wouldn't be too wrong) but if the majority of the tenants were to invite 4 guests, some tenants would not be able to invite guests at all as the pool would be too full. In other words, policies are there to define faireness of usage.
The second step is to create a designer that restrict the customization of the application to only allow policy-compliant behaviors. It would be great to generate the designer directly from the policies description, but we are not there yet. An important aspect you should consider is to offer multiple designers (each with a different level of expressiveness) based on the relationship you have with the customizer. For example, for your standard customers whom you do not know much of, you would offer the standard tool, which offer the most limited level of customization but, on the good side, minimizes the risk of bad behavior. Alternatively, if the customizer is a trusted partner of yours (who (a) knows your application well and therefore is less prone of doing errors and (b) has a vested interest in making sure your application works well) you could imagine to give him a more powerful designer (e.g. a fully fledge workflow designer). Of course this level of expressiveness could allow the introduction of bad behavior (e.g. infinite loop, processes that don't terminate etc.) but the level of trust in this partner is higher than the risk of bad behavior, which could justify this approach. Another possibility could be to offer yet another set of tools to customers who are willing to pay for additional isolation (i.e. their own server(s)) as having their own server(s) would mitigate the impact of malicious or mistake-driven bad behavior.
In a perfect world, limiting the expressiveness of the various customizers would be sufficient, as only good behavior will be generated out of these designers; but we all know that believing in perfect worlds leads often to disappointment. This is why runtime enforcement or "policing" is required in this architecture. "Runtime policing" makes sure that each tenant behaves in accordance to the rules describing proper behavior. You can think of the policing as the real life police patrolling the cities at night. In theory, if everybody obeyed the law no police would be needed; but since (apparently) sometime some people go beyond the boundaries imposed by the law, the police is necessary to bring these people back in the right track. Of course, like in real life, the degree of severity applied to the entities breaking the law is very diverse.
So in summary, shared environement (such as single instance multi-tenacy) introduce the need of templatized customization. To manage it, you should consider (a) offering multiple designers with different levels of expressiveness and (b) implement a "runtime police" making sure that everybody is running according to plan.
Although not completly novel, these concepts are usually not typical in the design of "on premise" applications where customization is often let loose, but critical in SaaS environements.
For those of you interested in some "behind the scene" material, here is a video showing the white board discussion I had with Fred and Alex Jack (from Southworks) which led to these concepts.