You're living in your own private AppDomain--or at least you want to be.
However, from the standpoint of Office development and add-ins for Word, Excel, etc., the best policy is to stay out of DefaultDomain (which is where you will land if you don't create a shim--see below) and load into your own AppDomain because even though you may understand when to call ReleaseComObject, chances are some other developer won't. You don't want to be in the same AppDomain as the developer that does this wrong. Also, being in the same AppDomain as another add-in has it's own additional set of problems--what if, for example, some other add-in in that domain has loaded a different version of the Office PIAs than the one you want?
Make sure your add-in loads in its own AppDomain. Today, a shim can make this happen. See these articles:
This is the right approach IMHO for the present. In the future, Microsoft is looking at providing a managed add-in model that will load you into your own AppDomain by default and do other goodness for add-ins.