Tuesday 26 July 2011

Why is ASP.net on mono like a new pet that hasn't been house trained?

...because it leaks all over the place.

We've been working for a while now on an MVC (initially MVC2, now MVC3) application running on mono 2.10.2 with FluentNHibernate 1.2 sitting on NHibernate 3.1, built as a web app but also available via a kiosked browser on the server. This effectively means we've got a web browser pointed at the server 24/7. In addition, it's a fairly heavily AJAX-ed application, pinging for data every second or so, which means we've got a browser sending requests to the application every second, 24/7. Still, not exactly heavy usage.

We've had a very, very hard time getting this working stably, and all of our troubles seem to be coming from the mono stack (possibly nhibernate as well, but even with that problem, it looks more likely that mono is the root cause). If you're putting together a web application using mono, you should probably look out for the following:


Memory leaks when running with sessions enabled

This seems to be something to do with CacheItems being created at a phenomenal rate without (apparently) ever being garbage collected. Even if you see no reason whatsoever why your application would be generating CacheItems, something under the hood will be there merrily churning them out. This will, eventually, bring your system to its knees before the mono process is killed. Switching garbage collectors to sgen (newer, better, see below...) doesn't help. This is almost certainly why mono web applications mysteriously stop working under heavy load according to the mono project.

Workarounds:

 You can, as suggested in the above link, restart your mono process every so often to release the memory. Not an ideal solution, obviously. Alternatively, if you don't need sessions you can run sessionless, which seems to stop this leak from happening (so it seems the CacheItems are being generated by something in the session code...). Or you could figure out what the fricking bug is and fix it.


An apparent race hazard when starting up a web app

If you choose to accept that you'll need to restart mono every so often, the next thing to watch out for is an intermittent exception when starting up. This occasionally manifests as an object not set to an instance of an object exception, but most commonly will appear as duplicate key exception thrown by the RoleManager - even when you aren't using roles for anything. To be honest, this being a race hazard is a total guess, I have no idea what the problem is, but something which is so intermittent just smacks of a race hazard. Since we didn't need sessions enabled, we simply switched to sessionless mode and stopped pursuing this one. Interestingly, since addressing all of the other problems listed in this post we've not seen any of these problems. So perhaps again something to do with the session code in mono...?

Workarounds:

 No idea. This seems to have just gone away as we've worked around all the rest of the weirdnesses. Best of luck to you if you're trying to solve this one. Sorry I can't help more.

Running the MVC sessionless is broken in mono

You should be able to run sessionlessly simply by adding the tag
< sessionstate mode="Off" > 
inside the system.web node in your web.config file. Providing you're not actually using TempData for anything, this should be fine. However, you'll get the following exception:

System.ArgumentNullException: Argument cannot be null.
Parameter name: httpSessionState
  at System.Web.HttpSessionStateWrapper..ctor (System.Web.SessionState.HttpSessionState httpSessionState) [0x00000] in <filename unknown>:0 
  at System.Web.HttpContextWrapper.get_Session () [0x00000] in <filename unknown>:0 
  at System.Web.Mvc.SessionStateTempDataProvider.LoadTempData (System.Web.Mvc.ControllerContext controllerContext) [0x00000] in <filename unknown>:0 
...

Workarounds:

According to the mighty Stack Overflow this has been fixed in .net. Alas not in mono. We worked around this by creating a dummy temp data provider
public class DummyTempDataProvider : ITempDataProvider
  {
   IDictionary ITempDataProvider.LoadTempData (ControllerContext controllerContext)
   {
    return null;
   }
   
   
   void ITempDataProvider.SaveTempData (ControllerContext controllerContext, IDictionary values)
   {
   }
  }

And then making sure all controllers overrode the CreateTempDataProvider function (since all our controllers inherit from a common base, we just stuck it in there).
protected override ITempDataProvider CreateTempDataProvider ()
  {
   //return base.CreateTempDataProvider();
   return new DummyTempDataProvider();
  }


Using the nHibernate WebSessionContext leaks memory

Our application uses nHibernate for persistence. Unfortunately, for some reason we've not been able to identify, if you use the WebSessionContext (which is what you should be using for a web application), you'll find that you are, once again, leaking memory. 

Workarounds:

 
We found that the ThreadStaticSessionContext has no such trouble and - on a hunch - tested out a hybrid session context we had lying around from an earlier project. This works fine, no leaks. The only real difference between this and the WebSessionContext seems to be that our HybridSessionContext uses the Items hashtable in the HttpContext directly to store the session whereas the WebSessionContext creates a new hashtable inside the items hashtable which uses the ISessionFactory as the key to the Session. Why this should cause a leak is a mystery; it may have nothing to do with the leak. But it's the only obvious difference.
internal class HybridSessionContext : CurrentSessionContext
 {
  [ThreadStatic]
  private static ISession threadSession = null;
  private const string webSession   = "YourNamespace.YourProject.WebContext";
  
  
  public HybridSessionContext(ISessionFactoryImplementor factory)
  {
  }
  
  
  protected override ISession Session
  {
   get 
   { 
    return (HttpContext.Current != null) ? HttpContext.Current.Items[webSession] as ISession : threadSession;  
   }
   
   set 
   { 
    if (HttpContext.Current != null)  HttpContext.Current.Items[webSession]  = value;
    else         threadSession        = value; 
   }
  }
 }


The boehm garbage collector causes further leaks

After all that, you may still be seeing leaks. We've found that the boehm garbage collector that mono uses by default is rather easily confused, especially, for some reason, when it comes to web applications. 

Workarounds:

Try switching to mono-sgen, the generational garbage collector (see the link above). We found this moved us from a still inexplicably-leaking application to one behaving normally... while testing on xsp. Because...



mod_mono causes even more leaks

After all of the above were resolved, and everything looked peachy testing locally through XSP, we pushed the system onto a target machine and ran through mod_mono instead.... and got yet more memory leaks.

Workarounds:

Run through xsp and use mod_proxy to serve it via apache. And you'll get back to not having leaks.

To summarise

Currently, I'd say that ASP.net on mono is a bit of a mess. You can work round the problems to some extent, but we've spent a ridiculous amount of time dealing with them. Unfortunately the mono mailing list on this subject seems to have gone very quiet (Dan in the list is the other part of the "we" that I refer to in this post), and we're sufficiently out of time that we simply cannot continue to bughunt for the underlying causes of these issues without at least some support from folks who know their way around the guts of mono better than we do.