I didn't find a way to automatically "recalculate" the layout when I open or close a group.
But I have tested the graph demo of yFiles for Silverlight where it seems this feature exists (at least when closing a group inside another group, the parent group is resized)
So does the IncrementalHierarchicLayouter exists directly in yEd? Or is it planned to add such a functionnality?