Circular layout does not support group nodes. I.e. the algorithm first removes all groups, then arranges the graph without groups, then reinserts the removed groups again. This is why you get the result in the second screenshot.
You can try to use circular layout's support for node types to get an arrangement close to the first screenshot, though:
- For each group node, assign the same fill color to each child node. Use a different color for each group node.
- In the circular layout settings on tab "General" set option "Layout Style" to "Single Cycle".
- In the circular layout settings on tab "Partition" set option "Node Types" to "Defined by Color".
Alternatively, you can use the same label text for all child nodes of a given group node and set "Node Types" to "Defined by Label".
While this approach might still result in overlapping groups (particularly if one group has many more child nodes than the other groups), it should reduce the likelihood of such overlaps.