When using a layout algorithm to arrange a graph, the quality of the result very much depends on the structure of the graph. And in surprisingly many cases, the graph structure is really all that matters. However, there are cases where semantics matter so much that it is very hard to get a good result from a general-purpose layout algorithm. In these cases, you need to use a domain-specific algorithm to get good results.
Your graph seems to be a family tree. And family trees fall in the "semantics matter" category. I.e. the only algorithm that really works well for family trees is, well, "Layout" -> "Family Tree", because this algorithm knows about "male", "female", and "family" nodes. (The family tree layout algorithm does not support the radial arrangement you seem to be looking for, though.)
In short, I think radial layout (or any other layout algorithm except for the family tree layout algorithm) is not a good choice for arranging your graph.
To answer your questions:
- The distance depends on the settings "Minimal Circle Distance" and "Minimal Node Distance" and the size and number of nodes on the circle. And while you can try to reduce the value of the two distance settings or the size of your nodes, you probably cannot do much about the number of nodes on the circle. The algorithm will always ensure that its circles are large enough to prevent the nodes on a given circle from overlapping each other. Moreover, the algorithm treats all nodes as circular disks when calculating the radius of a circle. As a result, the algorithm usually does not find the minimum radius that would satisfy the "a circle's nodes do not overlap each other" rule for rectangular nodes. (The algorithm's approximation for the minimum radius depends of the aspect ratio of your nodes. If said aspect ratio is close to 1:1, then the calculated radius will be close to the minimum radius. If the aspect ratio is very large or very small, the calculated radius might be a lot greater than the minimum radius.)
In your sample graph, there seem to be really a lot of nodes on the "daughter - son - son - son - son - wife" circle. As a result, I really doubt that said circle's radius can be reduced much. - Radial layout has no notion of "husband" or "wife". As such, there is no way to tell the algorithm to keep a husband node close to its wife node.
(Actually, this is a shortcoming of the yEd's settings UI. The algorithm supports the notion of node types and if you would assign a unique node type for all "married" nodes, the algorithm would indeed try to place these nodes close together. However, the settings UI does not offer a way to specify arbitrary node types. It only offers to derive node types from node color or node label. While the algorithm supports the notion of node types, it will only "group" nodes of the same type, if these nodes are on the same circle, too. This is not the case for the husband and wife nodes in your graph.)
This is part of why the family tree layout works well for family trees: it knows about husbands and wifes (in a way) and places them close to each other. - The "dots" are nodes that represent GEDCOM FAM tags. These are part of your GEDCOM data set. If you do not want to have those nodes, you may not use yEd's GEDCOM import to create your graph. Or, if you import your graph from a GEDCOM file, you will have to remove the family nodes manually afterwards. However, this will probably be a lot or work, because there is no tool that can retain and reconnect edges in a clever way when removing nodes.
Of course, if you remove the family nodes, the family tree layout algorithm will not work (well) for your graph anymore either.
