When label is not positioned on top of connector line, when the line is re-positioned the label keeps its relative position and moves with the line.
When the label is positioned on top of the line (in my case with white background), the re-positioned handle shows when clicking the edge, but moving the handle only moves the label. Maybe this could be changes so that 1st mouse click allows to grab the handle and then move both together, with double-click meaning to edit the label.
