Problems like these are very, very difficult to diagnose because we cannot reproduce these dramatic slow downs on our test machines.
One known cause for slow downs (that is not the result of a program error) is memory shortage. If you are working with large diagrams (several hundreds of nodes and edges or several large image nodes), this could be the case. The solution to this particular problem is assigning more memory to the yEd process as explained in How can I prevent 'Out of Memory' errors in yEd? (even if there are no errors).