The current yEd installers do not specify a maximum heap size, i.e. the executables produced by the yEd installers do not set the -Xmx option.
Thus I strongly suspect this to be either
a OpenJ9 specific problem or
a antivirus or firewall problem as stated in the error message.
In the latter case, the problem is specific to your system.
While I cannot help with OpenJ9 problems, I can think of two possible solutions:
Use the yEd installer with embedded Java 11 runtime environment. Said Java 11 JRE is an OpenJDK build and has been thoroughly tested on our Windows machines.
Use the Zipped yEd Jar file download, extract the archive to a directory of your choosing, and start yEd from command line with
[path to java binary]/java -jar [path to yEd directory]/yed.jar
where [path to java binary] is something like
and [path to yEd directory] is something like
but the actual values depend on your system.