The current yEd versions come bundled with Java 7 or Java 8. For these versions of Java it should no longer be necessary to explicitly specify the maximum amount of available memory. (Java 7 and Java 8 do come with auto-detection for available memory.)
If that does not work for you for some reason, you can try adding the following lines to your Info.plist file:
<key>JVMOptions</key>
<array>
<string>-Xmx768m</string>
</array>
<key>VMOptions</key>
<array>
<string>-Xmx768m</string>
</array>
right before the final occurrence of
</dict>