My NetWare 5.1 server crashes at the moment the installation starts the GUI

Because VMware does not support SVGA and there is no special configuration for NetWare 5 as there is for NetWare 6 you must choose VGA during installation of your NetWare 5.1 server. If you forget this you can restart the GUI with XINIT -nv (see XINIT /?).