rmic -Xnew is the so-called "new rmic", an effort in 2003 to develop a new implementation of rmic based on javadoc and the doclet API.
The implementation is incomplete, in particular it never implemented the CORBA options to generate IDL or IIOP stubs. Furthermore it is not documented or supported.
rmic -Xnew is problematic for Project Jigsaw because the doclet API is changing. It doesn't seem worth trying to get it working, instead it is time to just remove it.