After some investigation I was able to get things working, but I'm not sure if this is considered an approved method or not.
In the server.xml file found in /var/lib/vco/app-server/conf I added the following listener configuration to the server section. This will enable tc-server to expose jmx as a socket on a specific port. With authenticate set to false it doesn't use any of the configuration in the password file or access file parameters.
<Listener className="com.springsource.tcserver.serviceability.rmi.JmxSocketListener" port="6969" bind="192.168.0.100" useSSL="false" passwordFile="${catalina.base}/conf/jmxremote.password" accessFile="${catalina.base}/conf/jmxremote.access" authenticate="false"/>
Once I configured this things still weren't working, but I noticed this error when trying to restart vCO:
java.io.IOException: Cannot bind to URL [rmi://192.168.0.100:6969/jmxrmi]: javax.naming.NoInitialContextException: Cannot instantiate class: org.jnp.interfaces.NamingContextFactory [Root exception is java.lang.ClassNotFoundException: org.jnp.interfaces.NamingContextFactory]
This indicated a configuration problem for jndi, and it looks like the properties were set for using jboss based application configuration and jboss libs aren't included with vCO.
I modified the catalina.properties file in /var/lib/vco/app-server/conf and changed the naming configuration to use apache/tomcat libraries instead of jboss.
java.naming.factory.initial=org.jnp.interfaces.NamingContextFactory -> java.naming.factory.initial=org.apache.naming.java.javaURLContextFactory
java.naming.factory.url.pkgs=org.jboss.naming:org.jnp.interfaces -> java.naming.factory.url.pkgs=org.apache.naming
With these changes I was able to connect to jmx via jvisualvm without any issue.