Following code doesn't work, even if it compiles.

class Echo {
    public static void main (String[] arg) {
        System.out.println("Hello, world!");
    }
}

Perhaps it would be possible to look for other .class files, if the EvalSO.class wasn't found. This appears to be what ideone does already.