Cracking Java byte-code encryption

Why Java obfuscation schemes based on byte-code encryption won't work

Q: If I encrypt my .class files and use a custom classloader to load and decrypt them on the fly, will this prevent decompilation?

Featured Resource

Presented by Coverity

2013 Coverity Scan Open Source Report

Learn about the state of quality in the world's leading open source projects in the 2013 Coverity Scan
Learn More

A: The problem of preventing Java byte-code decompilation is almost as old the language itself. Despite a range of obfuscation tools available on the market, novice Java programmers continue to think of new and clever ways to protect their intellectual property. In this Java Q&A installment, I dispel some myths around an idea frequently rehashed in discussion forums.
The extreme ease with which Java .class files can be reconstructed into Java sources that closely resemble the originals has a lot to do with Java byte-code design goals and trade-offs. Among other things, Java byte code was designed for compactness, platform independence, network mobility, and ease of analysis by byte-code interpreters and JIT (just-in-time)/HotSpot dynamic compilers. Arguably, the compiled .class files express the programmer's intent so clearly they could be easier to analyze than the original source code.
Several things can be done, if not to prevent decompilation completely, at least to make it more difficult. For example, as a post-compilation step you could massage the .class data to make the byte code either harder to read when decompiled or harder to decompile into valid Java code (or both). Techniques like performing extreme method name overloading work well for the former, and manipulating control flow to create control structures not possible to represent through Java syntax work well for the latter. The more successful commercial obfuscators use a mix of these and other techniques.
Unfortunately, both approaches must actually change the code the JVM will run, and many users are afraid (rightfully so) that this transformation may add new bugs to their applications. Furthermore, method and field renaming can cause reflection calls to stop working. Changing actual class and package names can break several other Java APIs (JNDI (Java Naming and Directory Interface), URL providers, etc.). In addition to altered names, if the association between class byte-code offsets and source line numbers is altered, recovering the original exception stack traces could become difficult.

Then there is the option of obfuscating the original Java source code. But fundamentally this causes a similar set of problems.

Encrypt, not obfuscate?

Perhaps the above has made you think, "Well, what if instead of manipulating byte code I encrypt all my classes after compilation and decrypt them on the fly inside the JVM (which can be done with a custom classloader)? Then the JVM executes my original byte code and yet there is nothing to decompile or reverse engineer, right?"
Unfortunately, you would be wrong, both in thinking that you were the first to come up with this idea and in thinking that it actually works. And the reason has nothing to do with the strength of your encryption scheme.

A simple class encoder

To illustrate this idea, I implemented a sample application and a very trivial custom classloader to run it. The application consists of two short classes:

public class Main
{
    public static void main (final String [] args)
    {   
        System.out.println ("secret result = " + MySecretClass.mySecretAlgorithm ());
    }
} // End of class
package my.secret.code;
import java.util.Random;
public class MySecretClass
{
    /**
     * Guess what, the secret algorithm just uses a random number generator... 
     */
    public static int mySecretAlgorithm ()
    {
        return (int) s_random.nextInt ();
    }
    private static final Random s_random = new Random (System.currentTimeMillis ());
} // End of class

My aspiration is to hide the implementation of my.secret.code.MySecretClass by encrypting the relevant .class files and decrypting them on the fly at runtime. To that effect, I use the following tool (some details omitted; you can download the full source from Resources):

public class EncryptedClassLoader extends URLClassLoader
{    
    public static void main (final String [] args)
        throws Exception
    {        
        if ("-run".equals (args [0]) && (args.length >=  3))
        {
            // Create a custom loader that will use the current loader as
            // delegation parent:
            final ClassLoader appLoader =
                new EncryptedClassLoader (EncryptedClassLoader.class.getClassLoader (),
                new File (args [1]));
            
            // Thread context loader must be adjusted as well:
            Thread.currentThread ().setContextClassLoader (appLoader);
            
            final Class app = appLoader.loadClass (args [2]);
            
            final Method appmain = app.getMethod ("main", new Class [] {String [].class});
            final String [] appargs = new String [args.length - 3];
            System.arraycopy (args, 3, appargs, 0, appargs.length);
            
            appmain.invoke (null, new Object [] {appargs});
        }
        else if ("-encrypt".equals (args [0]) && (args.length >= 3))
        {
            ... encrypt specified classes ...
        }
        else
            throw new IllegalArgumentException (USAGE);        
    }
    
    /**
     * Overrides java.lang.ClassLoader.loadClass() to change the usual parent-child
     * delegation rules just enough to be able to "snatch" application classes
     * from under system classloader's nose.
     */
    public Class loadClass (final String name, final boolean resolve)
        throws ClassNotFoundException
    {
        if (TRACE) System.out.println ("loadClass (" + name + ", " + resolve + ")");
        
        Class c = null;
        
        // First, check if this class has already been defined by this classloader
        // instance:
        c = findLoadedClass (name);
        
        if (c == null)
        {
            Class parentsVersion = null;
            try
            {
                // This is slightly unorthodox: do a trial load via the
                // parent loader and note whether the parent delegated or not;
                // what this accomplishes is proper delegation for all core
                // and extension classes without my having to filter on class name: 
                parentsVersion = getParent ().loadClass (name);
                
                if (parentsVersion.getClassLoader () != getParent ())
                    c = parentsVersion;
            }
            catch (ClassNotFoundException ignore) {}
            catch (ClassFormatError ignore) {}
            
            if (c == null)
            {
                try
                {
                    // OK, either 'c' was loaded by the system (not the bootstrap
                    // or extension) loader (in which case I want to ignore that
                    // definition) or the parent failed altogether; either way I
                    // attempt to define my own version:
                    c = findClass (name);
                }
                catch (ClassNotFoundException ignore)
                {
                    // If that failed, fall back on the parent's version
                    // [which could be null at this point]:
                    c = parentsVersion;
                }
            }
        }
        
        if (c == null)
            throw new ClassNotFoundException (name);
        
        if (resolve)
            resolveClass (c);
        
        return c;
    }
        
    /**
     * Overrides java.new.URLClassLoader.defineClass() to be able to call
     * crypt() before defining a class.
     */
    protected Class findClass (final String name)
        throws ClassNotFoundException
    {
        if (TRACE) System.out.println ("findClass (" + name + ")");
        
        // .class files are not guaranteed to be loadable as resources;
        // but if Sun's code does it, so perhaps can mine...
        final String classResource = name.replace ('.', '/') + ".class";
        final URL classURL = getResource (classResource);
        
        if (classURL == null)
            throw new ClassNotFoundException (name);
        else
        {
            InputStream in = null;
            try
            {
                in = classURL.openStream ();
                final byte [] classBytes = readFully (in);
                
                // "decrypt":
                crypt (classBytes);
                if (TRACE) System.out.println ("decrypted [" + name + "]");
                
                return defineClass (name, classBytes, 0, classBytes.length);
            }
            catch (IOException ioe)
            {
                throw new ClassNotFoundException (name);
            }
            finally
            {
                if (in != null) try { in.close (); } catch (Exception ignore) {}
            }
        }
    }
    
    /**
     * This classloader is only capable of custom loading from a single directory. 
     */
    private EncryptedClassLoader (final ClassLoader parent, final File classpath)
        throws MalformedURLException
    {
        super (new URL [] {classpath.toURL ()}, parent);
        
        if (parent == null)
            throw new IllegalArgumentException ("EncryptedClassLoader" +
                " requires a non-null delegation parent");
    }
    
    /**
     * De/encrypts binary data in a given byte array. Calling the method again
     * reverses the encryption.
     */
    private static void crypt (final byte [] data)
    {
        for (int i = 8; i < data.length; ++ i) data [i] ^= 0x5A;
    }
    ... more helper methods ...
    
} // End of class

EncryptedClassLoader has two basic operations: encrypting a given set of classes in a given classpath directory and running a previously encrypted application. The encryption is very straightforward: it consists of basically flipping some bits of every byte in the binary class contents. (Yes, the good old XOR (exclusive OR) is almost no encryption at all, but bear with me. This is just an illustration.)
Classloading by EncryptedClassLoader deserves a little more attention. My implementation subclasses java.net.URLClassLoader and overrides both loadClass() and defineClass() to accomplish two goals. One is to bend the usual Java 2 classloader delegation rules and get a chance to load an encrypted class before the system classloader does it, and another is to invoke crypt() immediately before the call to defineClass() that otherwise happens inside URLClassLoader.findClass().
After compiling everything into the bin directory:

>javac -d bin src/*.java src/my/secret/code/*.java

I "encrypt" both Main and MySecretClass classes:

>java -cp bin EncryptedClassLoader -encrypt bin Main my.secret.code.MySecretClass
encrypted [Main.class]
encrypted [my\secret\code\MySecretClass.class]

These two classes in bin have now been replaced with encrypted versions, and to run the original application, I must run the application through EncryptedClassLoader:

>java -cp bin Main
Exception in thread "main" java.lang.ClassFormatError: Main (Illegal constant pool type)
        at java.lang.ClassLoader.defineClass0(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:502)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:123)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:250)
        at java.net.URLClassLoader.access00(URLClassLoader.java:54)
        at java.net.URLClassLoader.run(URLClassLoader.java:193)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:186)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:299)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:265)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:255)
        at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:315)
>java -cp bin EncryptedClassLoader -run bin Main
decrypted [Main]
decrypted [my.secret.code.MySecretClass]
secret result = 1362768201

TWEAKERZ OF UGANDA

Sunday, 15 June 2014