Spring Cloud Config and Self-Signed Certificates

My corporate IT security team decreed earlier this year that internal-facing, infrastructure servers must implement HTTPS using standard self-signed SSL certificates, replacing the official corporate certificates used previously.

At first glance, this sounds simple enough — register the certificates in your browser and continue on unaffected. Subsequently, we realized that configuration changes were not getting to our Docker instances using the Spring Cloud Config server, and we didn't understand why.

What Is a Spring Cloud Config?

From the Spring Cloud Config homepage:

"Spring Cloud Config provides server and client-side support for externalized configuration in a distributed system. With the Config Server, you have a central place to manage external properties for applications across all environments. The concepts on both client and server map identically to the Spring Environment and PropertySource abstractions, so they fit very well with Spring applications but can be used with any application running in any language. As an application moves through the deployment pipeline from dev to test and into production you can manage the configuration between those environments and be certain that applications have everything they need to run when they migrate. The default implementation of the server storage backend uses git so it easily supports labeled versions of configuration environments, as well as being accessible to a wide range of tooling for managing the content. It is easy to add alternative implementations and plug them in with Spring configuration."

So, in a nutshell, a Spring Cloud Config server clones a Git repo and provides configuration to applications. As new or updated configurations are pushed to the underlying repo, the server pulls the changes and provides the updated configurations.

What is especially useful is a Spring Cloud Config implemented as a Docker service, where other services deployed in the swarm reference the Spring Cloud Config server internally for the environment the swarm supports, e.g., production, testing, development, etc.

Investigation

Check Server Health

Now, to try and find out what's happening, the server has a /health endpoint. Perhaps, it'll provide something useful.

[ssosna@slartibartfast spring-config]$ curl http://slartibartfast:9101/health
{"status":"DOWN"}[ssosna@slartibartfast spring-config]$ 

This is not very helpful since I already know the service is done.

I learned that /health is secured but can be made unsecured by defining management.security.enabled=false. After adding this environment variable to my Docker service, I got this:

[ssosna@slartibartfast spring-config]$ curl http://slartibartfast:9101/health
{"status":"DOWN","diskSpace":{"status":"UP","total":52576092160,"free":37885374464,"threshold":10485760},"rabbit":{"status":"UP","version":"3.7.8"},"refreshScope":{"status":"UP"},"configServer":{"status":"DOWN","repository":{"application":"app","profiles":"default"},"error":"java.lang.IllegalStateException: Cannot clone or checkout repository"}}[ssosna@slartibartfast spring-config]$ 

Scroll right for the interesting bit: java.lang.IllegalStateException: Cannot clone or checkout repository. This is very helpful, but I have no idea why.

Server logs were no more helpful, logging the same IllegalStateException but with no additional context to help understand what's happening.

Search Spring Source

Since no breadcrumbs were provided at run-time, I cloned the project and dug into the code, finding the error string in the class JGitEnvironmentRepository. The method refresh() creates a Git client and attempts to clone or pull the repository. The biggest revelation is that Spring uses the JGit for its Git operations.

Test JGit Alone

JGit is an API-based, pure-Java implementation of Git, without any external dependencies on a separate Git installation or other OS-specific requirements. To simplify testing, I added a simple JGit repository clone to an existing project.

try {
    System.out.println ("************ Before cloning.");
    Git.cloneRepository().setURI("https://Scott_Sosna@stash.mycompany.com/scm/project/project-to-clone.git").call();
    System.out.println ("************ After cloning.");
} catch (Exception e) {
    System.out.println ("************ JGit Exception: " + e);
}

The Maven build created the Docker image and pushed it into Docker. I then created a container and started interactively, and voila!

************ Before cloning.
************ JGit Exception: org.eclipse.jgit.api.errors.TransportException: https://Scott_Sosna@stash.mycompany.com/scm/project/project-to-clone.git: Secure connection to https://Scott_Sosna@stash.mycompany.com/scm/project/project-to-clone.git c

The new self-signed certificates are not available inside Docker, causing the repository clone to fail.

Solution

The certificates need to be added to the Java keystore inside the Docker container.

Add Certificates to the Project

The certificates are resources added to the project wherever appropriate for your build tool. Maven projects would add the certificates in main/src/resources. I created a specific subdirectory for them:

Load Certificates to Java Keystore

The Dockerfile definition file needs to include steps for copying the certificates to the Docker image and updating the global keystore.

USER root
COPY src/main/resources/certs $JAVA_HOME/jre/lib/security/
RUN \
    cd $JAVA_HOME/jre/lib/security && \
    echo $JAVA_HOME && \
    keytool -keystore cacerts -storepass changeit -noprompt -trustcacerts -importcert -alias Dell101 -file "MyCompany Issuing Certificate Authority 101.crt" && \
    keytool -keystore cacerts -storepass changeit -noprompt -trustcacerts -importcert -alias Dell102 -file "MyCompany Issuing Certificate Authority 102.crt" && \
    keytool -keystore cacerts -storepass changeit -noprompt -trustcacerts -importcert -alias Dell301 -file "MyCompany Issuing Certificate Authority 301.crt" && \
    keytool -keystore cacerts -storepass changeit -noprompt -trustcacerts -importcert -alias Dell302 -file "MyCompany Issuing Certificate Authority 302.crt" && \
    keytool -keystore cacerts -storepass changeit -noprompt -trustcacerts -importcert -alias DellRoot2015 -file "MyCompany Root Certificate Authority 2015.crt"

Load and Go

Your newly-built Docker image should contain the self-signed certificates, and the Spring Cloud Config server should behave as expected.

Final Thoughts

• The environment variable spring.cloud.config.server.git.skipSslValidation is documented but did not affect how JGit functioned.
• JGit does almost no logging, making it difficult to do any debugging when problems like this arise.