Spring Boot

Source Code Investigation

Call Stack

  • org.springframework.boot.SpringApplication.run

    • org.springframework.boot.SpringApplication.prepareEnvironment

      • org.springframework.boot.SpringApplication.configureEnvironment

        • org.springframework.boot.SpringApplication.configurePropertySources

PropertySource

  • Testing

    • Precedence (high to low)

      • DynamicValuesPropertySource

        @DynamicPropertySource

      • @TestPropertySource - properties attribute

      • @TestPropertySource - locations attribute

  • PropertySourceLoader

    • PropertiesPropertySourceLoader for loading .properties files
    • YamlPropertySourceLoader for loading .yaml and .yml files

Default precedence order, from highest to lowest, in PropertySource name

  1. configurationProperties

    org.springframework.boot.context.properties.source.ConfigurationPropertySourcesPropertySource

    To be used with PropertyResolver or added to the Environment

  2. servletConfigInitParams

    In the case that the application is not a Java Servlet application, this PropertySource will be an instance of StubPropertySource.

  3. servletContextInitParams

    In the case that the application is not a Java Servlet application, this PropertySource will be an instance of StubPropertySource.

  4. systemProperties

    org.springframework.core.env.PropertiesPropertySource

  5. systemEnvironment

    SystemEnvironmentPropertySourceEnvironmentPostProcessor.OriginAwareSystemEnvironmentPropertySource

  6. random

    RandomValuePropertySource

Recipes

Config

  • Resources

  • Notes

    • Avoid using annotation scanning, which tends to spread bean definitions all over the place. Instead, centralize them in the @SpringBootApplication class file.
    • If there are really too many bean definitions to put into one single file, create a subdirectory under the root package named config, and dedicate it to @Configuration classes.

Initialize a new project

  • Spring Initializr - Website (opens in a new tab)

    Spring Initializr is also a REST web service, so if curl https://start.spring.io, a quickstart guide will display, or directly supply parameters to create a project:

    curl -G https://start.spring.io/starter.zip \
      -d dependencies=web \
      -d javaVersion=11 -o demo.zip

  • Spring Boot CLI (opens in a new tab)

    spring init \
 -d=data-rest,devtools,actuator,lombok,cloud-feign,prometheus,testcontainers \
 -j=<JAVA_VERSION> \
 --build=<maven/gradle> \
 -x \
 -g=<GROUP_ID> \
 -a=<ARTIFACT_ID> \
 -n=<PROJECT_NAME> \
 --package-name=<PACKAGE_NAME> \
 --description=<DESCRIPTION> \
 <DIRECTORY_NAME>

Override Spring config properties at runtime

  • Environment variable

    If the property is spring.xxx.yyy, the corresponding environment variable is SPRING_XXX_YYY

    e.g. SPRING_PROFILES_ACTIVE=<profile1[,profile2...]>

  • JVM system property

    Use -D JVM option, property name is the same as the target property

    e.g. -Dspring.profiles.active=<profile1[,profile2...]>

  • CLI argument

    Use --<property=value>

    e.g. --server.port=9000 --logging.level.org.springframework.boot=trace

Spring config locations

  • spring.config.name and spring.config.location are used very early to determine which files have to be loaded. They must be defined as an environment property (typically an OS environment variable, a system property, or a command-line argument).

    java -jar myproject.jar --spring.config.location=classpath:/default.properties,classpath:/override.properties

  • spring.config.name defaults to application.properties, while spring.config.location defaults to the following locations:

    • file:./config/
    • file:./
    • classpath:/config/
    • classpath:/

  • When custom config locations are configured by using spring.config.additional-location, they are used in addition to the default locations. Additional locations are searched before the default locations.

  • Resources

Verify if a property source file is loaded successfully

Change logging level to trace for org.springframework.boot.context.config package

logging:
  level:
    org.springframework.boot.context.config: trace

Look for Adding imported property source in the log output

Access Environment Variables

    1. Java SE:
    System.getenv("ENVIRONMENT_VARIABLE")
    1. Spel:
    @Value("#{systemEnvironment['ENVIRONMENT_VARIABLE']}")
    1. Spring Environment class:
    private final Environment environment;

public App(Environment environment) {
    this.environment = environment;
}

var envPath = environment.getProperty("ENVIRONMENT_VARIABLE");

Track which property is imported from which property source

  1. Use /actuator/env endpoint, it will list all PropertySource and their properties.

  2. org.springframework.boot.context.config.ConfigDataImporter.load

    // loaded variable contains PropertySource and the retrieved properties
ConfigData loaded = this.loaders.load(loaderContext, resource);
if (loaded != null) {
  this.loaded.add(resource);
  this.loadedLocations.add(location);
  result.put(candidate, loaded);
}

List all active Auto Configuration instances with interface types

  • #TODO

Bean lazy initialization

Global flag:

spring:
  main:
    lazy-initialization: true

Caveats:

  • Might cause AutoConfiguration not to be loaded

Exclude specific Auto Configuration classes

  • In Java code

  1. Use @SpringBootApplication(exclude = <AutoConfigurationClass.class>)

  2. Use config property spring.autoconfigure.exclude=<AutoConfigurationClass.class>

Activate profiles in the order of precedence

@ActiveProfiles({"db", "kafka", "aws", "ccom"})

The order of activation is from left to right, so the order of precedence is ccom, aws, kafka, db.

Loading Spring config properties from classpath

  • Classes under src/main/java directory and classes under src/test/java use different classloaders.

  • Classes under src/main/java will only load classpath Resources from src/main/resources directory.

  • Test classes under src/test/java will load classpath Resources both from src/main/resources and src/test/resources directory.

  • For test classes, test classpath Resources take precedence over classpath Resources if they have the same name.

    For example, suppose we have the following application config files with active Profiles: db, aws.

    src/main/resources/application.properties
src/main/resources/application-db.properties
src/test/resources/application.properties
src/test/resources/application-aws.properties

    The following application config files will be loaded in the following order:

    test-classes/application.properties
test-classes/application-aws.properties
classes/application-db.properties

Spring config management strategy

  • application.properties

    • Should contain the default configuration for the application including test cases.
    • There's only one application.properties file under src/main/resources by default and no application.properties file under src/test/resources.
    • Since only one Spring config with the same name is allowed, the application.properties file under src/test/resources will override the application.properties file under src/main/resources. Therefore you can't use application-aws.properties in src/test/resources to override application-aws.properties in src/main/resources. Instead you will lose config properties from application-aws.properties in src/main/resources when running test cases.

  • Solutions to have overriding effect for running test cases

    • Every profile config should have unique config properties without any overlapping

    • Prepare profiles and config as follows:

      • src/main/resources/application-aws.properties
      • src/test/resources/application-aws-test.properties

      And activate profiles like: @ActiveProfiles({"aws", "aws-test"})

      Then the properties from src/test/resources/application-aws-test.properties will override the properties from src/main/resources/application-aws.properties if they have the same name.

      Cons: more profiles to manage

    • Alternatively, put all overriding overriding properties into a single config file named application-test.properties under src/test/resources.

      Every test case must activate the profile test like: @ActiveProfiles("test")

      Cons: might override unintended properties, in that case, use inline config to override again.

Persistence

Database Initialization

  • Spring Boot - AutoConfiguration

    org.springframework.boot.autoconfigure.sql.init.SqlInitializationProperties

    @ConfigurationProperties("spring.sql.init")

  • Schema creation

    • In a JPA-based app, you can choose to let Hibernate create the schema or use Spring Boot to run DDL in schema-[spring.datasource.platform].sql, but you cannot do both. Make sure to disable spring.jpa.hibernate.ddl-auto if you use schema-[spring.datasource.platform].sql.

    • Using Spring Boot to create schema

      1. Turn off Hibernate auto DDL generation function

        # hibernate.hbm2ddl.auto
spring.jpa.hibernate.ddl-auto=none

      2. Enable initialization mode as always

        spring.datasource.initialization-mode=always
spring.sql.init.mode=always (Spring Boot 2.5.0+)

      3. Specify RDBMS provider

        # This is for Spring Boot, and must be specified for schema-${spring.datasource.platform}.sql and data-${spring.datasource.platform}.sql to be executed.
spring.datasource.platform=mysql
spring.sql.init.platform=mysql (Spring Boot 2.5.0+)
 
# This is for JPA
# spring.jpa.database=mysql

      4. Ensure database is created beforehand

        # DB_NAME cannot be specified if it does not exist before the web application starts as database initialiaztion script (schema-${spring.datasource.platform}.sql) is run after the connection is obtained.
spring.datasource.url=jdbc:mysql://${HOST}:${PORT}/${DB_NAME}

  • Data population

    • Using Spring Boot to populate data

      • spring.datasource.initialization-mode defaults to embedded in org.springframework.boot.autoconfigure.jdbc.DataSourceProperties

      • org.springframework.boot.sql.init.AbstractScriptDatabaseInitializer.isEnabled determines if data script is run or not.

      • Debug org.springframework.boot.autoconfigure.jdbc.DataSourceInitializer for possible issues. The statements in the a script file are run one by one, not altogether, therefore statement order matters and foreign key constraints will get in the way.

      • org.springframework.jdbc.datasource.init.ScriptUtils.executeSqlScript runs the initialization SQL script (data.sql), which separates statements by ;(semi-colon) by default, if missing \n is used instead. This could break SQL statements in an undesirable manner and cause SQL exceptions.

Turn on Hibernate Statistics

spring.jpa.properties.hibernate.generate_statistics=true
logging.level.org.hibernate.engine.internal.StatisticalLoggingSessionEventListener=info

Web

Add Servlet Filter

Serve static content

  • Add web and thymeleaf module

  • Store static content files under src/main/resources/static

  • Configure static serving paths to file locations

    @Configuration
@EnableWebMvc
public class MvcConfig implements WebMvcConfigurer {
    @Override
    public void addResourceHandlers(ResourceHandlerRegistry registry) {
        registry.addResourceHandler("/static/js/**").addResourceLocations("classpath:/static/js/");
        registry.addResourceHandler("/static/css/**").addResourceLocations("classpath:/static/css/");
        registry.addResourceHandler("/static/media/**").addResourceLocations("classpath:/static/media/");
        registry.addResourceHandler("/static/**").addResourceLocations("classpath:/static/");
    }
}

Content Negotiation

Testing

Testing - Integration

Testing - Web

  • Enable @MockMvc

    @AutoConfigureMockMvc
@WebMvcTest(UserVehicleController.class)
class MyControllerTests {
    @Autowired
    private MockMvc mvc;
...

  • @SpringBootTest

    • webEnvironment=MOCK

      • Web layer unit testing with mocked dependencies using MockMvc and MockBean

    • webEnvironment=RANDOM_PORT / DEFINED_PORT

      • Web layer integration testing with real servers using RestTemplate or WebTestClient as the client (real HTTP requests)

  • WebClient is the reactive equivalent of RestTemplate. WebTestClient is a wrapper of WebClient for testing purposes.

  • MockMvc allows you to test Spring MVC controller endpoints without real HTTP communications (a mocked Servlet environment).

  • Use WireMock to mock unneeded HTTP services.

  • Resources

Mockito

Mock a generic type object

  • Use @Mock annotation to mock a generic type object, the mock will be populated with Spring Boot test support

    @Mock
private ListenableFuture<SendResult<String, Log>> listenableFuture;
 
@Test
void publish() throws Exception {
    var logJson = """
        {
          "id": "%s",
          "message": "test",
          "loggedAt": "2021-01-01T00:00:00",
          "addedAt": "2021-01-01T00:00:00"
        }
        """.formatted(UUID.randomUUID());
    when(kafkaTemplate.sendDefault(any(Log.class))).thenReturn(listenableFuture);
    mvc.perform(post("/log/publish/v1")
            .contentType(APPLICATION_JSON)
            .content(logJson))
        .andExpect(status().isOk());
}

Development

Spring Boot DevTools

  • Tips

    • With Spring Boot DevTools enabled, the application will restart automatically when a file in the classpath is changed. This is useful for development but should be disabled in production.
    • In IntelliJ IDEA, when the application is running, make code changes and hit Ctrl + Shift + F9 will compile the source code file and trigger an automatic restart.

Deployment

Change the Name of Project and Spring Boot JAR

  • Create a settings.gradle (or settings.gradle.kts) under the project directory if it doesn't exist
  • Add rootProject.name = '<Desired Project Name>' to it

Convert a Spring Boot executable JAR to a deployable WAR File

Containerization

systemd

Observability

Spring Boot Actuator

Implementing Custom JMX Endpoints

  • Notes

    • If you add a @Bean annotated with @Endpoint, any methods annotated with @ReadOperation, @WriteOperation, or @DeleteOperation are automatically exposed over JMX and, in a web application, over HTTP as well.
    • For reference, check any Spring bean annotated with @Endpoint, such as HealthEndpoint.

  • Implementing Custom JMX Endpoints (opens in a new tab)

Examine beans at runtime

  • Use Actuator endpoint: /actuator/beans

    • Use jq

      curl -s -G http://<ROOT>/actuator/beans | jq '.contexts[].beans | keys'

    • Use VS Code to traverse the JSON response

      curl -s -G http://<ROOT>/actuator/beans | jq | code -n -

      This will open a new VS Code window. Use Ctrl + Shift + O to navigate

List all bean names
curl -s -G http://$CONTEXT_PATH/actuator/beans | jq '.contexts[].beans | keys' | grep -Eo '\w+'
List all bean classes
curl -s -G http://$CONTEXT_PATH/actuator/beans | jq '.contexts[].beans[].type' | grep -Eo '(\w+\.?\$*/?)+' | sort
Examine loggers at runtime

  • Use Actuator endpoint: /actuator/loggers

    All loggers and their logging level are listed. Check the package you are interested in.

Check if auto configurations are loaded

  • Use Actuator endpoint: /actuator/conditions

    All auto configuration classes are listed with their condition evaluation result.

List all loaded PropertySource

  • Use Actuator endpoint: /actuator/env

    curl -s -G http://localhost:8080/actuator/env | jq '.propertySources[].name'

List all system properties

  • Use Actuator endpoint: /actuator/env

    curl -s -G http://localhost:8080/actuator/env | jq '.propertySources[] | select(.properties != null) | .properties' | jq -s add

    1. Defaults only display values
    2. Non-defaults also display origin of the value
Examine all web mappings

  • Use Actuator endpoint: /actuator/mappings

    curl -s -G http://localhost:8080/actuator/mappings | jq '.contexts|..|.mappings | select(. != null)'

  • List all URL patterns via dispatcherServlet

    curl -s -G http://localhost:8080/actuator/mappings | jq '..|.patterns? | select(. != null)' | jq -s add | jq '. | unique'

List all loaded beans at runtime
  • /actuator/beans
List all loaded config properties at runtime
  • /actuator/env
Enable all Actuator endpoints
  • Set management.endpoints.web.exposure.include=*
List all Actuator endpoints
  • Set management.endpoints.web.discovery.enabled=true, and access /actuator endpoint
Profile application startup

  1. Application class

    @SpringBootApplication
public class Application {
    public static void main(String[] args) {
        SpringApplication application = new SpringApplication(Application.class);
        application.setApplicationStartup(new FlightRecorderApplicationStartup());
        application.run(args);
    }
}

  2. Ensure the Spring config properties are set as below

    management.endpoints.web.exposure.include=startup
management.endpoint.startup.enabled=true

  3. Run the applicationTurn with VM flag

    -XX:StartFlightRecording=duration=30s,settings=profile,filename=startup.jfr (Java 11+)

  4. The JFR recording file will be saved under the directory where the Java application is run. When running the application, you may see info similar to shown below in the stdout:

    Started recording 1. The result will be written to:
 
C:\Users\Takechiyo\workspace\gitlab\startup.jfr

  5. Open it with JDK Mission Control, and navigate to Event Browser -> Spring Application -> Startup Step, and you will see all the recorded events.

List all metrics

  • Use /actuator/metrics endpoint to list all available metrics

  • Use /actuator/metrics/{metricName} endpoint to get the value of a specific metric

  • Metrics

    • Application Info

      • application.ready.time
      • application.started.time

    • Disk Metrics

      • disk.free
      • disk.total

    • Executor Metrics

      • executor.active
      • executor.completed
      • executor.pool.core
      • executor.pool.max
      • executor.pool.size
      • executor.queue.remaining
      • executor.queued

    • HTTP Metrics

      • http.server.requests
      • http.server.requests.active

    • JVM Metrics

      • Memory & Buffers

        • jvm.buffer.count
        • jvm.buffer.memory.used
        • jvm.buffer.total.capacity
        • jvm.memory.committed
        • jvm.memory.max
        • jvm.memory.usage.after.gc
        • jvm.memory.used

      • Garbage Collection

        • jvm.gc.live.data.size
        • jvm.gc.max.data.size
        • jvm.gc.memory.allocated
        • jvm.gc.memory.promoted
        • jvm.gc.overhead
        • jvm.gc.pause

      • Classes & Compilation

        • jvm.classes.loaded
        • jvm.classes.unloaded
        • jvm.compilation.time

      • Threads

        • jvm.threads.daemon
        • jvm.threads.live
        • jvm.threads.peak
        • jvm.threads.started
        • jvm.threads.states

      • Logging

        • log4j2.events

      • General

        • jvm.info

    • Process Metrics

      • process.cpu.time
      • process.cpu.usage
      • process.files.max
      • process.files.open
      • process.start.time
      • process.uptime

    • Resilience4j Metrics

      • Bulkhead

        • resilience4j.bulkhead.active.thread.count
        • resilience4j.bulkhead.available.concurrent.calls
        • resilience4j.bulkhead.available.thread.count
        • resilience4j.bulkhead.core.thread.pool.size
        • resilience4j.bulkhead.max.allowed.concurrent.calls
        • resilience4j.bulkhead.max.thread.pool.size
        • resilience4j.bulkhead.queue.capacity
        • resilience4j.bulkhead.queue.depth
        • resilience4j.bulkhead.thread.pool.size

      • Circuit Breaker

        • resilience4j.circuitbreaker.buffered.calls
        • resilience4j.circuitbreaker.calls
        • resilience4j.circuitbreaker.failure.rate
        • resilience4j.circuitbreaker.not.permitted.calls
        • resilience4j.circuitbreaker.slow.call.rate
        • resilience4j.circuitbreaker.slow.calls
        • resilience4j.circuitbreaker.state

      • Rate Limiter

        • resilience4j.ratelimiter.available.permissions
        • resilience4j.ratelimiter.waiting_threads

      • Retry & Time Limiter

        • resilience4j.retry.calls
        • resilience4j.timelimiter.calls

    • Spring Integration

      • spring.integration.channels
      • spring.integration.handlers
      • spring.integration.sources
      • spring.kafka.template

    • System Metrics

      • system.cpu.count
      • system.cpu.usage
      • system.load.average.1m

Micrometer

Micrometer Observation
Micrometer Tracing

OpenTelemetry

Security

SSL/TLS certificate

  • The entry in the Keystore must be a PrivateKeyEntry in PKCS12 format or a KeyEntry in JKS format, both containing a private key and corresponding certificate chain. Check sun.security.provider.JavaKeyStore for entry types.

    # SSL
server.port=8443
server.ssl.key-store=${keystore.p12}
server.ssl.key-store-password=${EXPORT_PASSWORD}
 
# Use: keytool -list -storetype PKCS12 -keystore ${keystore.p12} -storepass $EXPORT_PASSWORD -v | grep 'Alias name'
# Keystore entry alias
server.ssl.key-alias=${alias_name}
 
# JKS or PKCS12
server.ssl.keyStoreType=PKCS12
 
# Spring Security
security.require-ssl=true

Logging

  • org.springframework.boot.logging.logback.LogbackLoggingSystem

  • org.springframework.boot.logging.logback.defaults.xml

    Default logback configuration XML file

  • org.springframework.boot.logging.logback.DefaultLogbackConfiguration

    Default logback configuration used by Spring Boot.

  • org.springframework.boot.logging.logback.ColorConverter

    Color output converter

  • org.springframework.boot.logging.logback.LogbackConfigurator

    Allows programmatic configuration of logback which is usually faster than parsing XML.

  • Use environment variable or system property at runtime to change logging level of specific packages

    No source code changes needed, therefore no build and deployment, suitable for troubleshooting in any environment

    • Environment variable

      Non-intrusive, only need to restart Spring Boot application to pick up the environment variable, best for production troubleshooting

      export LOGGING_LEVEL_ORG_SPRINGFRAMEWORK_BOOT_CONTEXT_CONFIG=trace

    • System property

      Need to modify the CLI used to run the Spring Boot application by attaching the system property in the following form

      -Dlogging.level.org.springframework.boot.context.config=trace

    • IntelliJ IDEA

      1. Open Run/Debug Configurations, and set the property in Override configuration properties
      2. Rerun the Spring Boot application in IntelliJ.

  • Resources

Building

Spring Boot Maven Plugin

Spring Boot Gradle Plugin

Build a containerized native image

Add custom JVM flags

  • build.gradle

    bootRun {
    jvmArgs = [
        '-Dspring.profiles.active=dev',
        '-Dspring.config.location=classpath:/application-dev.properties'
    ]
}

  • build.gradle.kts

    project.tasks.withType<BootRun> {
    jvmArgs(
      "-XX:+DisableExplicitGC",
      "-XX:+IgnoreUnrecognizedVMOptions",
      "-XX:InitialHeapSize=4g"
    )
}

Troubleshooting

General method

  • Use logging
  1. Identify the Spring classes to be debugged
  2. turn on TRACE logging level for the package of the Spring classes
  • Use IDE debugging
  1. Identify the Spring classes to be debugged
  2. Use IntelliJ's "evaluate and log" function of debugger to print values
  3. To make step 2 easier, encapsulate the logic in a custom debugging util class such as IntellijDebugUtil.java (opens in a new tab)

Database table name does not match what is defined by JPA @Table annotation

Spring CLI

  • spring init --list

    To list all the capabilities of the service

Gradle

Specify system properties on command line

gradle bootRun --args='--spring.profiles.active=dev --spring.config.location=classpath:/application-dev.properties'

Kubernetes

Graceful shutdown

  • Anatomy
  1. Receive SIGTERM or PreStop Lifecycle Hook
  2. Fail ReadinessProbe
  3. Serve requests until Kubernetes detects ReadinessProbe failure
  4. Pod Endpoint removed from Service
  5. Finish serving in-flight requests
  6. Shutdown

Open Questions

  1. How to load DB initialization scripts in different profile conditionally?

Resources

Google CloudJupyter Notebook