Spring Boot – Nextra

Source Code Investigation

Call Stack

org.springframework.boot.SpringApplication.run
- org.springframework.boot.SpringApplication.prepareEnvironment
  - org.springframework.boot.SpringApplication.configureEnvironment
    - org.springframework.boot.SpringApplication.configurePropertySources

PropertySource

Testing
- Precedence (high to low)
  - DynamicValuesPropertySource
    
    @DynamicPropertySource
  - @TestPropertySource - properties attribute
  - @TestPropertySource - locations attribute
PropertySourceLoader
- PropertiesPropertySourceLoader for loading .properties files
- YamlPropertySourceLoader for loading .yaml and .yml files

Default precedence order, from highest to lowest, in PropertySource name

configurationProperties

org.springframework.boot.context.properties.source.ConfigurationPropertySourcesPropertySource

To be used with PropertyResolver or added to the Environment
servletConfigInitParams

In the case that the application is not a Java Servlet application, this PropertySource will be an instance of StubPropertySource.
servletContextInitParams

In the case that the application is not a Java Servlet application, this PropertySource will be an instance of StubPropertySource.
systemProperties

org.springframework.core.env.PropertiesPropertySource
systemEnvironment

SystemEnvironmentPropertySourceEnvironmentPostProcessor.OriginAwareSystemEnvironmentPropertySource
random

RandomValuePropertySource

Recipes

Config

Resources
- Spring Blog - Spring Tips: Configuration (opens in a new tab)
Notes
- Avoid using annotation scanning, which tends to spread bean definitions all over the place. Instead, centralize them in the @SpringBootApplication class file.
- If there are really too many bean definitions to put into one single file, create a subdirectory under the root package named config, and dedicate it to @Configuration classes.

Initialize a new project

Spring Initializr - Website (opens in a new tab)

Spring Initializr is also a REST web service, so if curl https://start.spring.io, a quickstart guide will display, or directly supply parameters to create a project:
```
curl -G https://start.spring.io/starter.zip \
      -d dependencies=web \
      -d javaVersion=11 -o demo.zip
```

Spring Boot CLI (opens in a new tab)

spring init \
 -d=data-rest,devtools,actuator,lombok,cloud-feign,prometheus,testcontainers \
 -j=<JAVA_VERSION> \
 --build=<maven/gradle> \
 -x \
 -g=<GROUP_ID> \
 -a=<ARTIFACT_ID> \
 -n=<PROJECT_NAME> \
 --package-name=<PACKAGE_NAME> \
 --description=<DESCRIPTION> \
 <DIRECTORY_NAME>

Override Spring config properties at runtime

Environment variable

If the property is spring.xxx.yyy, the corresponding environment variable is SPRING_XXX_YYY

e.g. SPRING_PROFILES_ACTIVE=<profile1[,profile2...]>
JVM system property

Use -D JVM option, property name is the same as the target property

e.g. -Dspring.profiles.active=<profile1[,profile2...]>
CLI argument

Use --<property=value>

e.g. --server.port=9000 --logging.level.org.springframework.boot=trace

Spring config locations

spring.config.name and spring.config.location are used very early to determine which files have to be loaded. They must be defined as an environment property (typically an OS environment variable, a system property, or a command-line argument).

java -jar myproject.jar --spring.config.location=classpath:/default.properties,classpath:/override.properties
spring.config.name defaults to application.properties, while spring.config.location defaults to the following locations:
- file:./config/
- file:./
- classpath:/config/
- classpath:/
When custom config locations are configured by using spring.config.additional-location, they are used in addition to the default locations. Additional locations are searched before the default locations.
Resources
- Spring Boot Documentation - 2.3. Application Property Files (opens in a new tab)

Verify if a property source file is loaded successfully

Change logging level to trace for org.springframework.boot.context.config package

logging:
  level:
    org.springframework.boot.context.config: trace

Look for Adding imported property source in the log output

Access Environment Variables

1. Java SE:
```
System.getenv("ENVIRONMENT_VARIABLE")
```

Spel:

@Value("#{systemEnvironment['ENVIRONMENT_VARIABLE']}")

Spring Environment class:

private final Environment environment;

public App(Environment environment) {
    this.environment = environment;
}

var envPath = environment.getProperty("ENVIRONMENT_VARIABLE");

Track which property is imported from which property source

Use /actuator/env endpoint, it will list all PropertySource and their properties.

org.springframework.boot.context.config.ConfigDataImporter.load

// loaded variable contains PropertySource and the retrieved properties
ConfigData loaded = this.loaders.load(loaderContext, resource);
if (loaded != null) {
  this.loaded.add(resource);
  this.loadedLocations.add(location);
  result.put(candidate, loaded);
}

Resources
- Spring Blog - Config file processing in Spring Boot 2.4 (opens in a new tab)

List all active Auto Configuration instances with interface types

#TODO

Bean lazy initialization

Global flag:

spring:
  main:
    lazy-initialization: true

Caveats:

Might cause AutoConfiguration not to be loaded

Exclude specific Auto Configuration classes

In Java code

Use @SpringBootApplication(exclude = <AutoConfigurationClass.class>)
Use config property spring.autoconfigure.exclude=<AutoConfigurationClass.class>

Activate profiles in the order of precedence

@ActiveProfiles({"db", "kafka", "aws", "ccom"})

The order of activation is from left to right, so the order of precedence is ccom, aws, kafka, db.

Rules of thumb
- @ActiveProfiles should be used with application entry point class, such as Application class (main method holder) or test cases.
- @Profile should be used with generic Spring components

Loading Spring config properties from classpath

Classes under src/main/java directory and classes under src/test/java use different classloaders.
Classes under src/main/java will only load classpath Resources from src/main/resources directory.
Test classes under src/test/java will load classpath Resources both from src/main/resources and src/test/resources directory.

For test classes, test classpath Resources take precedence over classpath Resources if they have the same name.

For example, suppose we have the following application config files with active Profiles: db, aws.

src/main/resources/application.properties
src/main/resources/application-db.properties
src/test/resources/application.properties
src/test/resources/application-aws.properties

The following application config files will be loaded in the following order:

test-classes/application.properties
test-classes/application-aws.properties
classes/application-db.properties

Spring config management strategy

application.properties
- Should contain the default configuration for the application including test cases.
- There's only one application.properties file under src/main/resources by default and no application.properties file under src/test/resources.
- Since only one Spring config with the same name is allowed, the application.properties file under src/test/resources will override the application.properties file under src/main/resources. Therefore you can't use application-aws.properties in src/test/resources to override application-aws.properties in src/main/resources. Instead you will lose config properties from application-aws.properties in src/main/resources when running test cases.
Solutions to have overriding effect for running test cases
- Every profile config should have unique config properties without any overlapping
- Prepare profiles and config as follows:
  - src/main/resources/application-aws.properties
  - src/test/resources/application-aws-test.properties
  And activate profiles like: @ActiveProfiles({"aws", "aws-test"})
  
  Then the properties from src/test/resources/application-aws-test.properties will override the properties from src/main/resources/application-aws.properties if they have the same name.
  
  Cons: more profiles to manage
- Alternatively, put all overriding overriding properties into a single config file named application-test.properties under src/test/resources.
  
  Every test case must activate the profile test like: @ActiveProfiles("test")
  
  Cons: might override unintended properties, in that case, use inline config to override again.

Persistence

Database Initialization

Spring Boot - AutoConfiguration

org.springframework.boot.autoconfigure.sql.init.SqlInitializationProperties

@ConfigurationProperties("spring.sql.init")

Schema creation

In a JPA-based app, you can choose to let Hibernate create the schema or use Spring Boot to run DDL in schema-[spring.datasource.platform].sql, but you cannot do both. Make sure to disable spring.jpa.hibernate.ddl-auto if you use schema-[spring.datasource.platform].sql.

Using Spring Boot to create schema

Turn off Hibernate auto DDL generation function

# hibernate.hbm2ddl.auto
spring.jpa.hibernate.ddl-auto=none

Enable initialization mode as always

spring.datasource.initialization-mode=always
spring.sql.init.mode=always (Spring Boot 2.5.0+)

Specify RDBMS provider

# This is for Spring Boot, and must be specified for schema-${spring.datasource.platform}.sql and data-${spring.datasource.platform}.sql to be executed.
spring.datasource.platform=mysql
spring.sql.init.platform=mysql (Spring Boot 2.5.0+)
 
# This is for JPA
# spring.jpa.database=mysql

Ensure database is created beforehand

# DB_NAME cannot be specified if it does not exist before the web application starts as database initialiaztion script (schema-${spring.datasource.platform}.sql) is run after the connection is obtained.
spring.datasource.url=jdbc:mysql://${HOST}:${PORT}/${DB_NAME}

Data population
- Using Spring Boot to populate data
  - spring.datasource.initialization-mode defaults to embedded in org.springframework.boot.autoconfigure.jdbc.DataSourceProperties
  - org.springframework.boot.sql.init.AbstractScriptDatabaseInitializer.isEnabled determines if data script is run or not.
  - Debug org.springframework.boot.autoconfigure.jdbc.DataSourceInitializer for possible issues. The statements in the a script file are run one by one, not altogether, therefore statement order matters and foreign key constraints will get in the way.
  - org.springframework.jdbc.datasource.init.ScriptUtils.executeSqlScript runs the initialization SQL script (data.sql), which separates statements by ;(semi-colon) by default, if missing \n is used instead. This could break SQL statements in an undesirable manner and cause SQL exceptions.

Turn on Hibernate Statistics

spring.jpa.properties.hibernate.generate_statistics=true
logging.level.org.hibernate.engine.internal.StatisticalLoggingSessionEventListener=info

Web

Add Servlet Filter

Use OncePerRequestFilter for filters to be executed only once for every HTTP request.
Resources
- Ways to add Servlet Filters in Spring Boot (opens in a new tab)

Serve static content

Add web and thymeleaf module
Store static content files under src/main/resources/static

Configure static serving paths to file locations

@Configuration
@EnableWebMvc
public class MvcConfig implements WebMvcConfigurer {
    @Override
    public void addResourceHandlers(ResourceHandlerRegistry registry) {
        registry.addResourceHandler("/static/js/**").addResourceLocations("classpath:/static/js/");
        registry.addResourceHandler("/static/css/**").addResourceLocations("classpath:/static/css/");
        registry.addResourceHandler("/static/media/**").addResourceLocations("classpath:/static/media/");
        registry.addResourceHandler("/static/**").addResourceLocations("classpath:/static/");
    }
}

Content Negotiation

Spring blog - Content Negotiation using Spring MVC (opens in a new tab)

Testing

Resources
Annotations
- @SpringBootTest
  - Used as an alternative to the standard spring-test @ContextConfiguration annotation when you need Spring Boot features. The annotation works by creating the ApplicationContext used in your tests through SpringApplication. In addition to @SpringBootTest a number of other annotations are also provided for testing more specific slices of an application.
- @TestConfiguration
Helper beans
- WebTestClient
- TestRestTemplate

Testing - Integration

Spring Boot reference - Auto-configured sliced testing (opens in a new tab)
Test Auto-configuration for sliced testing (opens in a new tab)

Instead of the whole ApplicationContext, Only the beans required for a particular slice of an application are used when that slice is being tested.

Use @MockBean to replace a bean with a mock

@SpringBootTest
class MyTests {
    @MockBean
    private SomeService service;
...

Testing - Web

Enable @MockMvc

@AutoConfigureMockMvc
@WebMvcTest(UserVehicleController.class)
class MyControllerTests {
    @Autowired
    private MockMvc mvc;
...

@SpringBootTest
- webEnvironment=MOCK
  - Web layer unit testing with mocked dependencies using MockMvc and MockBean
- webEnvironment=RANDOM_PORT / DEFINED_PORT
  - Web layer integration testing with real servers using RestTemplate or WebTestClient as the client (real HTTP requests)
WebClient is the reactive equivalent of RestTemplate. WebTestClient is a wrapper of WebClient for testing purposes.
MockMvc allows you to test Spring MVC controller endpoints without real HTTP communications (a mocked Servlet environment).
Use WireMock to mock unneeded HTTP services.
Resources
- Testing Java Applications Made Simple - Test Your Spring MVC Controller with the WebTestClient and MockMvc (opens in a new tab)

Mockito

Mock a generic type object

Use @Mock annotation to mock a generic type object, the mock will be populated with Spring Boot test support

@Mock
private ListenableFuture<SendResult<String, Log>> listenableFuture;
 
@Test
void publish() throws Exception {
    var logJson = """
        {
          "id": "%s",
          "message": "test",
          "loggedAt": "2021-01-01T00:00:00",
          "addedAt": "2021-01-01T00:00:00"
        }
        """.formatted(UUID.randomUUID());
    when(kafkaTemplate.sendDefault(any(Log.class))).thenReturn(listenableFuture);
    mvc.perform(post("/log/publish/v1")
            .contentType(APPLICATION_JSON)
            .content(logJson))
        .andExpect(status().isOk());
}

Development

Spring Boot DevTools

Tips
- With Spring Boot DevTools enabled, the application will restart automatically when a file in the classpath is changed. This is useful for development but should be disabled in production.
- In IntelliJ IDEA, when the application is running, make code changes and hit Ctrl + Shift + F9 will compile the source code file and trigger an automatic restart.

Deployment

Change the Name of Project and Spring Boot JAR

Create a settings.gradle (or settings.gradle.kts) under the project directory if it doesn't exist
Add rootProject.name = '<Desired Project Name>' to it

Observability

Actuator

Resources
- Actuator Endpoints (opens in a new tab)
- docs.spring.io - Spring Boot Actuator Web API Documentation (opens in a new tab)
  
  All Actuator endpoints reference

Actuator - Implementing Custom JMX Endpoints

Notes
- If you add a @Bean annotated with @Endpoint, any methods annotated with @ReadOperation, @WriteOperation, or @DeleteOperation are automatically exposed over JMX and, in a web application, over HTTP as well.
- For reference, check any Spring bean annotated with @Endpoint, such as HealthEndpoint.
Implementing Custom JMX Endpoints (opens in a new tab)

Actuator - Examine beans at runtime

Use Actuator endpoint: /actuator/beans
- Use jq
  
  curl -s -G http://<ROOT>/actuator/beans | jq '.contexts[].beans | keys'
- Use VS Code to traverse the JSON response
  
  curl -s -G http://<ROOT>/actuator/beans | jq | code -n -
  
  This will open a new VS Code window. Use Ctrl + Shift + O to navigate

Actuator - List all bean names

curl -s -G http://$CONTEXT_PATH/actuator/beans | jq '.contexts[].beans | keys' | grep -Eo '\w+'

Actuator - List all bean classes

curl -s -G http://$CONTEXT_PATH/actuator/beans | jq '.contexts[].beans[].type' | grep -Eo '(\w+\.?\$*/?)+' | sort

Actuator - Examine loggers at runtime

Use Actuator endpoint: /actuator/loggers

All loggers and their logging level are listed. Check the package you are interested in.

Actuator - Check if auto configurations are loaded

Use Actuator endpoint: /actuator/conditions

All auto configuration classes are listed with their condition evaluation result.

Actuator - List all loaded PropertySource

Use Actuator endpoint: /actuator/env

curl -s -G http://localhost:8080/actuator/env | jq '.propertySources[].name'

Actuator - List all system properties

Use Actuator endpoint: /actuator/env

curl -s -G http://localhost:8080/actuator/env | jq '.propertySources[] | select(.properties != null) | .properties' | jq -s add
1. Defaults only display values
2. Non-defaults also display origin of the value

Actuator - Examine all web mappings

Use Actuator endpoint: /actuator/mappings

curl -s -G http://localhost:8080/actuator/mappings | jq '.contexts|..|.mappings | select(. != null)'
List all URL patterns via dispatcherServlet

curl -s -G http://localhost:8080/actuator/mappings | jq '..|.patterns? | select(. != null)' | jq -s add | jq '. | unique'

Actuator - List all loaded beans at runtime

/actuator/beans

Actuator - List all loaded config properties at runtime

/actuator/env

Actuator - Enable all Actuator endpoints

Set management.endpoints.web.exposure.include=*

Actuator - List all Actuator endpoints

Set management.endpoints.web.discovery.enabled=true, and access /actuator endpoint

Actuator - Profiling application startup

Application class

@SpringBootApplication
public class Application {
    public static void main(String[] args) {
        SpringApplication application = new SpringApplication(Application.class);
        application.setApplicationStartup(new FlightRecorderApplicationStartup());
        application.run(args);
    }
}

Ensure the Spring config properties are set as below

management.endpoints.web.exposure.include=startup
management.endpoint.startup.enabled=true

Run the applicationTurn with VM flag

-XX:StartFlightRecording=duration=30s,settings=profile,filename=startup.jfr (Java 11+)
The JFR recording file will be saved under the directory where the Java application is run. When running the application, you may see info similar to shown below in the stdout:
```
Started recording 1. The result will be written to:
 
C:\Users\Takechiyo\workspace\gitlab\startup.jfr
```
Open it with JDK Mission Control, and navigate to Event Browser -> Spring Application -> Startup Step, and you will see all the recorded events.

Actuator - List all metrics

Use /actuator/metrics endpoint to list all available metrics
Use /actuator/metrics/{metricName} endpoint to get the value of a specific metric
Metrics
- Application Info
  - application.ready.time
  - application.started.time
- Disk Metrics
  - disk.free
  - disk.total
- Executor Metrics
  - executor.active
  - executor.completed
  - executor.pool.core
  - executor.pool.max
  - executor.pool.size
  - executor.queue.remaining
  - executor.queued
- HTTP Metrics
  - http.server.requests
  - http.server.requests.active
- JVM Metrics
  - Memory & Buffers
    - jvm.buffer.count
    - jvm.buffer.memory.used
    - jvm.buffer.total.capacity
    - jvm.memory.committed
    - jvm.memory.max
    - jvm.memory.usage.after.gc
    - jvm.memory.used
  - Garbage Collection
    - jvm.gc.live.data.size
    - jvm.gc.max.data.size
    - jvm.gc.memory.allocated
    - jvm.gc.memory.promoted
    - jvm.gc.overhead
    - jvm.gc.pause
  - Classes & Compilation
    - jvm.classes.loaded
    - jvm.classes.unloaded
    - jvm.compilation.time
  - Threads
    - jvm.threads.daemon
    - jvm.threads.live
    - jvm.threads.peak
    - jvm.threads.started
    - jvm.threads.states
  - Logging
    - log4j2.events
  - General
    - jvm.info
- Process Metrics
  - process.cpu.time
  - process.cpu.usage
  - process.files.max
  - process.files.open
  - process.start.time
  - process.uptime
- Resilience4j Metrics
  - Bulkhead
    - resilience4j.bulkhead.active.thread.count
    - resilience4j.bulkhead.available.concurrent.calls
    - resilience4j.bulkhead.available.thread.count
    - resilience4j.bulkhead.core.thread.pool.size
    - resilience4j.bulkhead.max.allowed.concurrent.calls
    - resilience4j.bulkhead.max.thread.pool.size
    - resilience4j.bulkhead.queue.capacity
    - resilience4j.bulkhead.queue.depth
    - resilience4j.bulkhead.thread.pool.size
  - Circuit Breaker
    - resilience4j.circuitbreaker.buffered.calls
    - resilience4j.circuitbreaker.calls
    - resilience4j.circuitbreaker.failure.rate
    - resilience4j.circuitbreaker.not.permitted.calls
    - resilience4j.circuitbreaker.slow.call.rate
    - resilience4j.circuitbreaker.slow.calls
    - resilience4j.circuitbreaker.state
  - Rate Limiter
    - resilience4j.ratelimiter.available.permissions
    - resilience4j.ratelimiter.waiting_threads
  - Retry & Time Limiter
    - resilience4j.retry.calls
    - resilience4j.timelimiter.calls
- Spring Integration
  - spring.integration.channels
  - spring.integration.handlers
  - spring.integration.sources
  - spring.kafka.template
- System Metrics
  - system.cpu.count
  - system.cpu.usage
  - system.load.average.1m

Actuator - Cache TTL troubleshooting

Optimize Spring Boot Performance: Mastering Metrics Cache TTL (opens in a new tab)

OpenTelemetry

Spring Boot starter | OpenTelemetry (opens in a new tab)

Security

SSL/TLS certificate

The entry in the Keystore must be a PrivateKeyEntry in PKCS12 format or a KeyEntry in JKS format, both containing a private key and corresponding certificate chain. Check sun.security.provider.JavaKeyStore for entry types.

# SSL
server.port=8443
server.ssl.key-store=${keystore.p12}
server.ssl.key-store-password=${EXPORT_PASSWORD}
 
# Use: keytool -list -storetype PKCS12 -keystore ${keystore.p12} -storepass $EXPORT_PASSWORD -v | grep 'Alias name'
# Keystore entry alias
server.ssl.key-alias=${alias_name}
 
# JKS or PKCS12
server.ssl.keyStoreType=PKCS12
 
# Spring Security
security.require-ssl=true

Logging

org.springframework.boot.logging.logback.LogbackLoggingSystem
org.springframework.boot.logging.logback.defaults.xml

Default logback configuration XML file
org.springframework.boot.logging.logback.DefaultLogbackConfiguration

Default logback configuration used by Spring Boot.
org.springframework.boot.logging.logback.ColorConverter

Color output converter
org.springframework.boot.logging.logback.LogbackConfigurator

Allows programmatic configuration of logback which is usually faster than parsing XML.
Use environment variable or system property at runtime to change logging level of specific packages

No source code changes needed, therefore no build and deployment, suitable for troubleshooting in any environment
- Environment variable
  
  Non-intrusive, only need to restart Spring Boot application to pick up the environment variable, best for production troubleshooting
  
  export LOGGING_LEVEL_ORG_SPRINGFRAMEWORK_BOOT_CONTEXT_CONFIG=trace
- System property
  
  Need to modify the CLI used to run the Spring Boot application by attaching the system property in the following form
  
  -Dlogging.level.org.springframework.boot.context.config=trace
- IntelliJ IDEA
  1. Open Run/Debug Configurations, and set the property in Override configuration properties
  2. Rerun the Spring Boot application in IntelliJ.
Resources
- Coralogix - Spring Boot Logging Best Practices Guide (opens in a new tab)

Building

Containerized
- How to Build Java Apps with Paketo Buildpacks (opens in a new tab)

Spring Boot Maven Plugin

Spring Boot Gradle Plugin

Spring Boot Docs - Spring Boot Gradle Plugin Reference (opens in a new tab)

Build a containerized native image

Spring Boot Docs - Reacting to the GraalVM Native Image Plugin (opens in a new tab)

Using gradle bootBuildImage to build a containerized native image

Add custom JVM flags

build.gradle

bootRun {
    jvmArgs = [
        '-Dspring.profiles.active=dev',
        '-Dspring.config.location=classpath:/application-dev.properties'
    ]
}

build.gradle.kts

project.tasks.withType<BootRun> {
    jvmArgs(
      "-XX:+DisableExplicitGC",
      "-XX:+IgnoreUnrecognizedVMOptions",
      "-XX:InitialHeapSize=4g"
    )
}

Troubleshooting

General method

Use logging

Identify the Spring classes to be debugged
turn on TRACE logging level for the package of the Spring classes

Use IDE debugging

Identify the Spring classes to be debugged
Use IntelliJ's "evaluate and log" function of debugger to print values
To make step 2 easier, encapsulate the logic in a custom debugging util class such as IntellijDebugUtil.java (opens in a new tab)

Database table name does not match what is defined by JPA `@Table` annotation

Because Spring Boot uses its own naming strategy (org.springframework.boot.orm.jpa.hibernate.SpringPhysicalNamingStrategy starting from Spring Boot 1.4) to generate database table names
Stack Overflow - Spring Boot + JPA : Column name annotation ignored (opens in a new tab)

Spring CLI

spring init --list

To list all the capabilities of the service

Gradle

Spring Boot Gradle Plugin Reference Guide (opens in a new tab)

Specify system properties on command line

gradle bootRun --args='--spring.profiles.active=dev --spring.config.location=classpath:/application-dev.properties'

Kubernetes

Graceful shutdown

Anatomy

Receive SIGTERM or PreStop Lifecycle Hook
Fail ReadinessProbe
Serve requests until Kubernetes detects ReadinessProbe failure
Pod Endpoint removed from Service
Finish serving in-flight requests
Shutdown

Open Questions

How to load DB initialization scripts in different profile conditionally?

Resources

Spring Boot Application Properties Reference

docs.spring.io - Common Application Properties (latest) (opens in a new tab)

Google Cloud Jupyter Notebook