NPE in GitSCM.onLoaded() on startup: descriptor is null when calling getOldGitExe()

[meetgoti07]

Jenkins and plugins versions report

Environment

apache-httpcomponents-client-4-api:4.5.14-269.vfa_2321039a_83
asm-api:9.9.1-189.vb_5ef2964da_91
bouncycastle-api:2.30.1.83-289.v8426fcd19371
caffeine-api:3.2.3-194.v31a_b_f7a_b_5a_81
commons-lang3-api:3.20.0-109.ve43756e2d2b_4
commons-text-api:1.15.0-218.va_61573470393
credentials:1491.v6d6145e96e1c
credentials-binding:717.v951d49b_5f3a_a_
display-url-api:2.217.va_6b_de84cc74b_
git:5.10.0
git-client:6.5.0
gson-api:2.13.2-173.va_a_092315913c
instance-identity:203.v15e81a_1b_7a_38
ionicons-api:94.vcc3065403257
jakarta-activation-api:2.1.4-1
jakarta-mail-api:2.1.5-1
mailer:525.v2458b_d8a_1a_71
mina-sshd-api-common:2.16.0-167.va_269f38cc024
mina-sshd-api-core:2.16.0-167.va_269f38cc024
plain-credentials:199.v9f8e1f741799
scm-api:728.vc30dcf7a_0df5
script-security:1399.ve6a_66547f6e1
ssh-credentials:361.vb_f6760818e8c
structs:362.va_b_695ef4fdf9
variant:70.va_d9f17f859e0
workflow-scm-step:466.va_d69e602552b_
workflow-step-api:710.v3e456cc85233

What Operating System are you using (both controller, and any agents involved in the problem)?

Controller: macOS 15.2 (darwin 25.2.0), Apple Silicon
Agents: none involved (single-node setup)

Reproduction steps

Build Jenkins core from source (current main branch, version 2.1328.vfd10e2339a_19).
Start Jenkins using mvn -pl war jetty:run from the Jenkins core workspace.
Install Git plugin 5.10.0 and Git client plugin 6.5.0 via Manage Jenkins → Plugins (or place the .jpi files in JENKINS_HOME/plugins/).
Restart Jenkins.

Expected Results

Jenkins starts successfully. The Git plugin initializes without error and is available for use.

Actual Results

Jenkins fails to start with a NullPointerException in GitSCM.onLoaded(), causing HudsonFailedToLoad:

java.lang.NullPointerException: Cannot invoke "hudson.plugins.git.GitSCM$DescriptorImpl.getOldGitExe()" because "desc" is null
    at hudson.plugins.git.GitSCM.onLoaded(GitSCM.java:2174)
    at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
Caused: java.lang.reflect.InvocationTargetException
    ...
Caused: java.lang.Error
    at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:115)
    at hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:185)
    ...
Caused: org.jvnet.hudson.reactor.ReactorException
    ...
Caused: hudson.util.HudsonFailedToLoad
    at hudson.WebAppMain$3.run(WebAppMain.java:274)

Anything else?

Root cause: GitSCM.onLoaded() is called as a plugin initializer before GitSCM's descriptor (GitSCM$DescriptorImpl) has been registered by the extension system. The method then calls desc.getOldGitExe() on a null desc, crashing startup. This appears to be an initializer ordering issue exposed by changes in the Jenkins core development version.

Are you interested in contributing a fix?

Yes

[MarkEWaite]

This appears to be an initializer ordering issue exposed by changes in the Jenkins core development version.

I think that indicates an issue in the Maven hpi plugin, not an issue in the git plugin. Could you open an issue in that repository, rather than in the git plugin repository?

[meetgoti07]

Hello @MarkEWaite ,

I investigated this issue by examining both the maven-hpi-plugin and the git-plugin codebases. maven-hpi-plugin is purely a build-time tool with no runtime presence in Jenkins, so it has no involvement in this bug.

The root cause is in GitSCM.java lines 2169-2174. The onLoaded() method is annotated with @Initializer(after=PLUGINS_STARTED), but it calls jenkins.getDescriptorByType(DescriptorImpl.class) which returns null at that point because @Extension-annotated descriptors aren't registered until the EXTENSIONS_AUGMENTED milestone -- which comes after PLUGINS_STARTED. Line 2174 then calls desc.getOldGitExe() on the null reference, causing the NPE.

Additionally, getOldGitExe() (line 1818) always returns null, so the entire onLoaded() method body is effectively dead code -- the if block can never execute. This is legacy migration logic that no longer serves a purpose.

A minimal fix would be adding a null guard (if (desc != null && desc.getOldGitExe() != null)), though changing the milestone to after=EXTENSIONS_AUGMENTED or removing the dead code entirely would also be appropriate.

Your suggestion would be greatly appreciated.

[MarkEWaite]

I investigated this issue by examining both the maven-hpi-plugin and the git-plugin codebases. maven-hpi-plugin is purely a build-time tool with no runtime presence in Jenkins, so it has no involvement in this bug.

Thanks for investigating further. I did not read your initial description carefully enough. My apologies for that mistake.

I assumed you were running Jenkins from a development directory of the Git plugin with mvn hpi:run. That's not what you said. You did not use the Maven hpi plugin at all. You said that you are running Jenkins from a development directory of Jenkins core using the jetty plugin with jetty:run. Your steps were:

1. Build Jenkins core from source (current main branch, version 2.1328.vfd10e2339a_19).
2. Start Jenkins using mvn -pl war jetty:run from the Jenkins core workspace.
3. Install Git plugin 5.10.0 and Git client plugin 6.5.0 via Manage Jenkins → Plugins (or place the .jpi files in JENKINS_HOME/plugins/).
4. Restart Jenkins.

I tried the following steps on my Red Hat Enterprise Linux 8.10 computer running on an AMD Ryzen 5 5600X 6-Core Processor with Apache Maven 3.9.14 and Eclipse Temurin Java 21.0.10. That is the fastest computer that I have with the most recent Apache Maven and Eclipse Temurin Java. It is not as fast as your macOS 15.2 (darwin 25.2.0), Apple Silicon, but it is the fastest available to me.

1. cd ~/hub/core/jenkins && git checkout master && git pull to checkout 659b4380bf
2. mvn -am -pl war,bom -Pquick-build clean install to compile Jenkins from source code as described in the contributing guide
3. MAVEN_OPTS='--add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.io=ALL-UNNAMED --add-opens java.base/java.util=ALL-UNNAMED' mvn -pl war -Dhost=0.0.0.0 jetty:run to launch a development instance as described in the contributing guide (with the addition of --host:0.0.0.0 because my primary desktop computer is Windows and my primary Jenkins development computers are Linux)
4. Open http://rhel-8-a.markwaite.net:8080/jenkins/manage/pluginManager/ and use the plugin manager to install the Git plugin
5. Restart Jenkins and confirm there is no initialization error message reported and no null pointer exception
6. Create a freestyle project that clones a git repository and confirm the git plugin works as expected to checkout the repository

I generally don't run my Jenkins development environment from jetty:run. I usually use JENKINS_HOME=work java -jar war/target/jenkins.war based on the user documentation linked from the building the war file section of the contributing guide. However, the jetty:run technique should work for you as it does for me. We may need to ask for help from other Jenkins developers that are running Apple Silicon, if we can't find any indication why your commands are failing to load the git plugin and mine are succeeding.

Did you provide the MAVEN_OPTS argument to the command line? If not, could you try that, since that is the documented way to use jetty:run?

The contributing guide includes a warning about the jetty:run mode. It says:

(Beware that maven-plugin builds will not work in this mode, due to class loading conflicts.)

The jetty:run mode has some differences compared to java -jar jenkins.war and I prefer to avoid those differences by always running with java -jar jenkins.war. However, I think that jetty:run should either work or it should be removed from the documentation.

The details of my Maven and Java versions are reported as:

Apache Maven 3.9.14 (996c630dbc656c76214ce58821dcc58be960875b)
Maven home: /home/mwaite/tools/apache-maven-3.9.14
Java version: 21.0.10, vendor: Eclipse Adoptium, runtime: /opt/jdk-21
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "4.18.0-553.92.1.el8_10.x86_64", arch: "amd64", family: "unix"

[meetgoti07]

Hello @MarkEWaite,

I tried to reproduce the issue again, but I was unable to replicate it. I will continue investigating to identify the root cause.

[MarkEWaite]

I've found two other surprising behaviors from jetty:run that have prompted me to ask some questions in the Jenkins developer mailing list.

[meetgoti07]

I tried to debug the issue and also looked into why I was not able to reproduce the bug again.
Here are my findings.

The NPE seems to come from a race condition during the Jenkins startup reactor. The reactor runs @Initializer tasks concurrently using a thread pool. GitSCM.onLoaded() expects the descriptor to already be registered. In the original run, the thread executing onLoaded() likely ran before the descriptor registration finished, so desc was still null.

On later attempts with a fresh JENKINS_HOME, rebuilds, or repeated restarts, the descriptor happened to be registered earlier, so the NPE did not occur again. Since thread scheduling is non deterministic, this makes the issue intermittent.

Probable cause

GitSCM.onLoaded() at line 2174 calls desc.getOldGitExe() without checking if desc is null. Descriptor registration and @Initializer execution both run as concurrent reactor tasks, so there is no strict ordering between them.

When running with mvn -pl war jetty:run in Jetty dev mode, the classloader setup and task submission timing are slightly different from running Jenkins with java -jar jenkins.war. That difference may increase the chance of hitting the race condition, but it is still not deterministic.

[MarkEWaite]

When running with mvn -pl war jetty:run in Jetty dev mode, the classloader setup and task submission timing are slightly different from running Jenkins with java -jar jenkins.war. That difference may increase the chance of hitting the race condition, but it is still not deterministic.

I'm hesitant to change that code because it has been that way since 2018. That means it has continued to be used through the transition to Java 8, to Java 11, to Java 17, and to Java 21. It has been used on amd64, arm64, s390x, and riscv64. If the race condition has not been repeatedly exposed in that many years, then it is unlikely to be a significant issue.

If you find it concerning that desc might be null, the entire block could be deleted, since desc.getOldGitExe() is always null. The code block:

        if (desc.getOldGitExe() != null) {
            String exe = desc.getOldGitExe();
            String defaultGit = GitTool.getDefaultInstallation().getGitExe();
            if (exe.equals(defaultGit)) {
                return;
            }
            System.err.println("[WARNING] you're using deprecated gitexe attribute to configure git plugin. Use Git installations");
        }

calls:

        public String getOldGitExe() {
            return null;
        }

That was changed in 2023 to always return null because the value was never initialized in the descriptor.

[meetgoti07]

Hi, I also encountered the issue only once. Repeating the same steps did not reproduce the NPE again. So I agree that since we do not have solid evidence of the bug, it is probably fine to leave it as is for now.

Also, I have created a PR to clean up the code you mentioned: #3936

[daniel-beck]

git-plugin/src/main/java/hudson/plugins/git/GitSCM.java

Line 2169 in a6df1a4

@Initializer(after=PLUGINS_STARTED)

should probably be https://github.com/jenkinsci/jenkins/blob/659b4380bf1829fcb114665618a9c665e329d0e0/core/src/main/java/hudson/init/InitMilestone.java#L89 cf. https://github.com/jenkinsci/jenkins/blob/659b4380bf1829fcb114665618a9c665e329d0e0/core/src/main/java/hudson/Functions.java#L271-L272 (although this is admittedly not an area I understand well).

I wonder whether a patched core that takes longer to get from PLUGINS_LOADED to EXTENSIONS_AUGMENTED would reveal this more consistently.