2016-10-18

Using Maven with OSGi Part 3

Introduction

In this installment I'll try to explain concepts behind SNAPSHOT artifact handling in Maven and Aether. How this is done in pax-url-aether will be described in next part of the series.

SNAPSHOT concept seems simple, it may however be very confusing at the same time - especially if multiple repositories are involved or Aether is used as part of wider solution - like pax-url-aether or Apache Karaf.

Ideally we may start with these definitions:

  • non-SNAPSHOT version (like e.g., 1.2.0.RELEASE) never changes - each copy of an artifact with non-SNAPSHOT version should be exactly the same, no matter what repository does it come from
  • SNAPSHOT version (like e.g., 1.2.0.BUILD-SNAPSHOT) may always change, so there's a need to verify if we may get newer build of the artifact.

This is usually obvious when dealing with 3rd party libraries and general rule is:

Do not use SNAPSHOT dependencies for 3rd party libraries when releasing a product containing such library

This general rule doesn't apply during development - before releasing a product, some 3rd party (or internal) dependency libraries may use SNAPSHOT versions. I this case it's helpful to know when exactly SNAPSHOT versions are updated, when Maven/Aether checks for new SNAPSHOT build and when it's actually downloaded and written as current SNAPSHOT version inside Maven's local repository.

I'd like to provide detailed information on the subject in this part of Maven in OSGi series.

Metadata

The appealing idea behind SNAPSHOTs is to make easier to just depend on latest/current version of dependency without specifying the exact version number.

To be able to achieve such goal, Maven uses the concept of metadata. Instead of referring to external documentation, I'll present the metadata concept by example.

Let's start with simplest Maven project, without any dependencies, using SNAPSHOT version.

<project
    xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">

    <modelVersion>4.0.0</modelVersion>

    <groupId>grgr</groupId>
    <artifactId>universalis-api</artifactId>
    <version>0.1.0.BUILD-SNAPSHOT</version>

</project>

This POM declares exactly three things:

  • groupId = grgr
  • artifactId = universalis-api
  • version = 0.1.0.BUILD-SNAPSHOT

There's no need to have any code inside such project. Let's see what happens when we simply install such project in local repository by invoking mvn clean install.

$ mvn clean install
[INFO] Scanning for projects...
...
[INFO] --- maven-install-plugin:2.4:install (default-install) @ universalis-api ---
[INFO] Installing /data/ggrzybek/sources/_testing/grgr-universalis-api/target/universalis-api-0.1.0.BUILD-SNAPSHOT.jar ↵
    to /home/ggrzybek/.m2/repository/grgr/universalis-api/0.1.0.BUILD-SNAPSHOT/universalis-api-0.1.0.BUILD-SNAPSHOT.jar
[INFO] Installing /data/ggrzybek/sources/_testing/grgr-universalis-api/pom.xml ↵
    to /home/ggrzybek/.m2/repository/grgr/universalis-api/0.1.0.BUILD-SNAPSHOT/universalis-api-0.1.0.BUILD-SNAPSHOT.pom
...

I've extracted the information about maven-install-plugin execution, but in fact, the two above artifacts are not the only files that are stored in ~/.m2/repository/grgr/universalis-api directory. Remaining artifacts are metadata files. Before describing them, we'll mvn clean install again to see what has changed (git init; git add .; git commit trick inside ~/.m2/repository/grgr/universalis-api).

~/.m2/repository/grgr/universalis-api/0.1.0.BUILD-SNAPSHOT/_remote.repositories

This is Aether-specific file (we've described it shortly in first part of the series). It contains: 

#NOTE: This is an Aether internal implementation file, its format can be changed without prior notice.
#Tue Oct 18 09:29:14 CEST 2016
universalis-api-0.1.0.BUILD-SNAPSHOT.pom>=
universalis-api-0.1.0.BUILD-SNAPSHOT.jar>=

This is *.properties file with two properties. Only the keys are relevant, values are empty. By looking at this file, we can't say much, but after looking at other _remote.repositories files in our ~/.m2/repository, we can say:

  • It's written in org.eclipse.aether.internal.impl.TrackingFileManager#update()
  • Keys are constructed with: file name + > + repository ID
  • Most _remote.repositories files in ~/.m2/repository contain xyz.jar>central= property, because central repository is used by default in Maven.
  • This file is one of the most important sources of confusion when we operate in multi repository environment. I'm pretty sure we've all seen the following: 
    [ERROR] Failed to execute goal on project universalis-api: Could not resolve dependencies for project grgr:universalis-api:jar:0.1.0.BUILD-SNAPSHOT: ↵
      Failure to find commons-pool:commons-pool:jar:1.6.0.redhat-9 in https://repo.maven.apache.org/maven2 was cached in the local repository, ↵
      resolution will not be reattempted until the update interval of central has elapsed or updates are forced -> [Help 1]
    The version shown is arbitrary, but the problem is that Aether, when using org.eclipse.aether.internal.impl.EnhancedLocalRepositoryManager, not only checks if artifact is available locally - it also checks if it's downloaded from one of the repositories that are currently configured (in current Maven build or Aether session).
    The above error occurs, because we don't use the same repository (that we've downloaded the artifact from originally) in current build. Maven, by default uses EnhancedLocalRepositoryManager, pax-url-aether uses SimpleLocalRepositoryManagerFactory and doesn't have this problem.
  • Due to the above, it's important to carefully identify remote repositories used in parent POMs. To not end up with this: 
    $ find ~/.m2/repository -name _remote.repositories|xargs grep '>..'|cut -d '>' -f 2|sort|uniq
...
apache-snapshots=
apache.snapshots=
...
ea=
EA=
...
jboss.public=
jboss-public-repository=
jboss-public-repository-group=
...
ops4j.snapshot=
ops4j-snapshots=
...
sonatype=
sonatype-nexus-snapshots=
sonatype-snapshots=
...

~/.m2/repository/grgr/universalis-api/0.1.0.BUILD-SNAPSHOT/maven-metadata-local.xml

This file describes information about particular groupId:artifactId:version combination. For non-SNAPSHOT versions, we don't have this file created - there are simply no additional builds within single GAV where version is non-SNAPSHOT.

For SNAPSHOT versions however this file is used to list all different SNAPSHOT builds.

Here's the file after initial mvn clean install:

<?xml version="1.0" encoding="UTF-8"?>
<metadata modelVersion="1.1.0">
  <groupId>grgr</groupId>
  <artifactId>universalis-api</artifactId>
  <version>0.1.0.BUILD-SNAPSHOT</version>
  <versioning>
    <snapshot>
      <localCopy>true</localCopy>
    </snapshot>
    <lastUpdated>20161018072914</lastUpdated>
    <snapshotVersions>
      <snapshotVersion>
        <extension>jar</extension>
        <value>0.1.0.BUILD-SNAPSHOT</value>
        <updated>20161018072914</updated>
      </snapshotVersion>
      <snapshotVersion>
        <extension>pom</extension>
        <value>0.1.0.BUILD-SNAPSHOT</value>
        <updated>20161018072914</updated>
      </snapshotVersion>
    </snapshotVersions>
  </versioning>
</metadata>

And here's the diff after 2nd mvn clean install:

--- a/0.1.0.BUILD-SNAPSHOT/maven-metadata-local.xml
+++ b/0.1.0.BUILD-SNAPSHOT/maven-metadata-local.xml
@@ -7,17 +7,17 @@
     <snapshot>
       <localCopy>true</localCopy>
     </snapshot>
-    <lastUpdated>20161018072914</lastUpdated>
+    <lastUpdated>20161018075358</lastUpdated>
     <snapshotVersions>
       <snapshotVersion>
         <extension>jar</extension>
         <value>0.1.0.BUILD-SNAPSHOT</value>
-        <updated>20161018072914</updated>
+        <updated>20161018075358</updated>
       </snapshotVersion>
       <snapshotVersion>
         <extension>pom</extension>
         <value>0.1.0.BUILD-SNAPSHOT</value>
-        <updated>20161018072914</updated>
+        <updated>20161018075358</updated>
       </snapshotVersion>
     </snapshotVersions>
   </versioning>

Here's summary of important info about this file:

  • There's only one build tracked. Each installed artifact (like *.pom and *.jar) has its <snapshotVersion>.
  • versioning/snapshot/localCopy = true is an indication of local origin of the artifact.
  • There are timestamps for each artifact (and the metadata itself)

~/.m2/repository/grgr/universalis-api/maven-metadata-local.xml

This file describes information about particular groupId:artifactId across different versions. After several mvn clean install invocations for single SNAPSHOT versions, this file has only timestamp updated and it's neither particularly interesting nor complex: 

<?xml version="1.0" encoding="UTF-8"?>
<metadata>
  <groupId>grgr</groupId>
  <artifactId>universalis-api</artifactId>
  <versioning>
    <versions>
      <version>0.1.0.BUILD-SNAPSHOT</version>
    </versions>
    <lastUpdated>20161018075358</lastUpdated>
  </versioning>
</metadata>

It becomes however more interesting after our simple project evolves and other versions are installed. After we mvn clean install versions 0.1.0, 0.1.1, 0.2.0, 0.1.2 (in that order), we can see:

<?xml version="1.0" encoding="UTF-8"?>
<metadata>
  <groupId>grgr</groupId>
  <artifactId>universalis-api</artifactId>
  <versioning>
    <release>0.1.2</release>
    <versions>
      <version>0.1.0.BUILD-SNAPSHOT</version>
      <version>0.1.0</version>
      <version>0.1.1</version>
      <version>0.2.0</version>
      <version>0.1.2</version>
    </versions>
    <lastUpdated>20161018085259</lastUpdated>
  </versioning>
</metadata>

  • It's a kind of index of versions for particular combination of groupId:artifactId
  • versioning/release may not exactly point to latest version (more about LATEST … later).

Low level, canonical example

All right. Having the metadata background in mind, let's actually try to resolve this grgr:universalis-api:0.1.0.BUILD-SNAPSHOT artifact using Aether code.

(I've reverted back to the state where we had only 0.1.0.BUILD-SNAPSHOT available in ~/.m2/repository/grgr/universalis-api).

DefaultServiceLocator locator = MavenRepositorySystemUtils.newServiceLocator();
locator.setService(LocalRepositoryManagerFactory.class, SimpleLocalRepositoryManagerFactory.class);
locator.setService(org.eclipse.aether.spi.log.LoggerFactory.class, Slf4jLoggerFactory.class);
RepositorySystem system = locator.getService(RepositorySystem.class);

RepositorySystemSession session = MavenRepositorySystemUtils.newSession();
LocalRepositoryManager lrm = system.newLocalRepositoryManager(session, new LocalRepository("/home/ggrzybek/.m2/repository"));
((DefaultRepositorySystemSession)session).setLocalRepositoryManager(lrm);

ArtifactRequest req = new ArtifactRequest();
req.setArtifact(new DefaultArtifact("grgr", "universalis-api", "jar", "0.1.0.BUILD-SNAPSHOT"));
ArtifactResult res = system.resolveArtifact(session, req);

Nothing strange here. We're resolving an artifact without any reference to remote repositories - we expect to have SNAPSHOT artifact in local repository.

What Aether does internally?

org.eclipse.aether.resolution.VersionRequest is executed in version resolver. This request concerns grgr:universalis-api:jar:0.1.0.BUILD-SNAPSHOT artifact. The result of version resolution may be a change of version in original ArtifactRequest - usually after checking that remote SNAPSHOT metadata contains newer build than we already have.

If aether.versionResolver.noCache config option is not true, Aether's session cache is consulted (this is more important in such cases as full Maven build).

If version is not a release version (i.e., it is equal to RELEASE, LATEST or ends with SNAPSHOT), then Aether executes another request (or collection of such requests) - org.eclipse.aether.resolution.MetadataRequest.

MetadataRequest is prepared for local repository and for each remote repository added to ArtifactRequest in our code (we've added none).

Metadata is resolved by org.eclipse.aether.impl.MetadataResolver

Only one MetadataRequest - for local repository - is executed. This is handled by org.eclipse.aether.internal.impl.SimpleLocalRepositoryManager#find() method.

For local repository, grgr/universalis-api/0.1.0.BUILD-SNAPSHOT/maven-metadata-local.xml file is examined. If we had remote repositories added to initial ArtifactRequest, grgr/universalis-api/0.1.0.BUILD-SNAPSHOT/maven-metadata-central.xml (for repository with ID=central) would have been checked as well.

For each metadata found (for local and remote repositories. In our case - only for local one), versions are read. Versions correspond to /metadata/versioning XPath element from maven-metadata-<ID>.xml file.

Versions from each examined metadata file are merged to find latest version (based on timestamp from metadata). Latest version found from all MetadataResults is used as version in VersionResult

If aether.versionResolver.noCache config option is not true, Aether's session cache is updated.

Version from VersionResult is set as the version of Artifact inside our ArtifactRequest. In our case nothing has changed - 0.1.0.BUILD-SNAPSHOT was left untouched. We'll see how it may change when dealing with LATEST version and with remote metadata later.

org.eclipse.aether.internal.impl.SimpleLocalRepositoryManager#find() is used again - this time to find locally available grgr:universalis-api:jar:0.1.0.BUILD-SNAPSHOT artifact. Simple java.io.File#isFile() call is enough to determine whether the artifact is locally available. In case of EnhancedLocalRepositoryManager, _remote.repositories file would be examined as well.

ArtifactResult is returned and its artifact.file points to locally available artifact.

Here's important information about the above process:

  • We didn't mention about policies, but it's important to be aware that in case of local repositories, no policies are enforced. Metadata about SNAPSHOTs is read unconditionally.
  • No version translation happens here. It'd happen if we had remote repositories. (more about this later).

RELEASE and LATEST versions

To explain these concepts, let's mvn clean install some more versions of grgr:universalis-api. In order: 0.1.0, 0.1.1 and 0.2.0.BUILD-SNAPSHOT

Here's current grgr/universalis-api/maven-metadata-local.xml:

<?xml version="1.0" encoding="UTF-8"?>
<metadata>
  <groupId>grgr</groupId>
  <artifactId>universalis-api</artifactId>
  <versioning>
    <release>0.1.1</release>
    <versions>
      <version>0.1.0.BUILD-SNAPSHOT</version>
      <version>0.1.0</version>
      <version>0.1.1</version>
      <version>0.2.0.BUILD-SNAPSHOT</version>
    </versions>
    <lastUpdated>20161018102447</lastUpdated>
  </versioning>
</metadata>

We can try resolving grgr:universalis-api:RELEASE artifact:

ArtifactRequest req = new ArtifactRequest();
req.setArtifact(new DefaultArtifact("grgr", "universalis-api", "jar", "RELEASE"));
ArtifactResult res = system.resolveArtifact(session, req);

  • org.eclipse.aether.internal.impl.SimpleLocalRepositoryManager#find() is used by metadata resolver to fetch ~/.m2/repository/grgr/universalis-api/maven-metadata-local.xml file - which is metadata per groupId:artifactId - for all versions.
  • /metadata/versioning/release and /metadata/versioning/latest are taken into account (in that order)
  • ArtifactResult refers to 0.1.1 version after resolution

We can also try resolving grgr:universalis-api:LATEST artifact:

ArtifactRequest req = new ArtifactRequest();
req.setArtifact(new DefaultArtifact("grgr", "universalis-api", "jar", "LATEST"));
ArtifactResult res = system.resolveArtifact(session, req);

  • ~/.m2/repository/grgr/universalis-api/maven-metadata-local.xml file is read as above
  • /metadata/versioning/release and /metadata/versioning/latest are taken into account (in that order)
  • ArtifactResult refers to 0.1.1 version after resolution...

Hey - what's the difference then between RELEASE and LATEST? None - at least when only local repositories are involved.

What's more, if we mvn clean install versions 0.2.0 and 0.1.2 (in that order), resolving grgr:universalis-api:LATEST (or RELEASE) would result in … version 0.1.2 instead of 0.2.0.

Remote repositories

Let's start fresh and this time use remote repository. We can use Sonatype Nexus as our remote repository manager (even if installed locally). We'll use mvn clean deploy after correct configuration of server credentials.

Our POM contains now:

<distributionManagement>
    <repository>
        <id>local-nexus</id>
        <url>http://localhost:8081/nexus/content/repositories/releases</url>
    </repository>
    <snapshotRepository>
        <id>local-nexus</id>
        <url>http://localhost:8081/nexus/content/repositories/snapshots</url>
    </snapshotRepository>
</distributionManagement>

Deploying a project:

$ mvn clean deploy
[INFO] Scanning for projects...
...
[INFO] --- maven-install-plugin:2.4:install (default-install) @ universalis-api ---
[INFO] Installing /data/ggrzybek/sources/_testing/grgr-universalis-api/target/universalis-api-0.1.0.BUILD-SNAPSHOT.jar ↵
    to /home/ggrzybek/.m2/repository/grgr/universalis-api/0.1.0.BUILD-SNAPSHOT/universalis-api-0.1.0.BUILD-SNAPSHOT.jar
[INFO] Installing /data/ggrzybek/sources/_testing/grgr-universalis-api/pom.xml ↵
    to /home/ggrzybek/.m2/repository/grgr/universalis-api/0.1.0.BUILD-SNAPSHOT/universalis-api-0.1.0.BUILD-SNAPSHOT.pom
...
[INFO] --- maven-deploy-plugin:2.7:deploy (default-deploy) @ universalis-api ---
Downloading: http://localhost:8081/nexus/content/repositories/snapshots/grgr/universalis-api/0.1.0.BUILD-SNAPSHOT/maven-metadata.xml
Uploading: http://localhost:8081/nexus/content/repositories/snapshots/grgr/universalis-api/0.1.0.BUILD-SNAPSHOT/universalis-api-0.1.0.BUILD-20161018.111155-1.jar
Uploaded: http://localhost:8081/nexus/content/repositories/snapshots/grgr/universalis-api/0.1.0.BUILD-SNAPSHOT/universalis-api-0.1.0.BUILD-20161018.111155-1.jar (3 KB at 12.8 KB/sec)
Uploading: http://localhost:8081/nexus/content/repositories/snapshots/grgr/universalis-api/0.1.0.BUILD-SNAPSHOT/universalis-api-0.1.0.BUILD-20161018.111155-1.pom
Uploaded: http://localhost:8081/nexus/content/repositories/snapshots/grgr/universalis-api/0.1.0.BUILD-SNAPSHOT/universalis-api-0.1.0.BUILD-20161018.111155-1.pom (1020 B at 15.8 KB/sec)
Downloading: http://localhost:8081/nexus/content/repositories/snapshots/grgr/universalis-api/maven-metadata.xml
Uploading: http://localhost:8081/nexus/content/repositories/snapshots/grgr/universalis-api/0.1.0.BUILD-SNAPSHOT/maven-metadata.xml
Uploaded: http://localhost:8081/nexus/content/repositories/snapshots/grgr/universalis-api/0.1.0.BUILD-SNAPSHOT/maven-metadata.xml (787 B at 19.2 KB/sec)
Uploading: http://localhost:8081/nexus/content/repositories/snapshots/grgr/universalis-api/maven-metadata.xml
Uploaded: http://localhost:8081/nexus/content/repositories/snapshots/grgr/universalis-api/maven-metadata.xml (285 B at 9.0 KB/sec)
...

maven-deploy-plugin:

  • Downloads maven-metadata.xml for groupId:artifactId:version - if Nexus is empty, none is found
  • Uploads *.jar and *.pom artifacts with SNAPSHOT version changed to <timestamp>-<build number>
  • Downloads maven-metadata.xml for groupId:artifactId - if Nexus is empty, none is found
  • Uploads both groupId:artifactId and groupId:artifactId:version metadata - after changing them. This is important information - repository managers don't generate metadata - it's Maven that does it (after examining currently deployed metadata - by previous mvn deploy) and uploads changed version back to Nexus.

Example with remote repository

Let's run our example again - after removing local copy of SNAPSHOT. We'll try to resolve SNAPSHOT from remote repository.

DefaultServiceLocator locator = MavenRepositorySystemUtils.newServiceLocator();
locator.setService(RepositoryConnectorFactory.class, BasicRepositoryConnectorFactory.class);
locator.addService(TransporterFactory.class, HttpTransporterFactory.class);
locator.setService(LocalRepositoryManagerFactory.class, SimpleLocalRepositoryManagerFactory.class);
locator.setService(org.eclipse.aether.spi.log.LoggerFactory.class, Slf4jLoggerFactory.class);
RepositorySystem system = locator.getService(RepositorySystem.class);

RepositorySystemSession session = MavenRepositorySystemUtils.newSession();
LocalRepositoryManager lrm = system.newLocalRepositoryManager(session, new LocalRepository("/home/ggrzybek/.m2/repository"));
((DefaultRepositorySystemSession)session).setLocalRepositoryManager(lrm);

RemoteRepository.Builder b = new RemoteRepository.Builder("local-nexus", "default", "http://localhost:8081/nexus/content/repositories/snapshots");
RepositoryPolicy enabledPolicy = new RepositoryPolicy(true, RepositoryPolicy.UPDATE_POLICY_NEVER, RepositoryPolicy.CHECKSUM_POLICY_IGNORE);
RepositoryPolicy disabledPolicy = new RepositoryPolicy(false, RepositoryPolicy.UPDATE_POLICY_NEVER, RepositoryPolicy.CHECKSUM_POLICY_IGNORE);
b.setReleasePolicy(disabledPolicy);
b.setSnapshotPolicy(enabledPolicy);

ArtifactRequest req = new ArtifactRequest();
req.addRepository(b.build());

req.setArtifact(new DefaultArtifact("grgr", "universalis-api", "jar", "0.1.0.BUILD-SNAPSHOT"));
ArtifactResult res = system.resolveArtifact(session, req);
LOG.info("Result: " + res);

We've added one RemoteRepository to ArtifactRequest. This repository has two policies set:

  • disabled releases
  • enabled snapshots, ignoring checksum verification errors and no-update policy (more about this soon).

Assuming that there's currently no ~/.m2/repository/grgr/univrsalis-api directory, what are the additional operations performed by Aether? Let's trace the metadata resolution fragment:

MetadataRequest is prepared for local repository and for each remote repository - we have two such requests now.

MetadataRequest for local repository uses grgr/universalis-api/0.1.0.BUILD-SNAPSHOT/maven-metadata-local.xml path - there's no such file.

MetadataRequest for remote repository uses grgr/universalis-api/0.1.0.BUILD-SNAPSHOT/maven-metadata-local-nexus.xml path (local-nexus is ID of our remote repository in pom.xml) - there's no such file (yet). This request has favorLocalRepository set to true.

For each remote repository, org.eclipse.aether.impl.UpdateCheck is started.

org.eclipse.aether.impl.UpdateCheck#policy is set to never (as we've requested).

org.eclipse.aether.impl.UpdateCheckManager is invoked to verify if update is needed

touch file is checked (~/.m2/repository/grgr/universalis-api/0.1.0.BUILD-SNAPSHOT/resolver-status.properties) - there's no such file (yet)

Because we don't have any metadata file available locally, the check is marked as required. Even if we've specified "never" as update policy.

For each required check, org.eclipse.aether.internal.impl.DefaultMetadataResolver.ResolveTask is scheduled.

RepositoryConnector is used to download metadata. After downloading, we have first file: ~/.m2/repository/grgr/universalis-api/0.1.0.BUILD-SNAPSHOT/maven-metadata-local-nexus.xml:

<?xml version="1.0" encoding="UTF-8"?>
<metadata modelVersion="1.1.0">
  <groupId>grgr</groupId>
  <artifactId>universalis-api</artifactId>
  <version>0.1.0.BUILD-SNAPSHOT</version>
  <versioning>
    <snapshot>
      <timestamp>20161018.111155</timestamp>
      <buildNumber>1</buildNumber>
    </snapshot>
    <lastUpdated>20161018111155</lastUpdated>
    <snapshotVersions>
      <snapshotVersion>
        <extension>jar</extension>
        <value>0.1.0.BUILD-20161018.111155-1</value>
        <updated>20161018111155</updated>
      </snapshotVersion>
      <snapshotVersion>
        <extension>pom</extension>
        <value>0.1.0.BUILD-20161018.111155-1</value>
        <updated>20161018111155</updated>
      </snapshotVersion>
    </snapshotVersions>
  </versioning>
</metadata>

touch file is updated. Aether writes the following properties to ~/.m2/repository/grgr/universalis-api/0.1.0.BUILD-SNAPSHOT/resolver-status.properties:

#NOTE: This is an Aether internal implementation file, its format can be changed without prior notice.
#Tue Oct 18 14:12:00 CEST 2016
maven-metadata-local-nexus.xml.lastUpdated=1476792720594

For each metadata found (for local and remote repositories), versions are read. Versions correspond to /metadata/versioning XPath element from maven-metadata-<ID>.xml file.

Versions from each examined metadata file are merged to find latest version (based on timestamp from metadata). Latest version found from all MetadataResults is used as version in VersionResult

Our version result is 0.1.0.BUILD-20161018.111155-1 - from remote metadata.

This time such artifact is not available locally (yet).

Artifact is downloaded.

If aether.artifactResolver.snapshotNormalization is true (which is default), downloaded universalis-api-0.1.0.BUILD-20161018.111155-1.jar artifact is copied to universalis-api-0.1.0.BUILD-SNAPSHOT.jar as well.

Update policy

As we've seen, the key information required to resolve SNAPSHOT artifacts is contained in maven-metadata.xml. There are different scenarios now:

  • Someone else may have deployed newer version of SNAPSHOT artifact to Remote repository
  • We may have installed (mvn clean install) newer version of SNAPSHOT ourselves.

With SNAPSHOTs, the two questions are:

  • When (if at all) maven-metadata.xml is downloaded from remote repository?
  • When (if at all) actual artifact is downloaded from remote repository?

The answer is "it depends".

Let's assume that Nexus contains two builds of an artifact: 0.1.0.BUILD-20161018.111155-1 and 0.1.0.BUILD-20161018.124558-2:

  • We may have no local version of remote maven-metadata-local-nexus.xml.
  • We may have local version of remote maven-metadata-local-nexus.xml where 0.1.0.BUILD-20161018.111155-1 is the latest - older than remote.
  • We may have local version of remote maven-metadata-local-nexus.xml where 0.1.0.BUILD-20161018.124558-2 is the latest - up to date with remote.

We also may have local maven-metadata-local.xml describing locally mvn clean installed SNAPSHOTs.

Here are the rules for determining when remote maven-metadata.xml is downloaded:

  • Initial resolution request (ArtifactRequest) must have one or more remote repositories added.
  • Assuming we're resolving with remote repository having local-nexus ID, Aether will look for maven-metadata-local-nexus.xml file.
  • If there's no ~/.m2/repository/grgr/universalis-api/0.1.0.BUILD-SNAPSHOT/maven-metadata-local-nexus.xml file, it'll be downloaded.
  • If there's such file, its timestamp (java.io.File#lastModified()) is verified against effective update policy:
    • never - remote metadata won't be downloaded
    • interval:N - remote metadata will be downloaded if the timestamp is older than N minutes
    • daily - remote metadata will be downloaded if the timestamp is not created today (even if it was created a minute before midnight and it's 00:00:01 now).
    • always - remote metadata will be downloaded

That's all! These rules are quite straightforward and clean. There's however another resolution performed by Aether - to determine which version will actually be used.

org.eclipse.aether.impl.VersionResolver uses org.eclipse.aether.impl.MetadataResolver. Metadata is gathered using the above rules. In simplest case, when artifact resolution is performed with one remote repository, we have at most two metadata files:

  • ~/.m2/repository/grgr/universalis-api/0.1.0.BUILD-SNAPSHOT/maven-metadata-local.xml - from mvn clean install, may not exist
  • ~/.m2/repository/grgr/universalis-api/0.1.0.BUILD-SNAPSHOT/maven-metadata-local-nexus.xml - from remote repository, must exist, but may be not up to date with remote version

Now, Aether checks /metadata/snapshotVersions/snapshotVersion elements (case when at 11:11:55 artifact was deployed to nexus and at 12:10:33 we've invoked mvn clean install):

  • In maven-metadata-local.xml:
    <snapshotVersions>
  <snapshotVersion>
    <extension>jar</extension>
    <value>0.1.0.BUILD-SNAPSHOT</value>
    <updated>20161018121033</updated>
  </snapshotVersion>
  <snapshotVersion>
    <extension>pom</extension>
    <value>0.1.0.BUILD-SNAPSHOT</value>
    <updated>20161018121033</updated>
  </snapshotVersion>
</snapshotVersions>
  • In maven-metadata-local-nexus.xml:
    <snapshotVersions>
  <snapshotVersion>
    <extension>jar</extension>
    <value>0.1.0.BUILD-20161018.111155-1</value>
    <updated>20161018111155</updated>
  </snapshotVersion>
  <snapshotVersion>
    <extension>pom</extension>
    <value>0.1.0.BUILD-20161018.111155-1</value>
    <updated>20161018111155</updated>
  </snapshotVersion>
</snapshotVersions>

What Aether does is simply comparing snapshotVersion/updated timestamps, so org.eclipse.aether.impl.VersionResolver after examining all MetadataResults, returns latest version, which is 0.1.0.BUILD-SNAPSHOT. VersionResult has:

  • versionResult.version = 0.1.0.BUILD-SNAPSHOT
  • versionResult.repository = "/home/ggrzybek/.m2/repository (simple)" (LocalRepository)

Let's now force download of current remote metadata (using e.g., UPDATE_POLICY_ALWAYS), to get:

<snapshotVersions>
  <snapshotVersion>
    <extension>jar</extension>
    <value>0.1.0.BUILD-20161018.124558-2</value>
    <updated>20161018124558</updated>
  </snapshotVersion>
  <snapshotVersion>
    <extension>pom</extension>
    <value>0.1.0.BUILD-20161018.124558-2</value>
    <updated>20161018124558</updated>
  </snapshotVersion>
</snapshotVersions>

Now, after examining all MetadataResults, VersionResult has:

  • versionResult.version = 0.1.0.BUILD-20161018.124558-2
  • versionResult.repository = "local-nexus (http://localhost:8081/nexus/content/repositories/snapshots, default, snapshots)" (RemoteRepository)

A quick summary now - so far, update policy was used to determine whether to download remote version of maven-metadata-local-nexus.xml and entire resolution process updates requested version inside ArtifactResult.artifact.version.

After examining all available metadata files, resolution process changed our initial GAV from grgr:universalis-api:jar:0.1.0.BUILD-SNAPSHOT to grgr:universalis-api:jar:0.1.0.BUILD-20161018.124558-2. Now usual resolution process continues:

  • local repository is checked if it contains grgr/universalis-api/0.1.0.BUILD-SNAPSHOT/universalis-api-0.1.0.BUILD-20161018.124558-2.jar - it doesn't (yet)
  • artifact is scheduled to be downloaded. It's written to universalis-api-0.1.0.BUILD-20161018.124558-2.jar, and copied (because aether.artifactResolver.snapshotNormalization is true) to universalis-api-0.1.0.BUILD-SNAPSHOT.jar.

Update policy is not used at all for actual artifacts! It's used only for metadata.

Summary

Whew. It was long material. Let's summarize SNAPSHOT handling with Aether:

  • If resolution is performed without remote repository, local SNAPSHOT artifact must be available.
  • If resolution is performed with remote repository, update policy is taken into account.
  • update policy is used to determine whether metadata from remote repositories is downloaded.
  • After possible download of metadata we have following maven-metadata-<ID>.xml files:
    • one for each remote repository used
    • possibly one related to locally mvn clean installed SNAPSHOT
  • Using timestamps, newest SNAPSHOT version is passed for further artifact resolution
    • if latest SNAPSHOT comes from one of remote metadata files, SNAPSHOT is translated to <timestamp>-<build number>.
    • if latest SNAPSHOT comes from local metadata file, version is unchanged
  • The effect of version resolution and metadata resolution is possible change of version for pending artifact resolution.
  • If translated artifact coordinates (groupId and artifactId don't change, version may be changed) describe locally available artifact, it's returned.
  • If the artifact is not available, it's being downloaded from repository which was related to metadata file that declared latest SNAPSHOT.
  • Update policy is not used at all for artifact download.

In next installment I'll describe how update policies may be configured with pax-url-aether.

Autor:
Etykiety: , , , ,

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)