Programming, contributing

2021-11-23

Pax Web 8 - Whiteboard services and Karaf commands

Introduction

Pax Web 8 was released on October 20th. While the main blog post summarizing all the changes is still work-in-progress, let me show you how it works with Karaf 4.4.

Here's a list of Karaf command invocations, where the commands are provided by two different bundles:

pax-web-karaf - this is the bundle that provides Karaf commands to check the information about installed Web Applications. This bundle is supposed to replace all the commands that were so far available in Karaf itself (see Karaf's web:list and http:list commands)
org.ops4j.pax.web.samples/showcase - this bundle provides sample:hs and sample:whiteboard commands that can be used to register/unregister various Web elements and contexts

Detailed presentation

First `web:context-list` presents the contexts

A context is very broad term. Contexts can be:

implementations of javax.servlet.ServletContext - these objects are identified by a context path path and in JavaEE sense, there's one such context per Java Web Application
objects that implement org.osgi.service.http.HttpContext interface (HttpService scenario) or extend org.osgi.service.http.context.ServletContextHelper abstract class (Whiteboard scenario) - these contexts are the the objects that OSGi developers deal with directly (or reference directly during Whiteboard registration)
objects of org.ops4j.pax.web.service.spi.model.OsgiContextModel class - this is an internal Pax Web 8 class that's used to unify the experience related to org.osgi.service.http.HttpContext and org.osgi.service.http.context.ServletContextHelper contexts. These objects are how the user-facing OSGi services (HttpContext and ServletContextHelper) are tracked internally.

When Karaf starts with Pax Web 8, there's only one OSGi context:

karaf@root()> web:context-list
Bundle ID │ Symbolic Name                                 │ Context Path │ Context Name │ Rank │ Service ID │ Type       │ Scope   │ Registration Properties
──────────┼───────────────────────────────────────────────┼──────────────┼──────────────┼──────┼────────────┼────────────┼─────────┼──────────────────────────────────────────
77        │ org.ops4j.pax.web.pax-web-extender-whiteboard │ /            │ default      │ 0    │ 0          │ Whiteboard │ static* │ osgi.http.whiteboard.context.name=default
          │                                               │              │              │      │            │            │         │ osgi.http.whiteboard.context.path=/

*) This context is using ServletContextHelper/HttpContext without resolving an org.osgi.framework.ServiceReference.

There's quite a lot of information in this single line:

The bundle that has registered this OSGi context is pax-web-extender-whiteboard and that's the behavior defined in 140.2 The Servlet Context chapter of OSGi CMPN R7 specification. The context path is / and the name is default.
Service rank and Service ID are 0, so if one wants to override default Whiteboard context, it's enough to register a org.osgi.service.http.context.ServletContextHelper with higher ranking.
Type == Whiteboard means that the OSGi context comes from Whitboard part of Pax Web. It also means that there are other types of contexts (namely - HttpService and WAB).
Scope == static* effectively means that this OSGi context is special and treated as default in Pax Web.
Finally we can see (some) registration properties that can be used when targetting given OSGi context.

Quite low-level command shows how interesting bundles reference Pax Web services. The `Usage Count` is obtained via reflection (for now - Felix specific).

The purpose of this command is to verify if the bundles properly get and unget Pax Web services, so we have no resource leaks.

Initially, only two Pax Web extenders obtained an instance of org.osgi.service.http.HttpService, org.ops4j.pax.web.service.WebContainer.

karaf@root()> web:meta --web-usage-count 
Registering Bundle: org.ops4j.pax.web.pax-web-runtime [71]
Service ID: 109
Service Scope: bundle

Usage Counts for bundles referencing the service:
Bundle ID │ Symbolic Name                                 │ Service ID │ Scope     │ Service objectClass                                                         │ Usage Count
──────────┼───────────────────────────────────────────────┼────────────┼───────────┼─────────────────────────────────────────────────────────────────────────────┼────────────
75        │ org.ops4j.pax.web.pax-web-extender-war        │ 90         │ singleton │ [org.ops4j.pax.web.service.spi.model.events.WebApplicationEventListener]    │ 1
75        │ org.ops4j.pax.web.pax-web-extender-war        │ 109        │ bundle    │ [org.osgi.service.http.HttpService, org.ops4j.pax.web.service.WebContainer] │ 1
77        │ org.ops4j.pax.web.pax-web-extender-whiteboard │ 109        │ bundle    │ [org.osgi.service.http.HttpService, org.ops4j.pax.web.service.WebContainer] │ 1
111       │ org.ops4j.pax.web.pax-web-karaf               │ 109        │ bundle    │ [org.osgi.service.http.HttpService, org.ops4j.pax.web.service.WebContainer] │ 0

Let's use sample command to Whiteboard-register a `org.osgi.service.http.context.ServletContextHelper` with custom name and non-default context path.

karaf@root()> sample:whiteboard context c1 /c1
>> Using context for bundle org.ops4j.pax.web.samples.showcase [110]
>> Registering org.osgi.service.http.context.ServletContextHelper with "c1" name, "/c1" context path and for org.ops4j.pax.web.samples.showcase [110].

Let's register a `javax.servlet.http.HttpServlet` Whiteboard service, targetting the above context.

karaf@root()> sample:whiteboard servlet s1 /s1/* c1
>> Using context for bundle org.ops4j.pax.web.samples.showcase [110]
>> Registering a servlet with "s1" name, "/s1/*" pattern, "(osgi.http.whiteboard.context.name=c1)" context selector and for org.ops4j.pax.web.samples.showcase [110].
>>>> Registered successfully. You can test it using `curl -i http://127.0.0.1:8181/c1/s1/anything`

The thing to highlight is that the servlet was registered with osgi.http.whiteboard.context.select=(osgi.http.whiteboard.context.name=c1) registration property. That's the Whiteboard-way to select a target OSGi context, where this servlet should be registered to.

This is exactly the reason why Pax Web 8 was started in the first place! A servlet may target more contexts and it should be available in all of them.

We can check that the servlet is working directly from Karaf

karaf@root()> exec curl -i http://127.0.0.1:8181/c1/s1/anything % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 225 0 225 0 0 5032 0 --:--:-- --:--:-- --:--:-- 5113 HTTP/1.1 200 OK Date: Tue, 23 Nov 2021 14:27:06 GMT Content-Type: text/plain;charset=utf-8 Transfer-Encoding: chunked Servlet: s1 Servlet Config: org.ops4j.pax.web.service.spi.servlet.OsgiInitializedServlet$1@332abd96 Servlet Context: org.ops4j.pax.web.service.spi.servlet.OsgiScopedServletContext@34453e72 Configured name (Whiteboard): /s1/*

As expected, there are now more contexts

karaf@root()> context-list Bundle ID │ Symbolic Name │ Context Path │ Context Name │ Rank │ Service ID │ Type │ Scope │ Registration Properties ──────────┼───────────────────────────────────────────────┼──────────────┼──────────────┼──────┼────────────┼────────────┼───────────┼────────────────────────────────────────── 77 │ org.ops4j.pax.web.pax-web-extender-whiteboard │ / │ default │ 0 │ 0 │ Whiteboard │ static* │ osgi.http.whiteboard.context.name=default │ │ │ │ │ │ │ │ osgi.http.whiteboard.context.path=/ 110 │ org.ops4j.pax.web.samples.showcase │ /c1 │ c1 │ 0 │ 117 │ Whiteboard │ singleton │ osgi.http.whiteboard.context.name=c1 │ │ │ │ │ │ │ │ osgi.http.whiteboard.context.path=/c1 *) This context is using ServletContextHelper/HttpContext without resolving an org.osgi.framework.ServiceReference.

And the usage count has changed

karaf@root()> web:meta --web-usage-count Registering Bundle: org.ops4j.pax.web.pax-web-runtime [71] Service ID: 109 Service Scope: bundle Usage Counts for bundles referencing the service: Bundle ID │ Symbolic Name │ Service ID │ Scope │ Service objectClass │ Usage Count ──────────┼───────────────────────────────────────────────┼────────────┼───────────┼─────────────────────────────────────────────────────────────────────────────┼──────────── 75 │ org.ops4j.pax.web.pax-web-extender-war │ 90 │ singleton │ [org.ops4j.pax.web.service.spi.model.events.WebApplicationEventListener] │ 1 75 │ org.ops4j.pax.web.pax-web-extender-war │ 109 │ bundle │ [org.osgi.service.http.HttpService, org.ops4j.pax.web.service.WebContainer] │ 1 77 │ org.ops4j.pax.web.pax-web-extender-whiteboard │ 109 │ bundle │ [org.osgi.service.http.HttpService, org.ops4j.pax.web.service.WebContainer] │ 1 110 │ org.ops4j.pax.web.samples.showcase │ 109 │ bundle │ [org.osgi.service.http.HttpService, org.ops4j.pax.web.service.WebContainer] │ 1 110 │ org.ops4j.pax.web.samples.showcase │ 117 │ singleton │ [org.osgi.service.http.context.ServletContextHelper] │ 2 110 │ org.ops4j.pax.web.samples.showcase │ 119 │ singleton │ [javax.servlet.http.HttpServlet] │ 1 110 │ org.ops4j.pax.web.samples.showcase │ 111 │ singleton │ [org.osgi.service.http.runtime.HttpServiceRuntime] │ 0 111 │ org.ops4j.pax.web.pax-web-karaf │ 109 │ bundle │ [org.osgi.service.http.HttpService, org.ops4j.pax.web.service.WebContainer] │ 0

Now there's more to explain:

The interesting rows are the ones related to bundle org.ops4j.pax.web.samples.showcase, as it's the bundle that is used to register Whiteboard services.

The usage count for org.osgi.service.http.HttpService/org.ops4j.pax.web.service.WebContainer is exactly 1, as the instance of the service is cached for entire Whiteboard Web Application
The usage count for javax.servlet.http.HttpServlet OSGi service is exactly 1. What's more, it was 0 before servlets init() method was called.

The usage count for org.osgi.service.http.context.ServletContextHelper OSGi service is 2. That's because it was dereferenced twice:

To obtain default OSGi context for requests that do not use target servlet (for example - only a request chain containing filters)

To obtain an OSGi context for the servlet. This is required, so the servlet (which may be registered by different bundle than the one registering the ServletContextHelper) gets its own bundle-scoped instance, so Whiteboard specification is satisfied.

The usage count for org.osgi.service.http.runtime.HttpServiceRuntime is 0 because it was only used by the command itself to provide user with example curl invocation.

A Whiteboard Web Application is a collection of Whiteboard-registered Web elements and contexts, registered by single bundle. One bundle may register web elements targetting different context paths and one context path (equivalent of JavaEE Web Application) may be populated by web elements coming from different bundles.

Now let's unregister the context, which was targetted by the servlet.

karaf@root()> sample:whiteboard -d -s 117 context >> Using context for bundle org.ops4j.pax.web.samples.showcase [110] >> Unregistering org.osgi.service.http.context.ServletContextHelper with 117 service.id and for org.ops4j.pax.web.samples.showcase [110]. >>>> Unregistered org.apache.felix.framework.ServiceRegistrationImpl@5840e742 successfully. karaf@root()> exec curl -i http://127.0.0.1:8181/c1/s1/anything % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 371 100 371 0 0 65443 0 --:--:-- --:--:-- --:--:-- 74200 HTTP/1.1 404 Not Found Cache-Control: must-revalidate,no-cache,no-store Content-Type: text/html;charset=iso-8859-1 Content-Length: 371

See? We've got expected 404

What's more, we can see this information in the logs:

(WhiteboardExtenderContext.java:446) : Unregistering ServletModel{id=ServletModel-5,name='s1',urlPatterns=[/s1/*],contexts=[{WB,OCM-3,c1,/c1}]} because its context selection filter doesn't match any context

No worries - when proper context is registered again, the servlet will move to this new context. We'll see it later.

context-list command shows again only one context

karaf@root()> context-list Bundle ID │ Symbolic Name │ Context Path │ Context Name │ Rank │ Service ID │ Type │ Scope │ Registration Properties ──────────┼───────────────────────────────────────────────┼──────────────┼──────────────┼──────┼────────────┼────────────┼─────────┼────────────────────────────────────────── 77 │ org.ops4j.pax.web.pax-web-extender-whiteboard │ / │ default │ 0 │ 0 │ Whiteboard │ static* │ osgi.http.whiteboard.context.name=default │ │ │ │ │ │ │ │ osgi.http.whiteboard.context.path=/ *) This context is using ServletContextHelper/HttpContext without resolving an org.osgi.framework.ServiceReference.

The usage counts has changed too

karaf@root()> web:meta --web-usage-count Registering Bundle: org.ops4j.pax.web.pax-web-runtime [71] Service ID: 109 Service Scope: bundle Usage Counts for bundles referencing the service: Bundle ID │ Symbolic Name │ Service ID │ Scope │ Service objectClass │ Usage Count ──────────┼───────────────────────────────────────────────┼────────────┼───────────┼─────────────────────────────────────────────────────────────────────────────┼──────────── 75 │ org.ops4j.pax.web.pax-web-extender-war │ 90 │ singleton │ [org.ops4j.pax.web.service.spi.model.events.WebApplicationEventListener] │ 1 75 │ org.ops4j.pax.web.pax-web-extender-war │ 109 │ bundle │ [org.osgi.service.http.HttpService, org.ops4j.pax.web.service.WebContainer] │ 1 77 │ org.ops4j.pax.web.pax-web-extender-whiteboard │ 109 │ bundle │ [org.osgi.service.http.HttpService, org.ops4j.pax.web.service.WebContainer] │ 1 110 │ org.ops4j.pax.web.samples.showcase │ 109 │ bundle │ [org.osgi.service.http.HttpService, org.ops4j.pax.web.service.WebContainer] │ 1 110 │ org.ops4j.pax.web.samples.showcase │ 119 │ singleton │ [javax.servlet.http.HttpServlet] │ 0 110 │ org.ops4j.pax.web.samples.showcase │ 111 │ singleton │ [org.osgi.service.http.runtime.HttpServiceRuntime] │ 0 111 │ org.ops4j.pax.web.pax-web-karaf │ 109 │ bundle │ [org.osgi.service.http.HttpService, org.ops4j.pax.web.service.WebContainer] │ 0

Important observation:

Usage count for javax.servlet.http.HttpServlet dropped to 0, as the servlet has been stopped (destroy() was called on it).

There are no more references for org.osgi.service.http.context.ServletContextHelper, because this service was unregistered completely.

There's still one usage for org.osgi.service.http.HttpService/org.ops4j.pax.web.service.WebContainer because the Whiteboard Web Application is still there. It'll be removed completely when the bundle 110 is stopped.

Let's see Pax Web 8 magic in action. Registering new OSGi context that can satisfy already registered servlet, automatically redeploys the servlet to new matching context!

karaf@root()> sample:whiteboard context c1 /c2 >> Using context for bundle org.ops4j.pax.web.samples.showcase [110] >> Registering org.osgi.service.http.context.ServletContextHelper with "c1" name, "/c2" context path and for org.ops4j.pax.web.samples.showcase [110].

We've registered a c1 context (the name used before), but with different context path: /c2 instead of /c1.

Let's check the usage counts

karaf@root()> web:meta --web-usage-count Registering Bundle: org.ops4j.pax.web.pax-web-runtime [71] Service ID: 109 Service Scope: bundle Usage Counts for bundles referencing the service: Bundle ID │ Symbolic Name │ Service ID │ Scope │ Service objectClass │ Usage Count ──────────┼───────────────────────────────────────────────┼────────────┼───────────┼─────────────────────────────────────────────────────────────────────────────┼──────────── 75 │ org.ops4j.pax.web.pax-web-extender-war │ 90 │ singleton │ [org.ops4j.pax.web.service.spi.model.events.WebApplicationEventListener] │ 1 75 │ org.ops4j.pax.web.pax-web-extender-war │ 109 │ bundle │ [org.osgi.service.http.HttpService, org.ops4j.pax.web.service.WebContainer] │ 1 77 │ org.ops4j.pax.web.pax-web-extender-whiteboard │ 109 │ bundle │ [org.osgi.service.http.HttpService, org.ops4j.pax.web.service.WebContainer] │ 1 110 │ org.ops4j.pax.web.samples.showcase │ 109 │ bundle │ [org.osgi.service.http.HttpService, org.ops4j.pax.web.service.WebContainer] │ 1 110 │ org.ops4j.pax.web.samples.showcase │ 119 │ singleton │ [javax.servlet.http.HttpServlet] │ 0 110 │ org.ops4j.pax.web.samples.showcase │ 111 │ singleton │ [org.osgi.service.http.runtime.HttpServiceRuntime] │ 0 110 │ org.ops4j.pax.web.samples.showcase │ 120 │ singleton │ [org.osgi.service.http.context.ServletContextHelper] │ 2 111 │ org.ops4j.pax.web.pax-web-karaf │ 109 │ bundle │ [org.osgi.service.http.HttpService, org.ops4j.pax.web.service.WebContainer] │ 0

The new org.osgi.service.http.context.ServletContextHelper is already referenced twice (the same reasons as above), but the javax.servlet.http.HttpServlet is referenced 0 times - because the servlet wasn't called yet (thus no init() was called on it).

Servlet is available in new context and not available in the old one

karaf@root()> exec curl -i http://127.0.0.1:8181/c1/s1/anything % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 371 100 371 0 0 201k 0 --:--:-- --:--:-- --:--:-- 362k HTTP/1.1 404 Not Found Cache-Control: must-revalidate,no-cache,no-store Content-Type: text/html;charset=iso-8859-1 Content-Length: 371 karaf@root()> exec curl -i http://127.0.0.1:8181/c2/s1/anything % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 225 0 225 0 0 124k 0 --:--:-- --:--:-- --:--:-- 219k HTTP/1.1 200 OK Date: Tue, 23 Nov 2021 14:29:32 GMT Content-Type: text/plain;charset=utf-8 Transfer-Encoding: chunked Servlet: s1 Servlet Config: org.ops4j.pax.web.service.spi.servlet.OsgiInitializedServlet$1@2e0bd609 Servlet Context: org.ops4j.pax.web.service.spi.servlet.OsgiScopedServletContext@6b095dac Configured name (Whiteboard): /s1/*

context-list at this stage

karaf@root()> context-list Bundle ID │ Symbolic Name │ Context Path │ Context Name │ Rank │ Service ID │ Type │ Scope │ Registration Properties ──────────┼───────────────────────────────────────────────┼──────────────┼──────────────┼──────┼────────────┼────────────┼───────────┼────────────────────────────────────────── 77 │ org.ops4j.pax.web.pax-web-extender-whiteboard │ / │ default │ 0 │ 0 │ Whiteboard │ static* │ osgi.http.whiteboard.context.name=default │ │ │ │ │ │ │ │ osgi.http.whiteboard.context.path=/ 110 │ org.ops4j.pax.web.samples.showcase │ /c2 │ c1 │ 0 │ 120 │ Whiteboard │ singleton │ osgi.http.whiteboard.context.name=c1 │ │ │ │ │ │ │ │ osgi.http.whiteboard.context.path=/c2 *) This context is using ServletContextHelper/HttpContext without resolving an org.osgi.framework.ServiceReference.

Usage counts still shows no leakage

karaf@root()> web:meta --web-usage-count Registering Bundle: org.ops4j.pax.web.pax-web-runtime [71] Service ID: 109 Service Scope: bundle Usage Counts for bundles referencing the service: Bundle ID │ Symbolic Name │ Service ID │ Scope │ Service objectClass │ Usage Count ──────────┼───────────────────────────────────────────────┼────────────┼───────────┼─────────────────────────────────────────────────────────────────────────────┼──────────── 75 │ org.ops4j.pax.web.pax-web-extender-war │ 90 │ singleton │ [org.ops4j.pax.web.service.spi.model.events.WebApplicationEventListener] │ 1 75 │ org.ops4j.pax.web.pax-web-extender-war │ 109 │ bundle │ [org.osgi.service.http.HttpService, org.ops4j.pax.web.service.WebContainer] │ 1 77 │ org.ops4j.pax.web.pax-web-extender-whiteboard │ 109 │ bundle │ [org.osgi.service.http.HttpService, org.ops4j.pax.web.service.WebContainer] │ 1 110 │ org.ops4j.pax.web.samples.showcase │ 109 │ bundle │ [org.osgi.service.http.HttpService, org.ops4j.pax.web.service.WebContainer] │ 1 110 │ org.ops4j.pax.web.samples.showcase │ 119 │ singleton │ [javax.servlet.http.HttpServlet] │ 1 110 │ org.ops4j.pax.web.samples.showcase │ 111 │ singleton │ [org.osgi.service.http.runtime.HttpServiceRuntime] │ 0 110 │ org.ops4j.pax.web.samples.showcase │ 120 │ singleton │ [org.osgi.service.http.context.ServletContextHelper] │ 2 111 │ org.ops4j.pax.web.pax-web-karaf │ 109 │ bundle │ [org.osgi.service.http.HttpService, org.ops4j.pax.web.service.WebContainer] │ 0

Let's unregister the servlet this time

karaf@root()> sample:whiteboard -d servlet s1 >> Using context for bundle org.ops4j.pax.web.samples.showcase [110] >>>> Unregistered org.apache.felix.framework.ServiceRegistrationImpl@68ed9dd9 successfully. karaf@root()> web:meta --web-usage-count Registering Bundle: org.ops4j.pax.web.pax-web-runtime [71] Service ID: 109 Service Scope: bundle Usage Counts for bundles referencing the service: Bundle ID │ Symbolic Name │ Service ID │ Scope │ Service objectClass │ Usage Count ──────────┼───────────────────────────────────────────────┼────────────┼───────────┼─────────────────────────────────────────────────────────────────────────────┼──────────── 75 │ org.ops4j.pax.web.pax-web-extender-war │ 90 │ singleton │ [org.ops4j.pax.web.service.spi.model.events.WebApplicationEventListener] │ 1 75 │ org.ops4j.pax.web.pax-web-extender-war │ 109 │ bundle │ [org.osgi.service.http.HttpService, org.ops4j.pax.web.service.WebContainer] │ 1 77 │ org.ops4j.pax.web.pax-web-extender-whiteboard │ 109 │ bundle │ [org.osgi.service.http.HttpService, org.ops4j.pax.web.service.WebContainer] │ 1 110 │ org.ops4j.pax.web.samples.showcase │ 109 │ bundle │ [org.osgi.service.http.HttpService, org.ops4j.pax.web.service.WebContainer] │ 1 110 │ org.ops4j.pax.web.samples.showcase │ 111 │ singleton │ [org.osgi.service.http.runtime.HttpServiceRuntime] │ 0 110 │ org.ops4j.pax.web.samples.showcase │ 120 │ singleton │ [org.osgi.service.http.context.ServletContextHelper] │ 1 111 │ org.ops4j.pax.web.pax-web-karaf │ 109 │ bundle │ [org.osgi.service.http.HttpService, org.ops4j.pax.web.service.WebContainer] │ 0

See that the org.osgi.service.http.context.ServletContextHelper is now referenced only once - as the default OSGi context for /c2 context path.

Let's unregister the Whiteboard context

karaf@root()> sample:whiteboard -d -s 120 context >> Using context for bundle org.ops4j.pax.web.samples.showcase [110] >> Unregistering org.osgi.service.http.context.ServletContextHelper with 120 service.id and for org.ops4j.pax.web.samples.showcase [110]. >>>> Unregistered org.apache.felix.framework.ServiceRegistrationImpl@1ff977fd successfully. karaf@root()> web:meta --web-usage-count Registering Bundle: org.ops4j.pax.web.pax-web-runtime [71] Service ID: 109 Service Scope: bundle Usage Counts for bundles referencing the service: Bundle ID │ Symbolic Name │ Service ID │ Scope │ Service objectClass │ Usage Count ──────────┼───────────────────────────────────────────────┼────────────┼───────────┼─────────────────────────────────────────────────────────────────────────────┼──────────── 75 │ org.ops4j.pax.web.pax-web-extender-war │ 90 │ singleton │ [org.ops4j.pax.web.service.spi.model.events.WebApplicationEventListener] │ 1 75 │ org.ops4j.pax.web.pax-web-extender-war │ 109 │ bundle │ [org.osgi.service.http.HttpService, org.ops4j.pax.web.service.WebContainer] │ 1 77 │ org.ops4j.pax.web.pax-web-extender-whiteboard │ 109 │ bundle │ [org.osgi.service.http.HttpService, org.ops4j.pax.web.service.WebContainer] │ 1 110 │ org.ops4j.pax.web.samples.showcase │ 109 │ bundle │ [org.osgi.service.http.HttpService, org.ops4j.pax.web.service.WebContainer] │ 1 110 │ org.ops4j.pax.web.samples.showcase │ 111 │ singleton │ [org.osgi.service.http.runtime.HttpServiceRuntime] │ 0 111 │ org.ops4j.pax.web.pax-web-karaf │ 109 │ bundle │ [org.osgi.service.http.HttpService, org.ops4j.pax.web.service.WebContainer] │ 0 karaf@root()> context-list Bundle ID │ Symbolic Name │ Context Path │ Context Name │ Rank │ Service ID │ Type │ Scope │ Registration Properties ──────────┼───────────────────────────────────────────────┼──────────────┼──────────────┼──────┼────────────┼────────────┼─────────┼────────────────────────────────────────── 77 │ org.ops4j.pax.web.pax-web-extender-whiteboard │ / │ default │ 0 │ 0 │ Whiteboard │ static* │ osgi.http.whiteboard.context.name=default │ │ │ │ │ │ │ │ osgi.http.whiteboard.context.path=/ *) This context is using ServletContextHelper/HttpContext without resolving an org.osgi.framework.ServiceReference.

See - no more custom context and the usage counts show that there's only one extra reference for org.osgi.service.http.HttpService/org.ops4j.pax.web.service.WebContainer - kept by our showcase bundle.

The last reference should be gone when bundle 110 is stopped

karaf@root()> web:meta --web-usage-count Registering Bundle: org.ops4j.pax.web.pax-web-runtime [71] Service ID: 109 Service Scope: bundle Usage Counts for bundles referencing the service: Bundle ID │ Symbolic Name │ Service ID │ Scope │ Service objectClass │ Usage Count ──────────┼───────────────────────────────────────────────┼────────────┼───────────┼─────────────────────────────────────────────────────────────────────────────┼──────────── 75 │ org.ops4j.pax.web.pax-web-extender-war │ 90 │ singleton │ [org.ops4j.pax.web.service.spi.model.events.WebApplicationEventListener] │ 1 75 │ org.ops4j.pax.web.pax-web-extender-war │ 109 │ bundle │ [org.osgi.service.http.HttpService, org.ops4j.pax.web.service.WebContainer] │ 1 77 │ org.ops4j.pax.web.pax-web-extender-whiteboard │ 109 │ bundle │ [org.osgi.service.http.HttpService, org.ops4j.pax.web.service.WebContainer] │ 1 111 │ org.ops4j.pax.web.pax-web-karaf │ 109 │ bundle │ [org.osgi.service.http.HttpService, org.ops4j.pax.web.service.WebContainer] │ 0
Everything is nicely cleaned up!
Summary

I know I should've written much more about Pax Web 8, but the day will eventually come...

Autor: ggrzybek No comments:

2016-10-18

Using Maven with OSGi Part 3

Introduction

In this installment I'll try to explain concepts behind SNAPSHOT artifact handling in Maven and Aether. How this is done in pax-url-aether will be described in next part of the series.

SNAPSHOT concept seems simple, it may however be very confusing at the same time - especially if multiple repositories are involved or Aether is used as part of wider solution - like pax-url-aether or Apache Karaf.

Ideally we may start with these definitions:

non-SNAPSHOT version (like e.g., 1.2.0.RELEASE) never changes - each copy of an artifact with non-SNAPSHOT version should be exactly the same, no matter what repository does it come from

SNAPSHOT version (like e.g., 1.2.0.BUILD-SNAPSHOT) may always change, so there's a need to verify if we may get newer build of the artifact.

This is usually obvious when dealing with 3rd party libraries and general rule is:

Do not use SNAPSHOT dependencies for 3rd party libraries when releasing a product containing such library

This general rule doesn't apply during development - before releasing a product, some 3rd party (or internal) dependency libraries may use SNAPSHOT versions. I this case it's helpful to know when exactly SNAPSHOT versions are updated, when Maven/Aether checks for new SNAPSHOT build and when it's actually downloaded and written as current SNAPSHOT version inside Maven's local repository.

I'd like to provide detailed information on the subject in this part of Maven in OSGi series.

Metadata

The appealing idea behind SNAPSHOTs is to make easier to just depend on latest/current version of dependency without specifying the exact version number.

To be able to achieve such goal, Maven uses the concept of metadata. Instead of referring to external documentation, I'll present the metadata concept by example.

Let's start with simplest Maven project, without any dependencies, using SNAPSHOT version.

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>grgr</groupId> <artifactId>universalis-api</artifactId> <version>0.1.0.BUILD-SNAPSHOT</version> </project>

This POM declares exactly three things:

groupId = grgr

artifactId = universalis-api

version = 0.1.0.BUILD-SNAPSHOT

There's no need to have any code inside such project. Let's see what happens when we simply install such project in local repository by invoking mvn clean install.

$ mvn clean install [INFO] Scanning for projects... ... [INFO] --- maven-install-plugin:2.4:install (default-install) @ universalis-api --- [INFO] Installing /data/ggrzybek/sources/_testing/grgr-universalis-api/target/universalis-api-0.1.0.BUILD-SNAPSHOT.jar ↵ to /home/ggrzybek/.m2/repository/grgr/universalis-api/0.1.0.BUILD-SNAPSHOT/universalis-api-0.1.0.BUILD-SNAPSHOT.jar [INFO] Installing /data/ggrzybek/sources/_testing/grgr-universalis-api/pom.xml ↵ to /home/ggrzybek/.m2/repository/grgr/universalis-api/0.1.0.BUILD-SNAPSHOT/universalis-api-0.1.0.BUILD-SNAPSHOT.pom ...

I've extracted the information about maven-install-plugin execution, but in fact, the two above artifacts are not the only files that are stored in ~/.m2/repository/grgr/universalis-api directory. Remaining artifacts are metadata files. Before describing them, we'll mvn clean install again to see what has changed (git init; git add .; git commit trick inside ~/.m2/repository/grgr/universalis-api).

~/.m2/repository/grgr/universalis-api/0.1.0.BUILD-SNAPSHOT/_remote.repositories

This is Aether-specific file (we've described it shortly in first part of the series). It contains:

#NOTE: This is an Aether internal implementation file, its format can be changed without prior notice. #Tue Oct 18 09:29:14 CEST 2016 universalis-api-0.1.0.BUILD-SNAPSHOT.pom>= universalis-api-0.1.0.BUILD-SNAPSHOT.jar>=

This is *.properties file with two properties. Only the keys are relevant, values are empty. By looking at this file, we can't say much, but after looking at other _remote.repositories files in our ~/.m2/repository, we can say:

It's written in org.eclipse.aether.internal.impl.TrackingFileManager#update()

Keys are constructed with: file name + > + repository ID

Most _remote.repositories files in ~/.m2/repository contain xyz.jar>central= property, because central repository is used by default in Maven.

This file is one of the most important sources of confusion when we operate in multi repository environment. I'm pretty sure we've all seen the following:
[ERROR] Failed to execute goal on project universalis-api: Could not resolve dependencies for project grgr:universalis-api:jar:0.1.0.BUILD-SNAPSHOT: ↵ Failure to find commons-pool:commons-pool:jar:1.6.0.redhat-9 in https://repo.maven.apache.org/maven2 was cached in the local repository, ↵ resolution will not be reattempted until the update interval of central has elapsed or updates are forced -> [Help 1]
The version shown is arbitrary, but the problem is that Aether, when using org.eclipse.aether.internal.impl.EnhancedLocalRepositoryManager, not only checks if artifact is available locally - it also checks if it's downloaded from one of the repositories that are currently configured (in current Maven build or Aether session).
The above error occurs, because we don't use the same repository (that we've downloaded the artifact from originally) in current build. Maven, by default uses EnhancedLocalRepositoryManager, pax-url-aether uses SimpleLocalRepositoryManagerFactory and doesn't have this problem.

Due to the above, it's important to carefully identify remote repositories used in parent POMs. To not end up with this:
$ find ~/.m2/repository -name _remote.repositories|xargs grep '>..'|cut -d '>' -f 2|sort|uniq ... apache-snapshots= apache.snapshots= ... ea= EA= ... jboss.public= jboss-public-repository= jboss-public-repository-group= ... ops4j.snapshot= ops4j-snapshots= ... sonatype= sonatype-nexus-snapshots= sonatype-snapshots= ...

~/.m2/repository/grgr/universalis-api/0.1.0.BUILD-SNAPSHOT/maven-metadata-local.xml

This file describes information about particular groupId:artifactId:version combination. For non-SNAPSHOT versions, we don't have this file created - there are simply no additional builds within single GAV where version is non-SNAPSHOT.

For SNAPSHOT versions however this file is used to list all different SNAPSHOT builds.

Here's the file after initial mvn clean install:

<?xml version="1.0" encoding="UTF-8"?> <metadata modelVersion="1.1.0"> <groupId>grgr</groupId> <artifactId>universalis-api</artifactId> <version>0.1.0.BUILD-SNAPSHOT</version> <versioning> <snapshot> <localCopy>true</localCopy> </snapshot> <lastUpdated>20161018072914</lastUpdated> <snapshotVersions> <snapshotVersion> <extension>jar</extension> <value>0.1.0.BUILD-SNAPSHOT</value> <updated>20161018072914</updated> </snapshotVersion> <snapshotVersion> <extension>pom</extension> <value>0.1.0.BUILD-SNAPSHOT</value> <updated>20161018072914</updated> </snapshotVersion> </snapshotVersions> </versioning> </metadata>

And here's the diff after 2nd mvn clean install:

--- a/0.1.0.BUILD-SNAPSHOT/maven-metadata-local.xml +++ b/0.1.0.BUILD-SNAPSHOT/maven-metadata-local.xml @@ -7,17 +7,17 @@ <snapshot> <localCopy>true</localCopy> </snapshot> - <lastUpdated>20161018072914</lastUpdated> + <lastUpdated>20161018075358</lastUpdated> <snapshotVersions> <snapshotVersion> <extension>jar</extension> <value>0.1.0.BUILD-SNAPSHOT</value> - <updated>20161018072914</updated> + <updated>20161018075358</updated> </snapshotVersion> <snapshotVersion> <extension>pom</extension> <value>0.1.0.BUILD-SNAPSHOT</value> - <updated>20161018072914</updated> + <updated>20161018075358</updated> </snapshotVersion> </snapshotVersions> </versioning>

Here's summary of important info about this file:

There's only one build tracked. Each installed artifact (like *.pom and *.jar) has its <snapshotVersion>.

versioning/snapshot/localCopy = true is an indication of local origin of the artifact.

There are timestamps for each artifact (and the metadata itself)

~/.m2/repository/grgr/universalis-api/maven-metadata-local.xml

This file describes information about particular groupId:artifactId across different versions. After several mvn clean install invocations for single SNAPSHOT versions, this file has only timestamp updated and it's neither particularly interesting nor complex:

<?xml version="1.0" encoding="UTF-8"?> <metadata> <groupId>grgr</groupId> <artifactId>universalis-api</artifactId> <versioning> <versions> <version>0.1.0.BUILD-SNAPSHOT</version> </versions> <lastUpdated>20161018075358</lastUpdated> </versioning> </metadata>

It becomes however more interesting after our simple project evolves and other versions are installed. After we mvn clean install versions 0.1.0, 0.1.1, 0.2.0, 0.1.2 (in that order), we can see:

<?xml version="1.0" encoding="UTF-8"?> <metadata> <groupId>grgr</groupId> <artifactId>universalis-api</artifactId> <versioning> <release>0.1.2</release> <versions> <version>0.1.0.BUILD-SNAPSHOT</version> <version>0.1.0</version> <version>0.1.1</version> <version>0.2.0</version> <version>0.1.2</version> </versions> <lastUpdated>20161018085259</lastUpdated> </versioning> </metadata>

It's a kind of index of versions for particular combination of groupId:artifactId

versioning/release may not exactly point to latest version (more about LATEST … later).

Low level, canonical example

All right. Having the metadata background in mind, let's actually try to resolve this grgr:universalis-api:0.1.0.BUILD-SNAPSHOT artifact using Aether code.

(I've reverted back to the state where we had only 0.1.0.BUILD-SNAPSHOT available in ~/.m2/repository/grgr/universalis-api).

DefaultServiceLocator locator = MavenRepositorySystemUtils.newServiceLocator(); locator.setService(LocalRepositoryManagerFactory.class, SimpleLocalRepositoryManagerFactory.class); locator.setService(org.eclipse.aether.spi.log.LoggerFactory.class, Slf4jLoggerFactory.class); RepositorySystem system = locator.getService(RepositorySystem.class); RepositorySystemSession session = MavenRepositorySystemUtils.newSession(); LocalRepositoryManager lrm = system.newLocalRepositoryManager(session, new LocalRepository("/home/ggrzybek/.m2/repository")); ((DefaultRepositorySystemSession)session).setLocalRepositoryManager(lrm); ArtifactRequest req = new ArtifactRequest(); req.setArtifact(new DefaultArtifact("grgr", "universalis-api", "jar", "0.1.0.BUILD-SNAPSHOT")); ArtifactResult res = system.resolveArtifact(session, req);

Nothing strange here. We're resolving an artifact without any reference to remote repositories - we expect to have SNAPSHOT artifact in local repository.

What Aether does internally?

org.eclipse.aether.resolution.VersionRequest is executed in version resolver. This request concerns grgr:universalis-api:jar:0.1.0.BUILD-SNAPSHOT artifact. The result of version resolution may be a change of version in original ArtifactRequest - usually after checking that remote SNAPSHOT metadata contains newer build than we already have.

If aether.versionResolver.noCache config option is not true, Aether's session cache is consulted (this is more important in such cases as full Maven build).

If version is not a release version (i.e., it is equal to RELEASE, LATEST or ends with SNAPSHOT), then Aether executes another request (or collection of such requests) - org.eclipse.aether.resolution.MetadataRequest.

MetadataRequest is prepared for local repository and for each remote repository added to ArtifactRequest in our code (we've added none).

Metadata is resolved by org.eclipse.aether.impl.MetadataResolver

Only one MetadataRequest - for local repository - is executed. This is handled by org.eclipse.aether.internal.impl.SimpleLocalRepositoryManager#find() method.

For local repository, grgr/universalis-api/0.1.0.BUILD-SNAPSHOT/maven-metadata-local.xml file is examined. If we had remote repositories added to initial ArtifactRequest, grgr/universalis-api/0.1.0.BUILD-SNAPSHOT/maven-metadata-central.xml (for repository with ID=central) would have been checked as well.

For each metadata found (for local and remote repositories. In our case - only for local one), versions are read. Versions correspond to /metadata/versioning XPath element from maven-metadata-<ID>.xml file.

Versions from each examined metadata file are merged to find latest version (based on timestamp from metadata). Latest version found from all MetadataResults is used as version in VersionResult

If aether.versionResolver.noCache config option is not true, Aether's session cache is updated.

Version from VersionResult is set as the version of Artifact inside our ArtifactRequest. In our case nothing has changed - 0.1.0.BUILD-SNAPSHOT was left untouched. We'll see how it may change when dealing with LATEST version and with remote metadata later.

org.eclipse.aether.internal.impl.SimpleLocalRepositoryManager#find() is used again - this time to find locally available grgr:universalis-api:jar:0.1.0.BUILD-SNAPSHOT artifact. Simple java.io.File#isFile() call is enough to determine whether the artifact is locally available. In case of EnhancedLocalRepositoryManager, _remote.repositories file would be examined as well.

ArtifactResult is returned and its artifact.file points to locally available artifact.

Here's important information about the above process:

We didn't mention about policies, but it's important to be aware that in case of local repositories, no policies are enforced. Metadata about SNAPSHOTs is read unconditionally.

No version translation happens here. It'd happen if we had remote repositories. (more about this later).

RELEASE and LATEST versions

To explain these concepts, let's mvn clean install some more versions of grgr:universalis-api. In order: 0.1.0, 0.1.1 and 0.2.0.BUILD-SNAPSHOT

Here's current grgr/universalis-api/maven-metadata-local.xml:

<?xml version="1.0" encoding="UTF-8"?> <metadata> <groupId>grgr</groupId> <artifactId>universalis-api</artifactId> <versioning> <release>0.1.1</release> <versions> <version>0.1.0.BUILD-SNAPSHOT</version> <version>0.1.0</version> <version>0.1.1</version> <version>0.2.0.BUILD-SNAPSHOT</version> </versions> <lastUpdated>20161018102447</lastUpdated> </versioning> </metadata>

We can try resolving grgr:universalis-api:RELEASE artifact:

ArtifactRequest req = new ArtifactRequest(); req.setArtifact(new DefaultArtifact("grgr", "universalis-api", "jar", "RELEASE")); ArtifactResult res = system.resolveArtifact(session, req);

org.eclipse.aether.internal.impl.SimpleLocalRepositoryManager#find() is used by metadata resolver to fetch ~/.m2/repository/grgr/universalis-api/maven-metadata-local.xml file - which is metadata per groupId:artifactId - for all versions.

/metadata/versioning/release and /metadata/versioning/latest are taken into account (in that order)

ArtifactResult refers to 0.1.1 version after resolution

We can also try resolving grgr:universalis-api:LATEST artifact:

ArtifactRequest req = new ArtifactRequest(); req.setArtifact(new DefaultArtifact("grgr", "universalis-api", "jar", "LATEST")); ArtifactResult res = system.resolveArtifact(session, req);

~/.m2/repository/grgr/universalis-api/maven-metadata-local.xml file is read as above

/metadata/versioning/release and /metadata/versioning/latest are taken into account (in that order)

ArtifactResult refers to 0.1.1 version after resolution...

Hey - what's the difference then between RELEASE and LATEST? None - at least when only local repositories are involved.

What's more, if we mvn clean install versions 0.2.0 and 0.1.2 (in that order), resolving grgr:universalis-api:LATEST (or RELEASE) would result in … version 0.1.2 instead of 0.2.0.

Remote repositories

Let's start fresh and this time use remote repository. We can use Sonatype Nexus as our remote repository manager (even if installed locally). We'll use mvn clean deploy after correct configuration of server credentials.

Our POM contains now:

<distributionManagement> <repository> <id>local-nexus</id> <url>http://localhost:8081/nexus/content/repositories/releases</url> </repository> <snapshotRepository> <id>local-nexus</id> <url>http://localhost:8081/nexus/content/repositories/snapshots</url> </snapshotRepository> </distributionManagement>

Deploying a project:

$ mvn clean deploy [INFO] Scanning for projects... ... [INFO] --- maven-install-plugin:2.4:install (default-install) @ universalis-api --- [INFO] Installing /data/ggrzybek/sources/_testing/grgr-universalis-api/target/universalis-api-0.1.0.BUILD-SNAPSHOT.jar ↵ to /home/ggrzybek/.m2/repository/grgr/universalis-api/0.1.0.BUILD-SNAPSHOT/universalis-api-0.1.0.BUILD-SNAPSHOT.jar [INFO] Installing /data/ggrzybek/sources/_testing/grgr-universalis-api/pom.xml ↵ to /home/ggrzybek/.m2/repository/grgr/universalis-api/0.1.0.BUILD-SNAPSHOT/universalis-api-0.1.0.BUILD-SNAPSHOT.pom ... [INFO] --- maven-deploy-plugin:2.7:deploy (default-deploy) @ universalis-api --- Downloading: http://localhost:8081/nexus/content/repositories/snapshots/grgr/universalis-api/0.1.0.BUILD-SNAPSHOT/maven-metadata.xml Uploading: http://localhost:8081/nexus/content/repositories/snapshots/grgr/universalis-api/0.1.0.BUILD-SNAPSHOT/universalis-api-0.1.0.BUILD-20161018.111155-1.jar Uploaded: http://localhost:8081/nexus/content/repositories/snapshots/grgr/universalis-api/0.1.0.BUILD-SNAPSHOT/universalis-api-0.1.0.BUILD-20161018.111155-1.jar (3 KB at 12.8 KB/sec) Uploading: http://localhost:8081/nexus/content/repositories/snapshots/grgr/universalis-api/0.1.0.BUILD-SNAPSHOT/universalis-api-0.1.0.BUILD-20161018.111155-1.pom Uploaded: http://localhost:8081/nexus/content/repositories/snapshots/grgr/universalis-api/0.1.0.BUILD-SNAPSHOT/universalis-api-0.1.0.BUILD-20161018.111155-1.pom (1020 B at 15.8 KB/sec) Downloading: http://localhost:8081/nexus/content/repositories/snapshots/grgr/universalis-api/maven-metadata.xml Uploading: http://localhost:8081/nexus/content/repositories/snapshots/grgr/universalis-api/0.1.0.BUILD-SNAPSHOT/maven-metadata.xml Uploaded: http://localhost:8081/nexus/content/repositories/snapshots/grgr/universalis-api/0.1.0.BUILD-SNAPSHOT/maven-metadata.xml (787 B at 19.2 KB/sec) Uploading: http://localhost:8081/nexus/content/repositories/snapshots/grgr/universalis-api/maven-metadata.xml Uploaded: http://localhost:8081/nexus/content/repositories/snapshots/grgr/universalis-api/maven-metadata.xml (285 B at 9.0 KB/sec) ...

maven-deploy-plugin:

Downloads maven-metadata.xml for groupId:artifactId:version - if Nexus is empty, none is found

Uploads *.jar and *.pom artifacts with SNAPSHOT version changed to <timestamp>-<build number>

Downloads maven-metadata.xml for groupId:artifactId - if Nexus is empty, none is found

Uploads both groupId:artifactId and groupId:artifactId:version metadata - after changing them. This is important information - repository managers don't generate metadata - it's Maven that does it (after examining currently deployed metadata - by previous mvn deploy) and uploads changed version back to Nexus.

Example with remote repository

Let's run our example again - after removing local copy of SNAPSHOT. We'll try to resolve SNAPSHOT from remote repository.

DefaultServiceLocator locator = MavenRepositorySystemUtils.newServiceLocator(); locator.setService(RepositoryConnectorFactory.class, BasicRepositoryConnectorFactory.class); locator.addService(TransporterFactory.class, HttpTransporterFactory.class); locator.setService(LocalRepositoryManagerFactory.class, SimpleLocalRepositoryManagerFactory.class); locator.setService(org.eclipse.aether.spi.log.LoggerFactory.class, Slf4jLoggerFactory.class); RepositorySystem system = locator.getService(RepositorySystem.class); RepositorySystemSession session = MavenRepositorySystemUtils.newSession(); LocalRepositoryManager lrm = system.newLocalRepositoryManager(session, new LocalRepository("/home/ggrzybek/.m2/repository")); ((DefaultRepositorySystemSession)session).setLocalRepositoryManager(lrm); RemoteRepository.Builder b = new RemoteRepository.Builder("local-nexus", "default", "http://localhost:8081/nexus/content/repositories/snapshots"); RepositoryPolicy enabledPolicy = new RepositoryPolicy(true, RepositoryPolicy.UPDATE_POLICY_NEVER, RepositoryPolicy.CHECKSUM_POLICY_IGNORE); RepositoryPolicy disabledPolicy = new RepositoryPolicy(false, RepositoryPolicy.UPDATE_POLICY_NEVER, RepositoryPolicy.CHECKSUM_POLICY_IGNORE); b.setReleasePolicy(disabledPolicy); b.setSnapshotPolicy(enabledPolicy); ArtifactRequest req = new ArtifactRequest(); req.addRepository(b.build()); req.setArtifact(new DefaultArtifact("grgr", "universalis-api", "jar", "0.1.0.BUILD-SNAPSHOT")); ArtifactResult res = system.resolveArtifact(session, req); LOG.info("Result: " + res);

We've added one RemoteRepository to ArtifactRequest. This repository has two policies set:

disabled releases

enabled snapshots, ignoring checksum verification errors and no-update policy (more about this soon).

Assuming that there's currently no ~/.m2/repository/grgr/univrsalis-api directory, what are the additional operations performed by Aether? Let's trace the metadata resolution fragment:

MetadataRequest is prepared for local repository and for each remote repository - we have two such requests now.

MetadataRequest for local repository uses grgr/universalis-api/0.1.0.BUILD-SNAPSHOT/maven-metadata-local.xml path - there's no such file.

MetadataRequest for remote repository uses grgr/universalis-api/0.1.0.BUILD-SNAPSHOT/maven-metadata-local-nexus.xml path (local-nexus is ID of our remote repository in pom.xml) - there's no such file (yet). This request has favorLocalRepository set to true.

For each remote repository, org.eclipse.aether.impl.UpdateCheck is started.

org.eclipse.aether.impl.UpdateCheck#policy is set to never (as we've requested).

org.eclipse.aether.impl.UpdateCheckManager is invoked to verify if update is needed

touch file is checked (~/.m2/repository/grgr/universalis-api/0.1.0.BUILD-SNAPSHOT/resolver-status.properties) - there's no such file (yet)

Because we don't have any metadata file available locally, the check is marked as required. Even if we've specified "never" as update policy.

For each required check, org.eclipse.aether.internal.impl.DefaultMetadataResolver.ResolveTask is scheduled.

RepositoryConnector is used to download metadata. After downloading, we have first file: ~/.m2/repository/grgr/universalis-api/0.1.0.BUILD-SNAPSHOT/maven-metadata-local-nexus.xml:

<?xml version="1.0" encoding="UTF-8"?> <metadata modelVersion="1.1.0"> <groupId>grgr</groupId> <artifactId>universalis-api</artifactId> <version>0.1.0.BUILD-SNAPSHOT</version> <versioning> <snapshot> <timestamp>20161018.111155</timestamp> <buildNumber>1</buildNumber> </snapshot> <lastUpdated>20161018111155</lastUpdated> <snapshotVersions> <snapshotVersion> <extension>jar</extension> <value>0.1.0.BUILD-20161018.111155-1</value> <updated>20161018111155</updated> </snapshotVersion> <snapshotVersion> <extension>pom</extension> <value>0.1.0.BUILD-20161018.111155-1</value> <updated>20161018111155</updated> </snapshotVersion> </snapshotVersions> </versioning> </metadata>

touch file is updated. Aether writes the following properties to ~/.m2/repository/grgr/universalis-api/0.1.0.BUILD-SNAPSHOT/resolver-status.properties:

#NOTE: This is an Aether internal implementation file, its format can be changed without prior notice. #Tue Oct 18 14:12:00 CEST 2016 maven-metadata-local-nexus.xml.lastUpdated=1476792720594

For each metadata found (for local and remote repositories), versions are read. Versions correspond to /metadata/versioning XPath element from maven-metadata-<ID>.xml file.

Versions from each examined metadata file are merged to find latest version (based on timestamp from metadata). Latest version found from all MetadataResults is used as version in VersionResult

Our version result is 0.1.0.BUILD-20161018.111155-1 - from remote metadata.

This time such artifact is not available locally (yet).

Artifact is downloaded.

If aether.artifactResolver.snapshotNormalization is true (which is default), downloaded universalis-api-0.1.0.BUILD-20161018.111155-1.jar artifact is copied to universalis-api-0.1.0.BUILD-SNAPSHOT.jar as well.

Update policy

As we've seen, the key information required to resolve SNAPSHOT artifacts is contained in maven-metadata.xml. There are different scenarios now:

Someone else may have deployed newer version of SNAPSHOT artifact to Remote repository

We may have installed (mvn clean install) newer version of SNAPSHOT ourselves.

With SNAPSHOTs, the two questions are:

When (if at all) maven-metadata.xml is downloaded from remote repository?

When (if at all) actual artifact is downloaded from remote repository?

The answer is "it depends".

Let's assume that Nexus contains two builds of an artifact: 0.1.0.BUILD-20161018.111155-1 and 0.1.0.BUILD-20161018.124558-2:

We may have no local version of remote maven-metadata-local-nexus.xml.

We may have local version of remote maven-metadata-local-nexus.xml where 0.1.0.BUILD-20161018.111155-1 is the latest - older than remote.

We may have local version of remote maven-metadata-local-nexus.xml where 0.1.0.BUILD-20161018.124558-2 is the latest - up to date with remote.

We also may have local maven-metadata-local.xml describing locally mvn clean installed SNAPSHOTs.

Here are the rules for determining when remote maven-metadata.xml is downloaded:

Initial resolution request (ArtifactRequest) must have one or more remote repositories added.

Assuming we're resolving with remote repository having local-nexus ID, Aether will look for maven-metadata-local-nexus.xml file.

If there's no ~/.m2/repository/grgr/universalis-api/0.1.0.BUILD-SNAPSHOT/maven-metadata-local-nexus.xml file, it'll be downloaded.

If there's such file, its timestamp (java.io.File#lastModified()) is verified against effective update policy:

never - remote metadata won't be downloaded

interval:N - remote metadata will be downloaded if the timestamp is older than N minutes

daily - remote metadata will be downloaded if the timestamp is not created today (even if it was created a minute before midnight and it's 00:00:01 now).

always - remote metadata will be downloaded

That's all! These rules are quite straightforward and clean. There's however another resolution performed by Aether - to determine which version will actually be used.

org.eclipse.aether.impl.VersionResolver uses org.eclipse.aether.impl.MetadataResolver. Metadata is gathered using the above rules. In simplest case, when artifact resolution is performed with one remote repository, we have at most two metadata files:

~/.m2/repository/grgr/universalis-api/0.1.0.BUILD-SNAPSHOT/maven-metadata-local.xml - from mvn clean install, may not exist

~/.m2/repository/grgr/universalis-api/0.1.0.BUILD-SNAPSHOT/maven-metadata-local-nexus.xml - from remote repository, must exist, but may be not up to date with remote version

Now, Aether checks /metadata/snapshotVersions/snapshotVersion elements (case when at 11:11:55 artifact was deployed to nexus and at 12:10:33 we've invoked mvn clean install):

In maven-metadata-local.xml:
<snapshotVersions> <snapshotVersion> <extension>jar</extension> <value>0.1.0.BUILD-SNAPSHOT</value> <updated>20161018121033</updated> </snapshotVersion> <snapshotVersion> <extension>pom</extension> <value>0.1.0.BUILD-SNAPSHOT</value> <updated>20161018121033</updated> </snapshotVersion> </snapshotVersions>

In maven-metadata-local-nexus.xml:
<snapshotVersions> <snapshotVersion> <extension>jar</extension> <value>0.1.0.BUILD-20161018.111155-1</value> <updated>20161018111155</updated> </snapshotVersion> <snapshotVersion> <extension>pom</extension> <value>0.1.0.BUILD-20161018.111155-1</value> <updated>20161018111155</updated> </snapshotVersion> </snapshotVersions>

What Aether does is simply comparing snapshotVersion/updated timestamps, so org.eclipse.aether.impl.VersionResolver after examining all MetadataResults, returns latest version, which is 0.1.0.BUILD-SNAPSHOT. VersionResult has:

versionResult.version = 0.1.0.BUILD-SNAPSHOT

versionResult.repository = "/home/ggrzybek/.m2/repository (simple)" (LocalRepository)

Let's now force download of current remote metadata (using e.g., UPDATE_POLICY_ALWAYS), to get:
<snapshotVersions> <snapshotVersion> <extension>jar</extension> <value>0.1.0.BUILD-20161018.124558-2</value> <updated>20161018124558</updated> </snapshotVersion> <snapshotVersion> <extension>pom</extension> <value>0.1.0.BUILD-20161018.124558-2</value> <updated>20161018124558</updated> </snapshotVersion> </snapshotVersions>

Now, after examining all MetadataResults, VersionResult has:

versionResult.version = 0.1.0.BUILD-20161018.124558-2

versionResult.repository = "local-nexus (http://localhost:8081/nexus/content/repositories/snapshots, default, snapshots)" (RemoteRepository)

A quick summary now - so far, update policy was used to determine whether to download remote version of maven-metadata-local-nexus.xml and entire resolution process updates requested version inside ArtifactResult.artifact.version.

After examining all available metadata files, resolution process changed our initial GAV from grgr:universalis-api:jar:0.1.0.BUILD-SNAPSHOT to grgr:universalis-api:jar:0.1.0.BUILD-20161018.124558-2. Now usual resolution process continues:

local repository is checked if it contains grgr/universalis-api/0.1.0.BUILD-SNAPSHOT/universalis-api-0.1.0.BUILD-20161018.124558-2.jar - it doesn't (yet)

artifact is scheduled to be downloaded. It's written to universalis-api-0.1.0.BUILD-20161018.124558-2.jar, and copied (because aether.artifactResolver.snapshotNormalization is true) to universalis-api-0.1.0.BUILD-SNAPSHOT.jar.

Update policy is not used at all for actual artifacts! It's used only for metadata.

Summary

Whew. It was long material. Let's summarize SNAPSHOT handling with Aether:

If resolution is performed without remote repository, local SNAPSHOT artifact must be available.

If resolution is performed with remote repository, update policy is taken into account.

update policy is used to determine whether metadata from remote repositories is downloaded.

After possible download of metadata we have following maven-metadata-<ID>.xml files:

one for each remote repository used

possibly one related to locally mvn clean installed SNAPSHOT

Using timestamps, newest SNAPSHOT version is passed for further artifact resolution

if latest SNAPSHOT comes from one of remote metadata files, SNAPSHOT is translated to <timestamp>-<build number>.

if latest SNAPSHOT comes from local metadata file, version is unchanged

The effect of version resolution and metadata resolution is possible change of version for pending artifact resolution.

If translated artifact coordinates (groupId and artifactId don't change, version may be changed) describe locally available artifact, it's returned.

If the artifact is not available, it's being downloaded from repository which was related to metadata file that declared latest SNAPSHOT.

Update policy is not used at all for artifact download.

In next installment I'll describe how update policies may be configured with pax-url-aether.

Autor: ggrzybek No comments:

Etykiety: aether, fuse, karaf, maven, osgi

2016-07-22

Using Maven with OSGi Part 2

Introduction

In previous installment I've described the foundation block of Maven - Eclipse Aether library. I've mentioned declarative usage of Maven using mvn: URI scheme.

In Karaf 4:

karaf@root()> bundle:install mvn:commons-io/commons-io/2.5 Bundle ID: 52
or in JBoss Fuse:
JBossFuse:karaf@root> osgi:install mvn:commons-io/commons-io/2.5 Bundle ID: 295

In this part I'll describe how Maven and Aether library are used in Karaf 4 and in JBoss Fuse 6.3.x (standalone mode). I'll leave JBoss Fuse fabric mode for next installment.

Let's recall some fundamental concepts:

local repository - accessed by Aether with the help of org.eclipse.aether.repository.LocalRepositoryManager interface and org.eclipse.aether.repository.LocalRepository class. Effectively local repository is a wrapper for locally accessible filesystem directory that follows specific structure (organization of Maven artifacts).

remote repository - accessed by Aether with the help of org.eclipse.aether.repository.RemoteRepository interface. Effectively remote repository is a wrapper for URI, a set of policies related to snapshot/release versions plus proxy, mirroring and authentication information.

pax-url-aether

The above commands rely on mvn: protocol implementation of java.net.URLStreamHandler interface. This implementation is provided by OPS4J PAX URL Aether library and uses Eclipse Aether to handle Maven artifact resolution.

When pax-url-aether bundle is installed and active in OSGi framework, it provides OSGi service with org.ops4j.pax.url.mvn.MavenResolver interface.

In Karaf 4:

karaf@root()> bundle:services -p 4 OPS4J Pax Url - aether: (4) provides: ------------------------------------- objectClass = [org.osgi.service.cm.ManagedService] service.bundleid = 4 service.id = 9 service.pid = org.ops4j.pax.url.mvn service.scope = singleton ---- objectClass = [org.osgi.service.url.URLStreamHandlerService] service.bundleid = 4 service.id = 10 service.scope = singleton url.handler.protocol = mvn ---- objectClass = [org.ops4j.pax.url.mvn.MavenResolver] service.bundleid = 4 service.id = 19 service.scope = singleton

In JBoss Fuse:

JBossFuse:karaf@root> ls 4 You are about to access system bundle 4. Do you wish to continue (yes/no): yes OPS4J Pax Url - aether: (4) provides: ------------------------------------- objectClass = [org.osgi.service.cm.ManagedService] service.id = 8 service.pid = org.ops4j.pax.url.mvn ---- objectClass = [org.osgi.service.url.URLStreamHandlerService] service.id = 9 url.handler.protocol = mvn ---- objectClass = [org.ops4j.pax.url.mvn.MavenResolver] service.id = 21

Configuration

This time I won't show any code examples. We don't need them. As with most of other OSGi services we can configure how pax-url-aether uses Aether with Configuration Admin service. Describing Configuration Admin is a good task for another blog post. There are some articles about Configuration Admin and if something's not clear, please refer to original documentation.

To configure org.ops4j.pax.url.mvn.MavenResolver service (the implementation is org.ops4j.pax.url.mvn.internal.AetherBasedResolver) we use org.ops4j.pax.url.mvn PID (persistent identifier). We can configure this PID manually using Configuration Admin API or using etc/org.ops4j.pax.url.mvn.cfg file.

Internally, pax-url-aether uses org.ops4j.pax.url.mvn.internal.config.MavenConfigurationImpl object. It contains information used by org.ops4j.pax.url.mvn.internal.AetherBasedResolver.

Let's see what's the default configuration (sightly formatted) provided by Karaf 4 (/work directory is a docker mount volume):

karaf@root()> property-list --pid org.ops4j.pax.url.mvn felix.fileinstall.filename = file:/work/etc/org.ops4j.pax.url.mvn.cfg org.ops4j.pax.url.mvn.defaultRepositories = \ file:/work/system@id=system.repository@snapshots, \ file:/work/data/kar@id=kar.repository@multi@snapshots, \ file:/work/system@id=child.system.repository@snapshots org.ops4j.pax.url.mvn.repositories = \ http://repo1.maven.org/maven2@id=central, \ http://repository.springsource.com/maven/bundles/release@id=spring.ebr.release, \ http://repository.springsource.com/maven/bundles/external@id=spring.ebr.external, \ http://zodiac.springsource.com/maven/bundles/release@id=gemini, \ http://repository.apache.org/content/groups/snapshots-group@id=apache@snapshots@noreleases, \ https://oss.sonatype.org/content/repositories/snapshots@id=sonatype.snapshots.deploy@snapshots@noreleases, \ https://oss.sonatype.org/content/repositories/ops4j-snapshots@id=ops4j.sonatype.snapshots.deploy@snapshots@noreleases, \ http://repository.springsource.com/maven/bundles/external@id=spring-ebr-repository@snapshots@noreleases org.ops4j.pax.url.mvn.useFallbackRepositories = false service.pid = org.ops4j.pax.url.mvn

And JBoss Fuse:

JBossFuse:karaf@root> config:proplist --pid org.ops4j.pax.url.mvn felix.fileinstall.filename = file:/data/servers/jboss-fuse-6.3.0.redhat-145/etc/org.ops4j.pax.url.mvn.cfg org.ops4j.pax.url.mvn.defaultRepositories = \ file:/data/servers/jboss-fuse-6.3.0.redhat-145/system@snapshots@id=karaf.system,\ file:/home/ggrzybek/.m2/repository@snapshots@id=local,\ file:/data/servers/jboss-fuse-6.3.0.redhat-145/local-repo@snapshots@id=karaf.local-repo,\ file:/data/servers/jboss-fuse-6.3.0.redhat-145/system@snapshots@id=child.karaf.system org.ops4j.pax.url.mvn.globalChecksumPolicy = warn org.ops4j.pax.url.mvn.globalUpdatePolicy = daily org.ops4j.pax.url.mvn.localRepository = /data/servers/jboss-fuse-6.3.0.redhat-145/data/repository org.ops4j.pax.url.mvn.repositories = \ http://repo1.maven.org/maven2@id=maven.central.repo, \ https://maven.repository.redhat.com/ga@id=redhat.ga.repo, \ https://maven.repository.redhat.com/earlyaccess/all@id=redhat.ea.repo, \ https://repository.jboss.org/nexus/content/groups/ea@id=fuseearlyaccess org.ops4j.pax.url.mvn.settings = /data/servers/jboss-fuse-6.3.0.redhat-145/etc/maven-settings.xml org.ops4j.pax.url.mvn.useFallbackRepositories = false service.pid = org.ops4j.pax.url.mvn

The above configurations differ a bit. JBoss Fuse provides more explicit configuration. Let's describe each used (and assumed) properties that can be used to configure pax-url-aether (org.ops4j.pax.url.mvn.internal.AetherBasedResolver).

org.ops4j.pax.url.mvn.defaultRepositories

this is a list of local repositories searched for an artifact in the first phase of artifact resolution. This repository should not contain URIs other than file://-based. Each repository from this list is treated as local repository. pax-url-aether iterates over this list and checks one repository at a time. If neither location contains the artifact being resolved, pax-url-aether switches to second phase that involves remote repositories.
Also these repositories do not require write access - Aether doesn't write any files there.

Access to these repositories can be presented using the following code (did I promise not to show any code example? sorry...):

RepositorySystem system = locator.getService(RepositorySystem.class); RepositorySystemSession session = MavenRepositorySystemUtils.newSession(); String basedir = singleRepositoryFromListOfDefaultRepositories; ((DefaultRepositorySystemSession)session).setLocalRepositoryManager(system.newLocalRepositoryManager(session, new LocalRepository(basedir))); ArtifactRequest req = new ArtifactRequest(); req.setArtifact(new DefaultArtifact("commons-io", "commons-io", "jar", "2.5")); ArtifactResult res = system.resolveArtifact(session, req);

We don't invoke any req.addRepository(repositoryBuilder.build());, so Aether doesn't try to go to any external location.
Line 4 shows that we're trying one of repositories from org.ops4j.pax.url.mvn.defaultRepositories at a time. Each such local repository is checked independently.

org.ops4j.pax.url.mvn.repositories

this is a list of remote repositories searched for an artifact in the second phase of artifact resolution. This repository may contain URIs with file: scheme, but it's better to add such repositories to org.ops4j.pax.url.mvn.defaultRepositories. Each repository is accessed using some configured connector (pax-url-aether uses a connector that underneath invokes httpclient 4.x library - that's why configuring org.apache.http.headers logger may be a good idea)

org.ops4j.pax.url.mvn.localRepository

this is a local repository that supports Aether in second phase of artifact resolution. Its role is a bit different than the role of org.ops4j.pax.url.mvn.defaultRepositories. When Aether actually resolves artifact in one of the remote repositories, it stores the downloaded artifact to org.ops4j.pax.url.mvn.localRepository. That's why write access is required for this location.

If not specified, this property defaults to: ${user.home}/.m2/repository! Be aware of this if Aether seems to find artifacts that are not expected.

Second phase of artifact resolution can be presented using the following code (I love clean code!):

RepositorySystem system = locator.getService(RepositorySystem.class); RepositorySystemSession session = MavenRepositorySystemUtils.newSession(); String basedir = localRepository; ((DefaultRepositorySystemSession)session).setLocalRepositoryManager(system.newLocalRepositoryManager(session, new LocalRepository(basedir))); ArtifactRequest req = new ArtifactRequest(); req.addRepository(new RemoteRepository.Builder("ID1", "default", "http://uri1").build()); req.addRepository(new RemoteRepository.Builder("ID2", "default", "http://uri2").build()); req.setArtifact(new DefaultArtifact("commons-io", "commons-io", "jar", "2.5")); ArtifactResult res = system.resolveArtifact(session, req);

Here, for each remote repository from the list of URIs in org.ops4j.pax.url.mvn.repositories property we call org.eclipse.aether.resolution.ArtifactRequest.addRepository().

Also, in line 4 we use local repository from org.ops4j.pax.url.mvn.localRepository property - the same local repository is used for each remote repository being searched.

org.ops4j.pax.url.mvn.useFallbackRepositories

If true, then Aether will always use http://repo1.maven.org/maven2 repository in addition to any remote repositories specified. I prefer explicit declaration of Maven Central repository (if needed), so it's better to say false here.

org.ops4j.pax.url.mvn.settings

Ah, big topic here. We can specify an explicit location of an XML document following Maven Settings XML Schema.

If not specified, pax-url-aether searches for settings file in the following locations:

${user.home}/.m2/settings.xml

${maven.home}/conf/settings.xml

$M2_HOME/conf/settings.xml

In Karaf 4, implicit location is used (most probably ${user.home}/.m2/settings.xml). In JBoss Fuse, explicit ${karaf.etc}/maven-settings.xml value is configured and default, commented template is shipped.

Why specify custom settings.xml file, when we have properties such as org.ops4j.pax.url.mvn.repositories? There are few things that can be specified only there:

HTTP proxies

custom HTTP headers added when accessing particular remote repositories

Here's the example of HTTP proxy configuration:

 <proxies> <proxy> <id>proxy</id> <host>127.0.0.1</host> <port>3128</port> <protocol>http</protocol> <username></username> <password></password> <nonProxyHosts>127.0.0.*|*.repository.corp</nonProxyHosts> </proxy> </proxies>

And here's the example of specifying custom HTTP headers:

 <servers> <server> <id>maven.central.repo</id> <configuration> <httpHeaders> <httpHeader> <name>User-Agent</name> <value>Karaf</value> </httpHeader> <httpHeader> <name>Secret-Header</name> <value>secret_value</value> </httpHeader> </httpHeaders> </configuration> </server> </servers>

With custom headers specification, we can see these in logs when accessing repository with ID=maven.central.repo (see below for repository URI specification):

17:30:44,590 | DEBUG | ... | http-outgoing-0 >> GET /maven2/commons-io/commons-io/2.7/commons-io-2.7.jar HTTP/1.1 17:30:44,590 | DEBUG | ... | http-outgoing-0 >> Cache-control: no-cache 17:30:44,590 | DEBUG | ... | http-outgoing-0 >> Cache-store: no-store 17:30:44,590 | DEBUG | ... | http-outgoing-0 >> Pragma: no-cache 17:30:44,591 | DEBUG | ... | http-outgoing-0 >> Expires: 0 17:30:44,591 | DEBUG | ... | http-outgoing-0 >> Accept-Encoding: gzip 17:30:44,591 | DEBUG | ... | http-outgoing-0 >> User-Agent: Karaf 17:30:44,591 | DEBUG | ... | http-outgoing-0 >> Secret-Header: secret_value 17:30:44,591 | DEBUG | ... | http-outgoing-0 >> Host: repo1.maven.org

org.ops4j.pax.url.mvn.repositories - again

After describing org.ops4j.pax.url.mvn.settings let's get back for a moment to org.ops4j.pax.url.mvn.repositories.

If a list of remote repositories in org.ops4j.pax.url.mvn.repositories is prefixed with + sign, all repositories available in all active profiles defined in settings.xml file are appended to effective list of remote repositories searched.

For example if we have this:

org.ops4j.pax.url.mvn.repositories= \ +http://repo1.maven.org/maven2@id=maven.central.repo

And this in settings.xml:

 <profiles> <profile> <id>default</id> <repositories> <repository> <id>private.repository</id> <url>http://localhost:8181/maven-repository</url> </repository> </repositories> </profile> </profiles> <activeProfiles> <activeProfile>default</activeProfile> </activeProfiles>

We can see this in logs during sample resolution:

18:10:17,734 | DEBUG | ... | Using transporter WagonTransporter with priority -1.0 for http://repo1.maven.org/maven2/ 18:10:17,736 | DEBUG | ... | Using connector BasicRepositoryConnector with priority 0.0 for http://repo1.maven.org/maven2/ 18:10:17,800 | DEBUG | ... | http-outgoing-8 >> GET /maven2/commons-io/commons-io/2.7/commons-io-2.7.jar HTTP/1.1 18:10:17,802 | DEBUG | ... | http-outgoing-8 >> Host: repo1.maven.org ... 18:10:17,872 | DEBUG | ... | Using transporter WagonTransporter with priority -1.0 for http://localhost:8181/maven-repository/ 18:10:17,873 | DEBUG | ... | Using connector BasicRepositoryConnector with priority 0.0 for http://localhost:8181/maven-repository/ 18:10:17,875 | DEBUG | ... | http-outgoing-9 >> GET /maven-repository/commons-io/commons-io/2.7/commons-io-2.7.jar HTTP/1.1 18:10:17,876 | DEBUG | ... | http-outgoing-9 >> Host: localhost:8181 ...

org.ops4j.pax.url.mvn.globalChecksumPolicy

When Aether fetches artifact from remote repository, it always tries to download SHA1/MD5 checksum for the artifact. It may fail to do so. If repository URI doesn't specify per-repository value, this global property's value is used. Actually if this global value is specified, per-repository values are ignored.

This property may have 3 values determining Aether's behavior:

fail - resolution fails

warn - information is printed at WARN level

ignore - nothing happens.

Note that there's no way to prevent fetching checksums.

org.ops4j.pax.url.mvn.globalUpdatePolicy

When Aether fetches SNAPSHOT artifacts, it needs to fetch maven-metadata.xml first. Before hitting org.ops4j.pax.url.mvn.repositories, Aether checks the presence of resolver-status.properties file in org.ops4j.pax.url.mvn.localRepository location (this status file is specific to given groupId, artifactId and version, for example: <REPOSITORY>/commons-io/commons-io/2.5-SNAPSHOT/resolver-status.properties). We can control whether Aether actually should refresh metadata information:

always - Aether always fetches maven-metadata.xml when resolving SNAPSHOTs

never - opposite of the above

daily - Aether fetches maven-metadata.xml if a day passed since timestamp written in maven-metadata-ID_OF_REPOSITORY.xml.lastUpdated property inside resolver-status.properties file.

interval:<NUMBER_OF_MINUTES> - Aether fetches maven-metadata.xml if given number of minutes passed.

Maven Repository URI

I've mentioned repository URI in few places. When specifying URI on a org.ops4j.pax.url.mvn.repositories list, we may use the following format:

http(s)://host:port/path@snapshots@noreleases@id=ID@other_options

Options that may be specified are:

id=ID - this option may (should) be specified to identify a repository. We may then refer to the repository for example when specifying custom headers.

snapshots - whether the repository should be used when resolving SNAPSHOT artifacts

noreleases - whether the repository should not be used when resolving non-SNAPSHOT artifacts

releasesUpdate=daily|never|always|interval:MINUTES - see description of org.ops4j.pax.url.mvn.globalUpdatePolicy property

snapshotsUpdate=daily|never|always|interval:MINUTES - see description of org.ops4j.pax.url.mvn.globalUpdatePolicy property

update=daily|never|always|interval:MINUTES - see description of org.ops4j.pax.url.mvn.globalUpdatePolicy property

releasesChecksum=fail|warn|ignore - see description of org.ops4j.pax.url.mvn.globalChecksumPolicy property

snapshotsChecksum=fail|warn|ignore - see description of org.ops4j.pax.url.mvn.globalChecksumPolicy property

checksum=fail|warn|ignore - see description of org.ops4j.pax.url.mvn.globalChecksumPolicy property

Other options

There are other properties that can be configured in org.ops4j.pax.url.mvn PID:

org.ops4j.pax.url.mvn.defaultLocalRepoAsRemote

Whether local repository specified in org.ops4j.pax.url.mvn.localRepository should be added as first remote repository inserted to the list configured with org.ops4j.pax.url.mvn.repositories property - it's a bad idea...

Caveats

Due to highly asynchronous nature of OSGi™ (and in particular - a slight race condition between pax-url-aether that configures org.ops4j.pax.url.mvn.MavenResolver service on one side and felix.fileinstall and felix.configadmin bundles that create configuration for org.ops4j.pax.url.mvn PID on other side), there's short period of time where other configuration may be used in org.ops4j.pax.url.mvn.MavenResolver service.

To prevent such interregnum, I suggest duplicating Maven/Aether properties from ${karaf.etc}/org.ops4j.pax.url.mvn.cfg in ${karaf.etc}/config.properties. If pax-url-aether can't find ConfigurationAdmin (yet), it defaults to bundle properties and these may be specified in etc/config.properties.

Summary

I hope the above information will clear all confusion related to pax-url-aether configuration in OSGi framework as JBoss Fuse or Karaf.

Autor: ggrzybek No comments:

Etykiety: aether, fuse, karaf, maven, osgi

Using Maven with OSGi Part 1

Introduction

(If you're interested in pax-url-aether configuration in JBoss Fuse standalone mode or in Karaf, please visit Part 2 of the series.)

In this short series of articles I'd like to show how Maven can be used inside OSGi environment. Both Apache Karaf and JBoss Fuse use Maven extensively and it's important to understand how it really works to be able to use it successfully.

I personally think that learning internals of any technology is the best way to use and maintain it in the longer period. Ultimately getting the official source code and reading it in your favourite IDE is much better than relying on official (or unofficial) documentation.

Of course sometimes (usually) there's no time to dig through the internals, so I hope this article will provide an alternative.

Runtime

JBoss Fuse is a technology based on Apache Karaf runtime and from OSGi point of view there's no big difference. Both runtimes allow to install OSGi bundles and Karaf features that provide OSGi services of the runtime and user applications. JBoss Fuse provide even higher abstraction of profiles that group bundles, features and other items.

Apache Karaf and JBoss Fuse may reference Maven artifacts directly using mvn:groupId/artifactId/version[/type[/classifier]] URIs. These may reference bundles, features and other artifacts.

Following my usual way of learning, I'll start with low level details and continue with higher level mechanism and concepts.

Maven

Although there are good alternatives (like Gradle), Apache Maven is still de-facto standard tool for build and dependency management. When we decompose Maven tool into parts, we can reuse the dependency management part in our code. Dependency management is one of the most important aspect of software development and inside OSGi runtime, Maven dependencies are only one of the layer of dependency management. However we won't cover OSGi bundle and Karaf feature dependencies here.

We would like to fetch any artifact stored in one of external Maven repositories and use it (as bundle, feature or configuration file) inside the runtime. We'd like to do it the Maven-way, i.e., declaratively. The best example is installation of external Maven artifact inside OSGi runtime. Like Karaf:

karaf@root()> bundle:install mvn:commons-io/commons-io/2.5 Bundle ID: 52
or Fuse:
JBossFuse:karaf@root> osgi:install mvn:commons-io/commons-io/2.5 Bundle ID: 295

These commands work out of the box, but usually there's a need to change the default configuration, e.g., configure additional remote repositories, change credentials, configure HTTP proxies, etc. Before describing configuration options, let's start with the basics.

Aether

Eclipse Aether is a set of libraries used internally by Maven for dependency resolution. There are various tasks that can be performed using Aether, like finding a closure of artifacts for a graph of dependencies, but even with this low-level library we'll focus on one particular task - getting artifacts from remote repositories.

Official Aether Wiki page is sufficient to get started and see how to use it in code. I'll provide more detailed information in order to describe important concepts.

Aether uses an interface-based API where actual implementations of the interfaces are configured using CDI. There are two most important interfaces used:

org.eclipse.aether.RepositorySystem - an entry to repository system that provides various resolution methods

org.eclipse.aether.RepositorySystemSession - provide additional information specific to operations performed on RepositorySystem

and a set of classes:

org.eclipse.aether.*.*Request - various request classes passed as commands to RepositorySystem. We'll focus mainly on org.eclipse.aether.resolution.ArtifactRequest.

RepositorySystem is configured in dependency-injection style - we can select concrete implementations of several SPI interfaces that alter some aspects of Aether, while RepositorySystemSession is configured using properties and directly set objects. Session alters a way in which repository deals with requests.

So let's check how these work together. First let's configure the repository system:
DefaultServiceLocator locator = MavenRepositorySystemUtils.newServiceLocator(); locator.setService(RepositoryConnectorFactory.class, BasicRepositoryConnectorFactory.class); locator.setService(TransporterFactory.class, FileTransporterFactory.class); locator.setService(TransporterFactory.class, HttpTransporterFactory.class); locator.setService(org.eclipse.aether.spi.log.LoggerFactory.class, Slf4jLoggerFactory.class); RepositorySystem system = locator.getService(RepositorySystem.class);

Nothing extraordinary: we'll have access to http: and file: based repositories and SLF4J API will be used for logging.

Now let's configure session. The configuration property is arbitrary and more properties will be described later.
RepositorySystemSession session = MavenRepositorySystemUtils.newSession(); ((DefaultRepositorySystemSession)session).setConfigProperty("aether.connector.basic.threads", "2"); LocalRepositoryManager localRepositoryManager = system.newLocalRepositoryManager(session, new LocalRepository("/home/user/.m2/repository")); ((DefaultRepositorySystemSession)session).setLocalRepositoryManager(localRepositoryManager);

And finally let's perform some operation - artifact resolution:
ArtifactRequest req = new ArtifactRequest(); req.setArtifact(new DefaultArtifact("commons-io", "commons-io", "jar", "2.5")); req.addRepository(new RemoteRepository.Builder("central", "default", "http://repo1.maven.org/maven2").build()); ArtifactResult res = system.resolveArtifact(session, req);

The above tells Aether to resolve artifact commons-io:commons-io:jar:2.5 using local repository inside /home/user/.m2/repository and if it's not found there, to search for the artifact inside http://repo1.maven.org/maven2 remote repository. We could (and it's usual practice) configure more remote repositories (using org.eclipse.aether.resolution.ArtifactRequest#addRepository()) to be searched if artifact isn't available locally.

The code above isn't needed to use Maven inside Karaf or JBoss Fuse, but it brings two super important concepts:

local repository - accessed by Aether with the help of org.eclipse.aether.repository.LocalRepositoryManager interface and org.eclipse.aether.repository.LocalRepository class. Effectively local repository is a wrapper for locally accessible filesystem directory that follows specific structure (organization of Maven artifacts).

remote repository - accessed by Aether with the help of org.eclipse.aether.repository.RemoteRepository interface. Effectively remote repository is a wrapper for URI, a set of policies related to snapshot/release versions plus proxy, mirroring and authentication information.

The key point is that if an artifact can't be found in local repository it is being searched for in (one of the) remote repositories. Proper code should ensure that local repositories are always searched before remote repositories.

Logs

For debugging purposes it is very helpful to see all the operations in logs. We can increase logging level for few loggers:
log4j.logger.org.eclipse.aether = DEBUG log4j.logger.org.apache.http.headers = DEBUG

Also, we'll add another remote repository to see how Aether checks them all:
req.addRepository(new RemoteRepository.Builder("jboss-public", "default", "https://repository.jboss.org/nexus/content/groups/public").build()); req.addRepository(new RemoteRepository.Builder("central", "default", "http://repo1.maven.org/maven2").build());

Here are the logs when commons-io:commons-io:2.5:jar artifact is resolved and it is not available in local repository:
11:13:47.181 DEBUG {main} [o.e.a.i.i.DefaultLocalRepositoryProvider] : Using manager EnhancedLocalRepositoryManager with priority 10.0 for target/repo-1469178827169 11:13:47.188 INFO {main} [g.t.m.a.AetherTest] : Request: commons-io:commons-io:jar:2.5 < [jboss-public (https://repository.jboss.org/nexus/content/groups/public, default, releases+snapshots), central (http://repo1.maven.org/maven2, default, releases+snapshots)] 11:13:47.631 DEBUG {main} [o.e.a.i.i.DefaultTransporterProvider] : Using transporter HttpTransporter with priority 5.0 for https://repository.jboss.org/nexus/content/groups/public 11:13:47.632 DEBUG {main} [o.e.a.i.i.DefaultRepositoryConnectorProvider] : Using connector BasicRepositoryConnector with priority 0.0 for https://repository.jboss.org/nexus/content/groups/public 11:13:49.015 DEBUG {main} [o.a.h.headers] : >> GET /nexus/content/groups/public/commons-io/commons-io/2.5/commons-io-2.5.jar HTTP/1.1 11:13:49.015 DEBUG {main} [o.a.h.headers] : >> Host: repository.jboss.org ... 11:13:49.385 DEBUG {main} [o.a.h.headers] : << HTTP/1.1 404 Not Found ... 11:13:49.572 DEBUG {main} [o.e.a.i.i.DefaultTransporterProvider] : Using transporter HttpTransporter with priority 5.0 for http://repo1.maven.org/maven2 11:13:49.572 DEBUG {main} [o.e.a.i.i.DefaultRepositoryConnectorProvider] : Using connector BasicRepositoryConnector with priority 0.0 for http://repo1.maven.org/maven2 11:13:49.704 DEBUG {main} [o.a.h.headers] : >> GET /maven2/commons-io/commons-io/2.5/commons-io-2.5.jar HTTP/1.1 11:13:49.705 DEBUG {main} [o.a.h.headers] : >> Host: repo1.maven.org ... 11:13:49.770 DEBUG {main} [o.a.h.headers] : << HTTP/1.1 200 OK ... 11:13:50.079 DEBUG {main} [o.a.h.headers] : >> GET /maven2/commons-io/commons-io/2.5/commons-io-2.5.jar.sha1 HTTP/1.1 11:13:50.079 DEBUG {main} [o.a.h.headers] : >> Host: repo1.maven.org ... 11:13:50.145 DEBUG {main} [o.a.h.headers] : << HTTP/1.1 200 OK ... 11:13:50.156 DEBUG {main} [o.e.a.i.i.EnhancedLocalRepositoryManager] : Writing tracking file /data/ggrzybek/sources/_testing/grgr-test-maven/target/repo-1469178827169/commons-io/commons-io/2.5/_remote.repositories 11:13:50.161 INFO {main} [g.t.m.a.AetherTest] : Result: commons-io:commons-io:jar:2.5 < central (http://repo1.maven.org/maven2, default, releases+snapshots)

As we can see here's the sequence of events:

Aether uses local repository at target/repo-1469178827169 location

https://repository.jboss.org/nexus/content/groups/public is checked first and we get HTTP 404

http://repo1.maven.org/maven2 is checked next and we get HTTP 200

Aether fetches SHA1 checksum then for found artifact

Aether writes tracking file at target/repo-1469178827169/commons-io/commons-io/2.5/_remote.repositories that looks like this:
#NOTE: This is an Aether internal implementation file, its format can be changed without prior notice. #Fri Jul 22 11:13:50 CEST 2016 commons-io-2.5.jar>central=
This file allows us to recall where the artifact was downloaded from.

SNAPSHOTs

Let's see how Aether works when resolving SNAPSHOT versions. We'll reuse the same remote repositories as before. By default using new RemoteRepository.Builder("central", "default", "http://repo1.maven.org/maven2").build() gives us remote repository that's enabled regardless of whether we use the repository to resolve SNAPSHOT or non-SNAPSHOT artifacts. We can of course change it:
RemoteRepository.Builder b1 = new RemoteRepository.Builder("central", "default", "http://repo1.maven.org/maven2"); RemoteRepository.Builder b2 = new RemoteRepository.Builder("jboss-public", "default", "https://repository.jboss.org/nexus/content/groups/public"); RepositoryPolicy enabledPolicy = new RepositoryPolicy(true, RepositoryPolicy.UPDATE_POLICY_ALWAYS, RepositoryPolicy.CHECKSUM_POLICY_FAIL); RepositoryPolicy disabledPolicy = new RepositoryPolicy(false, RepositoryPolicy.UPDATE_POLICY_ALWAYS, RepositoryPolicy.CHECKSUM_POLICY_FAIL); b1.setReleasePolicy(enabledPolicy); b1.setSnapshotPolicy(enabledPolicy); b2.setReleasePolicy(disabledPolicy); b2.setSnapshotPolicy(enabledPolicy); req.addRepository(b1.build()); req.addRepository(b2.build());

In the above example, we explicitly enable resolving SNAPSHOT artifacts in central and jboss-public repositories. We won't try to resolve non-SNAPSHOT artifacts in jboss-public. Here are the logs related to resolving commons-io:commons-io:2.5-SNAPSHOT:jar:
12:11:17.195 DEBUG {main} [o.e.a.i.i.DefaultLocalRepositoryProvider] : Using manager EnhancedLocalRepositoryManager with priority 10.0 for target/repo-1469182277187 12:11:17.201 INFO {main} [g.t.m.a.AetherTest] : Request: commons-io:commons-io:jar:2.5-SNAPSHOT < [central (http://repo1.maven.org/maven2, default, releases+snapshots), jboss-public (https://repository.jboss.org/nexus/content/groups/public, default, snapshots)] 12:11:17.851 DEBUG {DefaultMetadataResolver-0-1} [o.e.a.i.i.DefaultTransporterProvider] : Using transporter HttpTransporter with priority 5.0 for https://repository.jboss.org/nexus/content/groups/public 12:11:17.852 DEBUG {DefaultMetadataResolver-0-1} [o.e.a.i.i.DefaultRepositoryConnectorProvider] : Using connector BasicRepositoryConnector with priority 0.0 for https://repository.jboss.org/nexus/content/groups/public 12:11:17.853 DEBUG {DefaultMetadataResolver-0-0} [o.e.a.i.i.DefaultTransporterProvider] : Using transporter HttpTransporter with priority 5.0 for http://repo1.maven.org/maven2 12:11:17.854 DEBUG {DefaultMetadataResolver-0-0} [o.e.a.i.i.DefaultRepositoryConnectorProvider] : Using connector BasicRepositoryConnector with priority 0.0 for http://repo1.maven.org/maven2 12:11:18.158 DEBUG {DefaultMetadataResolver-0-0} [o.a.h.headers] : >> GET /maven2/commons-io/commons-io/2.5-SNAPSHOT/maven-metadata.xml HTTP/1.1 12:11:18.158 DEBUG {DefaultMetadataResolver-0-0} [o.a.h.headers] : >> Host: repo1.maven.org ... 12:11:18.225 DEBUG {DefaultMetadataResolver-0-0} [o.a.h.headers] : << HTTP/1.1 404 Not Found ... 12:11:18.245 DEBUG {DefaultMetadataResolver-0-0} [o.e.a.i.i.DefaultUpdateCheckManager] : Writing tracking file /data/ggrzybek/sources/_testing/grgr-test-maven/target/repo-1469182277187/commons-io/commons-io/2.5-SNAPSHOT/resolver-status.properties 12:11:19.332 DEBUG {DefaultMetadataResolver-0-1} [o.a.h.headers] : >> GET /nexus/content/groups/public/commons-io/commons-io/2.5-SNAPSHOT/maven-metadata.xml HTTP/1.1 12:11:19.332 DEBUG {DefaultMetadataResolver-0-1} [o.a.h.headers] : >> Host: repository.jboss.org ... 12:11:19.611 DEBUG {DefaultMetadataResolver-0-1} [o.a.h.headers] : << HTTP/1.1 200 OK ... 12:11:19.850 DEBUG {DefaultMetadataResolver-0-1} [o.a.h.headers] : >> GET /nexus/content/groups/public/commons-io/commons-io/2.5-SNAPSHOT/maven-metadata.xml.sha1 HTTP/1.1 12:11:19.850 DEBUG {DefaultMetadataResolver-0-1} [o.a.h.headers] : >> Host: repository.jboss.org ... 12:11:20.079 DEBUG {DefaultMetadataResolver-0-1} [o.a.h.headers] : << HTTP/1.1 200 OK ... 12:11:20.082 DEBUG {DefaultMetadataResolver-0-1} [o.e.a.i.i.DefaultUpdateCheckManager] : Writing tracking file /data/ggrzybek/sources/_testing/grgr-test-maven/target/repo-1469182277187/commons-io/commons-io/2.5-SNAPSHOT/resolver-status.properties 12:11:20.107 DEBUG {main} [o.e.a.i.i.DefaultTransporterProvider] : Using transporter HttpTransporter with priority 5.0 for https://repository.jboss.org/nexus/content/groups/public 12:11:20.107 DEBUG {main} [o.e.a.i.i.DefaultRepositoryConnectorProvider] : Using connector BasicRepositoryConnector with priority 0.0 for https://repository.jboss.org/nexus/content/groups/public 12:11:20.694 DEBUG {main} [o.a.h.headers] : >> GET /nexus/content/groups/public/commons-io/commons-io/2.5-SNAPSHOT/commons-io-2.5-20151119.212356-154.jar HTTP/1.1 12:11:20.694 DEBUG {main} [o.a.h.headers] : >> Host: repository.jboss.org ... 12:11:20.901 DEBUG {main} [o.a.h.headers] : << HTTP/1.1 200 OK ... 12:11:21.590 DEBUG {main} [o.e.a.i.i.EnhancedLocalRepositoryManager] : Writing tracking file /data/ggrzybek/sources/_testing/grgr-test-maven/target/repo-1469182277187/commons-io/commons-io/2.5-SNAPSHOT/_remote.repositories 12:11:21.591 INFO {main} [g.t.m.a.AetherTest] : Result: commons-io:commons-io:jar:2.5-20151119.212356-154 < jboss-public (https://repository.jboss.org/nexus/content/groups/public, default, snapshots)

here's the sequence of events:

Aether uses local repository at target/repo-1469182277187 location

commons-io/commons-io/2.5-SNAPSHOT/maven-metadata.xml metadata artifacts are fetched in parallel from both remote repositories.

Metadata is found only in jboss-public repository

Aether fetches metadata SHA1 checksum

target/repo-1469182277187/commons-io/commons-io/2.5-SNAPSHOT/resolver-status.properties is written to track the information about metadata

target/repo-1469182277187/commons-io/commons-io/2.5-SNAPSHOT/maven-metadata-jboss-public.xml file shows that 2.5-20151119.212356-154 is the latest version of SNAPSHOT artifact

Aether downloads commons-io/commons-io/2.5-SNAPSHOT/commons-io-2.5-20151119.212356-154.jar from jboss-public

Aether writes tracking file at target/repo-1469182277187/commons-io/commons-io/2.5-SNAPSHOT/_remote.repositories that looks like this:
#NOTE: This is an Aether internal implementation file, its format can be changed without prior notice. #Fri Jul 22 12:11:21 CEST 2016 commons-io-2.5-20151119.212356-154.jar>jboss-public=
This file allows us to recall where the artifact was downloaded from.

There's one more thing worth noting - this time Aether invokes some operations in separate threads (DefaultMetadataResolver-0-* threads in addition to main thread). Aether usually does that when doing more than one task at a time.

With non-SNAPSHOT artifact resolution, we checked one repository at a time, because we had one task - org.eclipse.aether.resolution.ArtifactRequest

With SNAPSHOT artifact resolution, Aether internally invokes two org.eclipse.aether.resolution.MetadataRequest tasks (one for each remote repository) to find the latest SNAPSHOT.

The number of threads used in this operation can be controlled with aether.metadataResolver.threads configuration property.

Summary

In this article I presented some of the internal details of Aether library. We've seen pure Java code examples. In next installment we'll enter OSGi world and see how can we use higher level libraries.

Autor: ggrzybek No comments:

Etykiety: aether, fuse, karaf, maven, osgi

Blog Archive

▼ 2021 (1)

▼ November (1)

Pax Web 8 - Whiteboard services and Karaf commands

► 2016 (3)

► October (1)

► July (2)

► 2015 (1)

► December (1)

► 2014 (3)

► August (3)

2021-11-23

Pax Web 8 - Whiteboard services and Karaf commands

Introduction

Detailed presentation

First web:context-list presents the contexts

Quite low-level command shows how interesting bundles reference Pax Web services. The Usage Count is obtained via reflection (for now - Felix specific).

Let's use sample command to Whiteboard-register a org.osgi.service.http.context.ServletContextHelper with custom name and non-default context path.

Let's register a javax.servlet.http.HttpServlet Whiteboard service, targetting the above context.

We can check that the servlet is working directly from Karaf

As expected, there are now more contexts

And the usage count has changed

Now let's unregister the context, which was targetted by the servlet.

context-list command shows again only one context

The usage counts has changed too

Let's see Pax Web 8 magic in action. Registering new OSGi context that can satisfy already registered servlet, automatically redeploys the servlet to new matching context!

Let's check the usage counts

Servlet is available in new context and not available in the old one

context-list at this stage

Usage counts still shows no leakage

Let's unregister the servlet this time

Let's unregister the Whiteboard context

The last reference should be gone when bundle 110 is stopped

Summary

2016-10-18

Using Maven with OSGi Part 3

Introduction

Metadata

~/.m2/repository/grgr/universalis-api/0.1.0.BUILD-SNAPSHOT/_remote.repositories

~/.m2/repository/grgr/universalis-api/0.1.0.BUILD-SNAPSHOT/maven-metadata-local.xml

~/.m2/repository/grgr/universalis-api/maven-metadata-local.xml

Low level, canonical example

RELEASE and LATEST versions

Remote repositories

Example with remote repository

Update policy

Summary

2016-07-22

Using Maven with OSGi Part 2

Introduction

pax-url-aether

Configuration

Maven Repository URI

Other options

Caveats

Summary

Using Maven with OSGi Part 1

Introduction

Runtime

Maven

Aether

Logs

SNAPSHOTs

Summary

First `web:context-list` presents the contexts

Quite low-level command shows how interesting bundles reference Pax Web services. The `Usage Count` is obtained via reflection (for now - Felix specific).

Let's use sample command to Whiteboard-register a `org.osgi.service.http.context.ServletContextHelper` with custom name and non-default context path.

Let's register a `javax.servlet.http.HttpServlet` Whiteboard service, targetting the above context.

`context-list` command shows again only one context

`context-list` at this stage

The last reference should be gone when bundle `110` is stopped