Camel Blueprint test support

It's been a long time since some issues with camel-test-blueprint based tests were resolved. Two other issues were clarified only recently. It was about time to describe these problems and applied solutions.

Background

org.apache.camel.blueprint.BlueprintCamelContext is one of the implementations of org.apache.camel.CamelContext interface, designed to work inside OSGi runtimes (like Karaf, ServiceMix or JBoss Fuse). It makes the integration with OSGi framework easier and introduces another DSL to configure Camel applications - the blueprint DSL. It's an XML language in the http://camel.apache.org/schema/blueprint namespace.

As with other DSLs, one of the most important aspects of developing Camel application is testability. Developer should be able to run/test Camel routes with minimal effort. In case of OSGi, this simply means running/testing the route without a need to start full OSGi framework.

Felix Connect

Felix Connect (formerly known as PojoSR) is:

A service registry that enables OSGi style service registry programs without using an OSGi framework.

This simplified OSGi runtime was chosen as testing framework for Camel Blueprint applications.

The idea is simple (examples from Camel's org.apache.camel.test.blueprint.SimpleMockTest):

1. Develop Camel route using Blueprint XML DSL:

   <blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0"
           xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
           xsi:schemaLocation="
                http://www.osgi.org/xmlns/blueprint/v1.0.0 http://www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd">
       <camelContext xmlns="http://camel.apache.org/schema/blueprint">
           <route>
               <from uri="direct:start" />
               <to uri="mock:result" />
           </route>
       </camelContext>
   </blueprint>
   

2. Create JUnit test class that extends from org.apache.camel.test.blueprint.CamelBlueprintTestSupport
3. Override org.apache.camel.test.blueprint.CamelBlueprintTestSupport#getBlueprintDescriptor method:

   @Override
   protected String getBlueprintDescriptor() {
       return "org/apache/camel/test/blueprint/simpleMockTest.xml";
   }
   

4. Write a @Test method that uses helper methods from base classes:

   @Test
   public void testHelloWorld() throws Exception {
       getMockEndpoint("mock:result").expectedBodiesReceived("Hello World");
       template.sendBody("direct:start", "Hello World");
       assertMockEndpointsSatisfied();
   }
   

5. Run it as normal JUnit test.

Property placeholder support in Camel

Before we describe how properties and placeholders are handled in Blueprint Camel contexts, let's do a quick review of properties support in plain (non-OSGi) Camel context.

org.apache.camel.impl.DefaultCamelContext uses org.apache.camel.component.properties.PropertiesComponent to resolve property placeholders in various definitions of Camel model elements (processors, data formats, route definitions, ...). By default, those placeholders are delimited by {{ and }} and the location of properties is defined in the PropertiesComponent itself.

Other implementations of org.apache.camel.core.xml.AbstractCamelContextFactoryBean may however override initPropertyPlaceholder() method, to integrate with other sources of properties:

• org.apache.camel.spring.CamelContextFactoryBean adds support for BridgePropertyPlaceholderConfigurer
• org.apache.camel.blueprint.CamelContextFactoryBean adds (by default, it may be disabled) support for fetching properties from blueprint container (see org.apache.aries.blueprint.ext.AbstractPropertyPlaceholder class for details)

Now, having in mind that Camel may delegate to Blueprint container (Aries Blueprint in particular) when resolving properties, it's time to see...

The beauty of OSGi™

Under the hood, Blueprint version of Camel context is an OSGi service exposed from Blueprint Container. In OSGi, everything is dynamic, each service may come and go any time as new bundles are installed, refreshed, updated or removed. Each change to a service may lead to cascade of changes to other services. It'd be good, to test at least some of those scenarios within our simple OSGi registry.

ConfigAdmin

Configuration Admin is an OSGi service designed to manage configuration data used by bundles and services. It is implemented by Felix ConfigAdmin subproject. It wouldn't be that interesting in itself - just another OSGi service with its specific interfaces and usage scenarios...

... The interesting aspect is that Blueprint integrates with ConfigAdmin and allows for updates/reloads of blueprint container as a result of ConfigAdmin configuration changes. And it is highly asynchronous, with several layers of threads involved.

ConfigAdmin / Blueprint integration

Aries Blueprint project contains a subproject called blueprint-cm that introduces an XML namespace for custom elements that may be used in Blueprint descriptors. These custom elements enable integration between Blueprint container and ConfigAdmin service. Blueprint CM is an extension to core Aries Blueprint functionality that relates to property placeholders.

Here's example Blueprint descriptor with such elements from cm and ext namespaces (example from Camel's own test suite):

<blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xmlns:cm="http://aries.apache.org/blueprint/xmlns/blueprint-cm/v1.1.0"
        xmlns:ext="http://aries.apache.org/blueprint/xmlns/blueprint-ext/v1.0.0"
        xsi:schemaLocation="
             http://aries.apache.org/blueprint/xmlns/blueprint-cm/v1.1.0 http://aries.apache.org/schemas/blueprint-cm/blueprint-cm-1.1.0.xsd
             http://www.osgi.org/xmlns/blueprint/v1.0.0 http://www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd">

  <cm:property-placeholder persistent-id="stuff" placeholder-prefix="{{" placeholder-suffix="}}"
      update-strategy="reload">
    <cm:default-properties>
      <cm:property name="my.resources.config.folder" value="src/test/resources" />
      <cm:property name="my.resources.config.file" value="framework.properties" />
      <!-- default value is true -->
      <!-- but we override this value in framework.properties where we set it to false -->
      <cm:property name="my.context.messageHistory" value="true" />
    </cm:default-properties>
  </cm:property-placeholder>

  <ext:property-placeholder id="my-blueprint-placeholder">
    <ext:default-properties>
      <ext:property name="my-version" value="framework_1.0" />
    </ext:default-properties>
    <!-- define location of properties file -->
    <ext:location>file:{{my.resources.config.folder}}/etc/{{my.resources.config.file}}</ext:location>
  </ext:property-placeholder>

  <bean id="myCoolBean" class="org.apache.camel.test.blueprint.MyCoolBean">
    <property name="say" value="${my.greeting}" />
  </bean>

  <camelContext messageHistory="{{my.context.messageHistory}}" xmlns="http://camel.apache.org/schema/blueprint"
      useBlueprintPropertyResolver="true">
    <route>
      <from uri="direct:start" />
      <bean ref="myCoolBean" method="saySomething" />
      <to uri="mock:result" />
    </route>
  </camelContext>

</blueprint>

All these property placeholders...

The above example shows many possible variants of property placeholders usage in blueprint XML declaration.

• {{...}} inside camelContext element - this is Camel's own notation for resolvable properties. When Camel context is initialized (constructed using org.apache.camel.core.xml.AbstractCamelContextFactoryBean), string values of properties are processed using org.apache.camel.CamelContext#resolvePropertyPlaceholders() call. This method uses org.apache.camel.component.properties.PropertiesComponent which knows how to deal with {{...}} syntax. The fact that {{ prefix and }} suffix may be changed is obvious, but let's not introduce any more confusion.
• {{...}} inside the text of ext:location element - these are handled by Blueprint parser which iterates through all available org.apache.aries.blueprint.ext.AbstractPropertyPlaceholder implementations and tries to resolve the property. {{...}} delimiters are declared explicitly on cm:property-placeholder element.
• ${...} in definition of say property of MyCoolBean - this again is handled by Blueprint parser, but this time default ${...} delimiters are used, so ext:property-placeholder will be the source of my.greeting property

All these property sources...

This example defines three different property sources (effectively hashmaps that contain name-value pairs) that are used to resolve the placeholders.

• <ext:property-placeholder> (implemented by org.apache.aries.blueprint.ext.PropertyPlaceholder) - resolves placeholders from system properties and/or from other sources (<ext:default-properties> and <ext:location>)
• <cm:property-placeholder> (implemented by org.apache.aries.blueprint.compendium.cm.CmPropertyPlaceholder which extends org.apache.aries.blueprint.ext.PropertyPlaceholder) - resolves placeholders from the content of ConfigAdmin configuration with named PID (specified by persistent-id attribute). If property can't be resolved from ConfigAdmin, it delegates to base class (which is the same class that implements <ext:property-placeholder>)
• Camel's own property resolver (org.apache.camel.component.properties.PropertiesComponent#propertiesResolver)

There are two important things to remember:

• <cm:property-placeholder> can be configured to reload entire Blueprint container (which effectively recreates Camel context) when ConfigAdmin configuration changes. This is done with update-strategy="reload" attribute of <cm:property-placeholder> (the default value is none)
• Camel's property resolver can be configured to delegate to all instances of org.apache.aries.blueprint.ext.AbstractPropertyPlaceholder found in Blueprint container. This is done with useBlueprintPropertyResolver="true" attribute of <camelContext> (which is true by default). So each Camel context defined using Blueprint XML DSL can resolve property placeholders using properties defined in Blueprint specific components (cm and ext property placeholders).

camel-test-blueprint

camel-test-blueprint is simply a set of helper classes for testing Camel routes defined with Blueprint XML DSL inside simplified OSGi registry (Felix Connect). org.apache.camel.test.blueprint.CamelBlueprintTestSupport base class handles all aspects of setting up OSGi registry, leaving implementation of @Test to developer.

One of the goals of well designed testing framework is to ensure that test runs are as predictable as possible. OSGi itself is highly dynamic and asynchronous, but it is not the reason to accept unpredictable Camel Blueprint tests. Because I believe that knowing history helps with understanding how and why something works, let's see how CamelBlueprintTestSupport based tests evolved over time.

Before Camel 2.15.3 (CAMEL-8948), no reloading of Blueprint container

Before resolving CAMEL-8948 issue, i.e., since introduction of camel-test-blueprint, tests that used ConfigAdmin updates were affected by race condition. Here's the sequence of events with race conditions highlighted. Synchronization points are marked as red lines. In between these lines, operations performed by different threads are completely unsynchronized. When describing threads, we'll cover only <cm:property-placeholder> element, not an ext version.

main thread	BP extender thread	CM Event Dispatcher thread	CM Configuration Updater thread
Creation of PojoSR registry - blueprint extender bundle starts threads that create Blueprint containers
1a if @Test class implements useOverridePropertiesWithPropertiesComponent(), OSGi service OverrideProperties is registered 2a if @Test class implements loadConfigAdminConfigurationFile(), org.osgi.service.cm.Configuration#update() is called with properties from specified file. – new properties are persisted by ConfigAdmin – 4 properties field is set to new value in org.apache.felix.cm.impl.ConfigurationImpl	Blueprint XML with Camel context is parsed and run. Camel context itself isn't started yet 2c <cm:property-placeholder> is initialized and ConfigAdmin configuration is fetched using PID configured with persistent-id="PID" attribute. These properties are initial properties of the resolver 3 <cm:property-placeholder> is registered to get notified when ConfigAdmin configuration for specified PID changes 1b setOverrideProperties() is called on PropertiesComponent if OverrideProperties service was found in OSGi registry OSGi service related to Blueprint container is published in OSGi registry
2b if @Test class implements useOverridePropertiesWithConfigAdmin(), org.osgi.service.cm.Configuration#update() is called with the overridden properties. – new properties are persisted by ConfigAdmin – 4 properties field is set to new value in org.apache.felix.cm.impl.ConfigurationImpl		listeners for ConfigurationEvent.CM_UPDATED event are notified (currently it's only fileinstall listener)	ManagedService#updated() is called with updated configuration on every registered ManagedService OSGi service, if location check is passed 5.
Camel context is started Camel routes are started 6 Camel endpoint URIs are parsed and if they use property placeholders, Camel resolves them. CamelBlueprintTestSupport waits for OSGi service related to Blueprint container		listeners for ConfigurationEvent.CM_UPDATED event are notified (currently it's only fileinstall listener)	ManagedService#updated() is called with updated configuration on every registered ManagedService OSGi service, if location check is passed 5.
@Test method is invoked

Explanation of possible problems:

• 1a → 1b Override properties of PropertiesComponent may be empty, if Blueprint container was initialized before main thread managed to register OverrideProperties OSGi service
• 2a2b → 2c Initial set of properties available in <cm:property-placeholder> may be empty, if Blueprint container was initialized before main thread managed to update ConfigAdmin configuration
• 3 If <cm:property-placeholder> had update-strategy="reload" attribute, this would ensure that entire Blueprint container was reloaded on ConfigAdmin configuration change
• 4 Only after this moment, 2c is able to find non-null properties
• 5 This update fails. The problem is bundle location which prevents propagating ConfigAdmin configuration change to registered ManagedServices. This is the reason why we've changed configAdmin.getConfiguration(pid) to configAdmin.getConfiguration(pid, null) here.
• 6 Properties used to resolve placeholders depend on the moment when <cm:property-placeholder> was initialized. If BP extender thread was quicker than main thread. We could have two problems:
  • 2c before 2a - we could get "Property with key [placeholder] not found in properties from text: {{placeholder}}"
  • 2c before 2b - we could resolve wrong property (before overriding)

Summary of covered scenarios

• Before Camel 2.15.3 Blueprint container was loaded only once. It was never reloaded as a result of ConfigAdmin configuration change. No Blueprint XML DSL based Camel context was tested with <cm:property-placeholder update-strategy="reload">
• Both loadConfigAdminConfigurationFile() and useOverridePropertiesWithConfigAdmin() methods were used to provide initial properties of ConfigAdmin configuration and as a result - of <cm:property-placeholder> resolver
• In most cases, main thread changed ConfigAdmin configuration before <cm:property-placeholder> was initialized

After Camel 2.15.3 (CAMEL-8948), reloading of Blueprint container

After fixing ARIES-1350 in blueprint-core-1.4.4 we could provide better synchronization of threads involved in Blueprint tests. I've added some tests that use <cm:property-placeholder update-strategy="reload">, but also I've effectively removed the distinction between loadConfigAdminConfigurationFile() and useOverridePropertiesWithConfigAdmin() (see CAMEL-9313).

Anyway, in Camel 2.15.3 we have better (or "different") synchronization between threads. This time reloading of Blueprint container is taken into account. Using BlueprintEvent.CREATED event listeners we could add synchronization between main and BP Extender threads.

main thread	BP extender thread	CM Event Dispatcher thread	CM Configuration Updater thread
Creation of Felix Connect registry - blueprint extender bundle starts threads that create Blueprint containers
expectReload flag is set if Blueprint XML contains <cm:property-placeholder update-strategy="reload">. This flag set to true means that each invocation of org.osgi.service.cm.Configuration#update() should be followed by waiting for Blueprint container reload event. 1a if @Test class implements useOverridePropertiesWithPropertiesComponent(), OSGi service OverrideProperties is registered CamelBlueprintTestSupport waits for BlueprintEvent.CREATED event - for full initialization of Blueprint container	Blueprint XML with Camel context is parsed and run. Camel context itself isn't started yet 2a <cm:property-placeholder> is initialized and ConfigAdmin configuration is fetched using PID configured with persistent-id="PID" attribute. These properties are initial properties of the resolver 3 <cm:property-placeholder> is registered to get notified when ConfigAdmin configuration for specified PID changes 1b setOverrideProperties() is called on PropertiesComponent if OverrideProperties service was found in OSGi registry OSGi service related to Blueprint container is published in OSGi registry BlueprintEvent.CREATED is emitted
2b if @Test class implements loadConfigAdminConfigurationFile(), org.osgi.service.cm.Configuration#update() is called with properties from specified file. – new properties are persisted by ConfigAdmin – properties field is set to new value in org.apache.felix.cm.impl.ConfigurationImpl
if expectReload was true, CamelBlueprintTestSupport waits for BlueprintEvent.CREATED event again, because we know that Blueprint container will be reloaded		listeners for ConfigurationEvent.CM_UPDATED event are notified (currently it's only fileinstall listener)	ManagedService#updated() is called with updated configuration on every registered ManagedService OSGi service. This time we have successful call, bundle location is correct and listener registered in 3 is called CmPropertyPlaceholder#updated() is called (in yet another thread, but it doesn't change the diagram). 5 CmPropertyPlaceholder (<cm:property-placeholder>) doesn't set properties field to updated properties, it just invokes blueprintContainer.reload()
	Blueprint container is reloaded 2c <cm:property-placeholder> is initialized and this time, ConfigAdmin configuration contains changed values BlueprintEvent.CREATED is emitted
2b if @Test class implements useOverridePropertiesWithConfigAdmin(), org.osgi.service.cm.Configuration#update() is called with the overridden properties. – new properties are persisted by ConfigAdmin – properties field is set to new value in org.apache.felix.cm.impl.ConfigurationImpl
if expectReload was true, CamelBlueprintTestSupport waits for BlueprintEvent.CREATED event again, because we know that Blueprint container will be reloaded (2nd time)		listeners for ConfigurationEvent.CM_UPDATED event are notified (currently it's only fileinstall listener)	ManagedService#updated() is called again and eventually blueprintContainer.reload() is invoked
	Blueprint container is reloaded 2c <cm:property-placeholder> is initialized and again, ConfigAdmin configuration contains changed values BlueprintEvent.CREATED is emitted
Camel context is started Camel routes are started 6 Camel endpoint URIs are parsed and if they use property placeholders, Camel resolves them. We don't have to wait for OSGi service related to Blueprint container because we've synchronized to BlueprintEvent.CREATED event already @Test method is invoked

Legend:

• 1a → 1b There's still race condition here. But useOverridePropertiesWithPropertiesComponent() method is part of org.apache.camel.test.junit4.CamelTestSupport class, not related to Blueprint. This method should not be used in Blueprint.
• 2a When <cm:property-placeholder> is initialized in first incarnation of Blueprint container, initial properties fetched from ConfigAdmin are always null. The only way of setting initial properties is to use <cm:default-properties>/<cm:property> subelements of <cm:property-placeholder>
• 2b → 2c This time we have correct synchronization, so this sequence of events is always correct. Blueprint container after reload picks up updated properties.
• 5 There's no need to set new properties in current instance of CmPropertyPlaceholder. Entire Blueprint container will be reloaded, so when new CmPropertyPlaceholder instance is initialized, it'll pick updated properties directly from ConfigAdmin.
• 6 Careful synchronization of ConfigAdmin configuration updates and Blueprint events fixed the problem with all tests. We always know what exact properties will be used when resolving placeholders.

Summary of new scenarios

• After Camel 2.15.3 Blueprint container was loaded at least once. ConfigAdmin configuration change could lead to reload of Blueprint container (<cm:property-placeholder update-strategy="reload">)
• Neither loadConfigAdminConfigurationFile() nor useOverridePropertiesWithConfigAdmin() methods were used to provide initial properties of ConfigAdmin configuration. Both methods, if implemented, lead to reload of Blueprint container and effectively perform the same thing. That's why CAMEL-9313 and CAMEL-9377 were created.

Problems

• We have two methods that do the same. We can't provide initial ConfigAdmin configuration (in other way than with <cm:default-properties>/<cm:property>). camel:run Maven goal doesn't work, as it relies on -pid and -pf options and call to org.osgi.service.cm.Configuration#update() after Blueprint container is loaded. If Blueprint XML DSL doesn't set update-strategy="reload" in <cm:property-placeholder>, updating ConfigAdmin configuration won't have any effect for property resolvers.

CAMEL-9313, CAMEL-9377, reloading of Blueprint container, initialization of ConfigAdmin configurations

Two above diagrams show two opposite approaches to synchronization (no synchronization vs. too much synchronization). So the only missing piece is to restore the purpose of loadConfigAdminConfigurationFile(). This method has to be used to provide initial configuration of ConfigAdmin, before Blueprint container (BP Extender thread) has chance to initialize <cm:property-placeholder>. It was achieved with OSGi listeners to initialize ConfigAdmin configurations just after felix configadmin bundle registers (service.pid=org.apache.felix.cm.ConfigurationAdmin) OSGi service but before blueprint.core bundle is started.

Even if OSGi specification says that relying on any order of events is bad idea, in camel-test-blueprint we used some tricks to force our listener to be called before any other listener waiting for initialization of (service.pid=org.apache.felix.cm.ConfigurationAdmin). This is how it works now:

main thread	BP extender thread	CM Event Dispatcher thread	CM Configuration Updater thread
Preparation of test bundle containing @Test-annotated classes Careful ordering of bundle descriptors to initialize Felix Connect OSGi Registry. Most of the bundles don't have any particular order, the only required sequence is: – Felix Connect itself (bundle "0") – test bundle – felix.configadmin – aries.blueprint.core this order is required to do correct synchronization of events and listener invocations to ensure proper ConfigAdmin initialization. 2a if @Test class implements loadConfigAdminConfigurationFile(), org.osgi.service.cm.Configuration#update() is called with properties from specified file. Call is made early enough and we're 100% sure that it happens before Blueprint containers are created. – new properties are persisted by ConfigAdmin – properties field is set to new value in org.apache.felix.cm.impl.ConfigurationImpl
Creation of Felix Connect registry - blueprint extender bundle starts threads that create Blueprint containers		listeners for ConfigurationEvent.CM_UPDATED event are notified (currently it's only fileinstall listener)	no ManagedService#updated() is called, or at least it's not relevant
expectReload flag is set if Blueprint XML contains <cm:property-placeholder update-strategy="reload">. This flag set to true means that each invocation of org.osgi.service.cm.Configuration#update() should be followed by waiting for Blueprint container reload event. if @Test class implements useOverridePropertiesWithPropertiesComponent(), OSGi service OverrideProperties is registered CamelBlueprintTestSupport waits for BlueprintEvent.CREATED event - for full initialization of Blueprint container	Blueprint XML with Camel context is parsed and run. Camel context itself isn't started yet 2b <cm:property-placeholder> is initialized and ConfigAdmin configuration is fetched using PID configured with persistent-id="PID" attribute. These properties are initial properties of the resolver <cm:property-placeholder> is registered to get notified when ConfigAdmin configuration for specified PID changes setOverrideProperties() is called on PropertiesComponent if OverrideProperties service was found in OSGi registry OSGi service related to Blueprint container is published in OSGi registry BlueprintEvent.CREATED is emitted
2c if @Test class implements useOverridePropertiesWithConfigAdmin(), org.osgi.service.cm.Configuration#update() is called with the overridden properties. – new properties are persisted by ConfigAdmin – properties field is set to new value in org.apache.felix.cm.impl.ConfigurationImpl
if expectReload was true, CamelBlueprintTestSupport waits for BlueprintEvent.CREATED event again, because we know that Blueprint container will be reloaded (2nd time)		listeners for ConfigurationEvent.CM_UPDATED event are notified (currently it's only fileinstall listener)	ManagedService#updated() is called with updated configuration on every registered ManagedService OSGi service. CmPropertyPlaceholder#updated() is called (in yet another thread, but it doesn't change the diagram). CmPropertyPlaceholder (<cm:property-placeholder>) doesn't set properties field to updated properties, it just invokes blueprintContainer.reload()
	Blueprint container is reloaded 2d <cm:property-placeholder> is initialized and again, ConfigAdmin configuration contains changed values BlueprintEvent.CREATED is emitted
Camel context is started Camel routes are started 6 Camel endpoint URIs are parsed and if they use property placeholders, Camel resolves them. We don't have to wait for OSGi service related to Blueprint container because we've synchronized to BlueprintEvent.CREATED event already @Test method is invoked

Important changes:

• 2a → 2b This time we have correct synchronization, not with Blueprint events, but with OSGi listeners (Bundle and Service) so this sequence of events is always correct and <cm:property-placeholder> always sees ConfigAdmin configuration prepared by loadConfigAdminConfigurationFile() (if implemented).
• 2c → 2d Correct synchronization. Blueprint container after reload will see updated ConfigAdmin configuration
• 2a When <cm:property-placeholder> is initialized in first incarnation of Blueprint container, initial properties fetched from ConfigAdmin are always null. The only way of setting initial properties is to use <cm:default-properties>/<cm:property> subelements of <cm:property-placeholder>
• 6 Careful synchronization of ConfigAdmin configuration updates and Blueprint events fixed the problem with all tests. We always know what exact properties will be used when resolving placeholders.

Summary of new scenarios

• We've restored distinction between loadConfigAdminConfigurationFile() (initialization of ConfigAdmin configuration) and useOverridePropertiesWithConfigAdmin() (reloading of BlueprintContainer if update-strategy="reload") methods.

Summary

I hope this presentation of camel-test-blueprint internals will clear more confusion than it introduces and will be a good source of information in case you have any problems with org.apache.camel.test.blueprint.CamelBlueprintTestSupport based JUnit tests.