Ben Isherwood

HCI Plugins: Introduction and Quick Start Guide

Blog Post created by Ben Isherwood Employee on Jun 9, 2017

One of the design goals of HCI was extensibility.

 

The team wanted to make sure that HCI could connect to any system that you would like to use today or in the future. We also wanted the system to easily adjust to changing technology trends and evolve it's capabilities and use cases over time. As a result, HCI is a highly pluggable system. This allows us to upgrade and evolve technologies simply by building new plugins - without requiring updates to HCI itself. It also means that we can continue to provide seamless compatibility to existing systems as we prototype with bleeding edge advancements in content processing/analysis as well as taking advantage of new data stores.


The HCI Plugin SDK allows all end users to extend the capabilities of their HCI system. With the plugin SDK, one can:

  • Build connections to various systems to get data into HCI
  • Support customized processing on the data

 

The plugin SDK is available as a separate download from HCI, and is available on the downloads page. It includes:

  • Multiple levels of documentation, including full interface javadoc and useful README files
  • Helpful utilities to use in plugin logic
  • Example connector and stage plugins
  • A full-featured, offline plugin test harness

 

Looking for simple steps to build an HCI plugin?   Here's the process in a nutshell:

 

Step 1. Read the documentation

 

To make developers lives easier, the plugin SDK contains lots of inline help:

  • See the top level README.txt file for an overview on HCI and plugins
  • See the examples/EXAMPLES.txt file for instructions on building the example code
  • See the plugin-test/TEST_AND_DEBUG.txt file for plugin test harness test and debugging instructions

 

HTML javadoc is also provided for all plugin interfaces (ConnectorPlugin, StagePlugin) and utility classes in the doc/javadoc folder.  Click on the index.html file in this directory to open it in your web browser.

 

b1.png

 

Step 2. Build the example code

 

Example HCI connector and stage plugins are immediately available for customization. Hacking on these examples is probably the best way to learn the plugin technologies.

 

The following instructions come straight from the HCI EXAMPLES.txt file.

 

To build the example plugins:

1. Unpack the HCI SDK package.  

2. Navigate to the examples directory:
   cd HCI-Plugin-SDK/examples 

3. Create the classes directory:
   mkdir classes 

4. Compile the java files for your plugin:
    javac -cp ../lib/plugin-sdk-<build-number>.jar -d classes/ \
         src/com/hds/ensemble/sdk/examples/connect/*.java \
         src/com/hds/ensemble/sdk/examples/stage/*.java 

5. Copy the plugin resource file:
    cp -R src/META-INF/ classes/ 

6. Create the final jar:
    cd classes && \
    jar -cf ../HCI-example-plugins.jar * && \
    cd ..

This process generates a new plugin jar file named "HCI-example-plugins.jar" that you can test in the plugin-test harness and upload directly to the HCI system to use right away!


Note: The plugin jar file must be a "fat" jar and contain all dependency libraries for everything it uses inside it. HCI provides plugin classloader isolation, allowing different plugins to use different versions of software within the same system. To make this work, you need to include all libraries inside the plugin jar file that your plugin will use. Make sure that you do NOT include the HCI plugin SDK library within your plugin jar. The HCI Plugin SDK jar should be on the compilation classpath per the example above (a "provided jar"), but NOT end up within the final plugin jar. This allows the HCI SDK in the product to evolve without breaking your plugin.

 

Step 3. Validate, test, and debug plugin code

 

The plugin SDK comes with a full featured test harness: "plugin-test".  You can use this tool to validate, test, and debug custom HCI plugins.

 

The plugin test harness can operate in one of three modes:

  • Validation mode
  • Test mode
  • Debug mode

 

In validation mode, the harness ensures that the plugin manifest file exists, is well-formed, and that the plugin interfaces have all been correctly implemented. If your plugin validates successfully, then you have correctly edited the manifest file and implemented the required interfaces.


In test mode, custom configuration for each plugin is specified in the plugin-test harness configuration file. The test harness will then utilize each configuration to exercise additional functionality, check for errors, and make recommendations. Use this mode to exercise your plugin logic, and identify any issues.


Debug mode allows for configured plugins to be further utilized in a debuggable environment, allowing plugin logic itself to be checked for errors as methods are executed by the test harness. Executing in debug mode will prompt the user to connect any Java debugger or IDE to the specified port (5903 by default). This allows the user to set breakpoints in their plugin code and debug the logic itself while the tests execute.

 

Plugin Interface Validation

 

To validate a plugin:

  1. Go to the plugin-test directory: 
      cd HCI-Plugin-SDK/plugin-test 

  2. Run the test harness in validation mode:
     ./plugin-test -j <path-to-your-plugin-bundle-name>.jar 

The test harness validates the implementation of your plugin and:

  • Indicates where errors exist
  • Makes recommendations to resolve any issues it identifies
  • Makes best practice recommendations

 

plugin-test output in validation mode:

b3.jpg

 

Once your plugin validates successfully, you're ready to test it out!

 

Plugin Instance Testing

 

Once your plugin has been validated, you can move on to deep testing and analysis of the plugin.

 

In test mode, custom configuration for each plugin is specified in the plugin-test harness configuration file. The test harness will then utilize each configuration to exercise additional functionality, check for errors, and make recommendations. Use this mode to exercise your plugin logic, and identify any issues.


To test your plugin, you will first need to configure the plugin within the test harness. This involves:

  • Defining a PluginConfig to use when testing your connector or stage plugin
  • Defining the fields and streams on an inputDocument to use when testing your (stage) plugin


Fortunately, the plugin-test tool makes life easy with an "autoconfigure" option, which allows you to generate a plugin-test configuration for any or all plugins found within the specified plugin jar file.


To generate a configuration file for all plugins in a bundle:

 ./plugin-test -j <path-to-your-plugin-bundle-name>.jar -a [output-config-file] 

To generate a configuration file for only one plugin:

 ./plugin-test -j <path-to-your-plugin-bundle-name>.jar -plugin <plugin name> \
      -a [output-config-file]

This automatically generated configuration file may then be saved and edited in order to fine tune the plugin configuration to be used while running the plugin test harness. The auto-generated config will reflect the plugin-defined default values for all properties.


Custom configuration values for each plugin may be applied. If a plugin requires user input for config properties in the default configuration, these values must be specified in the plugin-test config file before testing the plugin. This is accomplished by adding the proper "value" entries for all default config properties in the plugin configuration.

 

Example PluginConfig Property:

{

  "name": "com.hds.hci.plugins.myplugin.property1",

  "type": "TEXT",

  "userVisibleName": "Field Name",

  "userVisibleDescription": "The name of the field to process",

  "options": [],

  "required": true,

  "value": "" // <--- Add non-empty value here for all required fields

}

 

For stage plugin testing, the inputDocument fields and streams may also be customized in the automatically generated plugin-test configuration file.

 

Example Input Document declaration:

"inputDocument": {

       "fields": {

            // ---> ADD OR MODIFY FIELDS HERE <----

            "HCI_id": [

                "https://testDocument.example.com/testDocument1"

            ],

            "HCI_doc_version": [

                "1"

            ],

            "HCI_displayName": [

                "Test Document"

            ],

            "HCI_URI": [

                "https://testDocument.example.com/testDocument1"

            ]

        },

       "streams": {

          // ---> ADD OR MODIFY STREAMS HERE <----

           "HCI_content": {

               "HCI_local-path": "/tmp/tmp1904411799648787907.tmp"

               // Instructions: To present a stream, use it's file path as the value of HCI_local-path above

        }

    }

}

 

Auto-configure and test execution example:

b4.jpg

 

Plugin Instance Debugging

 

Debug mode allows for configured plugins to be further utilized in a debug environment, allowing plugin logic itself to be checked for errors as methods are executed by the test harness. Executing in debug mode will prompt the user to connect any Java debugger or IDE to the specified port (5903 by default). This allows the user to set breakpoints in their plugin code and debug the logic itself while the tests executed.

 

The only difference between test mode and debug mode is:

  • The user sets Java IDE/debugger breakpoints in the methods of their plugin source code
  • The user starts the plugin-test tool using the "-d" option
  • The plugin-test tool prints the debugging port to connect to (5903 by default) and waits for a connection
  • The user attaches their Java IDE/debugger to the specified port (5903 by default)
  • Tests begin to execute and breakpoints will be hit

 

To run a test in debug mode:

 ./plugin-test -j <path-to-your-plugin-bundle-name>.jar -a [output-config-file] -d 

To run a test for a specific plugin in debug mode:

 ./plugin-test -j <path-to-your-plugin-bundle-name>.jar -plugin <plugin name> \
     -a [output-config-file] -d

 

plugin-test waiting for the user to connect the Java IDE to "Remote at port 5903":

b5.jpg

 

After the user connects to the process using the Java IDE (e.g. IntelliJ/Eclipse), tests will begin executing and breakpoints within the plugin source code can be stepped through manually to debug the custom plugin:

b6.jpg


Note that you can build, test, and debug an HCI plugin without ever touching an HCI system!

 

Step 4. Upload to HCI

 

Once your plugin bundle JAR file is ready and tested, you are ready to use it in production.

 

Simply upload your new plugin(s) to the UI by dragging and dropping it into the upload pane from any file browser, or just click to browse for the file locally.


  b1.jpg

 

b2.jpg


After upload, HCI will automatically make your plugin code available to all system instances and you may begin using your custom "Data Connections" and processing pipeline "Stages" immediately. Because HCI UI is dynamically generated based on the plugins available, you will immediately see your plugin appear in the existing drop down options.


Hopefully this demonstrates how anyone can easily build, package, and test custom HCI plugins.

 

Thanks for reading!

-Ben

Outcomes