Paulo Pires

Real Time Analysis and Visualization with Pentaho

Blog Post created by Paulo Pires Employee on Dec 6, 2018

Pentaho CDE Real Time Analysis Dashboard

About a month ago Ken Wood posted a blog post sharing a real time IoT data stream with LiDAR data and challeging the Pentaho Community to create an Analysis and Visualization with it. A few days later added another real time IoT data stream with dust particulates sensor data.

 

Using the latest Pentaho 8.2.0 release, with PDI I was able to fetch the data from each sensor, read the values from the JSON format and write all the incoming lines to a stage table.

 

For the dust particulate sensor then all that was needed now was to create a Data Service to feed the CDE Dashboard's Charts that will show the last 10 minutes of data in a CCC Line Chart, and the most recent measure in a CCC Bar Chart.

 

As for the LiDAR Motion sensor data, some analysis was needed in order to determine the direction of the people detected by the sensor. For that, after writing the data in the stage table, a sub-transformation is called that will do that analysis and write the output in a fact table. In this fact table we'll only have 1 line per person that translates the behavior of the person crossing the entrance and hallway, namely, where from and to the person is coming/going, the timestamps of entering and leaving each area, if it's going in, out or crossing the hallway, and the lag time in each of the areas.

 

Here are the screenshots of the ETL Analysis Process created:

1dustDS.png

Image 1 - Particulate Sensor transformation

 

 

2lidarDS.pngImage 2 - LiDAR Motion Sensor transformation

 

3subLidar.pngImage 3 - LiDAR Motion Sensor sub-transformation

 

After this, I have all the data needed to make a cool visualization using CDE Dashboards.

 

 

The dashboard is divided in 3 sections:

  • The first for the LiDAR data, shows the KPIs and the Last 10 movements table, which is refreshed every time a person enters or leaves the hallway or entrance.
  • The second related with the dust particulate sensor, shows the last 10min of data in a line chart and the most recent measure received in a bar chart, and is updated every second, since we receive data from this sensor every second.
  • Finally the third section, show the correlation between the LiDAR data and the dust particulate sensor for the last 24h, and it is refreshed every minute.

 

Here is the final result:

LiDAR and Dust Dashboard 2.pngImage 4 - CDE Real Time Dashboard

 

 

We will be deploying this solution in a server in the near future so everyone can access and see it working and will update this blog post.

Outcomes