Pentaho

Hidden Results from Machine Learning Models – Blog 2 of 2

As mentioned in the first blog of this 2 part series, there are hidden results that needs to be interpreted and extrapolated from the initial or assumed prediction of your results. Just like the previous blog shows that a binary class can have in 3 results, YES, NO and a state that should be interpreted as UNKNOWN, regardless of the Class max prob answer. 

This is an area of research called "learning to abstain from making a prediction" which is closely related to "active learning machine learning" where, in a semi-supervised scenario, a classifier can identify which cases/examples it would like a human to label in order to improve its model.

This situation compounds itself in a multi-class problem, more than 2 classes. Let’s now examine one of the “Hey Ray!” demonstrations again (sorry, I love these demonstration applications). First, there are two deep learning models in the analytic pipeline of this @Pentaho Data Integration application. The first deep learning model is a binary class that detects an injury or anomaly in an x-ray image, and as we learned previously, there's more than two answers. The second deep learning model, identifies the body part in the x-ray in a 5 part multi-class classifier. The parts that are being identified come from the arm and include, the hand, forearm, elbow, humerus and shoulder.

In the demonstration, I always try to zoom in on a specific body part for injury detection and arm part identification as shown in the first image chart below. In this case, the dominant feature is the elbow and the model will produce a Class max prob and Class:elbow predicted prob result of "elbow" with a probability of 0.894 or 89% for this specific perspective of this image. Different zoom levels or angled views could result in different probabilities values, but they would all point to this image as being an "elbow" and maybe some insignificant pieces of other parts. The chart below shows what the model “sees” in this x-ray image. 

This this is important to the “Hey Ray!” demonstration is because we are only looking for one of five body parts exclusively. The PDI transformation prepares the results using filter steps to determine the confidence of the results. At 89%, this would be considered a pretty high confidence level and the test-to-speech results would be reflected in the high probability. If the probability were to be lower, like 60% (which is still significant compared to the other probabilities), then the dialog of the speech would reflect the questionable result. We take the Class max prob, regardless of the value to predict our body part identification and use filter steps in PDI to determine the confidence in the dialog.

What we haven’t implemented as of yet, interpreting what happens when there are multiple significant predictions. What happens when a Class max prob prediction is below 50%? All class probabilities must sum to 1 or 100%. In a resulting probability of less than 50%, there must be other classes that are significantly greater than zero.

Let use the exact same x-ray image and zoom out to show more of the image.The full x-ray image shows significantly more of the rest of the arm beyond just the elbow. We can see much more of the humerus and the forearm in addition to the elbow. So what would our deep learning body part identification model predict with this image? Checkout the chart image below.

As you can see in this second chart image, the humerus probability is now the Class max prob, but is less than 50% at 48%. More importantly, the Class:forearm predicted and Class:elbow predicted are at significantly higher probabilities of 26% and 25% respectively. The sum of the probabilities of these three body parts is 0.992. We could interpret this as our body part identification model is identifying the middle arm area – forearm, elbow and humerus.

 When we created this demonstration, we were not aware of this interesting artifact. That is, the the deep learning model could identify mulitple classes simultaneously, but it makes sense after exploring the results more closely for different types of image capture perspectives. We are looking to add a multi-class interpretation flow to more accurately identify the x-ray images where possibilities could include, but are not limited to the body part combinations of,

• Shoulder
• Shoulder and Humerus
• Shoulder and Humerus and Elbow
• Shoulder and Humerus and Elbow and Forearm
• Shoulder and Humerus and Elbow and Forearm and Hand
• Humerus
• Humerus and Elbow
• Humerus and Elbow and Forearm
• Humerus and Elbow and Forearm and Hand
• Elbow
• Elbow and Forearm
• Elbow and Forearm and Hand
• Forearm
• Forearm and Hand
• Hand

There are more combinations possible and should be accounted for. This includes x-ray images of a hand and shoulder, or just the shoulder and elbow, and so forth. While it may seem impossible to capture these types of x-ray images in a normal situation, the fact most of the reasons for having an x-ray taken is for injury purposes, so you never know what you might analyze. Now consider the scenario where the entire body is being identified. Yikes! that's a lot of body parts and combinations. We're going to need a bigger team.