Pentaho

 View Only

 PMI Deep Learning network step - The following engines are unavailable (check configuration): Python - Scikit-learn R - MLR Spark - MLlib DL4j

  • Pentaho
  • Pentaho
Ingo Levin's profile image
Ingo Levin posted 11-16-2018 16:18

Having upgrade to the latest PMI 1.4 from Marketplace, I am getting the following error message opening the PMI Deep Learning network step:

"The following engines are unavailable (check configuration): Python - Scikit-learn R - MLR Spark - MLlib DL4j "

 

This only happens on the Deep Learning step, the other PMI steps work fine. I suspect it is not installing/loading the DJ4j lib correctly.

 

 

On the console

java.lang.ClassNotFoundException: weka.python.PythonSession

        at java.net.URLClassLoader.findClass(Unknown Source)

        at java.lang.ClassLoader.loadClass(Unknown Source)

        at java.lang.ClassLoader.loadClass(Unknown Source)

        at org.pentaho.di.core.plugins.KettleURLClassLoader.loadClassFromParent(

KettleURLClassLoader.java:94)

        at org.pentaho.di.core.plugins.KettleURLClassLoader.loadClass(KettleURLC

lassLoader.java:114)

        at java.lang.ClassLoader.loadClass(Unknown Source)

        at java.lang.Class.forName0(Native Method)

        at java.lang.Class.forName(Unknown Source)

        at weka.core.WekaPackageClassLoaderManager.forName(WekaPackageClassLoade

rManager.java:206)

        at weka.core.WekaPackageClassLoaderManager.forName(WekaPackageClassLoade

rManager.java:186)

        at org.pentaho.pmi.engines.PythonEngine.<clinit>(PythonEngine.java:58)

        at org.pentaho.pmi.PMIEngine.init(PMIEngine.java:86)

        at org.pentaho.pmi.PMIEngine.getEngineNames(PMIEngine.java:100)

        at org.pentaho.di.ui.trans.steps.pmi.BaseSupervisedPMIStepDialog.addConf

igTab(BaseSupervisedPMIStepDialog.java:719)

        at org.pentaho.di.ui.trans.steps.pmi.BaseSupervisedPMIStepDialog.open(Ba

seSupervisedPMIStepDialog.java:358)

        at org.pentaho.di.ui.spoon.delegates.SpoonStepsDelegate.editStep(SpoonSt

epsDelegate.java:120)

        at org.pentaho.di.ui.spoon.Spoon.editStep(Spoon.java:8949)

 

Tried these troubleshooting steps:

1) remove the ~\wekafiles directory

 

2) Re-Installed all R packages. Now have

> packageVersion('JavaGD')

[1] ‘0.6.1’

> packageVersion('XML')

[1] ‘3.98.1.9’

> packageVersion('LiblineaR')

[1] ‘2.10.8’

> packageVersion('e1071')

[1] ‘1.6.8’

> packageVersion('randomForest')

[1] ‘4.6.12’

> packageVersion('nnet')

[1] ‘7.3.12’

> packageVersion('gbm')

[1] ‘2.1.3’

 

3) Ran pyCheck.py. No errors.

C:\Users\ingo>python wekafiles/packages/wekaPython/resources/py/pyCheck.py

 

C:\Users\ingo>

 

4) Started Spoon. Having removed the wekafiles folder earlier, it forced a fresh download of all the packages.

 

5) Restarted Spoon. Created new Transformation and added the Deep Learning step:

Still getting the same error when I attempt to open the step.


#Pentaho
Mark Hall's profile image
Mark Hall

Did you remove your old version of PMI from <PDI install directory>/plugins/steps? The old PMI plugin directory is called "pmi". Version 1.4 installs into a new directory called "plugin-machine-intelligence". The install docs specify that it is necessary to remove the old one first. If ~/wekafiles has been cleaned out and pmi removed, then the install procedure will be: 1) Install PMI from marketplace; 2) restart Spoon; 3) PMI will automatically install all Weka-related packages; 4) restart Spoon to ensure that the Weka packages are loaded correctly.

Note that with the new version of the R integration in Weka (RPlugin), it is no longer necessary to have the R_HOME and R_LIBS_USER environment variables set. As long as R is in the registry, and available on the PATH, then Weka's R integration should be able to locate the necessary libraries etc. automatically.

Cheers,

Mark.

Ingo Levin's profile image
Ingo Levin

Mark, thank you! That was indeed the problem: I had not removed the pmi folder in the plugins/steps folder.

After having done so, and re-installig the plugin + the wekafiles, it is working now.

However I could not find this in the install docs. I was following PMI_v_1_4_Installation__Windows_.pdf from OCt 23, 2018 and in Chapter 2.2 INSTALL PENTAHO MACHINE INTELLIGENCE (PMI) it doesn't say anything about removing the pmi folder:

PMI is available from the PDI Marketplace. Simply open the Marketplace view in PDI and click

the Install button alongside the Plugin Machine Intelligence entry.

The first time PDI is launched after installing the PMI plugin there will be several minutes

added to the start-up time during which WEKA will install the packages necessary to support

the various execution engines. Again, this requires a connection to the internet. For the curious,

the packages include:

• wekaPython

(http://weka.sourceforge.net/packageMetaData/wekaPython/index.html)

• RPlugin

(http://weka.sourceforge.net/packageMetaData/RPlugin/index.html)

• distributedWekaBase

(http://weka.sourceforge.net/packageMetaData/distributedWekaBase/index.html)

• distributedWekaSparkDev

(http://weka.sourceforge.net/packageMetaData/distributedWekaSparkDev/index.

html)

• wekaDeeplearning4j

(https://deeplearning.cms.waikato.ac.nz/)

It is recommended to run PDI Spoon using the SpoonConsole batch file, so that you can verify

that the WEKA packages are being downloaded and installed successfully (as shown in Figure

7).

It is also important to re-start PDI after the initial WEKA package installation process completes

and the Spoon user interface appears. This is to ensure that the newly installed WEKA packages are loaded properly. Note that this re-start is required just once—i.e. after the initial

installation of WEKA packages.

Mark Hall's profile image
Mark Hall

Hi Ingo,

We only officially released PMI version 1.4 last week. October 23 would have been a draft of the install documents :-) Did you access this from the github repository for PMI? If so, grab another copy and you will find the following (in bold) on page 9:

NOTE: If you have an older version of PMI already installed, please remove it first by deleting the pmi sub-folder from plugins\steps in your PDI installation.

Cheers,

Mark.

Ingo Levin's profile image
Ingo Levin

Mark, I've got it from here: PMI Installation, Developer Guide and Sample/Demos > PMI_v_1_4_Installation__Windows_.pdf   ... and I can't find the paragraph you are quoting in there..

Gosforth Kubu's profile image
Gosforth Kubu

I Have same problem. Every step from "Data mining" group generate "The following engines are unavailable (check configuration): Python - Scikit-learn R - MLR DL4j" error when I click on it. Then step menu opens. Anyway, these steps do not see streems from previous steps (I think there is another problem with these steps not only scikit-learn).

I followed your instruction. R is installed, Python (Anaconda) is installed (there should be better way to obtain Python - Anaconda is a huge amount of sh...). I installed scikit-learn in Anaconda (I think by default this is not installed) - "conda install scikit-learn". I added R_HOME (not true that this is not needed - without that Pentaho PDI will NOT start).

Yet for some reason this error keeps display. What it means exactly? Maybe I need to manually copy some library into PDI?

What this error exactly means? How to verify if environment is working?

This is fresh installation of PDI so there was no old plugin.

Mark Hall's profile image
Mark Hall

Hmm. Perhaps the Weka packages (wekaPython, RPlugin, wekaDeeplearning4j) did not get installed the first time you re-started PDI after installing PMI? You can check by looking in ${user.home}/wekafiles/packages. If you have been using Weka as a stand-alone tool in the past, then I'd suggest blowing away ${user.home}/wekafiles and allowing the PMI plugin to re-install the required Weka packages (in case there are version clashes). Weka's R integration no longer requires R_HOME or R_LIBS_USER to be set. This has been verified on all three major OS's with the most recent release of the RPlugin Weka package. PDI's R script executor still requires (I believe) R_HOME to be set. However, if you are not using this step then it should not cause any issues. Also make sure that you are using a fairly recent version of R (>= 3.5). We have kept up with changes to the MLR R package, but maintaining full backwards compatibility is not feasible. Anaconda ships with scikit-learn, so there is no need to install this again. Which version of Anaconda are you using? Note that python virtual environments are not supported. It is possible to use other python distributions as long as the required dependencies are installed. Although huge, Anaconda is an easy way to get started.

If you continue to have issues, then PDI logs, stack traces etc. are useful when trying to provide help - otherwise I'm just speculating.

Cheers,

Mark.

Gosforth Kubu's profile image
Gosforth Kubu

This content was either too long or contained formatting that did not work with our migration. A PDF document is attached that contains the original representation

 

Hi Mark,thank you for your reply.Once again I installed this plugin - on another PC. I had not installed neither Weka nor PMI plugin before on this PC. PDI 8.2. No other plugins installed. I installed Anaconda 3.7 version and R 3.6.0. This time I could run PDI without adding R_HOME.I start PDI using SpoonConsole.bat. Just like before I see Data Mining steps yet after clicking on one of them there is an error: "The following engines are unavailable (check configuration): Python - Scikit-learn"What I see in spoon.log is:maj 20, 2019 8:27:03 PM org.apache.karaf.main.Main$KarafLockCallback lockAquiredINFO: Lock acquired. Setting startlevel to 1002019-05-20 20:27:06.047:INFO:oejs.Server:jetty-8.1.15.v201404112019-05-20 20:27:06.075:INFO:oejs.AbstractConnector:Started NIOSocketConnectorWrapper@0.0.0.0:9051maj 20, 2019 8:27:06 PM org.apache.cxf.bus.blueprint.NamespaceHandlerRegisterer registerINFO: Registered blueprint namespace handler forhttp://cxf.apache.org/blueprint/core <http://cxf.apache.org/blueprint/core>maj 20, 2019 8:27:06 PM org.apache.cxf.bus.blueprint.NamespaceHandlerRegisterer registerINFO: Registered blueprint namespace handler forhttp://cxf.apache.org/configuration/beans <http://cxf.apache.org/configuration/beans>maj 20, 2019 8:27:06 PM org.apache.cxf.bus.blueprint.NamespaceHandlerRegisterer registerINFO: Registered blueprint namespace handler forhttp://cxf.apache.org/configuration/parameterized-types <http://cxf.apache.org/configuration/parameterized-types>maj 20, 2019 8:27:06 PM org.apache.cxf.bus.blueprint.NamespaceHandlerRegisterer registerINFO: Registered blueprint namespace handler forhttp://cxf.apache.org/configuration/security <http://cxf.apache.org/configuration/security>maj 20, 2019 8:27:06 PM org.apache.cxf.bus.blueprint.NamespaceHandlerRegisterer registerINFO: Registered blueprint namespace handler forhttp://schemas.xmlsoap.org/wsdl/ <http://schemas.xmlsoap.org/wsdl/>maj 20, 2019 8:27:06 PM org.apache.cxf.bus.blueprint.NamespaceHandlerRegisterer registerINFO: Registered blueprint namespace handler forhttp://www.w3.org/2005/08/addressing <http://www.w3.org/2005/08/addressing>maj 20, 2019 8:27:06 PM org.apache.cxf.bus.blueprint.NamespaceHandlerRegisterer registerINFO: Registered blueprint namespace handler forhttp://schemas.xmlsoap.org/ws/2004/08/addressing <http://schemas.xmlsoap.org/ws/2004/08/addressing>maj 20, 2019 8:27:06 PM org.apache.cxf.bus.osgi.CXFExtensionBundleListener addExtensionsINFO: Adding the extensions from bundle org.apache.cxf.cxf-rt-management (195) [org.apache.cxf.management.InstrumentationManager]maj 20, 2019 8:27:06 PM org.apache.cxf.bus.osgi.CXFExtensionBundleListener addExtensionsINFO: Adding the extensions from bundle org.apache.cxf.cxf-rt-wsdl (198) [org.apache.cxf.wsdl.WSDLManager]maj 20, 2019 8:27:06 PM org.apache.cxf.bus.osgi.CXFExtensionBundleListener addExtensionsINFO: Adding the extensions from bundle org.apache.cxf.cxf-rt-bindings-xml (200) [org.apache.cxf.binding.xml.XMLBindingFactory, org.apache.cxf.binding.xml.wsdl11.XMLWSDLExtensionLoader]maj 20, 2019 8:27:06 PM org.apache.cxf.bus.osgi.CXFExtensionBundleListener addExtensionsINFO: Adding the extensions from bundle org.apache.cxf.cxf-rt-bindings-soap (201) [org.apache.cxf.binding.soap.SoapBindingFactory, org.apache.cxf.binding.soap.SoapTransportFactory]maj 20, 2019 8:27:06 PM org.apache.cxf.bus.blueprint.NamespaceHandlerRegisterer registerINFO: Registered blueprint namespace handler forhttp://cxf.apache.org/blueprint/bindings/soap <http://cxf.apache.org/blueprint/bindings/soap>maj 20, 2019 8:27:06 PM org.apache.cxf.bus.osgi.CXFExtensionBundleListener addExtensionsINFO: Adding the extensions from bundle org.apache.cxf.cxf-rt-transports-http (202) [org.apache.cxf.transport.http.HTTPTransportFactory, org.apache.cxf.transport.http.HTTPWSDLExtensionLoader, org.apache.cxf.transport.http.policy.HTTPClientAssertionBuilder, org.apache.cxf.transport.http.policy.HTTPServerAssertionBuilder, org.apache.cxf.transport.http.policy.NoOpPolicyInterceptorProvider]maj 20, 2019 8:27:06 PM org.apache.cxf.bus.osgi.CXFExtensionBundleListener addExtensionsINFO: Adding the extensions from bundle org.apache.cxf.cxf-rt-ws-policy (220) [org.apache.cxf.ws.policy.PolicyEngine, org.apache.cxf.policy.PolicyDataEngine, org.apache.cxf.ws.policy.AssertionBuilderRegistry, org.apache.cxf.ws.policy.PolicyInterceptorProviderRegistry, org.apache.cxf.ws.policy.PolicyBuilder, org.apache.cxf.ws.policy.PolicyAnnotationListener, org.apache.cxf.ws.policy.attachment.ServiceModelPolicyProvider, org.apache.cxf.ws.policy.attachment.external.DomainExpressionBuilderRegistry, org.apache.cxf.ws.policy.attachment.external.EndpointReferenceDomainExpressionBuilder, org.apache.cxf.ws.policy.attachment.external.URIDomainExpressionBuilder, org.apache.cxf.ws.policy.attachment.wsdl11.Wsdl11AttachmentPolicyProvider, org.apache.cxf.ws.policy.mtom.MTOMAssertionBuilder, org.apache.cxf.ws.policy.mtom.MTOMPolicyInterceptorProvider]maj 20, 2019 8:27:06 PM org.apache.cxf.bus.blueprint.NamespaceHandlerRegisterer registerINFO: Registered blueprint namespace handler forhttp://cxf.apache.org/transports/http/configuration <http://cxf.apache.org/transports/http/configuration>maj 20, 2019 8:27:06 PM org.apache.cxf.bus.blueprint.NamespaceHandlerRegisterer registerINFO: Registered blueprint namespace handler forhttp://cxf.apache.org/blueprint/simple <http://cxf.apache.org/blueprint/simple>maj 20, 2019 8:27:06 PM org.apache.cxf.bus.osgi.CXFExtensionBundleListener addExtensionsINFO: Adding the extensions from bundle org.apache.cxf.cxf-rt-frontend-jaxws (204) [org.apache.cxf.jaxws.context.WebServiceContextResourceResolver]maj 20, 2019 8:27:06 PM org.apache.cxf.bus.blueprint.NamespaceHandlerRegisterer registerINFO: Registered blueprint namespace handler forhttp://cxf.apache.org/blueprint/jaxws <http://cxf.apache.org/blueprint/jaxws>maj 20, 2019 8:27:06 PM org.apache.cxf.bus.blueprint.NamespaceHandlerRegisterer registerINFO: Registered blueprint namespace handler forhttp://cxf.apache.org/blueprint/jaxrs <http://cxf.apache.org/blueprint/jaxrs>maj 20, 2019 8:27:06 PM org.apache.cxf.bus.blueprint.NamespaceHandlerRegisterer registerINFO: Registered blueprint namespace handler forhttp://cxf.apache.org/blueprint/jaxrs-client <http://cxf.apache.org/blueprint/jaxrs-client>maj 20, 2019 8:27:06 PM org.apache.cxf.bus.blueprint.NamespaceHandlerRegisterer registerINFO: Registered blueprint namespace handler forhttp://cxf.apache.org/binding/coloc <http://cxf.apache.org/binding/coloc>maj 20, 2019 8:27:06 PM org.apache.cxf.bus.osgi.CXFExtensionBundleListener addExtensionsINFO: Adding the extensions from bundle org.apache.cxf.cxf-rt-transports-local (216) [org.apache.cxf.transport.local.LocalTransportFactory]maj 20, 2019 8:27:06 PM org.apache.cxf.bus.osgi.CXFExtensionBundleListener addExtensionsINFO: Adding the extensions from bundle org.apache.cxf.cxf-rt-bindings-object (217) [org.apache.cxf.binding.object.ObjectBindingFactory]maj 20, 2019 8:27:06 PM org.apache.cxf.bus.blueprint.NamespaceHandlerRegisterer registerINFO: Registered blueprint namespace handler forhttp://cxf.apache.org/blueprint/binding/object <http://cxf.apache.org/blueprint/binding/object>maj 20, 2019 8:27:06 PM org.apache.cxf.bus.blueprint.NamespaceHandlerRegisterer registerINFO: Registered blueprint namespace handler forhttp://cxf.apache.org/policy <http://cxf.apache.org/policy>maj 20, 2019 8:27:06 PM org.apache.cxf.bus.blueprint.NamespaceHandlerRegisterer registerINFO: Registered blueprint namespace handler forhttp://www.w3.org/ns/ws-policy <http://www.w3.org/ns/ws-policy>maj 20, 2019 8:27:06 PM org.apache.cxf.bus.blueprint.NamespaceHandlerRegisterer registerINFO: Registered blueprint namespace handler forhttp://www.w3.org/2006/07/ws-policy <http://www.w3.org/2006/07/ws-policy>maj 20, 2019 8:27:06 PM org.apache.cxf.bus.blueprint.NamespaceHandlerRegisterer registerINFO: Registered blueprint namespace handler forhttp://schemas.xmlsoap.org/ws/2004/09/policy <http://schemas.xmlsoap.org/ws/2004/09/policy>maj 20, 2019 8:27:06 PM org.apache.cxf.bus.blueprint.NamespaceHandlerRegisterer registerINFO: Registered blueprint namespace handler forhttp://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd <http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd>maj 20, 2019 8:27:06 PM org.apache.cxf.bus.blueprint.NamespaceHandlerRegisterer registerINFO: Registered blueprint namespace handler forhttp://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd <http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd>maj 20, 2019 8:27:06 PM org.apache.cxf.bus.blueprint.NamespaceHandlerRegisterer registerINFO: Registered blueprint namespace handler forhttp://www.w3.org/2000/09/xmldsig# <http://www.w3.org/2000/09/xmldsig#>maj 20, 2019 8:27:06 PM org.apache.cxf.bus.blueprint.NamespaceHandlerRegisterer registerINFO: Registered blueprint namespace handler forhttp://docs.oasis-open.org/ws-sx/ws-securitypolicy/200702 <http://docs.oasis-open.org/ws-sx/ws-securitypolicy/200702>maj 20, 2019 8:27:06 PM org.apache.cxf.bus.osgi.CXFExtensionBundleListener addExtensionsINFO: Adding the extensions from bundle org.apache.cxf.cxf-rt-ws-addr (237) [org.apache.cxf.ws.addressing.policy.AddressingAssertionBuilder, org.apache.cxf.ws.addressing.policy.UsingAddressingAssertionBuilder, org.apache.cxf.ws.addressing.policy.AddressingPolicyInterceptorProvider, org.apache.cxf.ws.addressing.impl.AddressingWSD...
Gosforth Kubu's profile image
Gosforth Kubu

When I try to launch example 'build_models' transformation:

2019/05/20 21:14:42 - Spoon - Using legacy execution engine

2019/05/20 21:14:42 - Spoon - Transformation opened.2019/05/20 21:14:42 - Spoon - Launching transformation [build_models]...2019/05/20 21:14:42 - Spoon - Started the transformation execution.2019/05/20 21:14:42 - build_models - Dispatching started for transformation [build_models]2019/05/20 21:14:43 - PMI Gradient Boosted Trees (Python).0 - ERROR (version 8.2.0.0-342, build 8.2.0.0-342 from 2018-11-14 10.30.55 by buildguy) : 2019/05/20 21:14:43 - PMI Gradient Boosted Trees (Python).0 - Engine not available: Python - Scikit-learn

2019/05/20 21:14:43 - PMI Gradient Boosted Trees (Python).0 - ERROR (version 8.2.0.0-342, build 8.2.0.0-342 from 2018-11-14 10.30.55 by buildguy) : org.pentaho.di.core.exception.KettleException:

2019/05/20 21:14:43 - PMI Gradient Boosted Trees (Python).0 - Engine not available: Python - Scikit-learn

2019/05/20 21:14:43 - PMI Gradient Boosted Trees (Python).0 -

2019/05/20 21:14:43 - PMI Gradient Boosted Trees (Python).0 -     at org.pentaho.di.trans.steps.pmi.BaseSupervisedPMIStep.init(BaseSupervisedPMIStep.java:107)2019/05/20 21:14:43 - PMI Gradient Boosted Trees (Python).0 -     at org.pentaho.di.trans.step.StepInitThread.run(StepInitThread.java:69)2019/05/20 21:14:43 - PMI Gradient Boosted Trees (Python).0 -     at java.lang.Thread.run(Thread.java:748)2019/05/20 21:14:43 - PMI Gradient Boosted Trees (Python).0 - ERROR (version 8.2.0.0-342, build 8.2.0.0-342 from 2018-11-14 10.30.55 by buildguy) : Error initializing step [PMI Gradient Boosted Trees (Python)]2019/05/20 21:14:43 - build_models - ERROR (version 8.2.0.0-342, build 8.2.0.0-342 from 2018-11-14 10.30.55 by buildguy) : Step [PMI Gradient Boosted Trees (Python).0] failed to initialize!2019/05/20 21:14:43 - Spoon - ERROR (version 8.2.0.0-342, build 8.2.0.0-342 from 2018-11-14 10.30.55 by buildguy) : build_models: preparing transformation execution failed2019/05/20 21:14:43 - Spoon - ERROR (version 8.2.0.0-342, build 8.2.0.0-342 from 2018-11-14 10.30.55 by buildguy) : org.pentaho.di.core.exception.KettleException: 2019/05/20 21:14:43 - Spoon - We failed to initialize at least one step.  Execution can not begin!2019/05/20 21:14:43 - Spoon - 2019/05/20 21:14:43 - Spoon - 2019/05/20 21:14:43 - Spoon -     at org.pentaho.di.trans.Trans.prepareExecution(Trans.java:1272)2019/05/20 21:14:43 - Spoon -     at org.pentaho.di.ui.spoon.trans.TransGraph$30.run(TransGraph.java:4287)2019/05/20 21:14:43 - Spoon -     at java.lang.Thread.run(Thread.java:748)2019/05/20 21:14:43 - build_models - ERROR (version 8.2.0.0-342, build 8.2.0.0-342 from 2018-11-14 10.30.55 by buildguy) : Errors detected!2019/05/20 21:14:43 - build_models - ERROR (version 8.2.0.0-342, build 8.2.0.0-342 from 2018-11-14 10.30.55 by buildguy) : Errors detected!

Mark Hall's profile image
Mark Hall

OK, this is an improvement - you only have one engine unavailable now :-) Is the python executable in your PATH? Did you install Anaconda system wide or just for your user? In the past, there have been some issues with system-wide installations of python (something to do with file permissions and not due to PMI/Weka as such).

Cheers,

Mark.

Gosforth Kubu's profile image
Gosforth Kubu

Yes. I added two variables to user PATH:

C:\Users\xxxx\AppData\Local\Continuum\anaconda3

C:\Program Files\R\R-3.6.0\bin\x64

This time I installed (another PC) Anaconda only for user.

Still error 'The following engines are unavailable (check configuration): Python - Scikit-learn'

Mark Hall's profile image
Mark Hall

Argh! It appears that Anaconda have complicated things in their most recent release. It used to be the case that the installer would add a single entry to your PATH variable for you automatically (and this could be seen when viewing/editing environment variables in System Properties). This is no longer the case and, in fact, it doesn't seem to add any entries to the PATH now! Instead, you are expected to use an "Anaconda prompt" instead of just a a "Command prompt" when using python from a console. As you can see in my screenshot, when you start an "Anaconda prompt" it adds a whole bunch of (temporary) entries to your PATH. You will need to copy these entries and add them to the PATH via System Properties/Environment variables. After I did this, the python engine (and the example transformation) was fully operational.

Cheers,

Mark.screen-shot-2019-05-23-at-12

Gosforth Kubu's profile image
Gosforth Kubu

Did not help at all. Still " The following engines are unavailable (check configuration): Python - Scikit-learn"

What I added to Paht for user (not system):

C:\Users\SomeUser\AppData\Local\Continuum\anaconda3;

C:\Users\SomeUser\AppData\Local\Continuum\anaconda3\Library\mingw-w64\bin;C:\Users\SomeUser\AppData\Local\Continuum\anaconda3\Library\usr\bin;C:\Users\SomeUser\AppData\Local\Continuum\anaconda3\Library\bin;C:\Users\SomeUser\AppData\Local\Continuum\anaconda3\Scripts;C:\Users\SomeUser\AppData\Local\Continuum\anaconda3\bin;C:\Users\SomeUser\AppData\Local\Continuum\anaconda3\condabin;C:\Program Files (x86)\Common Files\Oracle\Java\javapath;C:\Program Files (x86)\Intel\iCLS Client\;C:\Program Files\Intel\iCLS Client\;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\IPT;C:\Program Files\Intel\Intel(R) Management Engine Components\IPT;C:\WINDOWS\System32\OpenSSH\;C:\Users\SomeUser\AppData\Local\Microsoft\WindowsApps;C:\Users\SomeUser\AppData\Local\Continuum\anaconda3;C:\Program Files\R\R-3.6.0\bin\x64;

Mark Hall's profile image
Mark Hall

Hmm. I'm running out of suggestions. I figured that I'd reproduced your problem with the latest version of Anaconda python, and adding the additional paths got things working for me again under Windows 8. Can you verify that python starts and that various packages can be imported from a Command prompt (not an Anaconda prompt) - i.e:

python

>>> import socket

>>> import struct

>>> import os

>>> import json

>>> import base64

>>> import math

>>> import traceback

>>> import numpy as np

>>> import pandas as pd

>>> from sklearn import *

>>> import matplotlib.pyplot as plt

>>> from io import StringIO

>>> from io import BytesIO

Which version of PDI are you running?

Cheers,

Mark.

Gosforth Kubu's profile image
Gosforth Kubu

Hi Mark,

thanks to your patience.

I think this is because of this crap conda or anaconda... you name it.

When I run Python from command line:

C:\Users\SimeUser>python

Python 3.7.3 (default, Mar 27 2019, 17:13:21) [MSC v.1915 64 bit (AMD64)] :: Anaconda, Inc. on win32

Warning:

This Python interpreter is in a conda environment, but the environment hasnot been activated.  Libraries may fail to load.  To activate this environmentplease see https://conda.io/activation

Type "help", "copyright", "credits" or "license" for more information.

>>>

This sh... requires some 'activation'. God only knows for what. But anyway it does not work after 'activation'.Managing environments — conda 4.7.0.post7+18c4e5dc documentation I tried 'conda activate path-to-this-s..'Executed without any error but changed nothing.So I tried another trick according what they say:conda create -n py37 python=3.7.3 anacondaactivate py37Again, it exectued without problem but did nothing.

Maybe this 'activation' should be in Pentaho startup script?

Win restart changes nothing... cries that is not activated.

I'm runing PDI 8.2

Regards

Gosforth Kubu's profile image
Gosforth Kubu

I made another try. When I type 'activate myenv' im command prompt it starts conda. Then 'python' - still OK. I type 'import sklearn' - still OK.

But Pentaho plugin says 'Engine not available: Python - Scikit-learn'.

Conclusion; this plugin if was intended to work with this condasomething must be able to run 'activate ..' during startup. If it is not able to activate this condasometing env - it will never work.

I still believe this conda software is just crap. Plugin should work with pure Python or/and R.

Mark Hall's profile image
Mark Hall

As I said earlier, PMI's python engine is not compatible with python virtual environments (as managed by conda). I don't use conda - just python directly from the Anacoda prompt (and standard Command prompt once the necessary entries to the PATH variable were added). When I tested using the latest version of Anaconda recently, I did not do *anything* beyond installing Anaconda (locally for my user) and ascertaining which entries were required in the PATH and adding those via System properties/Environment variables.

Cheers,

Mark.

Data Conversion's profile image
Data Conversion
Attachment  View in library
66739.pdf 281 KB