Hitachi Content Platform​

 Multiple pipelines

  • Object Storage
  • Hitachi Content Intelligence HCI
Alison Apel's profile image
Alison Apel posted 11-08-2018 17:42

Would you please take a look at the following animation sequence:

https://hcpanywhere.hitachivantara.com/u/zEWvE6NFr1DeRa8O/HDID%20Content%20Intelligence%20Animation%20V2.mp4?l

and let me know if it accurately reflects what creating multiple pipelines is? This is for a video.
#HitachiContentIntelligenceHCI
Benjamin Isherwood's profile image
Benjamin Isherwood

Mostly.  When multiple pipelines are added to a workflow, they are executed sequentially. This allows pipelines to be reused effectively across workflows for different types of processing. To execute multiple parallel pipelines, you run multiple workflows.

-Ben

Data Conversion's profile image
Data Conversion

The animation looks like two pipeline are executed in parallel, with some documents going through the top pipeline, and others through the bottom one.

That is not the case. All documents flow through all pipelines, sequentially.

Alison Apel's profile image
Alison Apel

How would we do that? Two lines, one for each pipe? Or something different?

Alison Apel's profile image
Alison Apel

Is it more like this (see attachment)pipes-example-2

Alison Apel's profile image
Alison Apel

Or is it more like this?pipes-example-3

Jared Cohen's profile image
Jared Cohen

Pipelines are executed sequentially, i.e. a document exiting the first pipeline will then enter the next pipeline at the start.

Input -> Pipeline 1 -> Pipeline 2 -> ... -> Pipeline N -> Output

If you want to illustrate it with pipelines on multiple lines, I would recommend drawing arrows from the end of one pipeline to the beginning of the next on the next line.

Hope this helps,

-Jared

Troy Myers's profile image
Troy Myers

I like the webex but think this is more terminology issue, of Workflows versus pipelines.  Have seen/watched the Webexes we created, link below to the library of them.   Several go into specific pipelines and stages, along with the workflow settings. 

HCI Webexs and Demos

HCI workflow settings-20171226 1359-1.mp4

Alison Apel's profile image
Alison Apel

When you say all documents flow through all pipelines, do you mean you can have multiple pipelines with different stages.? In other words, Pipeline #1 HCP>Content Class Extraction>Snippet Extraction>Filter>Content Intelligence. Pipeline #2 HCP->MIME Type Detection>PST Expansion>Mbox Expansion>Content Intelligence. Pipeline #3 HCP>Zip Expansion>Tar Expansion>Date Conversion>Content Intelligence?

Data Conversion's profile image
Data Conversion

Yes, you can have multiple pipelines with different stages.

Alison Apel's profile image
Alison Apel

Yury - is this the correct diagram?

Data Conversion's profile image
Data Conversion

I don’t quite understand what little gray thingies represent. Are they pipelines or stages within a pipeline?