Hitachi Content Platform​

 View Only

HCI to audit HCP access and internal logs

By Hein Beeck posted 09-11-2020 07:54

Use Case:  Customer wants to be able to track who accessed when a specific tenant and did what action for whatever object.

This customer has no HCM instance deployed.  If HCM would have been deployed, the procedure would be even more simple as HCM is already collecting all syslog information from HCP.  HCM is not making use of the data we need as HCM will only visualise performance metrics and will drop the audit data we need.

If no HCM is installed, configure syslog of all HCP's to be audited to send syslog output to the HCI server on port 6901, actually the same as you would do for HCM (logstash is listening on port 9601) 
If HCM is deployed, you don't need to do this step as you can consume the data directly from the HCM instance.

Data connector:
In HCI, create a data connection for every HCP to be monitored. Use the 'HCP Syslog Kafka Queue' as data connector.

Enter the following details:
1/ Kafka server:
-if HCM is deployed on enter the IP of the HCM instance, port 9092
-if no HCM is installed, enter the IP of the actual HCI instance you deployed, port 9092

2/ Kafka Topic is 'syslog-queue'  <-- data will be consumed from here

3/ the HCP system name  (as highlighted, you need to make a data connection per HCP to be audited)

The pipeline is very simple, pick the HCP syslog parser. After this stage, drop entries which are not required, depending on the use case.  By default all internal messages like all HCP services will be reported here as well (for instance garbage collection starting, stopping etc.) 

In my case, the customer is not willing to see this internal info in the search index AND only want entries for 1 tenant, in this example, the tenant is hcpgateway.

example of my drop stage:

  • Input: data connections of 3 HCP's to be audited
  • pipeline: pipeline with the HCP syslog parser & drop document stages
  • output: index (see below)
In the edit settings, enable 'check for updates' in document discovery, I gave it an interval of 1800 seconds to fetch updates

sample of my index config:
1/ disable 'show HCI_id field', this field doesn't bring value in the search index for this use case (new option to disable this in HCI 1.6.3)


  • only 2 HCP's are shown as the hcpgateway tenant is only on 2 HCP's (replicated) out of the 3.
  • no internal housekeeping items are listed on the solr index as they are dropped as well while processing the flow.
  • easy to drill down based upon the search criteria.




04-26-2022 13:34

Very informative

09-11-2020 11:10

Great Work!