Nichole Schmitt

UCP Director (3.5, 4.0, 4.1.1)- Jobs, Events, Logs - Use CLI to help you gather intel on a failed job.

Blog Post created by Nichole Schmitt Employee on Feb 17, 2016

Hi All, this is my first attempt at blogging. This is a section from a handy UCP Notebook I helped author last year.

 

These tips are applicable to UCP 3.5, 4.0, 4.1.1. Note, however, an improvement in UCP 4.1.1 - It has its own Jobs page! Now in SCVMM you can see the events while the job is still running. In previous versions of UCP for Hyper-V, you had to wait for the job to complete before seeing the events. Otherwise, you'd have to use the CLI to watch the events while the job is running.

 

Jobs – Events – Logs

Architectural Overview

Orchestrator

On UCPManagement VM, all IIS services report to API trace logs. Logs are located in C:\Program Data\Hitachi\UCP\Logs.  (Program Data is ‘Hidden’. Use the View menu to unhide.)

The log is regularly archived, so finding which zip file contains a particular failure is a key trouble-shooting step.

UCP Jobs

Jobs are comprised of Events. They are viewable through:

  • Platform manager (SCVMM\vCenter) Jobs or Tasks panel
  • CLI
  • API

The events from every job are also contained in the Orchestrator log along with much more detail about what UCP was doing to accomplish the job. Finding the job’s stack trace in the Orchestrator log helps in troubleshooting.

Monitoring

The Monitoring service collects information from storage, switches, and servers. The details are reported in the UCPMonitor_APITrace.log and the aggregated health\performance status is reported to UCP.

Replication

When installed, the replication service runs at each of the paired UCP sites. Replication activity is reported in the ReplicationManager_APITrace.log

Scheduler

Scheduled tasks are reported here. For example, inventory refrshes.

Trace32

A third party tool recommended for viewing logs.

 

The following diagram displays the list of application pools in IIS on the UCPManagement VM.

  Each has its own log.

 

 

Exercise – A – Gather events from failed job

This exercise guides you through the following:

 

Goal #1:

Identify the failing job in the platform manager.

Goal #2

Get events from the failed job into a single text file.

Goal #3

Find the Orchestrator or Monitor log(s) containing the failed job.

 

 

View the Jobs in vCenter

If the job has not already been purged, find it in the Tasks and Events panel as shown below:

 

 

This tells us the string to search for in the Orchestrator logs. For example: “Unexpected error with error code: 5057 occurred on the device manager.”

Use the attached tool "ExceptionFinder" to find the Orchestrator log that contains the exception string

 

 

Viewing Jobs in SCVMM

To view the jobs panel in SCVMM:

 

Click Jobs in the left-hand panel, at the bottom:

 

  1. At the top, click History:

 

Scroll through the jobs till you see the failure:

 

 

 

 

Here we see the exact string to search for in the Orchestrator logs. The following is just one example:

“exception: 11023395”

 

Use the attached tool "ExceptionFinder" to find the Orchestrator log that contains the exception string.


 

Using Hitachi.UCP.CLI to gather events

Since some of the tasks must be done on the UCPManagement VM (log-gathering, for example), you can go ahead and run all of the following tasks from there. So launch a remote-desktop session to the UCPManagement VM.

The Hitachi CLI is already installed on the UCPManagement VM. But if you need to install it in some other location (like the serviceVM), see subheading “Supplemental – Downloading and Installing CLI”.

To establish a connection with UCP (“Get-UCPDirector cmdlet) you need to launch PowerShell with ucpadmin credentials. This will happen automatically if you’ve logged into the system as ucpadmin.

After opening PowerShell with ucpadmin credentials:

  1. Load the UCP CLI module with:

Import-module Hitachi.ucp.cli

  1. Connect to UCP Director with CLI session

Get-UCPDirector Ucpmanagement.domain.local

  1. If there aren’t too many failed jobs to parse through, you could use the following. But note, if the appliance has MANY failed jobs, this could take too long and bog down the system:

Get-UCPJob –JobStatus Error

  1. Get the ID of an object associated with the failed job (Cluster Template? Server? Switch?)

Get-UCPServer,

Get-UCPEthernetSwitch,

Get-UCPAllServiceTemplate,

(For more ideas, type: Get-Help Get-UCP*)

  1. Get jobs associated with that object ID. There are many ways to do this. To read more about it, do get-help get-ucpjob –full.

(See next page for some examples.)

Examples:

These examples help you find the failed job ID.

 

All failed jobs where the job Name starts with “Apply”:

get-ucpjob -JobStatus error | where {$_.Name -like "Apply *"} > c:\temp\FailedDeploymentJobs.txt

All failed jobs where a particular service template failed deployment:

get-ucpjob -JobStatus error | where {$_.Name -like "Apply *"} > c:\temp\FailedDeploymentJobs.txt

 

Or:

 

$t = Get-UCPLinuxServiceTemplate -ServiceTemplateId 83

 

get-ucpjob -JobStatus error | where {$_.TargetId -eq $t.GlobalResourceId}

 

All failed jobs where the type is TemplateApply and Status is Error

Get-UCPJob -JobType TemplateApply | where {$_.Status -eq "Error"}

Get job where user knows the target server is in chassis slot 0 and the job type was service template deployment (only one chassis in this environment):

$s=get-ucpserver | where {$_.PrimarySlot -eq 0}

Get-UCPEvent -TargetId $s.GlobalResourceId | where {$_.EventType -eq "ServerApplyTemplateStartedInfoEvent"}

Get all failed jobs for a given server, when the server’s Id is known:

$s=Get-UCPServer | where {$_.ServerId -like "*c6bf"}

get-ucpjob -JobStatus error | where {$_.TargetId -eq $s.GlobalResourceId}

 

  1. With the known job ID, collect the events from it and output to a text file which you will provide to the engineer who is supporting your help ticket.

The following command is a single line of text:

Get-UCPEvent -JobId 10163 | Select-Object EventId, Description, Severity | Format-Table -Autosize > c:\temp\Job10163.txt

Find the log with ExpnFinder:

To find the Orchestrator log(s) that contain the failed job. Use ExpnFinder tool as follows:

  1. Copy the ExpnFinder tool to c:\Program Data\Hitachi\UCP\Logs on the UCPManagement VM
  2. In powerShell, navigate to the above location and launch the script as follows:
  3. PS C:\ProgramData\Hitachi\UCP\logs\ExpnFinder> .\ExpnFinder.ps1
  4. The script asks: Enter the exception no. please- such as 1234567:
  5. Type:  JobId=<xxxx> (Example: JobId=4092)
  6. The output tells you which log(s) contain the failed job. This\these should be sent to UCP Engineering if the problem needs escalation.

View stack trace in Orchestrator log

Trace32 is the most popular log-viewing tool in UCP Engineering, but you can use other.

  1. After identifying which log contains your error, open it with Trace32
  2. In Tools \ Filter, enable the first Filter, choose ‘contains’ as criteria.
  3. If you don’t know the job ID yet, type some text from the error event as seen in the Job or Task panel in the platform manager. Click OK.
  4. In the filtered results, look for JobId=<ID>
  5. Filter again for text ‘containing’ JobId=<id>. Click OK.
  6. The Log text changes in the main panel to display only the activity related to that JobId

 

 

Supplemental – Download and Install CLI

If the UCP CLI is not installed on the machine you’re using, download and install it as follows:

  1. Open a browser to https://<ucpmanagement.domain.local>/ui.
  2. Then go to the Help menu and click Download CLI.
  3. Find the executable (usually in C:\users\MyName\Downloads)
  4. Double click the executable file: UCP.CLI.Installer.msi

 

  1. Launch PowerShell with ucpadmin credentials. You must either be logged into the workstation with the ucpadmin domain account, or do one of the following:
    1. In Windows Server 2012, navigate to the location of the PowerShell exe at C:\Windows\System32\WindowsPowerShell\v1.0. Hold and right-click the .exe, then choose “Run as different user.”

 

    1. In Windows Server 2008, the PowerShell icon is on the task bar. Shift + Right-click the icon, choose “Run as different user” and input the domain credentials for ucpadmin.

 

  If you frequently use Hitachi.UCP.CLI from a workstation that is not joined to the same domain as UCP, you could add a connection in your Credential Manager. Click the Start button and search for Credential Manager. Add an entry where the IP is UCPManagement IP address and the credentials are the domain account for ucpadmin. Now, simply launch PowerShell (without the ‘run as’). When you do Get-UCPDirector, the connection will be established with the ucpadmin account.

Attachments

Outcomes