Troubleshooting HNAS Not Finding a Suitable Domain Controller After Successful EVS Domain Join

Issue Description

During the implementation of VSP One File, the newly created EVS gets successfully joined to the Microsoft Active Directory (AD) Domain Controller. However, the Hitachi NAS (HNAS) system was unable to find a suitable Domain Controller (DC).

Environment

• Product: VSP One File, HNAS
• Components: EVS, Microsoft AD Domain Controller
• Version: 0.1

Symptoms

• EVS successfully joins the AD Domain Controller via cifs-name add without reporting any errors.
• A computer object corresponding to the EVS gets created in the AD Domain Controller as well.
• A trust account can also be seen with full control within the security settings of the computer object.
• HNAS fails to locate a suitable DC while the DNS servers are configured properly.
• cifs-dc get shows "Failed to connect to SAMR Pipe" followed by "No connection established"
• cifs-dc-errors shows increase in error counts every 5 seconds.
• cifs-dc list -v showed "Protocol Errors" with extremely high latencies.

Troubleshooting Steps

1. Verify Network Configuration
• Ensure that the HNAS and the AD Domain Controller are on the same network or have proper routing between them.
• Check for any network firewalls that might be blocking communication.
2. Check DNS Settings
• Verify that the DNS settings on the HNAS are correctly configured to point to the AD DNS servers.
3. Examine cifs-dc commands
• Review the "cifs-dc list -v" for error messages related to domain controller discovery.
• Look for specific error codes in "cifs-dc-errors"
4. Test Connectivity
• Use tools like ping and tracert to test connectivity between the HNAS and the AD Domain Controller.
• Ensure that there are no intermittent network issues affecting communication.
5. Packet Capture
• Initiate a packet capture
• Trigger a DC cache refresh from "cifs-dc prod" command. This will force HNAS to re-trigger the process to look for a suitable domain controller.
• Keep issuing "cifs-dc get" every second and observe the changing states. It is important to issue the command continuously until you get "No connection established" since the error changes in this command output.
• Stop the packet capture and analyze it.

SMB2	192	Session Setup Request
		###### Logs Truncated######
SMB2	198	Create Request File: samr
SMB2	222	Create Response File: samr
DCERPC	262	Bind: call_id: 876137, Fragment: Mid, 1 context items: SAMR V1.0 (32bit NDR)
DCERPC	250	Bind_ack: call_id: 876137, Fragment: Single, max_xmit: 4280 max_recv: 4280, 1 results: Acceptance
SAMR	270	Connect4 request
DCERPC	214	Fault: call_id: 876138, Fragment: Single, Ctx: 0, status: nca_s_fault_access_denied
TCP	66	62466 → 445 [FIN, ACK] Seq=4717 Ack=3373 Win=262140 Len=0 TSval=1900203217 TSecr=2120687154
TCP	66	445 → 62466 [ACK] Seq=3373 Ack=4718 Win=2096640 Len=0 TSval=2120687155 TSecr=1900203217
TCP	60	445 → 62466 [RST, ACK] Seq=3373 Ack=4718 Win=0 Len=0
		###### Logs Truncated######
SMB2	326	Session Setup Response
		###### Logs Truncated######
RPC_NETLOGON	218	NetrServerReqChallenge response
RPC_NETLOGON	342	NetrServerAuthenticate2 request
RPC_NETLOGON	222	NetrServerAuthenticate2 response
DCERPC	295	Alter_context: call_id: 876142, Fragment: Single, 1 context items: RPC_NETLOGON V1.0 (32bit NDR)
DCERPC	258	Alter_context_resp: call_id: 876142, Fragment: Single, max_xmit: 4280 max_recv: 4280, 1 results: Acceptance
RPC_NETLOGON	390	NetrLogonGetCapabilities request
DCERPC	214	Fault: call_id: 876143, Fragment: Single, Ctx: 0, status: nca_s_fault_sec_pkg_error

• The nca_s_fault_access_denied is observed corresponding to "Failed to connect to SAMR pipe" in cifs-dc get output whereas nca_s_fault_access_denied is observed along with "No connection established."

 

Resolution

Both the errors in packet capture are the responses being received from the Domain Controller. The errors being seen, nca_s_fault_access_denied, typically indicates that the server is being denied access due to insufficient permissions or incorrect configurations.

The customer needs to validate their AD environment and confirm that SAMR is working properly. To resolve this, they can:

1.       Verify that the file server's account has the necessary permissions in Active Directory.

2.       Ensure that the account is not restricted by any security policies.

3.       Check the firewall and network settings to ensure that SAMR traffic is not being blocked.

Scenario #2 applied to our case where there existed a Security policy "Network access: Restrict clients allowed to make remote calls to SAM". To resolve the issue, the AD support team needed to explicitly allow the HNAS trust account in the "Network access: Restrict clients allowed to make remote calls to SAM" policy. After making this change, the HNAS was able to locate a suitable Domain Controller with no issues. There were no errors observed in cifs-dc errors either.

Additional Information

Network access - Restrict clients allowed to make remote calls to SAM - Windows 10 | Microsoft Learn

Below is the basic Kerberos authentication process in a setup involving a file server, a Windows end user, and a Windows AD domain controller. Here is an overview of how the authentication flow works:

1.       User Login: The end user logs into their Windows machine using their credentials (username and password).

2.       Initial Authentication Request: The user's machine sends an authentication request to the Key Distribution Center (KDC) on the AD domain controller. This request is for a Ticket Granting Ticket (TGT).

3.       KDC Response: The KDC verifies the user's credentials against the Active Directory. If the credentials are valid, the KDC issues a TGT, which is encrypted with the user's password hash and sent back to the user's machine.

4.       Service Ticket Request: When the user tries to access the file server, their machine uses the TGT to request a Service Ticket from the KDC. This request includes the TGT and the Service Principal Name (SPN) of the file server.

5.       Service Ticket Issuance: The KDC verifies the TGT and issues a Service Ticket for the file server. This ticket is encrypted with the file server's secret key and sent back to the user's machine.

6.       Accessing the File Server: The user's machine presents the Service Ticket to the file server. The file server decrypts the ticket using its secret key and verifies the user's identity.

7.       Access Granted: If the Service Ticket is valid, the file server grants the user access to the requested resources.

This process ensures secure and efficient authentication without repeatedly asking the user for their credentials.

 

How does the file server get its secret key in the first place?

The file server gets its secret key through the process of joining the Active Directory (AD) domain.

1.       Joining the Domain: When the file server is joined to the AD domain, it registers itself with the domain controller. During this process, the server's account is created in the AD database.

2.       Key Generation: The domain controller generates a secret key for the file server. This key is derived from the server's password, which is stored securely in the AD database.

3.       Key Storage: The secret key is stored on the file server and is used for decrypting Kerberos tickets issued to it. The key is also stored in the AD database, allowing the Key Distribution Center (KDC) to encrypt tickets for the file server.

This secret key is crucial for the Kerberos authentication process, as it ensures that only the intended server can decrypt and validate the service tickets issued to it.

The Security Account Manager Remote (SAMR) protocol is used for managing user and group information on Windows systems. Here is a breakdown of its operations and how they fit into our scenario:

1.       SAMR Protocol: SAMR is a protocol used by Windows systems to perform operations related to user and group management. It allows remote clients to manage user accounts, groups, and other security-related information on a Windows server.

2.       Common SAMR Operations:

•                          Connect: Establishes a connection to the SAMR server.

•                          Enumerate Domains: Lists all domains available on the server.

•                          Lookup Domain: Retrieves information about a specific domain.

•                          Enumerate Users: Lists all user accounts in a domain.

•                          Create User: Creates a new user account.

•                          Delete User: Deletes an existing user account.

•                          Enumerate Groups: Lists all groups in a domain.

•                          Add Member to Group: Adds a user to a group.

•                          Remove Member from Group: Removes a user from a group.

3.       Access Control: SAMR operations require appropriate permissions. If a client does not have the necessary permissions, it will receive an access denied error, as seen in our packet capture.

4.       Usage in our Scenario: When the file server attempts to perform SAMR operations, it needs to authenticate and have the necessary permissions to access and modify user and group information on the domain controller. The nca_s_fault_access_denied error suggests that the file server's account might not have the required permissions to perform the requested operation.

 

 

------------------------------
Abhishek Saxena
Hitachi Vantara
------------------------------