Network Attached Storage​

 View Only

  • 1.  Why does HNAS need SMB-Client-Barring whereas other products in Market do not seem to have a need of the same functionality?

    Posted 05-16-2023 14:50

    HNAS 13.9 introduced SMB-client-barring, a list where clients are automatically added if they generated enough unauthenticated requests to HNAS. Why is this functionality needed?

    Other NAS products out there do not seem to have a need of similar blocking in place.
     



    ------------------------------
    Abhishek Saxena
    Systems Engineer
    S&P Global Market Intelligence
    ------------------------------


  • 2.  RE: Why does HNAS need SMB-Client-Barring whereas other products in Market do not seem to have a need of the same functionality?

    Posted 05-16-2023 16:36
    Hi

    I have the same question. Why is this really needed ? ( security ? , performance ?... maybe some sites
    have so much bad pw activity that it consumes resources and saps performance ? )

    Last month I wasted a day battling a "mystery problem" where all users on an important host
    ( a large Terminal server ) were un-able to access the NAS.

    The cause of the "mystery problem" ( partial service outage ) was the new on-by-default "auto barring" feature.

    Most of the auto-bars I have seen have been legit ( broken clients ...etc ) ;however,
    it appears that multi-user terminal servers are frequent accidental ( not legit ) auto-bars.

    Currently, I believe that the trigger for barring is:
    "If , from a given IP Address, there are 21 bad logon attempts within 2 seconds,
    THEN, that IP Address will be barred. "

    The thresholds of this can be tuned ; however, this can be problematic since
    de-sensitizing "auto-barring" to prevent non-legit auto-bars also
    prevents legit "auto-bars".

    The "auto-bar" feature needs a "do not bar" list .
    It should be possible to add the following to the "do not bar" list
    - IP Addresses
    -IP Address ranges ( 142.205.0.0/16 .... etc )
    - host names

    The "auto-bar" feature also needs an auto-clear feature so that when the client-side issue is
    resolved the bar is removed after N hours. ( the NAS admin should not have to babysit
    this feature and be forced to manually remove the "bars" )

    There may be an actual good reason for implementing this feature; however,
    having it active-by-default was probably not a good idea since it results
    in , partial, service outages. As currently implemented the feature causes some disruption
    and causes some extra work for NAS admins.

    Thanks

    Andy






    >
    Original Message


  • 3.  RE: Why does HNAS need SMB-Client-Barring whereas other products in Market do not seem to have a need of the same functionality?

    Posted 05-16-2023 16:45

    Also HNAS seems to be the only NAS solution out there implementing this. We have several DELL EMC NAS, along with few NetApps and Nasuni NAS. None of them seems to have this feature.
    I understand there could be technical reasons behind this implementation with HNAS, but other systems seem to be handing this with a better implementation. Not sure how though?



    ------------------------------
    Abhishek Saxena
    Systems Engineer
    S&P Global Market Intelligence
    ------------------------------

    Original Message


  • 4.  RE: Why does HNAS need SMB-Client-Barring whereas other products in Market do not seem to have a need of the same functionality?

    Posted 06-19-2023 10:34
      |   view attached

    Abhishek, HNAS Product Manager (Patrick Allaire) spent some time on this request and has provided the attached document for your consumption.

    We hope this helps.



    ------------------------------
    Albert Hagopian
    Software Development Engineer - Specialist
    Hitachi Vantara
    ------------------------------

    Attachment(s)

    docx
    SMB Barring Overview.docx   418 KB 1 version
    Original Message


  • 5.  RE: Why does HNAS need SMB-Client-Barring whereas other products in Market do not seem to have a need of the same functionality?

    Posted 06-19-2023 11:57

    Hi  Al,  Hi Patrick

    Thanks for putting together and posting the document about the auto-barring feature.

    We have a set of multi-user terminal-servers that are getting clobbered by auto-barring.
    Auto-barring is itself acting like a DOS attack service.  The common scenario (several times per week ) is a user with a session on terminal server X will , in a session on Laptop Y, change his / her password. The user's session on terminal server X will not posses the new credentials and will , in interactions with EVS E, start generating bad passwords.  EVS E will then bar terminal server X.  All users, not just  the absent minded user, with sessions on terminal server X are denied CIFS/SMB service by EVS E.

    I have attempted / considered these solutions:

    A. Declare that users must never be absent minded. Before changing your password (from your laptop) , log-off of all terminal server sessions , then change your password , then start another terminal server session.  Or change your password from your terminal server session.  This obviously isn't going to work.  ( for the same reason that it is impossible to herd kittens )

    B. Have the terminal-server sys-admin kill all detached terminal server sessions.  This is not possible, as users are running long runtime processes.

    C.  Monitor HNAS logs, identify culprits,  kill their terminal-server sessions and un-bar the barred terminal server.  This is what I am doing now,  and it is  not  effective and it consumes lots of admin time throughout the day. And its reactive,  it does not prevent bars.

    D. Turn off auto-barring .  This could cause HNAS to get barred by AD domain controllers; thus, is not a good option

    IMPORTANT QUESTION:

    User's definitely use Kerberos when logging on to the Terminal Servers. However, based on the barring behavior ,  Kerberos is likely not being used for CIFS/SMB operations.  The document Patrick created appears to imply that if all UNC path references, used by users,  were of the form
    \\server.corp.com\sharename ( rather than \\server\sharename ) , then Kerberos auth would be used for CIFS/SMB operations and ( to make a long story short)  bad credentials used for these CIFS/SMB operations won't cause the computer ( that the user's session is running on ) to be barred by the HNAS.    Please go to the mountain and ask the guru: "Will using exclusively FQDN UNC paths eliminate my terminal-server auto-barring woes ?"

    Thanks 


    Andy

     



    ------------------------------
    Andrew Romero
    Storage Administrator
    Fermi National Accelerator Laboratory
    ------------------------------

    Original Message


  • 6.  RE: Why does HNAS need SMB-Client-Barring whereas other products in Market do not seem to have a need of the same functionality?

    Posted 06-20-2023 08:27

    Andy, I spoke to our HNAS global GSC lead and you presume correctly, FQDN would do it and help you out with option "C".
    (exclusively authenticating SMB clients with Kerberos would eliminate auto-barring woes)

    Caveat emptor - based on other customer escalations, even using FQDN UNC paths still isn't foolproof as some applications 'fall back' to NTLM or are notoriously difficult to configure to use FQDN in.

    Hopefully the information posted by Skip last night with links to the SupportConnect article can be leveraged (and yes, some of the information in that article and in yesterday's info post from PM are the same).

    If you still have some issues with using FQDN/barring, please don't hesitate to opening a case with GSC.



    ------------------------------
    Albert Hagopian
    Software Development Engineer - Specialist
    Hitachi Vantara
    ------------------------------

    Original Message


  • 7.  RE: Why does HNAS need SMB-Client-Barring whereas other products in Market do not seem to have a need of the same functionality?

    Posted 06-19-2023 17:22

    Abhishek,
    There is a document on the Hitachi Vantara Knowledge Portal called HNAS SMB Auto-Barring (https://knowledge.hitachivantara.com/Knowledge/Storage/Network_Attached_Storage/Hitachi_NAS_Platform/HNAS_SMB_Auto-Barring.

    That document goes into a discussion of some of the workarounds to avoid having clients barre, but also cautions against totally disable SMB Auto-Barring..

    At the end of the document it provides a procedure and script on automating the identification of "bad actors" on your network, from the HNAS perspective.

    Automated Procedure for Customers to collect diagnostics and search for the events above from the downloaded diagnostics.

    1. In the vSMU GUI, verify that the system has been configured to send emails.
      1. Navigate to Home > Status & Monitoring > Email Alerts Setup
        Check that an SMTP Server and the From Address have been configured to send emails.
    2. Create a cron job on one of the nodes in the cluster to download daily node diagnostics, using the following example.
      crontab add -m <customer maillist>(Multiple users can be specified, in quotes, in the form "user1;user2" "00 08 * * *" "pn all diagemail  <customer email address>"
    3. Download and copy the attached PowerShell script to a directory on a Windows machine.
      FAILED-LOGIN_v4
    4. Copy the diagnostics into the same directory.
    5. Run the following PowerShell scripts: (You need 7zip installed on your PC).
      1. FAILED-LOGIN_v4
    6. This script produces two .txt files
      1. failed-login.txt to get the user name and IP address of the users with failed logins.
      2. wrong-login-per-day.txt to see the trend of the failed logins per day of both nodes.

    While this will not prevent bad actors from getting barred, it will assist you in identifying them , hopefully before they do get barred.



    ------------------------------
    Howard Dobrin
    Field Solutions Engineer (FSE)
    Hitachi Vantara
    ------------------------------

    Original Message


  • 8.  RE: Why does HNAS need SMB-Client-Barring whereas other products in Market do not seem to have a need of the same functionality?

    Posted 06-20-2023 13:03

    Abhishek, our Product Manager is providing the following information on the topic.

    What is SMB barring?

     It is a highly requested customer feature that was introduced with HNAS OS 13.9.  This feature enable administrator to maintain a list of rogue client IP addresses that are barred from SMB/SMB2.x/SMB3 access to the server.  Clients that cause SMB NTLM authentication failures by providing an incorrect password are automatically added to the list if the rate of failure is sufficient. Automatic barring of clients is enabled by default, and an event log is generated when a client is barred.

    No initial configuration of the feature is required, however the barred clients list can be managed if necessary using the following commands:

    smb-barred-client-add

    smb-barred-client-remove

    smb-barred-clients-list

    smb-barred-clients-clear

    Clients are barred based on their IP address so each IPv4 and IPv6 (if configured) address will need to be considered a separate entry. Once a client is barred, it is not possible for that client to connect over SMB regardless of the credentials being used – manual removal from the 'barred' list would be required.

    What is SMB barring benefits: 

    HNAS OS SMB barring feature is a free security enhancement (available to all customer with support contract) to protect HNAS cluster against denial of service (DoS) attack initiated by SMB client.  When used conjunction with NTLM authentication, it also improves SMB client quality of service.

    Netapp offers similar capability to protect against DoS attack at extra cost if you subscribe to their services; see NetApp Cloud Insights documentation

    Why is this SMB barring was introduced?

    • COVID-19 pandemic changes the work practice in the enterprise, more employees work remote which in turn, increase the risk for SMB client sessions to be exploited by "man-in-the-middle" attack.
    • With IT budget squeeze and shrinking maintenance windows, we have seen rise in customer incidents that could have been avoided if authentication best practices had been implemented; IT operation too frequently deferred good hygiene due to its disruptive nature to avoid negotiating needed outages.
    • As an example, customers that are still using NTLM authentication mechanism whereby HNAS authenticates users on their behalf, used when clients mount SMB shares using a short hostname \\evs\share (in contrast to Kerberos authentication, when clients authenticate directly with AD server, if they access HNAS using long FQDN as in \\evs.acme.com\share). 
    • Legacy NTLM authentication is supported with HNAS but is not as resilient with edge scenarios since HNAS acts as a proxy server in this method. 
      1. NTLM authentication requires redirect of SMB client session authentication to local Microsoft Active domain controller; a burst of these requests with incorrect password (e.g. boot storm) could be perceived by the domain controller as DoS attack (if incident exceed lockout attribute threshold) which in turn will lockout the HNAS cluster and interrupt all SMB clients (see Microsoft Active Directory account lockout). 
      1. If a client-supplied password is incorrect, the local Microsoft Active Directory domain controller will tear down a connection and HNAS will have to retry and reconnect to the domain controller for every bad password attempt. Meanwhile If that happens a lot, a backlog of valid client connection requests doesn't get authenticated and will experience slower response time.
    • Since NAS and active domain administration responsibility often fall in different departments, this risk often falls below the radar, and little is done proactively to avoid these edge cases. 
    • With NTLM authentication, SMB barring feature enables NAS administrator to eliminate the above-mentioned DoS attack risk, improve valid SMB client session quality of service and ensure communication resilience between HNAS cluster and local domain controller.
    • Alternatively, for environment as described above, administrator can monitor SMB client sessions traffic with invalid password is below local domain controller account lockout threshold and/or increase threshold value to avoid any service interruption. 

    SMB sessions with incorrect password can occur for a large number of reasons – some of which include:

    • Attempted brute-force attacks;
    • Service accounts with passwords that have been recently changed or have expired;
    • Issues with Active Directory replication;
    • Cached user credentials saved in certain programs;
    • Stored login details that contain overlapping credentials;
    • Scheduled tasks using expired credentials;
    • Shared drives using expired credentials



    ------------------------------
    Albert Hagopian
    Software Development Engineer - Specialist
    Hitachi Vantara
    ------------------------------

    Original Message


Related Content