Isilon Troubleshooting Guide: Networking: SmartConnect

Isilon Troubleshooting Guide: Networking - SmartConnect
Before you begin Revised June 11, 2019
This troubleshooting guide applies to OneFS 7.2 - 8.2
This TSG has the following related customer troubleshooting guides

(cTSGs):
- EMC Isilon Customer Troubleshooting Guide: Troubleshoot Your
SmartConnect Configuration
- EMC Isilon Customer Troubleshooting Guide: SmartConnect Client
Connections are not Balanced Between Nodes
CAUTION!
Before beginning this procedure, ensure you have an alternate means to
connect to the cluster other than the pool or subnet that you are currently
working on. For example, ensure you can connect via another subnet or
pool, or that you or the customer has physical access to the cluster.
The reason for this is so that if you make a mistake and the pool you are
working on goes down, you will be able to immediately access the cluster
by another means. If there is only one subnet or pool, and it goes down, if
you or the customer is not onsite, the cluster might be down for several
hours until someone can physically access the cluster, resulting in data
unavailability. For instructions on making a physical connection to the
cluster, see Isilon: How to connect to the management port of a node,
article 304071.
IMPORTANT!
Restarting a suspect or problematic process is not a fix! It temporarily puts
the process back into a working state, but if you do not fix the conditions
that caused the problem initially, they will probably occur again and the
problem will return. Instead, determine the root cause and permanently fix
the problem.
CAUTION!
Before you restart a service, confirm that the customer understands that
client connections and other cluster operations might be temporarily
disrupted while the service restarts.
How to use this document:

Start at the beginning and perform the steps in order. Although you will
probably not need to perform every step in this flowchart, each step builds
on the ones that come before it.
For an overview of the conventions used in this flowchart, see Appendix A:
How to use this flowchart.
What is the SSIP?

The SSIP is the SmartConnect Service IP address. In this guide, we will
refer to the SSIP in the following ways:
- SmartConnect Service IP (SSIP)
- SmartConnect IP (SCIP)
- Virtual IP (VIP)
- Isilon DNS IP
We appreciate your help in improving this document.

___________________
Submit your feedback at http://bit.ly/isilon-docfeedback.
Page 1 of 35 Dell - Internal Use - Confidential

Routing Revised June 11, 2019
Start
Note
In OneFS 8.2, clients are unable to
access the cluster over the
SmartConnect Service IP address.
Record this hashtag in the case

Is the customer's
notes: #tsgsmartconnect
initial complaint that the
Yes Consult a coach. If a coach is not available,
SmartConnect Service IP (SSIP)
consult your supervisor or manager for further
is on the wrong node?
direction.
No
INITIATE CODE RED

Is the cluster Consult your supervisor or manager
available to production No to initiate the Code Red process.
users? Only supervisors or above are allowed to
initiate the Code Red process.
Yes
Collect the following information and record it in the

case notes before you start troubleshooting.
You will use this information in the steps that follow.
Information to record How to find the information
The name of the affected SmartConnect zone on the Isilon

cluster. The fully qualified domain name (FQDN) is preferred Get this information from the customer.
but the short name is acceptable.
The SmartConnect Service IP (SSIP). For instructions, see Appendix B.
Run the following command, where <SSI P> is the

SmartConnect service IP address:
The SmartConnect master.
i si _f or _ar r ay i f conf i g | gr ep <SSI P>
If there is no output, note that in the case notes and continue.
The pool and subnet that contain the SmartConnect zone. For instructions, see Appendix C.
The IP addresses of the Isilon cluster's externally configured

For instructions, see Appendix D.
DNS IP address.
From a Linux client, run:

cat / et c/ r esol v. conf
The IP addresses of the client's configured DNS servers.
From a Windows client command prompt, run:
i pconf i g / al l
Go to 3A

___________________

Verify SmartConnect zones and network pools Revised June 11, 2019
? Return to Page 2A
3A
Check whether the affected SmartConnect zone exists in the

list of network pools and zone names.
OneFS 8.0 - 8.2

i si net wor k pool s l i st
OneFS 7.2.1
i si net wor ks l i st pool s
If the answer is no, the customer probably never configured

the SmartConnect zone name and settings.
Tell the customer to create the SmartConnect zone name in

Does the the appropriate pool, and make sure that the associated
affected SmartConnect records are in their DNS servers.
zone exist in the list of No
network pools and For instructions, see EMC Isilon External
zone names? NetworkConnectivity Guide: Routing, Network Topologies,
and Best Practices for SmartConnect.
If the customer has any questions while using this guide,

consult a coach.
Yes
Is the issue
No
resolved?
Yes
Record this hashtag in

the case notes: #tsgsmartconnect
End
Does the customer

report an intermittent
SmartConnect issue? Intermittent
Yes Go to 28A
means that sometimes the connection
works and sometimes it
does not work.
No
Go to 4A

___________________

Validate SmartConnect address configuration Revised June 11, 2019
? Return to Page 3A
? Return to Page 13A
4A
Open an SSH connection to any node in the cluster EXCEPT FOR

the node that is the SmartConnect master, and log in as root. (On a
compliance-mode cluster, log in as the administrative user.)
OneFS 8.2 supports multiple SmartConnect service IPs. You

will need to identify if the customer's cluster is single- or
Is the cluster multi-SSIP environment by running the following command and
running OneFS Yes noting the value in the SC Service Addrs: field:
8.2?
i si net wor k subnet s l i st - v
See__________
Appendix E for example output.
No
Check whether the customer's name server correctly returns SmartConnect addresses.
From the OneFS command-line interface, run the following command, where
<zonename> is the name of the affected SmartConnect zone on the Isilon cluster, and
where is the cluster's externally configured DNS IP address.
For OneFS 8.2, will be each IP address listed in the
SC Ser vi ce Addr s field from the output of i si net wor k subnet s l i st - v.
nsl ookup <zonename> 
For example:
nsl ookup i si l on. exampl e. or g 192. 168. 1. 1
Record the output in the case notes.
See__________
Appendix E for example output.
Do the
commands return IP Yes Go to 6A
addresses?
No
Go to 5A

___________________

Attempt to ping the DNS server Revised June 11, 2019
? Return to Page 4A
5A
Ping the DNS server from the cluster by running the

following command, where is the
cluster's externally configured DNS IP address that
failed the nslookup on page 4A.
_______
pi ng 

Do any of the The cluster cannot reach the DNS server. notes: #tsgsmartconnect
pings No This is likely caused by an issue on the Consult a coach. If a coach is not
succeed? customer's network or DNS server. available, consult your supervisor or
manager for further direction.
Yes
Go to 7A

___________________

Validate SmartConnect address configuration (2) Revised June 11, 2019
? Return to Page 4A
6A
What type of IP
address does the output
contain?
The SmartConnect An IP address that is within Anything else

Service IP address the range of a pool
(see the note at the
bottom of the page)

Go to 7A Does the customer notes: #tsgsmartconnect
report that client connections Consult a coach. If a coach is not
No available, consult your supervisor or
are not balanced properly
among the nodes? manager for further direction.
Yes

the case notes:
Go to 21A
#tsgsmartconnect
End
Note:
Run the following command to determine if the IP address is actually in a pool, where is the IP address you want to
check. If No pool s f ound is returned, the IP address is not in a pool.
OneFS 8.0 - 8.2

i si _cl assi c net wor ks l i st pool s - - has- addr =
Note: The isi_classic commands will be deprecated at some point. If the command does not work properly, consult a coach.
OneFS 7.2.1
i si net wor ks l i st pool s - - has- addr =
Output example 1 - The IP address is in a pool.

Subnet Pool Smar t Connect Zone Ranges Al l oc
--------------- --------------- ---------------------- ---------------------- -------
subnet 0 pool 0 company. exampl e. l ocal 192. 111. 33. 88- 192. 1. . . St at i c
Output example 2 - The IP address is not in a pool.

No pool s f ound

___________________

Ping the SmartConnect Service IP (SSIP) Revised June 11, 2019
? Return to Page 5A
? Return to Page 6A
? Return to Page 9A 7A
From the cluster, ping the SSIP:
pi ng <SSI P>
Did the ping

Yes Go to 8A
succeed?
No
Confirm that the SmartConnect Service IP address is set up on the node by running the following command,
where <SSI P> is the SmartConnect Service IP address:
i si _f or _ar r ay ' i f conf i g | gr ep <SSI P>'
For example:
i si _f or _ar r ay ' i f conf i g | gr ep 192. 168. 38. 80'
If the SSIP exists, the output looks similar to the following:

cl ust er - 1: i net 192. 168. 38. 80 net mask 0xf f f f f f 00 br oadcast 192. 168. 38. 255 zone 1
The SSIP exists, but you cannot reach it through the

network. Use the following guide to verify that the
network is working and is configured correctly:
Does the SSIP exist? Yes Isilon Troubleshooting Guide: Networking - Physical
Network/Customer Environment
No
This is probably due to a bug. Use following article to try to

resolve the issue. (Note that this issue was fixed in OneFS
7.2.0.4 and 7.2.1.1.):
Following the reboot of the SC Master, the SCIP disappears

from all interfaces and queries against SC fail, article 457063.

Record this hashtag
in the case notes: Does the article
Yes No Consult a coach. If a coach is not
#tsgsmartconnect resolve the issue?
available, consult your supervisor or
End

___________________

Verify the cluster is returning SmartConnect data Revised June 11, 2019
? Return to Page 7A
8A
Verify whether the cluster is returning SmartConnect data by running the following
command from the OneFS command-line interface, where <SSI P> is the SmartConnect
Service IP address of the subnet that the affected SmartConnect zone is in, and
<ZoneName> is the name of the affected SmartConnect zone on the Isilon cluster:
nsl ookup <ZoneName> <SSI P>
What does the

output contain?
The SmartConnect An IP address within the * * ser ver can' t There is

Service IP address pool range (see the note f i nd <ZoneName>: no output
at the bottom of _______
Page 6A REFUSED
for instructions on how to determine this)
Record this hashtag There is a problem with the customer's

in the case notes: DNS configuration. The customer's DNS Which version
#tsgsmartconnect administrator needs to identify and fix the of OneFS
Go to 9A are you running?
Consult a coach. If a coach is problem.
not available, consult your
Note: This is almost always an issue where
supervisor or manager for
there is both an A record and an NS record, or
further direction.
just an A record. Removing the A record and
having only an NS record will solve the
problem. Ask the customer to check their
configuration and make changes if needed.
7.2.1 8.0 - 8.2
If the customer has questions or needs help,
we have some documentation that can assist
you in helping them. See the "DNS" and
"SmartConnect" sections of
Isilon Troubleshooting Guide:
Networking - Configuration. Go to 13A Go to 10A

End

___________________

Error: ** server can't find <ZoneName>: REFUSED Revised June 11, 2019
? Return to Page 8A
9A

Did you reach notes: #tsgsmartconnect
this point in the process Yes Consult a coach. If a coach is not
before? available, consult your supervisor or
No
This refusal error message indicates that the IP address that the query is being
sent to is either not a SmartConnect Service IP address (SSIP) or is not a valid
IP address. The SmartConnect service will respond only if the query comes in
on the SSIP address for the SmartConnect zone. An SSIP address for another
zone will not respond for a SmartConnect name that does not belong to it.
Validate that the domain names on the cluster

match the domain names on the DNS server.
Recheck the correct SSIP for the affected zone. Follow the
instructions in Appendix B to find the SSIP, and then return here.
Use the SSIP that is returned when you retry the

command on Page 7A.
Go to 7A

___________________

No output after running nslookup <ZoneName><SSIP> Revised June 11, 2019
? Return to Page 8A
10A
Run
i si _gr oup_i nf o
See the box on this page for example output.
Is the
al l _enabl ed_pr ot ocol s No Go to 11A
section present
in the output?
Yes
In the
al l _enabl ed_pr ot ocol s
No Go to 12A
section, are all the cluster
nodes listed?
Yes
Go to 13A
Example output
This output indicates that the nfs protocol is not running on node 1, which results in node 1 not appearing in the
al l _enabl ed_pr ot ocol s list.
ef s. gmp. gr oup: <3, 18>: { 1- 3: 0- 5, smb: 1- 3, nf s: 2- 3, al l _enabl ed_pr ot ocol s: 2- 3 }
For additional information, see OneFS 8.0: SmartConnect stops handing out IPs for nodes if all enabled
protocols are not running, article 486580.

___________________

No output after running nslookup <ZoneName><SSIP>(2) Revised June 11, 2019
11A
Run the following command:

i si ser vi ces - a | gr ep i si _pr ot oheal t h_d
Is the
No i si _pr ot oheal t h_d Yes
service enabled?
Enable isi_protohealth_d:
i si ser vi ces - a i si _pr ot oheal t h_d enabl e
i si _gr oup_i nf o

Is the
al l _enabl ed_pr ot ocol s No
section present i si _f or _ar r ay - s ' ps auxw | gr ep - v gr ep | gr ep i si _pr ot oheal t h_d'
in the output?
Copy the output into the case notes.
Yes

In the
Consult a coach. If a coach is not
al l _enabl ed_pr ot ocol s No Go to 12A
section, are all the cluster available, consult your supervisor or
nodes listed? manager for further direction.
Yes
Go to 13A

___________________

Run the following command to check whether the i si _pr ot oheal t h_d service is running on all nodes:
i si _f or _ar r ay - s ' ps auxw | gr ep - v gr ep | gr ep i si _pr ot oheal t h_d'
Copy the output into the case notes.
If the service is running on all nodes, the output looks similar to the following:
cl ust er - 1: r oot 84639 0. 0 0. 6 140052 11360 - I s 10: 41AM 0: 00. 00 / usr / sbi n/ i si _pr ot oheal t h_d
If the service is not running on a node, the output looks like the following:
cl ust er - 1 exi t ed wi t h st at us 1
Using the output from the isi_group_info

command that you ran earlier, note which
protocols have nodes missing from their list of
Is the nodes in the output. See Page 10A for example
i si _pr ot oheal t h_d service Yes output. Troubleshoot those protocols on their
running on all nodes? corresponding nodes.
If you need assistance, consult a coach for the

relevant protocol.
No
Troubleshoot the nodes on which the

i si _pr ot oheal t h_d service is not running, so that the
service is running on all nodes.
Is the issue
No Go to 13A
resolved?
If you need assistance, consult a coach.
Yes

Is the issue
No Go to 10A the case notes: #tsgsmartconnect
resolved?
End
Yes

End

___________________

? Return to Page 8A
13A
Have you reached

this point in the flow Yes Go to 14A
previously?
No
Note
A SmartConnect service subnet was probably not configured when SmartConnect was By default, the SSIPs that are
originally configured, or it could be configured incorrectly. Configure the SmartConnect configured in a given pool's subnet
service subnet by using the OneFS web administration interface as follows: will be the only SSIPs that respond
for that pool. The SmartConnect
OneFS 8.0 - 8.2 Subnet can be used to change
1. Click Cluster Management > Network Configuration > External Network. which subnets SSIPs respond for
that pool.
2. In the Networks column, expand the groupnet and subnet that contain the affected Note: only one subnet can respond
SmartConnect zone. for a given pool. So if the
SmartConnect Subnet is set to
3. For the pool that contains the affected SmartConnect zone, click View/Edit. something other than the parent
subnet, the parent subnet will NOT
4. Click Edit. respond for the pool.
5. In the SmartConnect Basic section, from the SmartConnect service subnet For more information about the
drop-down list, select the name of the external network subnet whose SmartConnect SmartConnect service subnet, see
service answers DNS requests on behalf of the IP address pool. the OneFS Administration Guide
for your version of OneFS.
6. Click Save Changes.
OneFS 7.2.1
1. Click Cluster Management > Network Configuration.
2. In the External Network Settings area, click the link for the subnet that contains
the affected SmartConnect zone.
3. In the IP Address Pools area, for the pool for the zone in question, click the
SmartConnect settings Edit button.
4. In the SmartConnect Basic section, from the SmartConnect service subnet

drop-down list, select the name of the external network subnet whose SmartConnect
service answers DNS requests on behalf of the IP address pool.
5. Click Submit.
Go to 4A

___________________

Isilon Troubleshooting Guide: Networking - SmartConnect Revised June 11, 2019
Collect data for OneFS 7.2 - 8.1 14A

CAUTION!
Before you restart a service, confirm that the customer understands that
? Return to Page 13A client connections and other cluster operations might be temporarily
disrupted while the service restarts.
IMPORTANT!
The customer might want to restore service immediately by
restarting the SmartConnect service. It is important that you Is the cluster on
Yes Go to 17A
capture the data described on this page before restarting the OneFS 8.2?
service. Otherwise, it will be difficult to determine the root
cause and potential solutions.
No
Capture the state of the process, a pcap, and a truss as follows:

Note: Commands that wrap onto more than one line contain a backslash at the ends of lines. The backslash is not part of the command but is required because Adobe adds hard line breaks
and removes spaces at the ends of lines, which can break the command. When you copy and paste the commands below into the CLI, they will appear on multiple lines (exactly as they appear on
the page), but when you press Enter the command will run as it should.
1. Create a directory to capture the information:

mkdi r / i f s/ dat a/ I si l on_Suppor t / i si _smar t connect
2. Determine which node is the SmartConnect master (the node that contains the SmartConnect Service IP address on one of its interfaces). To determine the SmartConnect master, run the
following command, where <SSI P> is the SmartConnect Service IP address:
i si _f or _ar r ay - s ' i f conf i g | gr ep <SSI P>'
The command returns a single line preceded by the name of the node that contains the SmartConnect IP address. This node is the SmartConnect master.
3. Open two SSH connections to the SmartConnect master node.
4. Run the following two commands:

ki l l al l - WI NCH i si _dnsi q_d
ps - auwxx | gr ep i si _dnsi q_d | gr ep - v gr ep > / i f s/ dat a/ I si l on_Suppor t / i si _smar t connect / ` host name` . i si _dnsi q_d_st at e
5. In the first SSH window, copy and paste the following command to core the i si _dnsi q_d process:
pgr ep i si _dnsi q_d | xar gs gcor e - s - c / i f s/ dat a/ I si l on_Suppor t / i si _smar t connect / ` host name` . i si _dnsi q_d. $( dat e \ +%m%d%Y_%H%M) . cor e
6. In the first SSH window, copy and paste the following command at the command prompt to capture a truss:
pgr ep i si _dnsi q_d | xar gs - I % t r uss - p % - o / i f s/ dat a/ I si l on_Suppor t / i si _smar t connect / ` host name` . i si _dnsi q_d. $( dat e +%m%d%Y_%H%M) . t r uss - d - s 0
Continued on next page...
Go to 15A
We appreciate your help in improving this document. Submit your feedback at ___________________
http://bit.ly/isilon-docfeedback.
Collect data for OneFS 7.2 - 8.1 (2)

15A
Continued from previous page?
7. In the second SSH window, run the following command to obtain a packet capture:
i f conf i g | gr ep f l ags= | awk - F: ' { pr i nt $1} ' | egr ep - v ' l o0| i b0| i b1' | whi l e r ead i f con; do t cpdump - s 0 - C2048 - W5 - i " ${ i f con} " - w \
/ i f s/ dat a/ I si l on_Suppor t / i si _smar t connect / $( uname - n) . $( dat e +%m%d%Y_%H%M%S) . " ${ i f con} " . pcap &; done
Note: here is a simple example of correct SmartConnect activity: Smartconnect in pcaps

__________________
8. Attempt to connect to the cluster from a client via SmartConnect. From a client, run the following commands, where <DNS ser ver I P> is the IP address of the customer's DNS server,
<ZoneName> is the name of the affected SmartConnect zone on the Isilon cluster, and <SSI P> is the SmartConnect Service IP address. It is recommended that you use the fully qualified
domain name (FQDN) for the zone name, but the short name is acceptable.
nsl ookup <ZoneName> <DNS ser ver I P>
9. Note in the case notes the Name Server IP address that you used.
10. In the second SSH window, run the following command to stop the packet capture:
ki l l al l - SI GI NT t cpdump
11. In the first SSH window, press CTRL+C to end the truss.
12. In one of the SSH windows, open the / et c/ sysl og. conf file in a text editor.
13. Find the line that looks like this:

* . >=i nf o / var / l og/ i si _smar t connect
14. Change info to debug so that the line looks like this:
* . >=debug / var / l og/ i si _smar t connect
15. Save and exit the text editor.
Continued on next page...
Go to 16A
Collect data for OneFS 7.2 - 8.1 (3) Revised June 11, 2019
16A
16. Run this command:

ki l l al l - HUP sysl ogd
________
17. Rerun the following commands (see step 8 on Page 15A for more information):
18. In one of the SSH windows, open the / et c/ sysl og. conf file in a text editor.
19. Find the line that looks like this:

* . >=debug / var / l og/ i si _smar t connect
20. Change debug to info so that the line looks like this:
* . >=i nf o / var / l og/ i si _smar t connect
21. Save and exit the text editor.
22. Run this command:

ki l l al l - HUP sysl ogd
Go to 19A

___________________

Collect data for OneFS 8.2

Capture the state of the process, a packet capture (pcap), and a truss for OneFS 8.2 as follows:
Note: Commands that wrap onto more than one line contain a backslash at the ends of lines. The backslash is not part of the command but is required because Adobe adds hard line breaks and
removes spaces at the ends of lines, which can break the command. When you copy and paste the below commands into the CLI, they will appear on multiple lines (exactly as they appear on the page),
but when you press Enter the command will run as it should.
1. Create a directory to capture the information:

mkdi r / i f s/ dat a/ I si l on_Suppor t / i si _smar t connect
2. Determine which SmartConnect Service IPs are impacted, and for each do the following:
3. Determine which node has that SSIP configured. To determine the SmartConnect master, run the following command, where <SSI P> is the SmartConnect Service IP address:
i si _f or _ar r ay - s ' i f conf i g | gr ep <SSI P>'
The command returns a single line preceded by the name of the node that contains the SmartConnect IP address. This node is the SmartConnect master.
4. Open two SSH sessions to the SmartConnect master node and run the following two commands:
i si _dump_smar t connect
ps - auwxx | gr ep i si _smar t connect _d | gr ep - v gr ep > / i f s/ dat a/ I si l on_Suppor t / i si _smar t connect / ` host name` . i si _smar t connect _d_st at e
5. In the first SSH window, copy and paste the following command to core the i si _smar t connect _d process:
i si _kcor e ` pgr ep i si _smar t connect _d` / i f s/ dat a/ I si l on_Suppor t / i si _smar t connect / ` host name` . i si _smar t connect _d. $( dat e +%m%d%Y_%H%M) . cor e
6. In the first SSH window, at the command prompt, copy and paste the following command to capture a truss:
t r uss - p ` pgr ep i si _smar t connect _d` - o / i f s/ dat a/ I si l on_Suppor t / i si _smar t connect / ` host name` . i si _smar t connect _d. $( dat e +%m%d%Y_%H%M) . cor e
7. In the second SSH window, run the following command to obtain a packet capture:
i f conf i g | gr ep f l ags= | egr ep - v ' l o0| I SI I NTERNAL' | awk - F: ' { pr i nt $1} ' | whi l e r ead i f con; do \
t cpdump - s 0 - C2048 - W5 - i " ${ i f con} " - w \
/ i f s/ dat a/ I si l on_Suppor t / i si _smar t connect / $( uname - n) . $( dat e +%m%d%Y_%H%M) . " ${ i f con} " . pcap &; done
Note: here is a simple example of correct SmartConnect activity: Smartconnect in pcaps

__________________
Continued on next page?
Go to 18A
Collect data for OneFS 8.2 (2) Revised June 11, 2019
18A
8. Attempt to connect to the cluster from a client via SmartConnect. From a client, run the following
commands, where <DNS ser ver I P> is the IP address of the customer's DNS server, <ZoneName>
is the name of the affected SmartConnect zone on the cluster, and <SSI P> is the SmartConnect
Service IP address. It is recommended that you use the fully qualified domain name (FQDN) for the
zone name, but the short name is acceptable.
9. Record the DNS server IP address that you used in the case notes.
10. In the second SSH window, run the following command to stop the pcap:
ki l l al l - SI GI NT t cpdump
11. In the first SSH window, press CTRL+C to end the truss.
12. In one of the SSH windows, switch i si _smar t connect _d to debug logging:
i si _sc_l og_l evel - - l evel debug
13. Rerun the following commands (see step 8 for more information):
14. In one of the SSH windows, switch i si _smar t connect _d back to info log level:
i si _sc_l og_l evel - - l evel i nf o
Go to 19A

___________________

Truss process Revised June 11, 2019
19A
Confirm whether the truss process is running:
ps auxw | gr ep - v gr ep | gr ep t r uss
If the process is not running, there will be no output.
Is the
truss process No
running?
Yes
Kill the truss process:

ki l l al l t r uss
Confirm whether the truss process is running:

ps auxw | gr ep - v gr ep | gr ep t r uss
Restart the process by running the following command:

Is the
truss process No ki l l al l - 9 i si _dnsi q_d
running?
If the cluster is on OneFS 8.2, you will need to run this command, also:
ki l l al l - 9 i si _smar t connect _d
Yes
Record this hashtag in the case Confirm that the process restarted:
Consult a coach. If a coach is not ps auxw | gr ep - v gr ep | gr ep i si _dnsi q_d
manager for further direction. If the cluster is on OneFS 8.2, you will need to run this command, also:
ps auxw | gr ep - v gr ep | gr ep i si _smar t connect _d

Did the notes: #tsgsmartconnect
Go to 20A Yes processes No Consult a coach. If a coach is not
restart? available, consult your supervisor or

___________________

Determine severity of the case Revised June 11, 2019
20A
Gather logs, core, pcap, and truss by running the following command from the
OneFS command line interface:
i si _gat her _i nf o - f " / i f s/ dat a/ I si l on_Suppor t / i si _smar t connect "
Run the following commands:
Do both
commands return an IP
No Yes
address within the range
of the pool?
Adjust the severity of the case to Adjust the severity of the

1 or 2 (data unavailability). case to 3 or 4 (RCA).
Record this hashtag in the case Record this hashtag in the case
notes: #tsgsmartconnect notes: #tsgsmartconnect
Consult a coach. If a coach is not Consult a coach. If a coach is not
available, consult your supervisor or available, consult your supervisor or
manager for further direction. manager for further direction.

___________________

Validate the balance problem Revised June 11, 2019
? Return to Page 6A
21A
Note
You can use the OneFS Web administration
interface to view the number of clients
connected per node, but the command line
interface provides more data and helps you Does the
decide if a balance problem exists. The customer have
command line interface allows you to view the Yes Go to 22A
multiple SSIPs
number of connections by protocol. This allows configured?
you to see which protocols have the problem.
Many customers use a different pool for each
protocol; if your customer does this, then
knowing which protocols have problems
enables you to know which pools to look at. No
Obtain the total number of connections per node, by protocol.
On the OneFS command line interface, on any node in the cluster, run the following command for each protocol
that you are concerned with, where <pr ot ocol > is the protocol type (for example, nfs, smb, etc.):
OneFS 8.0 - 8.2

i si st at i st i cs quer y cur r ent l i st - - nodes=al l - - keys=node. cl i ent st at s. connect ed. <pr ot ocol >
OneFS 7.2.1
i si st at i st i cs quer y - - nodes=al l - - st at s=node. cl i ent st at s. connect ed. <pr ot ocol >
The protocol type options to use in the command are:
ci f s ht t p nf s si q
ftp i scsi nl m smb
hdf s ndmp papi
21B
The output looks similar to the following. In this example, nodes 8 and 10 appear to be imbalanced.
NodeI D node. cl i ent st at s. connect ed. smb

1 3
2 2
3 2
4 0
5 0
6 1
7 2
8 10
9 0
10 20
Aver age 4
Copy the output for each command into the case notes.
Go to 23A

___________________

Validate the balance problem Revised June 11, 2019
22A
When Multi-SSIP is in use, there is a tradeoff between fault tolerance and load balancing. SSIPs do not
synchronize their load balancing. So on failover, the responding SSIP will not know what the previous SSIP
responded with. This is most obvious with a round robin connection policy. With two SSIPs configured you
could see the following returned:
// DNS server uses SSIP1 -> 1. 2. 3. 4

// DNS server failovers to SSIP2 -> 1. 2. 3. 4
Client connections
If the customer has a few heavyweight clients connecting, using Multi-SSIP with a DNS server that
changes SSIPs frequently can impact load distribution.
To reduce this load impact, consider reducing the number of SSIPs.
If the customer has many lightweight clients connecting, the clients will be less affected by frequent SSIP
changes with a connection policy.
Connection policies
If the DNS server is round robining nameservers/SSIPs, SmartConnect will continue to follow the load
balancing policies. This means if round robin is configured on the pool, each IP address will be returned
<#> times, where <#> is the number of configured SSIPs. The other load balancing policies may not always
see the same IP address returned <#> about of times, but load balancing will still be affected. If the load
distribution is not acceptable to the customer, consider reducing the number of SSIPs.

Is the notes: #tsgsmartconnect
customer's DNS Consult a senior resource.
No
server configured for If a senior resource is not available, consult
recursion? your supervisor or manager for further
direction.
Yes
Go to 21B
Note
To see if the customer's DNS server is configured for recursion, run the following
command from a client:
# di g exampl e. com +nor ecur se | gr ep r a
; ; f l ags: qr r a; QUERY: 1, ANSWER: 1, AUTHORI TY: 13, ADDI TI ONAL: 27

___________________

Validate the balance problem (2) Revised June 11, 2019
23A
Determine which connection balance policy (SC Connect Policy) is Note

configured on each pool by running the following command: There are four connection load balancing policies:
- Round-robin
OneFS 8.0 - 8.2 - Connection count
i si net wor k pool s l i st - v - Network throughput
- CPU usage
OneFS 7.2.1
i si net wor ks l i st pool s - v
Ask the customer how clients connect to the cluster:

- Do all clients connect at the same time?
- Are bulk connections made at the same time?
- Do individual clients or small groups of clients connect
throughout the day?
How do
clients
connect?
Clients all connect Clients connect individually
at the same time, or or in small groups, or
bulk connections are made clients do not all connect
at the same time. at the same time.
Go to 24A Go to 25A

___________________

24A
Note
The round-robin connection policy is
recommended for systems in which all
clients connect at the same time.
Record this hashtag in the

case notes: #tsgsmartconnect
Is the
Consult a coach to analyze the
connection policy Yes
customer's workflow. If a coach is not
round-robin?
No
Advise the customer that the policy is unable to
meet customer expectations due to design
limitations.
Is the customer
willing to change their
No Refer to OneFS: How SmartConnect
connection policy to a
connection policies distribute client connections,
round-robin policy?
article 304462, for information about the
connection policies and how they distribute
client connections.
Yes
Record this hashtag in the case notes:

#tsgsmartconnect
End
If the customer has further questions, consult
a coach. If a coach is not available, consult your
supervisor or manager.
Change the connection policy to round-robin by running the following command, where <gr oupnet - name> is the name of the groupnet;
<subnet name> is the name of the subnet; and <pool name> is the name of the pool:
OneFS 8.0 - 8.2

i si net wor k pool s modi f y - - i d=<gr oupnet - name>. <subnet - name>. <pool - name> - - sc- connect - pol i cy=r ound_r obi n
For example:
i si net wor k pool s modi f y - - i d=gr oupnet 0. subnet 0. pool 0 - - sc- connect - pol i cy=r ound_r obi n
OneFS 7.2.1
i si net wor ks modi f y pool - - name=<subnet - name>: <pool - name> - - connect - pol i cy=r ound- r obi n
For example:
i si net wor k pool s modi f y - - name=subnet 0: pool 0 - - connect - pol i cy=r ound- r obi n
Go to 27A

___________________

Note
Any of the four connection policies 25A
work when clients connect on an
as-needed basis.
When customers complain, there is

often a perceived problem rather
than an actual problem. If the
customer does not understand how
the policies work, they might not
realize the policy is working as Explain to the customer how their current connection policy
designed. The focus of this section calculates how it distributes client connections. Refer to
is to: OneFS: How SmartConnect connection policies distribute
client connections, article 304462 for information about the
1. Validate that the connection connection policies and how they distribute client
policy is working as designed. connections.
2. Educate the customer about the
policy that they are using.
3. Determine whether the
customer is satisfied with the way
the policy is working. If not, help
the customer choose a different
policy.
Once the customer understands, ask the customer if they
are satisfied with the current client connection policy.
Is customer
satisfied with the current No Go to 26A
connection policy?
Yes

End

___________________

26A
Explain the different client connection policies to the

customer and ask customer which policy best fits their
workflow. If a different policy fits their workflow better,
ask if they would like to change to a different policy.

Does the customer notes: #tsgsmartconnect
want to change their client No Consult a coach. If a coach is not
connection policy? available, consult your supervisor or
Yes
Change the policy to a different policy that the customer chooses by running the following command for their
version of OneFS:
OneFS 8.x
- <gr oupnet > is the name of the groupnet
- <subnet > is the name of the subnet
- <pool > is the name of the pool
- <pol i cy> is one of the following policy types:
- r ound_r obi n
- cpu_usage
- conn_count
- t hr oughput
i si net wor k pool s modi f y - - i d=<gr oupnet >. <subnet >. <pool > - - sc- connect - pol i cy=<pol i cy>
OneFS 7.2
- <gr oupnet > is the name of the groupnet
- <subnet > is the name of the subnet
- <pool > is the name of the pool
- <pol i cy> is one of the following policy types:
- r ound- r obi n
- cpu- usage
- conn- count
- t hr oughput
i si net wor ks modi f y pool - - name=<subnet >: <pool > - - connect - pol i cy=<pol i cy>
Go to 27A

___________________

27A
Retest as follows:
1. Ask the customer to monitor the cluster for 24 hours to test the client distribution
resulting from the new policy. Waiting 24 hours allows the old connections to disconnect
and new connections to be established.
2. After the new connections have been established, the customer should verify
whether clients are distributed as expected. Ask the customer to contact you at the end
of the
24-hour period to let you know whether things are working as expected and if the
problem is resolved.
To perform the verification, the customer can use the protocol-based commands on
Page 21A, or, if there is only one pool, they can use the OneFS web administration
interface as follows:
a. Click Dashboard > Cluster Overview > Cluster Status.
b. View the Client Connection Summary section.

Is the issue
No Consult a coach. If a coach is not
resolved?
Yes

End

___________________

Known issues Revised June 11, 2019
? Return to Page 3A
28A
Review the following list of known issues to see if any of the documentation can help
you solve your issue:
- 7.2 Following the reboot of the SC Master, the SCIP disappears from all interfaces
and queries against SC fail, article 457063
- 8.0 isi_dnsiq_d goes unresponsive to queries due to network or security scanners,

article 468871
- 7.2.0.1 SmartConnect and IPs are unavailable when node is rebooted or

shutdown for up to 5 minutes, article 471768
- Fixed 8.0 - 8.1 Bug 155852: Smartconnect stops dispatching dns request when
vlan configuration fails in a node.

Did
the documentation
No Consult a coach. If a coach is not
resolve the
issue?
Yes

End

___________________

Appendix A: How to use this flowchart Revised June 11, 2019
Input circle: Colored circles help you navigate

through the flowchart. The first "input" circle on
each page is numbered with the current page Note
#A
number plus an A. Successive input circles on Provides context and
the same page are numbered with the page additional information.
number plus B, then C, etc. Sometimes a note box is
linked to a process step with
Directional
a colored dot.
arrows indicate
the path through
the process flow.
CAUTION!
Caution boxes warn that a Process step with
particular step needs to be Decision diamond Yes command:
performed with great care, to
prevent serious consequences. command xyz
No Output circle: "Output" circles

direct you to another page. Click
the link in the circle to go to the
Go to ## appropriate page. The color of the
output circle matches the color of
the input circle on the linked-to
#B Process step page.
Optional process step
Document Shape
Calls out supporting documentation for a
process step. When possible, these shapes
contain links to the reference document.
Sometimes linked to a process step with a
colored dot.
End point

___________________

Appendix B: Find the SmartConnect IP (SSIP) address in OneFS 8.2 Revised June 11, 2019
? Return to Page 2A
? Return to Page 9A
Directions for determining the SSIP address in OneFS 8.2 - for earlier versions of OneFS, go to the next
________
page.
To return the SSIP that is responsible for resolving a particular zone, follow the steps below.
1. Run the following command, where <zone> is the name of the zone you want to look at:
expor t zone=<zone>
Example output:
cl ust er - 2# expor t zone=cl ust er 1. exampl e. com
2. To find the ID of the pool where the DNS zone is configured, run the following command:
i si net wor k pool s l s - v | egr ep " ( $zone| I D) " | gr ep - B1 $zone
Example output:
cl ust er - 2# i si net wor k pool s l s - v | egr ep " ( $zone| I D) " | gr ep - B1 $zone
I D: gr oupnet 0. subnet 0. pool 0
SC Zone: cl ust er 1. exampl e. com
If there is no output from this command, the pool is not configured. Consult a coach. If a coach is not available, consult your supervisor or manager for
further direction.
3. Run the following command where is the zone ID obtained in step 1:
i si net wor k pool s vi ew - - i d= | egr ep " ( I D| Gr oupnet | Subnet | SC Subnet ) "
Example output:
cl ust er - 2# i si net wor k pool s vi ew - - i d=gr oupnet 0. subnet 0. pool 0 | egr ep " ( I D| Gr oupnet | Subnet | SC Subnet ) "
I D: gr oupnet 0. subnet 0. pool 0
Gr oupnet : gr oupnet 0
Subnet : subnet 0
SC Subnet :
If the SC Subnet field is populated, go to step 4.

If the SC Subnet field is not populated, go to step 5.
4. Run the following command, where <Gr oupnet > is the name of the groupnet that you obtained in step 3 and <SCsubnet > is the name of the SC
Subnet that you obtained in step 3:
i si net wor k subnet vi ew - - i d=<Gr oupnet >. <SCsubnet > | gr ep " SC Ser vi ce Addr s"
5. Run the following command, where <Gr oupnet > is the name of the groupnet that you obtained in step 3 and <Subnet > is the name of the Subnet that
you obtained in step 3:
i si net wor k subnet vi ew - - i d=<Gr oupnet >. <Subnet > | gr ep " SC Ser vi ce Addr s"
The IP addresses in the SC Ser vi ce Addr s field are the cluster's SSIPs.
Example output:
cl ust er - 2# i si net wor k subnet vi ew - - i d=gr oupnet 0. subnet 0 | gr ep " SC Ser vi ce Addr s"
SC Ser vi ce Addr s: 10. 11. 12. 132- 10. 11. 12. 135

___________________

Appendix B: Find the SmartConnect IP (SSIP) address in OneFS 7.2.1 - 8.1 Revised June 11, 2019
? Return to Appendix B
To return the SSIP that is responsible for resolving a particular zone, copy the following command and paste it into the command-line interface, where
<zonename> is the name of the affected SmartConnect zone on the Isilon cluster.
NOTE: When you copy and paste the command into the command-line interface, it will appear on multiple lines, but when you press Enter the
command will run as it should.
OneFS 8.0 - 8.1

echo " " ; zone=<zonename> ; i si net wor k subnet s l i st - v | gr ep - A15 $( i si net wor k pool s l i st - v | \
egr ep " $zone| SC Subnet " | gr ep - A1 $zone | gr ep " SC Subnet " | awk ' { pr i nt $NF } ' ) | \
egr ep " I D: | Sc Ser vi ce Addr : " | gr ep - v VLAN; echo " "
Example: Successful output showing the SCIP

I D: gr oupnet 0. subnet 0
Sc Ser vi ce Addr : 192. 168. 38. 90
Example: Unsuccessful output showing an error

Usage: gr ep [ OPTI ON] . . . PATTERN [ FI LE] . . .
Tr y ` gr ep - - hel p' f or mor e i nf or mat i on.
OneFS 7.2.1
zone=<zonename> ; i si net wor ks l i st subnet s | gr ep $( i si net wor ks l i st pool s - v | \
egr ep $zone\ | " Ser vi ce Subnet " | gr ep - A1 $zone | gr ep " Ser vi ce Subnet " | \
awk ' { pr i nt $NF } ' ) | awk ' { pr i nt $4 } '
Example: Successful output showing the SCIP

192. 168. 38. 80
Example: Unsuccessful output showing an error

Usage: gr ep [ OPTI ON] . . . PATTERN [ FI LE] . . .
Tr y ` gr ep - - hel p' f or mor e i nf or mat i on.
--------------------------------------------------------------------------------------------------------
IF YOU GET AN ERROR, the SmartConnect zone name that you used does not exist on the Isilon cluster.
To see which SmartConnect zone names are configured on the Isilon cluster, run the following command:
egr ep sc- zone- name\ | sc- zone- al i as\ > / i f s/ . i f svar / modul es/ f l exnet / f l x_conf i g. xml | cut - d " >" - f 2 | cut - d " <" - f 1
Example output:
paso- dobl e. name. l ocal
bachat a. name. l ocal

___________________

Appendix C: Locate the subnet and pool that contain the affected Revised June 11, 2019
SmartConnect zone
? Return to Page 2A
Procedure
To find the subnet and pool that contain the affected SmartConnect zone, run the following command. Find the row that
contains the affected SmartConnect zone and use that section of output. The example output in the boxes on this page
highlights such sections of output.
OneFS 8.0 - 8.2

i si net wor k pool s l i st - v | egr ep I D: \ | SC\ Zone: \ | SC\ Subnet
OneFS 7.2.1
i si net wor ks l i st pool s - v | gr ep - E ^\ [ a- zA- Z0- 9\ ] \ | " Zone " \ | " Ser vi ce Subnet "
Example output for OneFS 8.0 - 8.2
I D: gr oupnet 0. subnet 0. dat a01

SC Zone: dat a01. t est . exampl e. com
SC Subnet : subnet 0
I D: gr oupnet 0. subnet 0. f i l e01
SC Zone: f i l e01. t est . exampl e. com
I D: gr oupnet 0. subnet 0. f i l e23
SC Zone: f i l e23. t est . exampl e. com
I D: gr oupnet 0. subnet 0. nas01n1
SC Zone: nas01n1. t est . exampl e. com
Example output for 7.2.1
subnet 1: pool 0 - Def aul t ext - 1 pool

Zone : N/ A
Ser vi ce Subnet : N/ A
subnet 0: Pool 0 - Pool f or cel er a
Zone : uschdnas001p. aonus1. us. aon. bz
Ser vi ce Subnet : subnet 0
subnet 0: Pool 1 - Pool f or Fi l e Ser ver s
Zone : uschdi si 001. aonus1. us. aon. bz
subnet 0: Pool 2 - Pool f or ndmp backups
Zone : N/ A
Ser vi ce Subnet : N/ A
subnet 0: Pool 3 - r epl i cat i on
Zone : uchdi si r epl 1. aonus1. us. aon. bz

___________________

Appendix D: Locate the cluster's externally configured DNS IP address Revised June 11, 2019
? Return to Page 2A
Procedure
To find the IP addresses of the configured DNS servers, run the following command:
OneFS 8.0 - 8.2

i si net wor k gr oupnet s l i st - v | egr ep I D: \ | " DNS Ser ver s"
OneFS 7.2.1
i si net wor ks | gr ep " Domai n Name Ser ver "
Example output for OneFS 8.0 - 8.2
The I D field represents the name of the groupnet.
cl ust er - 1# i si net wor k gr oupnet s l i st - v | egr ep I D: \ | " DNS Ser ver s"
I D: gr oupnet 0
DNS Ser ver s: 10. 111. 252. 14, 10. 111. 252. 15
I D: t est
DNS Ser ver s: 1. 1. 1. 1
Example output for OneFS 7.2.1
cl ust er - 1# i si net wor ks | gr ep " Domai n Name Ser ver "
Domai n Name Ser ver : 10. 104. 128. 235

___________________

Appendix E: Locating the SmartConnect Service IP addresses Revised June 11, 2019
? Return to Page 4A
OneFS 8.2 Example output for isi network subnets list -v

cl ust er - 1# i si net wor k subnet s l i st - v
I D: gr oupnet 0. subnet 0
Name: subnet 0
Gr oupnet : gr oupnet 0
Pool s: pool 0
Addr Fami l y: i pv4
Base Addr : 10. 224. 36. 0
CI DR: 10. 224. 36. 0/ 22
Descr i pt i on: I ni t i al subnet
DSR Addr s: -
Gat eway: 10. 224. 36. 1
Gat eway Pr i or i t y: 10
MTU: 1500
Pr ef i xl en: 22
Net mask: 255. 255. 252. 0
SC Ser vi ce Addr s: 1. 2. 3. 4- 1. 2. 3. 6, 1. 2. 3. 8
SC Ser vi ce Name:
VLAN Enabl ed: Fal se
VLAN I D: -
If the environment is a single-SSIP, the value in the SC Ser vi ce Addr s field will look similar to this: 1. 2. 3. 4 - 1. 2. 3. 4
If the environment is a multi-SSIP, the value in the SC Ser vi ce Addr s field will look similar to this: 1. 2. 3. 4 - 1. 2. 3. 6, 1. 2. 3. 8- 1. 2. 3. 9
OneFS 7.2 - 8.2 Example output for nslookup <zone-name> <Isilon-DNS-IP>
The expected output contains a node's IP address from the named zone.
The output looks similar to the following. Note that there are two listings for Addr ess. You want the Addr ess that is associated with the zone
name that you used in the command (shown in blue, bold text here).
Ser ver : 192. 168. 1. 1

Addr ess: 192. 168. 1. 1#53
Name: i si l on. exampl e. or g

Addr ess: 10. 111. 112. 113

___________________

Revised June 11, 2019
Copyright © 2019 Dell Inc. or its subsidiaries. All rights reserved.
Dell believes the information in this publication is accurate as of its publication date. The
information is subject to change without notice.
THE INFORMATION IN THIS PUBLICATION IS PROVIDED "AS-IS." DELL MAKES NO REPRESENTATIONS

OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND
SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FORA
PARTICULAR PURPOSE. USE, COPYING, AND DISTRIBUTION OF ANY DELL SOFTWARE DESCRIBED IN
THIS PUBLICATION REQUIRES AN APPLICABLE SOFTWARE LICENSE.
Dell, EMC, and other trademarks are trademarks of Dell Inc. or its subsidiaries. Other trademarks
may be the property of their respective owners.
EMC Corporation
Hopkinton, Massachusetts 01748-9103
1-508-435-1000 in North America 1-866-464-7381
www.EMC.com

___________________

Isilon Troubleshooting Guide: Networking: SmartConnect

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Isilon Troubleshooting Guide: Networking: SmartConnect

Uploaded by

Copyright:

Available Formats

Isilon Troubleshooting Guide: Networking - SmartConnect

Before you begin Revised June 11, 2019

This troubleshooting guide applies to OneFS 7.2 - 8.2

This TSG has the following related customer troubleshooting guides

How to use this document:

What is the SSIP?

We appreciate your help in improving this document.

Page 1 of 35 Dell - Internal Use - Confidential

Record this hashtag in the case

INITIATE CODE RED

Collect the following information and record it in the

Information to record How to find the information

The name of the affected SmartConnect zone on the Isilon

The SmartConnect Service IP (SSIP). For instructions, see Appendix B.

Run the following command, where <SSI P> is the

If there is no output, note that in the case notes and continue.

The IP addresses of the Isilon cluster's externally configured

From a Linux client, run:

We appreciate your help in improving this document.

Page 2 of 35 Dell - Internal Use - Confidential

Check whether the affected SmartConnect zone exists in the

OneFS 8.0 - 8.2

If the answer is no, the customer probably never configured

Tell the customer to create the SmartConnect zone name in

If the customer has any questions while using this guide,

Record this hashtag in

Does the customer

We appreciate your help in improving this document.

Page 3 of 35 Dell - Internal Use - Confidential

Open an SSH connection to any node in the cluster EXCEPT FOR

OneFS 8.2 supports multiple SmartConnect service IPs. You

nsl ookup <zone- name> <I si l on- DNS- I P>

Record the output in the case notes.

We appreciate your help in improving this document.

Page 4 of 35 Dell - Internal Use - Confidential

Ping the DNS server from the cluster by running the

pi ng <I si l on- DNS- I P>

Record the output in the case notes.

Record this hashtag in the case

We appreciate your help in improving this document.

Page 5 of 35 Dell - Internal Use - Confidential

The SmartConnect An IP address that is within Anything else

Record this hashtag in the case

Record this hashtag in

OneFS 8.0 - 8.2

Output example 1 - The IP address is in a pool.

Output example 2 - The IP address is not in a pool.

We appreciate your help in improving this document.

Page 6 of 35 Dell - Internal Use - Confidential

From the cluster, ping the SSIP:

Did the ping

i si _f or _ar r ay ' i f conf i g | gr ep <SSI P>'

If the SSIP exists, the output looks similar to the following:

The SSIP exists, but you cannot reach it through the

This is probably due to a bug. Use following article to try to

Following the reboot of the SC Master, the SCIP disappears

Record this hashtag in the case

We appreciate your help in improving this document.

Page 7 of 35 Dell - Internal Use - Confidential

nsl ookup <ZoneName> <SSI P>

Record the output in the case notes.

What does the

The SmartConnect An IP address within the * * ser ver can' t There is

Record this hashtag There is a problem with the customer's