You are on page 1of 134

Troubleshooting and Debug Guide |TAC

Hillstone C/E/M/G/X/T Series

Device Troubleshooting and Debuge

Guide

TAC 2nd level support


Troubleshooting and Debug Guide |TAC
Troubleshooting and Debug Guide |TAC

Chapter 1 Interface

Aggregate Interface

Specification and Implementation

M platform and X platform support the same speed of optical port aggregation and
electrical port aggregation
M platform and X platform support the crossing board card interface aggregation
The maximum number of each agg inter face is 16 interfaces
Static binding support Gigabit and 10 Gigabit binding, as following
SG-6000(config-if-xe3/1)# show interface aggregate1
--------------------------------------------------------------------------------------------------------
Interface aggregate1
downstream bandwidth is 11000000000
upstream bandwidth is 11000000000

The interface total has 2 memeber interface


Member interface 1 ethernet4/1
Member interface 2 xethernet3/1

Work with CISCO static binding configuration

CISCO’ s convergent pattern need to configure as on model(For example, channel‐


group 1 mode on)
Troubleshooting and Debug Guide |TAC

Two modes of traffic load

Mode 1:
400-GuanWu(config)# flow
400-GuanWu(config-flow)# aggregate-distribute-mode session-based
After configured with session-based, the system will select the remainder in
termsof the current session ID to select the member interface
SG-6000-B(config-flow)# flow aggregate-distribute-mode flow-based
Select the member interface in terms of quintuple of flow, this is the system default
way

Mode 2:
400-GuanWu(config-if-agg1)# load-balance mode    
  flow              Flow-based load balance                       
Default Configuration. If configured this one, then it works as global configuration
of “flow aggregate-distribute-mode”.
tuple             Custom load balance                            
The highest priority. If configured this one, then it does not work as global
configuration of “flow aggregate-distribute-mode”.
400-GuanWu(config-if-agg1)# load-balance mode tuple
  <cr>             
  dest-ip           Destination ip address
  dest-mac          Destination mac address
  dest-port         Destination port
  protocol          Transfor protocol
  src-ip            Source ip address
  src-mac           Source mac address
  src-port          Source port
Troubleshooting and Debug Guide |TAC

Troubleshooting

Case one, the flow of device decreased after the aggregate interface

binded QOS, and packet loss happend

Use command “Show inter aggregate” to check if the bandwidth configuration


of interface is correct. In the old version, the bandwidth value of interface does not
change after the aggregate interface added new memebers, it makes the out
direction was impacted by the limitation of bandwidth value after the interface
binded QOS then started losing packet, but this problem has been fixed in the new
version, please update to 5.0R2P3 or advancer version.

Case two, Interface frequently DOWN/UP

In some cases, some of interface in the device (usually connect with other
manufacturer’s device by optical port or electrical port) are frequently DOWN/UP,
the time interval of DOWNUP is around 1S, actually R&D had troubleshooting this
problem before, but they never confirm that the reason was cause by Hillstone
device, below are the suggested troubleshooting steps for this problem:
1. Confirm the configuration of speed and duplex for each end interfaces are exactly
same.
2. If both end are auto mode, suggest both of end change to manual mode.
3. If the customer want to find the root cause, we suggest to add one switch(Better
be gateway) between the two device if the condition allowed, and then check the
interface status of both end of device to locate the problem device.
4. If the customer does not care the issue, then he can avoid it by using hold up
Troubleshooting and Debug Guide |TAC

command, below are the steps::


DEMO2(config)# interface ethernet0/1
DEMO2(config-if-eth0/1)# holdup ?
<1-3600> In increments of 500 milliseconds
The recommend value setting is 3 or 4.

Case Three, the optical port light of firewall does not turn on when the

firewall optical port connect with Cisco switch optical port.

1. Check if the Cisco switch using the original optical module, suggest to use remote
module.
2. Check the negotiation mode of both end, suggest to set the duplex and speed of
interface as duplex or half duplex mode.
3. If the problem still existed, delete all the configuration and configure both
interface again.

Case four, after configured the port trunk with opposite device, the

device report loop log for each 20 minutes.

Appearance : After the port trucking of hillstone device with H3C device, the
hillstone device report a loop log every twenty minutes after that.
Reason:The ARP digestion time of Hillstone device is 20 minutes, the firewall will
send ARP request again when the current ARP time out, but if the opposite device
send the ARP request back to firewall will cause the event.
Solution:
1. Check if H3C switch has configured “port bridge enable” this command, if it does,
please delete it.
Troubleshooting and Debug Guide |TAC

2. If the opposite device did not configure the corresponding command, please set
static ARP binding in hillstone device.。

Chapter 2 NAT

Configuration Examples

The deployment and configuration of transparent NAT

Requirement:The customer with an real IP of 10.1.1.0 is going to visit a server (real


IP is 20.1.1.2) with a external virtual IP 2.1.1.2, translate the client’s IP address as
1.1.1.0, and translate the destination IP address as 20.1.1.2, to hide bi-directional IP.

Configuration:address “src”
ip 10.1.1.0/24
exit
address “srcpool”
ip 1.1.1.0/24
exit
Troubleshooting and Debug Guide |TAC

interface ethernet0/5
zone “l2-untrust”
exit
interface ethernet0/6
zone “l2-trust”
exit
interface vswitchif1
zone “trust”
exit
vswitchif vswitchif1
nat-enable
exit
ip vrouter “trust-vr”
snatrule id 1 from “src”to “2.1.1.2”trans-to address-book“srcpool”mode
dynamicport
dnatrule id 1 from “src”to “2.1.1.2”trans-to “20.1.1.2”
exit
policy-global
rule from src to 2.1.1.2/32 service any permit
exit
Notes:
1. Does not support WebUI、HSM etc management way,only support CLI。
2. Does not support cone NAT function。
3. Does not support to monitor if DNAT server reachable by configuring track-tcp or
track-ping.
4. If you configured the rule that the next hop of flow is the address after
translation of VRouter and NAT as egress interface IP address, this rule does not
work in the transparent application mode.
5. If you configured the rule for NAT444 dynamicport fixedblock mode,it does not
work in the transparent application mode。
6. Does not support the root VSYS share the VSwitch to non-root VSYS, it cannot
share with the non-root VSYS after transparent NAT enabled.
7. The StoneOS version since after 5.0R3 supports this configuration.
Troubleshooting and Debug Guide |TAC

Troubleshooting

Case One、It does not work to do the SNAT and DNAT at the same time

after upgrade to StoneOS 5.0 version

Analysis : The reason is different packet handling procedure between StoneOS 5.0
version with 4.0 or 4.5version. The SNAT of 5.0 version is going to match with the
destination IP address before SNAT, but the SNAT of 4.0 and 4.5version was matching
the destination IP address after DNAT.。

WAN:192.168.1.1 LAN:10.88.16.157
Client:192.168.1.100 Server:10.88.16.181

As the below figure shown, it required to make client visiting server by through
WAN interface IP address, also translate the source address of client to LAN
interface IP address.
For the non-5.0 version, we need to add following dnat : from 192.168.1.100
to 192.168.1.1 translate to 10.88.16.181,also add snat : from 192.168.1.100 to
10.88.16.181 translate to 10.88.16.157,the source ip of policy is 192.168.100,the
dst ip is 192.168.1.1;
For the advancer version of 5.0, the dnat and policy keep the same,snat policy
from 192.168.1.100 to 192.168.1.1 translate to 10.88.16.157。

Case two、port/IP mapping does not work

1. Check the server to see if the service enabled normarlly


2. Check the corresponding port to see if it has been block by provider;
Troubleshooting and Debug Guide |TAC

3. If the internal server did not return the packet, try to add SNAT policy and
translate the source address to internal network interface IP address.
4. Use”show dnat server” to check the current status of server
5. Use “show session src-ip x.x.x.x” or “show session dst-ip x.x.x.x” to check if the
NAT translation is correct
6. Use “debug dp filter src-ip x.x.x.x” or “debug dp filter dst-ip x.x.x.x , debug dp
basic”(Confirm if the cpu utilization is overload before debug

Chapter 3 Policy

Case one、Configuration of longlife session

1. Self-define the longlife port, and setup the timeout.


2. Enable the longlife, suggest to make the settings less than 10% to avoid too much
longlife occupy the device resource.
The below is the command:
“longlife-sess-percent”

Troubleshooting

Case one 、 Policy refered the address book with domain name but

does match

1. Confirm the device configured with DNS server, and make sure the device can
Troubleshooting and Debug Guide |TAC

resolve the domain name of address book.


2. Confirm the DNS server of device and PC are the same one, otherwise it cannot
be matched.
3. If confirmed all the above issues are ok, it could be caused by the bug of
4.0version, then need to upgrade to 4.0R6P17 or advancer version.

Chapter 4 ALG

Configuration and Function

Case one 、 Two ways of accessing to non-standard port FTP by port

mapping

Solution one:
1.Self-define service FTP_2121 ,destination port is 2121, application is FTP,enable
the ALG of FTP.
2. Add destination NAT policy, mapping the 2121 port to 21 port.
3. Add policy from external security zone to internal security zone, with service
FTP_2121.
Solution two:
1. Self-define service FTP_2121,destination port is 2121, application is FTP,enable
the ALG of FTP.
2. Add destination NAT policy, mapping the 2121 port to 21 port.
3. Add policy from external security zone to internal security zone, with any to any
service.
4. Enable the application recognization function of internal security zone.
Troubleshooting and Debug Guide |TAC

5. Make sure to disable the tcp-other scan of IPS.

Troubleshooting

Case one、ALG function of non-standard port does not work

1. Check if the ALG function of corresponding protocol enabled;


2. Check if “alg auto” command enabled;
3. Make sure the application recognization function of security zone enabled.
4. Check if the port has been defined for other application

Case two、PC dial PPTP by firewall does not work,error code is 806

The firewall configured DNAT mapping internal PPTP server to external


address,self-defined the service book with destination port 1723, and set the
application as user-defined, which caused the firewall recognized these flow as
user-defined application but not PPTP, does not go through with the normal ALG
handling procedure, the message interaction is abnormal. The solution is to change
the appliocation of user-defined service as “none”. But also need to make sure that
the application recognization function of security zone enabled.

Case Three、ALG FTP is abnormal after reverser routing shut down

Topology : As the below figure show, because there is no route ahead to


10.148.136.0/24 in the three-layer switch, so we had configured SNAT in the 1.1.1.1
interface. Also because there are some other network segment on the left side of
firewall, but the firewall did not add the route for those networls, so the reverse
routing of 2.2.2.1 interfacee has been shut down also.
Troubleshooting and Debug Guide |TAC

Apperance:When the internal network use FTP client(10.148.136.61)to connect


with FTP server ( 172.25.101.24 ) , the disconnection situation happened
irregularly.

Analysis :Capturing the packet on the both side of firewall and client at the same
time, comparing the success packet and the failure packet.
Success capturing packet:

Failure capturing packet:


Troubleshooting and Debug Guide |TAC

We found the first packet is abnormal after consulting with passive mode. We can get
the syn ack from opposite end after we send out the syn packet when the
connection is successful. But we cannot get the syn ack when the connection failed.
User command “debug dp snoop,debug dp alg,debug dp basic” to analyze the
debug information of firewall, according to the seq of abnormal packet to find the
debug packet of firewall, comparing the handling process of these two packets(the
below left figure is the normal packet, the right figure is the failure packet):
Troubleshooting and Debug Guide |TAC

We found that ALG changed the mac address when it was handling the cache_mac
of packet, but when the connection is normal, the man address is the real mac
address, and the mac address turn to a wrong mac address when the connection
failed. This is the reason why the ack from server send to the wrong mac address
after we send the packet to the server. At the same time, the reverse routing of
firewall internal interface has been shut down, so the firewall did not check the
route information again after received ACK, but directly capture the packet in terms
of the wrong mac address, therefore, the client-end did not receive the SYN ack.。

Solution : Enable the reverse routing of internal network interface, also add the
route of aheading corresponding network segment.。
Troubleshooting and Debug Guide |TAC

Chapter 5 Route

FAQ

1. Does PBR drainage function works for the message drainage of self?
A:YES。
2. What Multicase Routing Protocol we supported?
A:For now, we only support static multicase protocol, and we don’t support mospf,
CBT, PIM-SM, DVMRP.

Troubleshooting

Case one、Policy routing does not work

1. Firstly, check if the network flow comes from the device itself. The flow of device
will not match with policy or rule.(Except for drainage function)
2. Check if the binding object binded with the ingress interface or security zone of
data
3. Check the service book was configured by the policy route, and check if the
corresponding security zone has been enabled the application recognization
function.
4. Check if the application feature library has been updated to the latest version,
check the If the real traffic has wrong recognization by statistics.
5. Check if the memory of device is too lack to handle the policy matching
6. Use “show config” to check if the policy route has been disabled, use “route
enable pbr” to enable it if it has been disabled.

Case two 、 Transparent deployment cannot unvarnished transmit

multicase data
1. Firstly we need to confirm if this issue was caused by Hillstone firewall, please
suggest customer skip the device test to check the application if it is available.
Troubleshooting and Debug Guide |TAC

2. If confirmed the application interruption was caused by hillstone device, then


capture the packet, check the if the session will be created, and what the
application has been recognized, for instances, the following application has been
recognized as RTSP.

3. In this case, the application might has been recognized by wrong, so suggest to
permit any to any in the policy of multicast source to multicast destination
address.;
4. If it is still does working, check if the NAT has been enabled, if it is, please delete
the NAT of multicast source and multicast destination address.
5. If it is still does workin, then we need to enable “igmp-snooping” function, and
selec “igmp-snooping” mode for the interface,usually the side is nearby router
that is host-mode, the other side is router-mode.
6. If it is still does working, use “no l4-vlidity-check” under flow mode, ignore the
forth layer application check.
7. If it is still does working, use virtual wire.
8. If it is still does working, collect debug information, submit case to 2 nd level. Use
command “debug dp basic, debug dp drop, debug dp snoop, debug dp policy
lookup, debug igmp, debug dp app, debug strmengine”。

Case Three、cannot unvarnished transmit multicast data under routing

mode
1. Debug in device, check if received multicast packet. If the packet has been
received, analyze if the session log of packet is normal. If there is no debug
information, please login the router to check the multicast status, if the pc is in
the multicast zone. Or skip to hillstone device check if any other thing good.
2. If confirmed that issue was caused by hillstone device, please confirm if the
configuration of mroute is right, try to enable “igmp-proxy”, appoint the mode.
3. If it is still does working, user “no l4-vlidity-check” to disable it under flow mode.
4. If it is still does working, please capture the packet in PC and device, submite case
to TAC 2nd level.
Troubleshooting and Debug Guide |TAC

Case Four、GRE over IPSEC cannot pass the multicast video


After the connection created of GRE over IPSEC, both side internal can ping each
other, but cannot transmit the multicase video, below is the topology:

1. Local side FW2 device debug: 


2014-07-11 22:07:05, DEBUG@FLOW: core 1 (sys up 0x30d525e ms): 51204: (i)
len=1358 3c97.0e6e.3b38->0100.5e14.0001/800 
192.168.3.100->224.20.0.1/17 
vhl=45, tos=00, id=23684, frag=0000, ttl=1, tlen=1344 
udp:ports 55644->1234, len=1324 
rx_handle_prepare: 3c97.0e6e.3b38->0100.5e14.0001, size 1358, type 0x800, vid 0,
port ethernet0/0 
dp_prepare_if_for_pak 
Switchid is 8(interface ethernet0/0) port ethernet0/0 
Switchid is 8(interface ethernet0/0) port ethernet0/0 ,pak iif=ethernet0/0 
rx_handle_prepare i_if is ethernet0/0 
rx_handle_prepare calling dp_sanity ethernet0/0 
Start l3 forward 
Packet: 192.168.3.100 -> 224.20.0.1, id: 23684, ip size 1344, prot: 17(UDP): 55644 ->
1234 
ad_vector_for_fast_flow: zonename trust, proto_flag[1] 0, proto 17 
dp_prepare_pak_lookup srcip: 192.168.3.100, dstip: 224.20.0.1,prot 17 
No session found, try to create session 
IP multicast packet from interface ethernet0/0. 
checking weather the packet is going to self... 
The to-self service is not registered 
MC-Dropped: begin to creating session for MC packet! 
Troubleshooting and Debug Guide |TAC

No DNAT configured for this VR 


begin lookup predefine prot:17 port:1234 
Identified as app UDP-ANY (prot=17). timeout 60. 
MC-Dropped: Multicast Forwarding Cache not found !! 
Droppped: failed to create session, drop the packet (action=0) 

2. Add multicast route in FW2, then debug: 


SG-6000[DBG](config)# show logging debug 
2014-07-11 22:30:33, DEBUG@FLOW: core 1 (sys up 0x322cfaf ms): 52613: (i)
len=1358 3c97.0e6e.3b38->0100.5e14.0001/800 
192.168.3.100->224.20.0.1/17 
vhl=45, tos=00, id=17711, frag=0000, ttl=1, tlen=1344 
udp:ports 58613->1234, len=1324 
rx_handle_prepare: 3c97.0e6e.3b38->0100.5e14.0001, size 1358, type 0x800, vid 0,
port ethernet0/0 
dp_prepare_if_for_pak 
Switchid is 8(interface ethernet0/0) port ethernet0/0 
Switchid is 8(interface ethernet0/0) port ethernet0/0 ,pak iif=ethernet0/0 
rx_handle_prepare i_if is ethernet0/0 
rx_handle_prepare calling dp_sanity ethernet0/0 
Start l3 forward 
Packet: 192.168.3.100 -> 224.20.0.1, id: 17711, ip size 1344, prot: 17(UDP): 58613 ->
1234 
ad_vector_for_fast_flow: zonename trust, proto_flag[1] 0, proto 17 
dp_prepare_pak_lookup srcip: 192.168.3.100, dstip: 224.20.0.1,prot 17 
Found the session 8 
session: id 8, prot 17, flag0 80000000,flag1 0, created 52556, life 3 
flow0(if id: 8 flow id: 16 flag: 40000810):192.168.3.100:58613 
->224.20.0.1:1234 
flow1(if id: 0 flow id: 17 flag: 10): 0.0.0.0:0 
->0.0.0.0:0 
Troubleshooting and Debug Guide |TAC

Dropped: TTL is too small. Send back icmp packet 


Dropped: Arp get fail, ip:0.0.0.0, interface ethernet0/0 

2. Change client TTL value of multicast source, change the 1 to 128: 


SG-6000[DBG](config-vrouter)# show logging debug 
2014-07-11 23:04:14, DEBUG@FLOW: core 1 (sys up 0x341a41b ms): 54633: (i)
len=1358 000c.29c5.785b->0100.5e14.0001/800 
192.168.3.100->224.20.0.1/17 
vhl=45, tos=00, id=44679, frag=0000, ttl=128, tlen=1344 
udp:ports 1274->1234, len=1324 
rx_handle_prepare: 000c.29c5.785b->0100.5e14.0001, size 1358, type 0x800, vid
0, port ethernet0/0 
dp_prepare_if_for_pak 
Switchid is 8(interface ethernet0/0) port ethernet0/0 
Switchid is 8(interface ethernet0/0) port ethernet0/0 ,pak iif=ethernet0/0 
rx_handle_prepare i_if is ethernet0/0 
rx_handle_prepare calling dp_sanity ethernet0/0 
Start l3 forward 
Packet: 192.168.3.100 -> 224.20.0.1, id: 44679, ip size 1344, prot: 17(UDP): 1274
-> 1234 
ad_vector_for_fast_flow: zonename trust, proto_flag[1] 0, proto 17 
dp_prepare_pak_lookup srcip: 192.168.3.100, dstip: 224.20.0.1,prot 17 
No session found, try to create session 
IP multicast packet from interface ethernet0/0. 
checking weather the packet is going to self... 
The to-self service is not registered 
MC-Dropped: begin to creating session for MC packet! 
No DNAT configured for this VR 
begin lookup predefine prot:17 port:1234 
Troubleshooting and Debug Guide |TAC

Identified as app UDP-ANY (prot=17). timeout 60. 


MC-Dropped: RPF checking failed !! 
Droppped: failed to create session, drop the packet (action=0) 

3. The RPF of multicast( reverse path forwarding) 。 Did not configure ingress
interface in the multicast route configuration of local FW2, the multicast packet
can be transmit as normal after the configuration done, then debug in the
opposite end of FW1: 
SG-6000[DBG](config-vrouter)# show logging debug 
2014-07-11 11:41:24, DEBUG@FLOW: core 1 (sys up 0x364066b ms): Finish
decap 
Packet: 192.168.3.100 -> 224.20.0.1, id: 44098, ip size 1344, prot: 17(UDP): 1395
-> 1234 
dp_prepare_pak_lookup srcip: 192.168.3.100, dstip: 224.20.0.1,prot 17 
No session found, try to create session 
IP multicast packet from interface tunnel1. 
checking weather the packet is going to self... 
The to-self service is not registered 
MC-Dropped: begin to creating session for MC packet! 
No DNAT configured for this VR 
begin lookup predefine prot:17 port:1234 
Identified as app UDP-ANY (prot=17). timeout 60. 
MC-Dropped: Multicast Forwarding Cache not found !! 
Droppped: failed to create session, drop the packet (action=0) 

4. The multicast communication works fine after the multicast route was
configured in the FW1. 
The static multicast route of local FW2: 
SG-6000[DBG](config-vrouter)# show ip mroute 
U:interface up D:interface down V:valid multicast entry I:invalid multicast entry 
=================================================================== 
source: 192.168.3.100 group : 224.20.0.1 vrouter: trust-vr 
status: V update time: - 
ingress interface: ethernet0/0(U) 
egress interface : tunnel1(U) 
=================================================================== 
Total: 1 

The static multicast route of opposite end FW1:: 


SG-6000# show ip mroute 
U:interface up D:interface down V:valid multicast entry I:invalid multicast entry 
=================================================================== 
Troubleshooting and Debug Guide |TAC

source: 192.168.3.100 group : 224.20.0.1 vrouter: trust-vr 


status: V update time: - 
ingress interface:  tunnel1(U)
egress interface : ethernet0/0(U)
=================================================================== 
Total: 1 

Chapter 6 FLOW

Configuration and Function statement

Case one、session rematch guide

Firstly, session rematch function guide


Session rematch function is configuration option of hillstone firewall’s layer two
forwarding function, no impact to layer three forwarding. In most of cases, the
routing of two directions of a session are the same, then we don’t need to enable
session rematch function.
When the two directions have different forwarding routing, or the forwarding
routing changed after the session created, we need to enable session rematch
function.
This function can solve the traffic obstruction issue which was caused by asymmetric
route and route changed when the upper stream and down stream is selecting
router.。
Secondly, Session rematch function theory
After enable the session rematch function, the traffic will be forward according to the
configuration of Virtual Wire when VSwitch is working on Virtual Wire mode, but not
in terms of egress interface that recorded in the session. When the VSwitch is not
Troubleshooting and Debug Guide |TAC

working on the Virtual Wire mode, it will compare the destination MAC address of
packet and session record, if the result is different, it will try to search the MAC
address list to find egress interface again according to the packet’s destination MAC,
if the new interface is same as the session’s, the VSwitch will directly forward the
packet, if it is not same, the VSwitch will update the egress interface information of
session and forward the packet if both two are in the same security zone, but if they
are in the different security zone, the VSWitch will delete session, dorp the packet,
and makes the future packet to trigger the session rebuilding.
Flow Chart:
Troubleshooting and Debug Guide |TAC

三、Session rematch Impact


Enable Session rematch function,has no impact to layer three forwarding, but only
affect to layer two forwarding, and those impacts include: The performance of layer
two forwarding will reduce 5% when the function enabled, because it need to do
some comparison.
Fourthly, this function is for 5.0R3 version, it will turn into automatic since after 5.0R4
version and 5.5version
五、Configuration example:
DEMO2(config)# flow
DEMO2(config-flow)# session rematch

Chapter 7 VPN

IPSEC VPN

Classic Configuration

Case one、Hillstone self signed IPSEC certification

Two Hillstone devices build an IPSEC VPN without third party CA server.

Configuration:
Troubleshooting and Debug Guide |TAC
Troubleshooting and Debug Guide |TAC

CRL can be ignore, click ok.


The final setting of trust-domain is below:

Use the same way to create trust domain “B device self-signed” , the final
configuration is below:
Troubleshooting and Debug Guide |TAC

The way to export device A certification:


Troubleshooting and Debug Guide |TAC

“B device self-signed certification”import device A


Creat trust domain in device A, then Import “ B device self-signed certification”
Troubleshooting and Debug Guide |TAC

VPN configuration:
Troubleshooting and Debug Guide |TAC

Same way to configure B device。


For the peer-id of device A,login to device B(recommend by CLI) to use “show
isakmp peer” to check the local id of device B,change the “/”to “,”in the ID, and
then fill up with the peer-id of device A,for example:
Device B:

peer-id of Device A:
Troubleshooting and Debug Guide |TAC

Same way to configure the peer-id of device B

Case two、L2TP over IPsec over GRE VPN using

Below is the topology:


The cellphone user dial in APN device through GSM network, the APN device connect
with Hillstone via GRE tunnel, the cellphone user create L2TP Over Ipsec VPN by GRE
tunnel and hillstone connections to access internal resource.
Troubleshooting and Debug Guide |TAC

From the perspective of client, this request can be satisfied (need to add one more
GRE packaging out of ipsec), the device need to be done the following configuration:
1. Configure the GRE tunnel between APN and Firewall
2. Configure L2tp Over Ipsec
But we have meet the below problems:
1. Hillstone received the decryption packet from the GRE tunnel,the source address
of ESP packet is 1.1.1.1—>5.5.5.1 ,in the normal case, the system should keep
decrypting the packet, and then we get the L2TP packet with destination port of
1701. However, the packet has been dropped after the first decryption in the
process of configuration.We suspect the reason is that the security zone of GRE
tunnel interface is different with the internal network interface’s security zone.By
default of hillstone device, the source security zone of VPN data need to be same
as the security zone of dial interface.
2. After we have configured the security zone of GRE tunnel interface and L2TP
tunnel interface as TRUST that same as that of internal network interface, the
data can be decrypt again, but it was still dropped after decryption done(Please
take the below figure as reference), according to the hint of Drop, we suspect the
reason is that the flow0’scorresponding tunnel interface is the tunnel2 of l2tp
when the session was created, but when we created flow1, the existing GRE VPN
route ofip route 1.1.1.1/24 tunnel4(the tunnel interface of GRE) caused the next
Troubleshooting and Debug Guide |TAC

hop to appoint to tunnel4, so the error happened.

Solution:According to the request of customer, we don’t have a best solution under


current topology, but we can avoid it via build more VR in Hillstone Device.We can
Troubleshooting and Debug Guide |TAC

built two VR, the first VR use to establish GRE with APN device, and configure L2TP
over IPsec function for the second VR. After test, we confirmed this solution is
working.

Collection of pediatric cases: Establish IPsec between Hillstone device

and Partner’s device

With CISCO

Configuration examples and points

场 景 一 、 Establish IPSecVPN configuration between Hillstone Firewall with Cisco


router
In daily work we usually meet the situation that we need to establish IPSecVPN
between hillstone firewall with Cisco Router or ASA device, below is a simple manual,
please take it as a reference.

Cisco device configuration


1. Configure the phase one IKE
crypto isakmp policy 1 Genrate iskamp policy number 1
encryption des Select DES or 3DES encryption
hash sha Using Hasing algorithm or smd5
authentication pre-share Authentication methods is using pre-share key, it is digital
signature authentication by default.
group 1 Specified Diffie-Hellman group,1 means 768 bit,2 means 1024 bit
lifetime 14400 Specified period of validity of security association, it will be
default value if did not set
Troubleshooting and Debug Guide |TAC

crypto isakmp key 0 hillstone address 202.102.1.6 Specified ip address of remote


router, use pre-share key:hillstone, key 0 means the pre-share key will be display by
plaintext in cisco side.

2. Configure the IKE policy of phase two


2.1 Configuration
Ip access-list extended vpn
Permit ip 1.1.1.0 0.0.0.255 2.2.2.0 0.0.0.255

2.2 Configure IPSec policy


Crypto ipsec transform-set Trans esp-des esp-md5-hmac Conversion name is
Trans,ues esp packaging and DES encryption, and use Md5 to check completeness

2.3 Configure Crypto map


crypto map map-hillstone 10 ipsec-isakmp map’s name is map-hillstone ,use id 10
to mark the map,one map can configure multiple ID, but one interface can only refer
one map.
set peer 202.102.1.6 set opposite end destination address
set transform-set Trans Quote Trans’s conversion collection
match address vpn
set pfs group2 enable PFS
set security-association lifetime second 1800 configure the TTL for IPsec sa
3. Apply Crypto map to interface
interface serial0
ip address 202.102.1.5 255.255.255.0
crypto map map-hillstone

Please notice the following problems:


1. Propose of two sides need to be the same
2. The IPsec of cisco was tigger by traffic, generally, hillstone device will be initiative
Troubleshooting and Debug Guide |TAC

during the process. But if the propose or proxy ID does not match, then hillstone
device won’t receive the ACK from cisco. Then we need to debug in cisco side,
collect some VPN debug information from cisco router.。
3. The proxy ID of hillstone device must be the same as Cisco’s, for example, if Cisco
side is 1.1.1.0 0.0.0.255 , then hillstone side should be 1.1.1.0/24 , not
1.1.1.1/24.

Case two、Configuration example for L2L VPN between Hillstone and ASA 8.4
1. Basic configuration of ASA
1.1 Clear the configuration of device
ciscoasa#clear config all //Clear all the configuration of device
1.2 Check related content
ciscoasa# dir //check files
Directory of disk0:/
92 -rwx 25214976 17:43:58 Jun 09 2014 asa844-k8.bin
80 -rwx 6889764 18:11:54 Aug 29 2012 asdm-602.bin
2 drwx 2048 18:13:30 Aug 29 2012 log
5 drwx 2048 18:13:46 Aug 29 2012 crypto_archive
82 -rwx 25159680 00:27:48 Nov 19 2012 asa842-k8.bin
83 -rwx 18927088 00:29:26 Nov 19 2012 asdm-649.bin
84 -rwx 0 00:32:20 Nov 19 2012 nat_ident_migrate
85 -rwx 2726 17:07:12 Nov 19 2012 8_0_2_0_startup_cfg.sav
86 drwx 2048 00:32:22 Nov 19 2012 coredumpinfo

1.3 Check directory of IOS


1.4 ciscoasa(config-if)# show ver // Check directory of IOS
Cisco Adaptive Security Appliance Software Version 8.0(2)
Device Manager Version 6.0(2)
Compiled on Fri 15-Jun-07 19:29 by builders
System image file is "disk0:/asa802-k8.bin"
Troubleshooting and Debug Guide |TAC

Config file at boot was "startup-config"

2. Upgrade IOS
2.1 Bakcup the current IOS at first:
ciscoasa# copy disk0:/asa802-k8.bin ftp:192.168.100.200
Source filename [asa802-k8.bin]?
Address or name of remote host []? 192.168.100.200
Destination filename [192.168.100.200]? asa802-k8.bin
Writing file ftp://192.168.100.200/asa802-k8.bin...

2.2 Delete the old IOS file after the backup done
ciscoasa# delete ?
/noconfirm Do not prompt for confirmation
/recursive Recursive delete
disk0: File to be deleted
flash: File to be deleted
ciscoasa# delete disk0:asa802-k8.bin
Delete filename [asa802-k8.bin]?
Delete disk0:/asa802-k8.bin? [confirm]

2.3 Update the new IOS:


ciscoasa# copy ftp://192.168.100.200 disk0:/asa844-k8.bin
Address or name of remote host [192.168.100.200 ]?
Source filename []? asa844-k8.bin
Destination filename [asa844-k8.bin]?
Accessing ftp://192.168.100.200 /asa844-k8.bin..

2.4 Reboot the device after everything done


Reload
ciscoasa# show version
Troubleshooting and Debug Guide |TAC

Cisco Adaptive Security Appliance Software Version 8.4(4)


Device Manager Version 6.0(2)
Compiled on Mon 21-May-12 10:48 by builders
System image file is "disk0:/asa844-k8.bin"
Config file at boot was "startup-config"

2.5 Check the IP of interface


ciscoasa# show ip
System IP Addresses:
Interface Name IP address Subnet mask Method
Vlan2 outside 192.168.100.1 255.255.255.0 manual
Vlan3 inside 192.168.50.1 255.255.255.0 manual
Current IP Addresses:
Interface Name IP address Subnet mask Method
Vlan2 outside 192.168.100.1 255.255.255.0 manual
Vlan3 inside 192.168.50.1 255.255.255.0 manual

3. Use main mode if both side are static IP


ciscoasa# show running-config
: Saved
:
ASA Version 8.4(4)
!
hostname ciscoasa
enable password 8Ry2YjIyt7RRXU24 encrypted
passwd 2KFQnbNIdI.2KYOU encrypted
names
!
interface Ethernet0/0
switchport access vlan 2
Troubleshooting and Debug Guide |TAC

!
interface Ethernet0/1
switchport access vlan 3
!
interface Vlan2
nameif outside
security-level 100
ip address 192.168.100.1 255.255.255.0
!
interface Vlan3
nameif inside
security-level 0
ip address 192.168.50.1 255.255.255.0
!
ftp mode passive
access-list 100 extended permit ip 192.168.50.0 255.255.255.0
192.168.200.0 255.255.255.0
access-list 101 extended permit ip any any
access-list 101 extended permit icmp any any
access-group 101 in interface outside
access-group 100 out interface outside
access-group 101 in interface inside
access-group 101 out interface inside
route outside 0.0.0.0 0.0.0.0 192.168.100.2 1
crypto ipsec ikev1 transform-set 10 esp-des esp-sha-hmac
crypto map to-hill 10 match address 100
crypto map to-hill 10 set peer 192.168.100.2
crypto map to-hill 10 set ikev1 transform-set 10
crypto map to-hill interface outside
crypto isakmp identity address //Use IP address as identity
Troubleshooting and Debug Guide |TAC

crypto ikev1 enable outside

crypto ikev1 policy 1


authentication pre-share
encryption des
hash sha
group 2
lifetime 86400

crypto ikev1 policy 10


authentication pre-share
encryption des
hash md5
group 2
lifetime 86400

tunnel-group 192.168.100.2 type ipsec-l2l //Notice: this name must be the


Ip address of peer, using main mode
tunnel-group 192.168.100.2 ipsec-attributes
ikev1 pre-shared-key ***** // //Fill up with share key

4. ASA dynamic address access, Hillstone static address ( using main mode ) ,
others are the same, only need to change the isakmp identity of ASA
crypto isakmp identity hostname
Specify the opposite end type as dynamic in hillstone device, also set peer-id as the
hostname of ASA.

5. ASA static address access, Hillstone dynamic address access ( Hillstone is the
initiator)
Troubleshooting and Debug Guide |TAC

Below is the configuration of Hillstone:Avvressive mode

to-pc(config)# show isakmp peer to-cisco


Name: to-cisco
Interface: ethernet0/4
Type: static
Mode: aggressive
Peer: 192.168.100.1
Connection-type: bidirectional
Peer id: ciscoasa
Local id: x7180
Proposals: p1
Nat-T: disabled
Accept-all-peer-id: disabled
DPD: disabled
PKI trust-domain:
AAA server:
Generate Route: disabled
Xauth-server: disabled
Xauth pool-name:
Description:

Configuration of ASA:
ASA Version 8.4(4)
!
hostname ciscoasa
enable password 8Ry2YjIyt7RRXU24 encrypted
passwd 2KFQnbNIdI.2KYOU encrypted
names
!
Troubleshooting and Debug Guide |TAC

interface Ethernet0/0
switchport access vlan 2
!
interface Ethernet0/1
switchport access vlan 3
!
interface Vlan2
nameif outside
security-level 0
ip address 192.168.100.1 255.255.255.0
!
interface Vlan3
nameif inside
security-level 100
ip address 192.168.50.1 255.255.255.0
!
access-list ipsec extended permit ip 192.168.50.0 255.255.255.0
192.168.200.0 255.255.255.0
access-list 101 extended permit ip any any
access-group 101 out interface outside
access-group 101 out interface inside
route outside 0.0.0.0 0.0.0.0 192.168.100.2 1
crypto ipsec ikev1 transform-set to-hill esp-des esp-sha-hmac
crypto dynamic-map to-x7180 10 set ikev1 transform-set to-hill
crypto map ipsec1 10 ipsec-isakmp dynamic to-x7180
crypto map ipsec1 interface outside
crypto isakmp identity hostname
crypto ikev1 enable outside
crypto ikev1 policy 10
authentication pre-share
Troubleshooting and Debug Guide |TAC

encryption des
hash md5
group 2
lifetime 86400
telnet 0.0.0.0 0.0.0.0 inside
telnet timeout 5
ssh timeout 5
ssh key-exchange group dh-group1-sha1
console timeout 0

tunnel-group x7180 type ipsec-l2l //x7180 equal to peer-id , and only for
aggressive mode
tunnel-group x7180 ipsec-attributes
ikev1 pre-shared-key ***** // Fill up with share key

Troubleshooting and Analysis

Trouble one、Establish IPSEC VPN with Cisco device, isakmp negotiation is fine, but
IPSECnegotiation is abnormal.
Deploy CISCO behind the NAT device,one to one address mapping; Hillstone device
connect with internet via public address. Because Cisco device need to go through
NAT by using aggressive mode, and set FQDN as its identity. 。 Below is the
configuration of CISCO:
crypto isakmp policy 10
encr 3des
hash md5
authentication pre-share
group 2
crypto isakmp key hillstone123 address 10.86.11.2
crypto isakmp nat keepalive 60 // unnecessary configurations, the NAT-t
Troubleshooting and Debug Guide |TAC

functionofCisco router is auto enabled.

crypto isakmp peer address 10.86.11.2


set aggressive-mode password hillstone123
set aggressive-mode client-endpoint fqdn cisco

crypto ipsec transform-set ipsec-hill esp-3des esp-sha-hmac

crypto map to-hill 5 ipsec-isakmp


 set peer 10.86.11.2
set transform-set ipsec-hill
 set pfs group2
match address to-hillstone

ip access-list extended to-hillstone


permit ip 192.168.100.0 0.0.0.255 192.168.101.0 0.0.0.255

interface FastEthernet0/0
ip address 192.168.1.1 255.255.255.0
duplex auto
speed auto
crypto map to-hill

Case one:
IPsec sa negotiation is not successful, using hillstone debug to analyze the first
received packet in the phase two, but we did not get the feedback from the opposite
end, and we keep re-transmiting after that.
According to the complete DEBUG analysis of hillstone:
2014-04-28 20:42:30, DEBUG@VPN: [118.140.3.55]: phase 1 (aggressive mode):
Troubleshooting and Debug Guide |TAC

remote supports DPD


2014-04-28 20:42:30, DEBUG@VPN: Received unknown Vendor ID
2014-04-28 20:42:30, DEBUG@VPN: [118.140.3.55]: phase 1 (aggressive mode):
Selected NAT-T version: draft-ietf-ipsec-nat-t-ike-02
 
2014-04-28 20:42:30, DEBUG@VPN: [118.140.3.55]: Compared: DB:Peer
2014-04-28 20:42:30, DEBUG@VPN: [118.140.3.55]: (lifetime = 86400:86400)
2014-04-28 20:42:30, DEBUG@VPN: [118.140.3.55]: (lifebyte = 0:0)
2014-04-28 20:42:30, DEBUG@VPN: [118.140.3.55]: enctype = 3DES-CBC:7
2014-04-28 20:42:30, DEBUG@VPN: [118.140.3.55]: (encklen = 0:256)
2014-04-28 20:42:30, DEBUG@VPN: [118.140.3.55]: hashtype = MD5:SHA
2014-04-28 20:42:30, DEBUG@VPN: [118.140.3.55]: authmethod = pre-shared
key:pre-shared key
2014-04-28 20:42:30, DEBUG@VPN: [118.140.3.55]: dh_group = 1024-bit MODP
group:1024-bit MODP group
2014-04-28 20:42:30, DEBUG@VPN: [118.140.3.55]: Compared: DB:Peer
2014-04-28 20:42:30, DEBUG@VPN: [118.140.3.55]: (lifetime = 86400:86400)
2014-04-28 20:42:30, DEBUG@VPN: [118.140.3.55]: (lifebyte = 0:0)
2014-04-28 20:42:30, DEBUG@VPN: [118.140.3.55]: enctype = 3DES-CBC:3DES-CBC
2014-04-28 20:42:30, DEBUG@VPN: [118.140.3.55]: (encklen = 0:0)
2014-04-28 20:42:30, DEBUG@VPN: [118.140.3.55]: hashtype = MD5:MD5

The NAT-t version of opposite end Cisco device supported is draft-ietf-ipsec-nat-t-ike-


02 , and our support NAT-t version is: RFC 3947 ,we have searched the following
version references: draft-ietf-ipsec-nat-t-ike-02 was published at April 2002, but
RFC3947 was published at January 2005. There are few version s between them, for
examples, 03,04,05,06,07,08. So we guess there are lots of difference between those
two versions, and the version of opposite end probably is the version before 2005.
We have test the new IOS version is ok in local side.
Troubleshooting and Debug Guide |TAC

Case two:
According to the feedback from customer, if we replace the hillstone device to Cisco
device, the negotiation will be fine no matter the IOS is old or new.。
According to the packet capturing of new IOS, the old version also will be send when
the new IOS is sending, then the new IOS and old IOS will be compatible.

Trouble two、Establish IPSEC with Cisco,DPD function of Hillstone failed,cannot


detect if the opposite side is alive automatically (Agreessive mode is normal)
According to the analysis of capture packet and debug, we acknowledged that the
cisco device has put the DPD information into the third packet of negotiation
procedure when Cisco is using main mode to negotiate. It makes hillstone device did
not get the DPD support information of Cisco. Usually, the firewall manufacturer will
notice the DPD support information in the first and second packet under main mode,
but Cisco notice it in the third packet.。
The StoneOS version after 5.0R3P6,5.0R4P2 ehanced the DPD information loading.
The aggressive mode has been verified that it can load the DPD information normally.

With HuaWei

Troubleshooting and Analysis

Trouble one:Establish IPsec with HUAWEI failed


Set HUAWEI as initiator of vpn,set Hillstone as respondor, according to debug, the
hillstone side hint:
Invalid flags 0x08
Troubleshooting and Debug Guide |TAC

According to capture the isakmp packet from HUAWEI , HUAWEI is using IKE V2.0
version:

But hillstone only support IKE V1.0 version:

The establishing works after HUAWEI change to IKE V1 version.

IPSEC FAQ

1. What is rmconf? What is sainfo?


A: Only if you configured an isakmp peer, it will generate a corresponding data
structure to save related information in the memory, any one of this data structure
is an rmconf. One isakmp peer is corresponding with an rmconf, it will search the
matchable isakmp peer/rmconf by peer ip or peer-id in the negotiation procedure.
Only if you configured an ipsec tunnel, it will generate a corresponding data
structure to save related information in the memory, any one of this data structure
is a sainfo. One ipsec sa is corresponding with an sainfo, it will search the matchable
ipsec tunnel/sainfo by proxy id in the negotiation procedure.
Troubleshooting and Debug Guide |TAC

2. Why the forwarding has been dropped after the OSPF established neibour
relation to learn the routing by ipsec vpn?
A:Usually this was caused by the tunnel binded to another tunnel but did not
spefcified GW. The routing of OSPF learned and issued to the route list will include
GW information, so for this application, we need to specify GW address as opposite
end tunnel interface address when we are binding tunnel.

3. Can hillstone device establish redundant VPN with other manufacturer’s device?
A: It cannot be achieved by VPN-track, but we can perform it by the redundancy of
OSPF route protocol.

4. There are two exits in the headquarters, one in branch, can we perform
redundant VPN by VPN-track?
A:Yes, but the branch need to initiate a connection and close the reverse-routing of
two exits in headquarters, and configure the redundant VPN configuration.

5. Why the tunnel is connected, but the status of vpn track is dead?
A: Check if the destination address of VPN is the tunnel’s interface without man
ping enable.
If we don’t configure “vpn track dst-ip” or “dst-ip” is the public address of
opposite end, the local side will encrypt the icmp packet and send to the other side
by tunnel, and if the opposite end find the destination address is the interface of
tunnel egress, they will send back the packet from the tunnel, so whever the
interface enabled “man ping” or not won’t affect the monitoring status.
If we configured “vpn track dst-ip” as the interface of opposite tunnel, the
opposite will excute d-plane process after decrypt icmp packet, but if the tunnel’s
interface did not enable “man ping”, then they won’t answer track packet.。

6. What are the functions of the “accept-all-peer-id” of phase one and the “accept-
all-proxy-id” of phase two
A: The function of “accept-all-peer-id” is to find an rmconf of usergroup which
do not ask verification for AAA, but only works for usergroup.
Troubleshooting and Debug Guide |TAC

Command “accept-all-proxy-id” is using to close the proxy check function of


phase two.

7. Is hillstone device support to establish GRE OVER IPSEC with Fortinet or other
manufacturer?
A:Yes,。Type in “ ip ospf network point-to-point” in the tunnel’s interface。

Troubleshooting

Case one 、 The Debug statement of the tunnel cannot be established

and no data

1. The peer id does not match in the phase one


2009-08-11 02:35:34, DEBUG@VPN: [40.0.0.10:500]: No ID match.

The peer id does not match in the phase one


Auth IKE ID failed!
/Check if it is dynamic mode, the device behind NAT must configure local ID, must
use aggressive mode, or check if the IKE ID of dial in VPN user is incorrect.

2. The share key does not match in the phase one


2009-08-11 03:17:12, DEBUG@VPN: [40.0.0.10:500]: Invalid payload or failed to ma
lloc buffer (pre-share key may mismatch).
// Check the third or fourth packet if it does not match
3. The encryption negotiation does match in the phase one
2009-08-11 04:48:18, DEBUG@VPN: [40.0.0.10:500]: enctype = 3DES-CBC:DES-
CBC //Encryption type
2009-08-11 04:48:18, DEBUG@VPN: [40.0.0.10:500]: (encklen = 0:0)
Troubleshooting and Debug Guide |TAC

2009-08-11 04:48:18, DEBUG@VPN: [40.0.0.10:500]: hashtype = SHA:SHA


2009-08-11 04:48:18, DEBUG@VPN: [40.0.0.10:500]: authmethod = pre-shared key:
pre-shared key
2009-08-11 04:48:18, DEBUG@VPN: [40.0.0.10:500]: dh_group = 768-bit MODP
group: 768-bit MODP group
2009-08-11 04:48:18, DEBUG@VPN: [40.0.0.10:500]: Rejected enctype:
DB(prop#1:trns#1):Peer(prop#1:trns#1) = 3DES-CBC:DES-CBC
2009-08-11 04:48:18, DEBUG@VPN: [40.0.0.10:500]: No suitable proposal found
2009-08-11 04:48:18, DEBUG@VPN: [40.0.0.10:500]: Phase 1 (main mode): failed to
get valid proposal!

4. The mode doest match in the phase two


2009-08-11 06:30:05, DEBUG@VPN: [40.0.0.10:500]: Error can not find ISAKMP peer.
2009-08-11 06:30:15, DEBUG@VPN: [40.0.0.10:500]: Peer Main mode, try to find
rmconf by IP and local if.

5. The group doest match in the phase two


2009-08-11 04:59:08, DEBUG@VPN: [40.0.0.10:500]: pfs group mismatched: my:1
peer :0 //group “Does not match”
2009-08-11 04:59:08, DEBUG@VPN: [40.0.0.10:500]: Not matched

6. The proxy-id doest match in the phase two


2009-08-11 05:22:44, DEBUG@VPN: [40.0.0.10:500]: failed to get sainfo.

7. Dropped:out tunnel doesn’t exist


Reason:Most of cases were caused by the incomplete configuration, the tunnel did
not issue the DP, need to check the configuration and finish it.

8. The system warning: “tunnel id (0) invalid” when the packet need to send by
encryption, or the system warning:“flow1’s tunnel id find by route is not the same
Troubleshooting and Debug Guide |TAC

with in tunnel’s,drop” when the packet need to be forward by decryption


Reason :Usually it was caused by the different configuration between routing and
tunnel binding with tunnel’s interface. If the configured route only specify the
tunnel’s egress interface without the next hop address, then we can change the
configuration and specify the next hop address and egress interface of tunnel; If we
configured the dial in vpn, then we use “generate-route” to issue the route
automatically, do not add them manually.。

9. The system warning: “Dropped: Route to x.x.x.x out interface zone is not the
same with tunnel's.” when the packet need to be send by encryption.
Reason : The security zone of selected egress interface and data packet in isakmp
peer is different.

Case two 、 Multi-egress VPN tunnel cannot establish or no data after

established

Analysis:In the case of default route, the egress interface of VPN negotiation packet
is according to search result of “show fib kernel”, the source address is the address of
“out-going interface” in the VPN’s configuration; the egress interface of tunnel
session was selected by hash of default route. (The egress interface is same as the
configuration of “out-going interface” since after StoneOS 5.0R3P4 version.
Also, there is no ISP route has been distribute by kernel, the DP included all the
routes. So the IKE negotiation packet won’t match ISP route, but the tunnel
session’s egress interface of DP will match ISP route.
According to the below theory to do the following troubleshooting:
1. Add the static route with 32 bit mask and destination address is opposite end, to
make sure the negotiation packet will be send from the specified port.
2. If the opposite end is dynamic address, then ask the other side initiate the
Troubleshooting and Debug Guide |TAC

connections, close the reverse-routing in local port.;


3. If the debug show that the packet still send from the unexpected interface after
the above operation, then delete then configuration of ipsec vpn and configure it
again.

Case three、Use windows dial in l2tp over ipsec failed

1. Client warning: “error 789”. “Attempt to connected failed, because the security
layer is initialized with the remote computer negotiation encountered an error
processing”.
Reason analysis:Did not enable the ipsec vpn of windows.
Solution:Open control pannel->Management tools->Service,Active IPSEC Services

2. win 7 dial in ipsec tunnel establish correct, but the PC did not initiate the
negotiation of L2TP
Open Regiistry:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\PolicyAgent , click
PolicyAgent, new create“DWORD(32- 位 )value”in the right frame, rename it as
“AssumeUDPEncapsulationContextOnSendRule”, notice the case matters, the fix
value is decimal 2, confirm and reboot PC.
Note:
1. In the "value data" box type in one of the following values:
Windows is configured to cannot be established with the server located behind a
network address translator security association.
)1 Windows is configured to can establish and the safety of the server located
behind a network address translator.
) 2 Windows configuration to the server and client based on Windows XP SP2 are
located behind a network address translator to establish security association.
Troubleshooting and Debug Guide |TAC

2. Click "ok", then exit registry editor note:

Case four、PNP VPN client cannot dial in, Server side Debug hint”but

user config is null”

The debug of client side shown that the first packet has been send out already,
the server side also received the first packet from the opposite end, but the debug
shown below:
2013-10-22 19:10:23, DEBUG@VPN: [192.168.100.1]: phase 1 (aggressive mode):
remote supports DPD
2013-10-22 19:10:23, DEBUG@VPN: [192.168.100.1]: phase 1 (aggressive mode):
remote is a smart vpn client
2013-10-22 19:10:23, DEBUG@VPN: [192.168.100.1]: but user config is null
Check the VPN configuration of server being careful:
Accept-all-peer-id: enabled
The server side has configured “Accept-all-peer-id” , PNP VPN need to check the
user’s ID information( pre-share key was generated by User ID), and the information
shown that do not check user information of AAA is conflicted, delete this setting and
function is working right again.

Case Five、IPSEC VPN re-connecting situations

IPsecVPN re-connecting happened in phase one and phase two.

Reason:There are lots of possibilities, now we have following:

1. The time out of phase one or phase two


Beacause IPsecVPN has no corresponding keeplive policy , so the system will
Troubleshooting and Debug Guide |TAC

reconnect once it time out in phase. Double check the log information of
reconnection, if the interval is almost the same as the timeout of two phases, it
should be this problem.。

2. VPNtrack failure cause the reconnection of IPSec


The VPN will reconnect to update the status of VPN at the moment of the status of
VPN track changed or the VPN track has been edited. If the reconnection was caused
by the status of VPNtrack , then we will find the following warning in the
corresponding log: IPSec tunnel XXXX turn to dead/alive。

3. DPD failure cause the reconnection of VPN


The VPN connection will be disconnect once the DPD detect the opposite end has no
respond, usually the warning informatil will like below when we debug VPN: :
[221.224.30.141]: DPD: remote peer [221.224.30.141:500] seems to be dead
Note: 2 and 3 in both cases are greatly influenced by the network environment,
usually if the above problems, you can modify the corresponding interval and the
number of times, as far as possible will decrease interval number increase。

4. The PPPoE address of egress changed


The system will reconnect when the egress interface of device is PPPoE and the IP
address of external interface has been changed.
Troubleshooting and Debug Guide |TAC

SCVPN

Configuration

Case one、To perform the PC’s domain authentication by SCVPN access

to internal network

The key is that to perform the SCVPN auto connection when the PC start up.
Configuration Steps:
1. Login to PC, install and login SCVPN as the general steps.
2. Active “auto login” of SCVPN client,and record the user ID and password of auto
login
3. In order to perform the auto-login when PC star up, we can do the follow settings
open control pannel>Administrative tools>System Configuration

L2TP

Troubleshooting

Case one、LNS user authentication failed

1. Network log reminding:


2013-10-17 19:30:251 debug1PPP user turn to LCP up
2013-10-17 19:30:261 debug 1PPP user yca6385404@jhyb.js turn to lcp down
2. Debug L2TP、AAA information:
Troubleshooting and Debug Guide |TAC

013-10-17 19:18:19, DEBUG@L2TPD: Failed to get AAA server for user


yca6553221@jhyb.js.
2013-10-17 19:18:19, DEBUG@L2TPD: PAP peer authentication failed for
yca6553221@jhyb.js
2013-10-17 19:18:19, DEBUG@L2TPD: slinkdown.
2013-10-17 19:18:19, DEBUG@L2TPD: Link down, if_up=0
3. The login user ID is yca6553221@jhyb.js , configure domain as jhyb.js of AAA in
L2TP tunnel.
4. If AAA is third party’s authentication server, suggest to configure the PPP
authentication as PAP protocol.

Case two 、 Hillstone device as LNS, provider’s VPDN device as LAC,

establish L2TP VPN between two device, parts of android system

cannot login

Excute a command “SG-6000-DX(config)# l2tp-include-ppp-acf”。By default the


PPP packet that hillstone device send out has been delete two byte “ff03” in the
head of packet, it might cause the opposite end cannot recognize our packet, but this
command can make our packet always bring ff03.

Case three、Cannot send email by dial in L2TP VPN

Trouble:
After dial in L2TP VPN , we can access email server 10.8.0.140 , and we can
receive email as usual, but we cannot send email, the warning remind that
Authentication Request。Internal network is okay to send email.
Troubleshooting and Debug Guide |TAC

Analysis:
After we dial in L2TP VPN,it is all right when we ping the port 110 and port 25
of 10.8.0.140 or telnet 10.8.0.140 , the network connectivity is fine, the problem
should happened in application layer. We have checked device configuration, there is
no configuration of IPS and application behavior control, so the SMTP cannot be
affect. Also, according to the debug analysis of device, the packet is all right without
any problem, so we suspect it is the server or medium device’ problem.
Because of the warning related authentication, so we need customer to confirm
if the server controlled the network segment. Customer confirmed that the address
of L2TP address pool and firewall internal interface are able to access mail server.
According to the capturing packet, because the configuration is L2TP Over
IPSec,the packet is envrypted, tyring to access by mapping. Also we found it is the
same to use mapping and L2TP. Below are the packet from the external network or
internal network access to server:
External network mapping capturing packet:

Internal network access capturing packet:

Comparing the capturing packet information, we found the TCP three-way


handshaking are the same during the SMTP protocol communication process, but
when we access from the external network, the data return from server after three-
way handshaking becomes messy code(all *), but feedback is fine if we access from
internal network. The subsequent client ask for authentication by ehlo, it reminds
Troubleshooting and Debug Guide |TAC

“Syntax error” when the external network access, but when we access from internal
netowork, it passed the authentication and works fine. The feedback of three-way
handshaking was initiated by the server, it has no relations with firewall, so we
suspect that some other medium device in the line changed SMTP protocol packet,
caused the messy code of feedback.
After the communication with customer, we found there is another PIX deivce in
the network, the pathway of internal network access would not pass the PIX. So we
suggest customer skip the PIX to test. And customer said the feedback is fine once he
skipped PIX, the mail of receving and sending works. So we confirmed that the root
cause is the PIX.

Solution:
The root cause is the PIX device, nothing wrong with Hillstone firewall.。

Case Four、Andriod phone cannot access to internal web server after

dial in by L2TP over ipsec

Analysis:
Customer said only android phone cannot access, after investigation, below is
the situation:
We use iphone to test, it is able to access web page by 3G or WiFi on hillstone
deivce, also we tried to test android phone, but it is unable to access the web page.
According to the above troubleshooting we found only the android user cannot
access, so we suspect that the root cause is between the negotiation of Andriod
phone and customer device.
During the process of debugging to access web server 192.168.0.4, we can see the
reply packet of server, no packet dropped, so we suspect the network between
Troubleshooting and Debug Guide |TAC

android phone and hillstone device is fine, the problem might caused by the
checksum value or or protocol beyond translation layer.
Because mobile phone cannot capture packet, so we used a wireless router to
connect with firewall, and mobile phone connected with the wireless network of the
wireless router, then we used mirroring of interface to capture packet. After that, we
found the checksum value of UDP packet is none before the traffic reached the
mobile phone, and the medium device had never been touched, so we confirmed
that the checksum is not the root cause.。
Finnaly, we changed the TCP MSS VPN value of firewall, reduce it to 1300, and
android phone passed the test, so we confirmed that the root cause is the value of
MSS, problem solved.

Chapter 8 AV

Troubleshooting

Case one、Solution to error identification of malicious website

Because we cannot edit the malicious website library, so we can only avoid it
bypolicy, below is the solution:
1.Create two av profile, test1 and test2 ,don’t open malicious webstie library
for test1, select http type ,record log only, open malicious website for test, select
http type,reconnect.
2. Create a new policy, the destination address is the domain name of website,
service is any, action permit, select test of profile.
3. Binding test2 to untrust security zone.
Troubleshooting and Debug Guide |TAC

It could match the av policy of policy and security zone at the same time, but the
priority of policy can avoid the problem.

Case three、 AV cannot detect virus

1. For the detection of ftp active mode, it has to bind profile to polcy.
2. For protocols of imap4 and pop3,usually the mail server provider will search and
kill the virus of attachment firstly, so sometime we cannot detect any virus the email
has been send to loca.

Case four、The network is getting slow after configured AV

Check if enabled the magic number fill up function。

Case Five、Reminding of update feature library: Update failed, invalid

head of feature library.

1. If it is online updating, which means the device got the feedback from the server,
but the received feature library is not right, usually it is should be the problem of
updating server.
2. If it is offline updating, which means the imported file is not right, please check if
the format of file is correc.
Troubleshooting and Debug Guide |TAC

Chapter 9 IPS

FAQ

1. Configured IPS profile and binded with policy, but did no check the “active” of
intrusions prevention global configuration, will the traffic go through the IPS
module handling process?
A:The device will resolve the protocol traffic of IPS profile that checked, but won’t
match with IPS feature, and control any of IPS. It will consume the performance of
device because it need to resolve the protocol packet.

Configuration and Statement

Case one、Use WEB ACL to control the access to specified website

WEB ACL using:


The object is to deny external user to access the t.hillstonenet.com:88/2012-10-
18/,/2012-10-18/.
Method:Using virtual host to solve
Configuration example:
ips sigset "http_http" template http
virtual-host t.hillstonenet.com:88 enable
web-acl-check enable action reset
web-acl /2012-10-18/ deny //Only need to fill with directory, no need to fill
host parts.

Test:
Troubleshooting and Debug Guide |TAC

2013-01-04 10:16:26, WARNING@IPS: From 10.88.16.172:50181to


122.193.30.101:88 , detected illegal deny access control, response action : reset
connections,URL:t.hillstonenet.com:88/2012-10-18/
Others:
Because the destination server use the non-standard port, seggest to set the
application type of self-defined service tcp88 as http in order to enhance the
application recognization
Rule id: 13
Status: E
From zone "l2-trust" to zone "l2-untrust"
Type: 0
Source addresses:
Any
Destination addresses:
Any
Services:
tcp-88
Action: PERMIT
Roles:
Users:
User-groups:
Profile(s): http
Hit 5 times
400-GuanWu[DBG](M)(config)# show service userdefined
tcp-88 TCP HTTP 88 - 1800
Troubleshooting and Debug Guide |TAC

Troubleshooting

Case one、The attack id was shown up on the log, but cannot find in

the feature list

The feature ID consist two parts, they are protocol ID(the first bit and the
second bit) and attack feature ID( the fifth bit), for examples, in
ID“600120”,“6”measn Telnet protocol,“00120”means attack featureID。If the
attack feature ID is larger than 60000 then it is abnormal, if it is less than 60000 then
it is normal attack feature, please refer to configuration manual for details. There are
some attack ID in the IPS log, but we cannot find it in the feature list, the reason is
that the last 5 bit of this ID is larger than 60000 which was detected by protocol
legality.

Chapter 10 AD

Configuration

Case one、ip-number-per-mac function explanation

Configure the number of “ip-number-per-mac as 1”, the behavior setting is


“drop” , one mac have two IP, they are 192.168.100.1 and 192.168.100.129
respectively.
1. At first, PING 192.168.100.1 is suceessful
Troubleshooting and Debug Guide |TAC

SOC-1008[DBG](config-zone-trust)# ping 192.168.100.1 count 1


Sending ICMP packets to 192.168.100.1
Seq ttl time (ms)
1 128 1.59
2. And then ( shorter time gap with step first ) ping 192.168.100.129 is
unsuccessful, also comes with security log alarm.
SOC-1008[DBG](config-zone-trust)# ping 192.168.100.129 count 1
Sending ICMP packets to 192.168.100.129
Seq ttl time (ms)
1 Destination Host Not Responding
Log:
2013-08-09 14:48:04, Security CRIT@FLOW: ARP spoof attack:drop! MAC address
001c.542d.a904 has 2 IP address on interface ethernet0/4
At the same time, the debug show:
2013-08-09 14:48:22, DEBUG@FLOW: core 1 (sys up 0x5e7bb2a ms): arp_handle,
ifname:ethernet0/4
ip number per mac is 1, is_drop is 1, port_id is 12
Exceed the ip number per mac limit, drop the ARP packet
The above two steps almost reached the target of configurations, one mac have only
one IP.
3. After a period, ping 192.168.100.129 is successful again.
SOC-1008[DBG](config-zone-trust)# ping 192.168.100.129 count 1
Sending ICMP packets to 192.168.100.129
Seq ttl time(ms)
1 128 1.12
Debug log:
2013-08-09 14:48:27, DEBUG@FLOW: core 1 (sys up 0x5e7cf72 ms): arp_handle,
ifname:ethernet0/4
ip number per mac is 1, is_drop is 1, port_id is 12
ARP packet check: interface ethernet0/4, port ethernet0/4
Troubleshooting and Debug Guide |TAC

Got ARP reply, update entry, if:12, ip:192.168.100.129, mac:001c.542d.a904

4. The function is only working for ARP attack by assistance, need to combine with
other ARP protection function together.

Troubleshooting

Case one、IP snoop attack protection cause the disconnection

1. Problems
Assume the device have two egress interfaces, they belong to security zone
untrust and untrust2, both closed reverse routing, if the device configured default
route for one interface, then the data won’t be received if it comes from interface 2.
2. Solution
In the attack defense function, close the IP snoop detection of untrust2 security
zone.
Troubleshooting and Debug Guide |TAC

Chapter 11 ARP Protection

Troubleshooting

Case one、Enabled IPMAC binding but the unbinded PC still can access

to network

1. Check if enabled the command “arp-disable-dynamic-entry” in the corresponding


interface;This command is using to skip checking dynamic ARP table
2. If the interface enabled revers-routing
3. Binded the MAC with user
4. The MAC passed 802.1X authnentication
5. Some network adapter has been set to mix mode, so the PC can change IP and
then still able to access network ( For example, if the IP address of PC is
192.168.1.100, mac is aaaa.aaaa.aaaa , the we bind 192.168.1.100 and
aaaa.aaaa.aaaa in the firewall,if we change the PC’s IP to 192.168.1.101 ,the
firewall would only check if the IP/MAC binding table include mac address of
forward data packet, but do not to check if the corresponding IP address is same,
so the firewall will forward the corresponding packet, but the network adapter
could handle the non-local IP packet under mix mode )In this case, if user used
vswitch interface to forward data, then need to configure the internal interface as
vswitch inter and enable ARP checking.
6. Check if the interface has been enabled hot-swappable function, which means
configured “proxy-arp and dns-proxy trans”.
Troubleshooting and Debug Guide |TAC

Chapter 12 QOS

Troubleshooting

Case one、IPQOS reserved bandwidth does not work.

Check if the configuration of interface bandwidth is same as the real bandwidth that
the provider provides

Case two、IPQOS maximum bandwidth limitation does not work

1. Check the time that the IP exceed the speed limitation, if it only happens in a
moment that would be fine.
2. Check if enabled flexible QoS.
3. Check if the IP in the QoS white list.
4. Excute “show qos interface ethXX | in x.x.x.x” to check if there is a line of this IP,
usually it should have one line on the upper line or bottom line(Or “show qos-
statistic”,the output will display by line ID, we can check the QoS line binded with
zone),for example:
SG-6000# show qos interface ethernet0/0 | include 192.168.1.218
Match IP: 192.168.1.218, IPqueue id 843
Match IP: 192.168.1.218, IPqueue id 830
 Two lines,one is input, another one is output.
5. If the policy binded with zone, check if the IP has any QoS policy binded with
interface, in the same situation, the QoS policy binded with interface has a higher
priority.
6. Check if the device have multiple publick network egress interface, but the IP has
Troubleshooting and Debug Guide |TAC

only been limited by one interface.


7. If the limited address of QoS policy is any, then change to detail IP address
segement or address book, to avoid the device line exceed the limitation of device to
cause the failure of QoS.
8. The number of entries in the address book for every policy do not over 10.

Case three、Apply QoS maximum bandwidth limitation does not work

1. Check if the security enabled application recognization


2. Make sure the application feature library has been updated to the latest version
3. Check the ALG of application

Case four、SX multi-QSM CPUs utilization load unbalance

If the egress interface bind with QoS profile, then its performance will lower than
that of ingress interface. Which means the QoS of input interface will consume more
CPU than that of output interface.
For instances, if cpu5 bind with input qos profile , cpu6 bind with Qos profile of
output interface,CPU5 consumed more utilization than that of CPU6.
We can manually adjust the QoS profile of interface and output interface to two CPU
in balance, then the performance will be balance.
Configuration Example:
SG-6000(config)# interface ethernet1/11
SG-6000(config-if-eth1/11)# qos-card-binding input qcpu
  qcpu5/0           Name of qos card
  qcpu6/0           Name of qos card
SG-6000# show qos-card-binding   
interface      type       qos card    status     flag
ethernet1/1    ingress    qcpu5/0     active     A
ethernet1/1    egress     qcpu6/0     active     A
Troubleshooting and Debug Guide |TAC

ethernet1/8    ingress    qcpu5/0     active     A


ethernet1/8    egress     qcpu6/0     active     A
ethernet1/9    ingress    qcpu5/0     active     A
ethernet1/9    egress     qcpu6/0     active     A
ethernet1/11   ingress    qcpu5/0     active     A
ethernet1/11   egress     qcpu6/0     active     A
aggregate1     ingress    qcpu5/0     active     A
aggregate1     egress     qcpu6/0     active     A

Chapter 13 Session Limitation

FAQ

1. Does the session limitation policy should select source security zone or destination
zone?
A : If the setting of IP limitation is “per-srcip” , then we need to select source
security zone; if the setting is “per-dstip” , then we need to select destination
security zone; as to other situations, it is okay to select source security zone or
destination zone.
Troubleshooting and Debug Guide |TAC

Configuration

Case one 、 The number of session is still large even configured with

session limit

The new session would match with the policy after configured with session
limit,we can clear session before test.

Chapter 14 AAA

Functions and configurations

Case 1. Configuration of Login dn and Base dn in some common

scenarios

Scenario 1:
Troubleshooting and Debug Guide |TAC

Admin is “administrator”, and all users in the root directory need to be synced:

Base dn:dc=hstest,dc=net
Login dn:cn=administrator,cn=users,dc=hstest,dc=net

Scenario 2:

Admin is “William”,the position of the admin is showing abouve, and all users in the root
directory need to be synced:
Base dn:dc=hstest,dc=net
Login dn:cn=William,ou=tac,dc=hstest,dc=net

Scenario 3:
Troubleshooting and Debug Guide |TAC

The structure is showing above, and only users in the OU “tac” need to be synced:
Base dn:ou=tac,dc=hstest,dc=net
Login dn:cn=administrator,cn=users,dc=hstest,dc=net

Case 2. The internal users would be authenticated randomly with one

of three AD servers, the firewall need to be deployed for SSO with AD

agent

Solution: Creat three AD server instances in the firewall for each AD server, and
install the AD agent on each AD server.

Case 3. Import and export users

1. For SonteOS 5.0, it’s OK to import user-password list from a backup file, or export
the list to a backup file.
2. There is an import/export button in the WebUI for StoneOS 5.0, but for the
importing job, you also need to create the users before the importing.
3. But in CLI, there is conmand could make you to import the user binding list
Troubleshooting and Debug Guide |TAC

without creating the users firstly:


SG-6000# import AAA user aaa-server local from     
ftp               FTP
tftp              Trivial FTP
usb0              USB0 storage
You could import the list file from ftp, tftp and usb storage. And there is a
requirement for the format of the imported list file which showing below.
user1,password1,group1,description1
user2,password2,group2,description2
user3,,,
As you can tell, the format is “username, password, usergroup, description”, you
could leave one of the attribute to null, but do not forget the symbol “,”.

Case 4. Deploy SCVPN and Webauth with Radius on Windows Server

2008

A. Install the services on Windows Server 2008


For this case, we need install “Active Directory Domain Services” and “Network
Policy and Access Services” on the Windows Server 2008.
Open the Server Manager and click “Add Roles”:
Troubleshooting and Debug Guide |TAC

Check the role Active Directory Domain Services and click next and install:

After the installation, run the dcpromo.exe to deploy the new domain controller,
Troubleshooting and Debug Guide |TAC

the following steps will be omitted:

After deployed the DC, open the Server Manager again to add role “Networ Policy
and Access Services”:
Troubleshooting and Debug Guide |TAC
Troubleshooting and Debug Guide |TAC

B. NPS configuration
Open NPS console:
Troubleshooting and Debug Guide |TAC

Register server in AD,


Troubleshooting and Debug Guide |TAC

Create new RADIUS client:


Troubleshooting and Debug Guide |TAC
Troubleshooting and Debug Guide |TAC

Create new connection request policy:


Troubleshooting and Debug Guide |TAC
Troubleshooting and Debug Guide |TAC
Troubleshooting and Debug Guide |TAC
Troubleshooting and Debug Guide |TAC
Troubleshooting and Debug Guide |TAC

Ensure that the order for this policy is 1

Create new network policy:


Troubleshooting and Debug Guide |TAC
Troubleshooting and Debug Guide |TAC
Troubleshooting and Debug Guide |TAC
Troubleshooting and Debug Guide |TAC
Troubleshooting and Debug Guide |TAC
Troubleshooting and Debug Guide |TAC

Also ensure that the order for this policy is 1:


Troubleshooting and Debug Guide |TAC

Run “gpupdate /force” in the CMD to update the User Policy:

C. Configuration on StoneOS
Add a Radius server is AAA server configuration page:
Troubleshooting and Debug Guide |TAC

After this configuration, you could be able to use this user database in the
Windows 2008 Server for authentication in SCVPN and Webauth.
Troubleshooting and Debug Guide |TAC

Troubleshooting

Case 1. Can’t sync users from the AD server

1. Check the connection between the firewall and the AD server.


2. Chech the configuration of the base-dn. You could get an error when debug
the AAA module with this mistake:
DEBUG@AAA: connection to ad once done, but base searching not complete

3. Check the configuration of the login-dn. You could get an error when debug
the AAA module with this mistake:
4. Chech the password of the login-dn. You could get an error when debug the
AAA module with this mistake:
DEBUG@AAA: searcher of ad bind failed

Case 2. Incomplete synced user information

The reason for this issue is that there usually a query limitation in the AD server.
The default number of returned user is for one query is 1000. To fix this issue, you
need to modify the value on the AD server. Follow these steps:
1) “Start”  “Run”  input command “ntdsutil” and press enter
2) Input “ldap policies” and press enter
3) Input “connections” and press enter
4) Input “connect to domain domain_name” and press enter
5) After connect to the domain successfully, input “quit” and press enter
6) Input “show value” to get current limitation value
7) Input “set maxpagesize to 5000” to set the value to 5000(this value should be set
according to actual case)
8) Input “commit changes” to save the modification
9) Imput “quit” to quit
Troubleshooting and Debug Guide |TAC

Case 3. Can’t sync users from the LDAP server

Check the configuration as decribed in the Case 1. Espacially the authentication


method. In most LDAP applications, the authentication is finished through plain text,
so choose it in the Authenticaiton Mode configuration.
And there are something very special in the LDAP configuration, it’s about the
Naming Attribute, Member Attribute and Group Class configuration.
The default value for these three value in SonteOS are “uid”, “uniqueMember” and
“groupofuniquenames”. Usually it would failed syncing the user information with
these defaut values, you need to adjust these values accdording to the actual case.
To get these values correctly, we need to analyze the structure of the directory on the
LDAP server. There is a tool called “LDAP Browser” which could give us some help.
Use the tool to connect to the LDAP server, and observe the dn of each user. Here is
an example:

Each entry of the directory has the same structure: attribute-value pair. “objectClass”
is the most important attribute, it’s used to define the basic category. Once the
category is confirmed, the necessary other attributes could be confirmed too. In the
image showing above, the entry has a objectClass named “person”. According to the
protocol LDAP v3, an entry with objectClass “person” require attribute “cn” and
“userPassword”, and you could use these two value for authentication. And if you
Troubleshooting and Debug Guide |TAC

need to use the LDAP server with the firewall, you could use the attribute “cn” as the
Naming Attrbute on the firewall.

And for group information syncing, in this case you could use the Member Attribute
as “member” and the Group Class as “groupOfNames” according to the image
showing above.

Case 4. Group information is missed when using local AAA server

When deploying the SCVPN, L2TP VPN or Webauth, you would need an
authentication server, and local AAA server is the most used one. But when run
“show auth-user” command in the CLI, sometimes the group information is missed.
Troubleshooting and Debug Guide |TAC

The reason is that the system would ignored this information by default to save
hardware resources. You could enable displaying this information by createing a
policy invoke the local AAA server.

Case 5. The system ptompt that failed to connect to the VPN server

when using L2TP VPN with AD AAA server used

By checking the debug log, we can ensure that the configuration is OK and the user
has been synced.
2013-12-10 17:06:18, DEBUG@AAA: response for auth_req (ID:77057, User 9102) fail

2013-12-10 17:06:18, DEBUG@AAA: record of ad1 receive BIND_ERROR message

After confirming with the customer, it seens like the server had been upgraded from
Windows 2003 to Windows 2008, and the users were created on the Windows 2003.
There may be some difference with the storage method between Windows 2003 and
Windows 2008. After reconfiguring the passwords of the users, the issue had been
fixed.

Chapter 15 HA

FAQ

1. Why the ports on master device flapping after a failure switch?


A:It’s designed as this so that the master device could clear the MAC address table.

2. When would the switch happen?


A : Preemption, Track failure, reload of the master device, unplug of reload of the
Troubleshooting and Debug Guide |TAC

SSM/IOM.

3. How deos the trial license affect the HA function?


The two device must have the same trial license type to set up a HA cluster, the
duration of trial doesn’t matter.

Troubleshooting

Case 1. Double Master device in a HA cluster

1. Platform and configuration issue: difference with the hardware platform,


software version, extend module version, HA helle time, HA cluster ID, HA group
ID, unreachable of HA link
2. License issue: AV, IPS, NBC, URLDB license also need to be the same
3. Function enabled: the two device need to run with the same function enabled to
set up a HA cluster. You could use the command “debug ha packet” to identify
the mismatch on the function enabled. For example:
2009-11-17 09:24:33, DEBUG@FLOW: core 0 (sys up 0xb72a0 ms): Received ha packet

type=2, seqno=5196

Receive a less ha version

Invalid license status license=0x35e peer license=0x3de

There are two HA license stream showing with Hexadecimal in the debug info.
Translate them to binary first, which turn to be 1101011110 and 1111011110, so
we can tell that the third bit is different. According to the offset table showing
below, we can confirm that this case is caused by the different configuration at
Policy_mode Enable.
Troubleshooting and Debug Guide |TAC

Case 2. The switch over doesn’t happen after unplug the tracking cable

The devices would take a long time to perform the cold synchronization, there would
be no switch over happen during this precess.

Case 3. Only local device information would be shown when excuting

command “show ha cluster”

1. Verify the cluster ID on both device;


2. Verify the hardware version, software version and license type on both device;
3. Run “debug ha” to see if there are HA heart beat packtes

Case 4. The sessions can’t be synced

1. Verify the zone binding relationship on both devices


2. Verify the data interface configuration on both devices
3. Verify the session limit configuration on both devices
4. Run “debug dp basic” and “debug haflow” to check the result

Case 5. Some of the ALG traffic which has been blocked by the policy

could directly go through

1. Run “show session” to verify the ALG traffic had been identified correctly
2. Run “show ha cluster” to verify both devices are working in asymmetric routing
mode
Troubleshooting and Debug Guide |TAC

3. Run “show ha traffic” to verify the ha traffic function has been enabled on both
device

Case 6. Some of the traffic can not go through in asymmetric routing

mode

1. Verify the routing entris on the device;


2. Verify the policy configuration on the device
3. Run “debug dp basic” to see that if the reverse packet had been dropped due to
mismatch of policy. Adjust the AH traffic delay time of enable HA traffic first packet
may fix the issue.

Case 7, Traffic interruption after the reload of the device

This issue is a flaw of the asymmetric routing mode. After the device reloaded, even
the device’s status is master, the device may have not finished learning the routing
information (routing mode) of ARP information (switch mode), so that all the packets
would be dropped during this time. For routing mode, running command “direct-
send default-nexthop x.x.x.x” could fix this issue.

Case 9. When using LACP with HA, the slave device canb’t be managed

When using the aggregate interface with LACP in a HA cluster, it turns out that we
can’t login to or ping the manage IP of the salve interface, also we can’t get the ARP
information for the interface in ther master device.
The reason for this issue is that after enable the LACP, there would be LCAP PDU
negotiation between interfaces, but the salve device will not send out any LACP PDU,
so that the negotiation would failed.
Troubleshooting and Debug Guide |TAC

Due to this reason, it’s impossible to manage the device via the IP address of the
aggregate interface.

Chapter 16 SNMP

Troubleshooting

Case 1. Actually there are lots of users online, but the SNMP tool get a

number of zero for the online users using the OID

(.1.3.6.1.4.1.28557.2.3.1.1)

OID.1.3.6.1.4.1.28557.2.3.1.1 is used to get the number of online ssh and telnet


administrators, currently the MID lib doesn’t support to get the number of online
users.

Case 2. The traffic showing in the SNMP tools have a great difference

with the traffic showing in the dashboard of the device

For the SNMP tools there are two nodes of public MIB for reading the traffic:
mgmt : mib-2: ifMIB :ifMIBObjects: ifXTable: ifXEntry: ifHCOutOctets ; OID:

1.3.6.1.2.1.31.1.1.1.10.X

These two values are corresponded to the “OutGoodOctets” value of the command
“show controller slot x port x statistics”
Troubleshooting and Debug Guide |TAC

Also there is another MID node:


mgmt : mib-2: interface: ifTable: ifOutOctets; OID: 1.3.6.1.2.1.2.2.1.16

It would provide the same vale of OID: 1.3.6.1.2.1.31.1.1.1.10.X, but only for
100Mbps interface and 1Gbps interface.

Case 3. No firewall information get after finish configuring the SNMP

function

There are common mistakes:


1. Mismatch on the community stream seeting between the firewall and SNMP
software
2. The SNMP management function of the interface is disabled
3. Wrong SNMP host IP address configuration. This mistake usually happened in the
NAT environment, you could run the Wireshark to get the actual IP address.

Chapter 17 LLB

Rules and limitations

1. The maximum number of routing enties created in outbound is 4096, and the reat
is 256/s.
2. The maximum number of inbound rule table is 64 for each domain name.
3. The maximum number of inboud rule talbe is 16 for each ISP addressbook entry or
each inbound interface.
4. If the inbound traffic can not match any rule table entry, only 16 address would
return.
Troubleshooting and Debug Guide |TAC

5. The capacity of DNS query rate is 1000/s in inbound rule table.

Chapter 18 WEBAUTH

Functions and configurations

Several types of timeout in WEBAUTH

webauth timeout: When authentication is successful, the system will automatically


refresh the login page before the configured timeout value ends in order to maintain
the login status. The value range is 10 to 3600*24 seconds. The default value is 60.
webauth force-timeout: If the forced re-login function is enabled, users must re-
login after the configured interval ends. The value range is 10 to 60*24*100 minutes.
Re-auth Interval: The system can re-authenticate a user after a successful
authentication. The value range is 10 to 60*24 minutes.
Redirect URL: The redirect URL function redirects the client to the specified URL after
successful authentication. You need to turn off the pop-up blocker of your web
browser to ensure this function can work properly.

Troubleshooting

Case 1. The pop-up window delay is high for some users

1. Run the ping on the client PC to confirm the network issue.


2. The Webauth function would used the resource of CORE0 itself. The delay would
be high if the utilization of the CORE0 is high. Some hack to reduce the utilization
of CORE0:
Troubleshooting and Debug Guide |TAC

a) Tune the timeout value for webauth to a lager one.


b) “no logging to buffer”
c) “Flow-on-core0 off”

Case 2. User is dropped out after the authentication in Windows 8

There are two modes of IE in windows, the normal mode and the metro mode. In
metro mode, when you minimize the IE window, the heart beat packet couldn’t sent
out, which caused the drop out. Use the normal mode of IE to avoid this issue.

Case 3. Notes for deploying Webauth in transparent mode

1. The webauth configuration is the same as the routing mode.


2. The vswitch interface must be configured with an IP address, and some static
routing entries would be added to ensure the connectivity for VLAN10-20.
Troubleshooting and Debug Guide |TAC

Chaper 19 License

FAQ

Q: Which type of License’s installation/uninstallation would need a reload of the


device?
A: AV, IPS, URLDB, VSYS, session, Performance, Platform-trail, Platform, Platform-
basic, APP DB, QOS, SSLVPN. And for VSYS license, you need to run the command
“exec feature-trial eable” before the reload.

Q: How to judge a license expired?


A: For trial License, the system would record the usage time every 15 minutes, and
this record would be saved on the EEPROM which could keep the data ever with
power lost. Then the system would compare the cumulative time with the license to
determine if the license expired. For service licenses, the expired time is an absolute
time such as 2016/9/9.

Q: How to get a usage time for a license?


A: Run command “show tech support”, there would be some information like
showing below would give you this kindle of information.
************** show license all ***************

The system has been used for 329220 minutes (about 228 days) by trial license

Q: What functions does Feature-trial license contain?


A: NBC, QOS, Vsys and SSL VPN with maximum number of user.

Q: Hwo does the license affect the session for the firewall?
Troubleshooting and Debug Guide |TAC

A:
1).For platform, platform trial license, no effect.
2).For session license, it could expand the maximum concurrent session for the
firewall.
3).For AV,IPS,URLDB, they would make the firewall to cut off half of the maximum
sessions.

Q: What would happen if install a formal in a device with a trial license installed?
A: The formal would replace the trial license to take effect, and the time of the trial
would not be count again.

Q: What will happen when the platform license expired?


A: The user can’t modify the configuration of the firewall. And the device would
running normally if you don’t reload it, but for every 5000 http connections, the
system would bre redirect one of them to a notice page which would remiand you
the license has expired. After the license expired for 3 months, all http connections
would redirected to the notice page. And if you reload the device after the license
expired, the device will not load any configuration files.

Q: If AV/IPS license expired, would they could continue takeing effect?


A: You can’t update the signature database after the liecnse expired, but the function
is still taking effect with the old version of the signature database.

Q: A device with a formal license installed and a device with a trial license installed,
could them perform a HA cluster?
A:Yes.

Q: A device with a feature trial license installed and a device without a feature trial
license installed, could them perform a HA cluster?
A:YES, but there would be a alarm generated:
Troubleshooting and Debug Guide |TAC

Alarm CRIT@FLOW: The licenses of the two HA devices are different in NBC license, NBC enable

Q: Could the use only brought the IPS and AV license when the formal platform
license expired?
A:YES. But the platform license also control the upgtade of the AV/IPS engine, we
suggest the customer to update the platform license also.

Q: Could the expand license used separately?


A:No, it is based on the platform.

Chapter 20 Log

Functions and configurations

Explaination for attack defense logs

1. UDP flood attack:DROP! trust::xethernet0/14 172.199.134.232->113.140.81.206


For the first time that the attack detected, some detail information would provided
such as interface and destination IP information.
2. UDP flood attack:DROP! srcip 172.245.111.75. Occurred 3 times in the last 14
seconds
When the attack lasts for 30 seonds or less, the log would be generated again, but
not as detail as the last one.
3. UDP flood attack:DROP! srcip 10.44.52.224. Occurred 533 times in the last
30conds
UDP flood attack:DROP! dstip 192.168.1.121.Occurred 652 times in the last 30
seconds
Troubleshooting and Debug Guide |TAC

If the attack lasts more than 30 seconds, the log would be generated every 30
seconds.
4. UDP flood:entry is not available:DROP!
The table used for handling the UPD flooe is overflow, this conditinon usually happen
when the UDP flood is very serious.

Configuraion of binary log

For StoneOS 5.0R2, 5.0R3 and above, HAS 1.0 R2 and above, the NAT, IM, URL log
support binary format.
CLI:
SG-6000(config)# logging traffic to syslog

<cr>

binary-format Enabe binary-format for syslog

custom-format Enabe custom-format for syslog(default)

Send NAT/URL/IM LOG to HSA

Generate the NAT log


Enalbe SNATlog when configurating the NAT rules.
Generate the URL log
Mothed 1(web+cli)
Create a URL filtering rule, and leave the control content to null.
Enter CLI to create a URL profile, and run command “web-serfing-record”
A(config)# url-profile url-f1

A(config-url-profile)# web-surfing-record

Verify the configuration by run command “show log traffic”


show log traffic

2012-08-01 11:18:54, INFO@FLOW: WEB: IP 192.168.50.200, user -, vrouter trust-vr, url

http://pos.baidu.com/ecom?di=u7920
Troubleshooting and Debug Guide |TAC

78&tm=BAIDU_CPRO_SETJSONADSLOT&fn=BAIDU_CPRO_SETJSONADSLOT&baidu_id=, url-

category -, method GET, action permit, reason Web surfing

 
Method 2(cli)
A(config)# url-profile url-f1

A(config-url-profile)# web-surfing-record

A(config-url-profile)# exit

A(config)# rule id 1

A(config-policy-rule)# url url-f1

Generate the IM log


A(config)# im qq log enable

A(config)# im msn log enable

Verifycation
A(config)# show log nbc

2012-08-01 11:02:23, INFO@FLOW: HSAIMLOGIN: IP

192.168.50.200:4000(10.88.16.163:4000)->119.147.45.109:8000(119.147.45.10

9:8000) (-) vrouter trust-vr, QQ, 303325243, Log off

 
Then configure a syslog and send the logs to it. And don’t forget to keep the format
of the log binary.

Explaination for end of session

In StoneOS 5.0 and above, we add the reason for end of session.
Log explaination comment
Ageout timeout  
Clear Clear session  
Block Blocked Blocked by application identification
Redirect redirected  
TCP-FIN TCP-FINpacket  
TCP-RST TCP Reset packet  
Troubleshooting and Debug Guide |TAC

TCP- Tcp shutdown Only use when SSL proxy is enabled


zapped
Close-ICMP ICMPclosed  
Sanity Sanity check

Chapter 21 IPv6

Functions and configurations

Case 1. Auto-config IP address for host

ipv6 address 2001:33:22:44::1/64


ipv6 address autoconfig
ipv6 enable
ipv6 nd ra interval 5
no ipv6 nd ra suppress
ipv6 address 2001::1/64

Case 2. Configuration of DNS64 and NAT64

1. Snat configuration:
id5 IPv6-any IPv6-any vswitchif1 10.88.16.163 Dyn-Pt

snatrule id 5 from "IPv6-any" to "IPv6-any" eif vswitchif1 trans-to 10.88.16.163 mode


dynamicport log
Troubleshooting and Debug Guide |TAC

2. Dnat:
400-GuanWu(config)# show dnat
-----------------------------------------------------------------------------------------------------------------
-------
vr name:trust-vr
=====================================================================
===================================
id from to service translate to port slb
--------------------------------------------------------------------------------------------------------
1 Any 10.88.16.163 HTTP 192.168.188.2
2 s-v61 d-v6 v4-mapped enabled
log enabled

dnatrule id 2 from "s-v61" to "d-v6" v4-mapped log

400-GuanWu(config)# show address s-v61


Name: s-v61
Address family: IPv6
Member count: 2
Members:
2003::1/128
2003::/64
Total IPv6 count: more than 2^32
IPv6 subnet in this entry: 1
2003::/64
400-GuanWu(config)# show add d-v6
Name: d-v6
Address family: IPv6
Troubleshooting and Debug Guide |TAC

Member count: 1
Members:
2005::/64
Total IPv6 count: more than 2^32
IPv6 subnet in this entry: 1
2005::/64
The address book used in the dns64 configuration is the IPv6 head used in the
fowrding.

3. DNS-4 configuration
ip dns-proxy domain any name-server 10.88.7.10 vrouter trust-vr
ipv6 dns64-proxy id 1 prefix 2005::/64 source IPv6-any trans-mapped-ip Any

prefix is used to tanslate IPv4 address to IPv6 address.

4. Policy
E 10 IPv6-any IPv6-any Any PERMIT

5. No IPv6 routing entyr needed

6. Enable the DNS proxy function in the IPv6 interface


2013-07-10 14:41:04, DEBUG@FLOW: core 1 (sys up 0x3e9f9bb7 ms): 1050647: (i)
len=94 0026.2df0.0f2a->001c.542d.a900/86dd
2003::E1E3:1326:A8A0:C20D->2005::3D87:A97D/58
ver=6, tc=0, fl=0, pl=40, hl=64
icmpv6:type=128, code=0

0026.2df0.0f2a->001c.542d.a900, size 94, type 0x86dd, vid 0, port ethernet0/0


Switchid is 8(interface ethernet0/0) port ethernet0/0
Switchid is 8(interface ethernet0/0) port ethernet0/0 ,pak iif=ethernet0/0
Troubleshooting and Debug Guide |TAC

dp_prepare_if_for_pak i_if is ethernet0/0


Start IPv6 l3 forward
Packet: 2003::E1E3:1326:A8A0:C20D -> 2005::3D87:A97D/58
ver=6, tc=0, fl=0, pl=40, hl=64
(ICMPv6)
dp_prepare_pak_lookup src:2003::E1E3:1326:A8A0:C20D,dst:2005::3D87:A97D prot
58
No session found, try to create session
-----------------First path creating new session-----------------
dp_route_first_create_session_v6 src 2003::e1e3:1326:a8a0:c20d dst
2005::3d87:a97d
--------VR:trust-vr start--------
[2003::E1E3:1326:A8A0:C20D]:1->[2005::3D87:A97D]:10
NAT: ICMP protocol type/code 8000
dp_first_lookup_dnat: search 2003::E1E3:1326:A8A0:C20D:1-
>2005::3D87:A97D:32768 prot 58
Order=1 dnat rule found
Destination ipv6 address: 3D87:A97D::3D87:A97D ipv4 address 61.135.169.125
New IP: 61.135.169.125, new port: 32768
Matched dst NAT: dnat rule id:2, new destination IP:Port->61.135.169.125:10
Get nexthop if_id: 7, flags: 20, nexthop: 10.88.16.1
Found the reverse route for force or prefer revs-route setting
NAT: ICMP protocol type/code 8000
idx=3 snat rule found
NAT: Use original port pool to allocate port. retval:42e
DYNAMIC PORT: allocated port:1070
Matched source NAT: snat rule id:5
Matched source NAT: source port1->port1070
--------VR:trust-vr end--------
IPv6 pak src zone trust, dst zone untrust, prot 58, dst-port 10.
Troubleshooting and Debug Guide |TAC

Policy 10 matches, ===PERMIT===


begin lookup predefine prot:58 port:10
Identified as app PING6 (prot=58). timeout 6.
flow0 src 2003::E1E3:1326:A8A0:C20D --> dst 2005::3D87:A97D with nexthop
10.88.16.1 ifindex 7
flow1 src 61.135.169.125 --> dst 10.88.16.163 nexthop not lookup or invalid
flow0's next hop: 2003::E1E3:1326:A8A0:C20D flow1's next hop: 10.88.16.1
crt_sess->revs_rres.nextop: 2003::E1E3:1326:A8A0:C20D, crt_sess->revs_rres.gw
10.88.16.1
Application 2049 hasn't been registered, don't need do ALG
APP inited for application PING6
The following session is installed
session: id 8434, prot 58, flag0 e,flag1 b, created 1050647, life 6
flow0(if id: 8 flow id: 16868 flag: 40008900):[2003::E1E3:1326:A8A0:C20D]:1
->[2005::3D87:A97D]:10
flow1(if id: 7 flow id: 16869 flag: 900): 61.135.169.125:10
->10.88.16.163:1070
Session installed successfully
-----------------------First path over---------------------
Found the session 8434
session: id 8434, prot 58, flag0 4e,flag1 b, created 1050647, life 6
flow0(if id: 8 flow id: 16868 flag: 40008910):[2003::E1E3:1326:A8A0:C20D]:1
->[2005::3D87:A97D]:10
flow1(if id: 7 flow id: 16869 flag: 910): 61.135.169.125:10
->10.88.16.163:1070
ICMP after translation: data1 42e, data2 a
L3 forward, out if is vswitchif1
vswitch if tx: try to lookup mac
vswitch if tx: got mac, send packet, egress port ethernet0/2
Troubleshooting and Debug Guide |TAC

Chapter 22 StoneOS upgarding

Make a backup of current configurations

A. WebUI:
Refer to the image below, click tab “System”  menu “Configuration File
Management”, check the box before the “Startup” configuration file, and then click
button “Export”. The web browser would popup a download dialog box to let you
download the configuration file, just save the file to your local disk.

B. CLI:
To save the configuration files to your computer via CLI, you need to set up a FTP or a
TFTP first. Then use the command “export configuration startup to tftp/ftp server
server_address” in the execution mode.
Troubleshooting and Debug Guide |TAC

Get the correct version of the firmware

Download the firmware file on the server ftp://ftp.hillstonenet.com/


(username/password: Partner/upgrade_stone_os) or
http://kb.hillstonenet.com/en/category/stoneos-image/ .
Please notice that for both SG-6000-E1700 and SG-6000-E1600, you need to
download the file with name SG6000-M-3-5.5R1Px.bin.
Besides, please verify the validity of the license installed in the device, ensure that
they are expired.

Setup a upgrading server (optional)

If you plan to perform the upgrading via WebUI, or using a USB drive via CLI, there is
no need to setup an upgrading server. But if you plan to perform the upgrading via
sysloader, or using ftp/tftp server via CLI, then you need to setup a ftp/tftp server
Troubleshooting and Debug Guide |TAC

before the upgrading. There are lots of server software you could use, choose one of
your favorite and install them follow the instructions provided by the developer.

Perform the upgrading

A. WebUI
Refer to the image below, click tab “System”  menu “Upgrade Management” 
tab “Upgrade Firmware”, then click the button “Browse” and locate the firmware file
you’ve downloaded to upload it to the device. Check the box “Reboot to make the
new firmware take effect” and click button “Apply”, then just wait the system to
upload the new firmware file and reboot. After the reboot complete, the system
would run with the new firmware. You could go to the same place showing below to
check the “current Version” to verify the upgrading.

B. CLI via USB driver


By using a USB driver in CLI, it’s very easy to perform the upgrading. For this method,
you need to prepare a USB driver and copy the firmware you’ve downloaded to the
driver’s root directory, and then just plug the driver to the slot on the front of the
device. Use the command “import image from USb0 file_name” in the execution
Troubleshooting and Debug Guide |TAC

mode to start the upgrading. Sometimes after plugging the USB driver, the system
would not identify the driver, because the system doesn’t have the hardware for this
particular USB driver. In this case you need to change to another USB driver or try to
perform the upgrading with another method. After the system finishing uploading
the firmware file, reboot the device to finish the upgrading. You could use command
“show version” to verify the upgrading.
Troubleshooting and Debug Guide |TAC

C. CLI via FTP/TFTP server


After setting up a FTP/TFTP upgrading server, you could perform the upgrading by
using them. Copy the firmware file to the servers, and then use command “import
image from tftp/ftp server server_address file_name” to perform the uploading, and
then reboot the device to finish the upgrading.
Troubleshooting and Debug Guide |TAC

D. Sysloader
In some extreme cases, maybe you cannot login to the device via the address of the
device’s interface, then you need to perform the upgrading using sysloader. You need
to prepare a RS232 console cable to connect to the console port of the device to use
sysloader. Power off the device first, connect the device with a console cable and
open the terminal client, then power on the device, you may see the following
outputs.
Troubleshooting and Debug Guide |TAC

When you can see the output stream “Press ESC to stop autoboot:
countdown_time”, press the ESC key in 5 seconds. Then you could launch the
sysloader. As you can find in the picture above, you could upload firmware to the
device via several methods: TFTP/FTP/USB driver.
Option 3 for using USB driver is similar with the method we mentioned before as
using USB driver in CLI, type number 3 and choose the filename.
But using the first two method is little different with using TFTP/FTP in CLI. Because in
sysloader, the device has not load the configuration file from the flash, there is no IP
address configured for the interfaces, you need to configure the temporary IP
address for the interface e0/0 so that the device could communicate with the server,
and certainly the server need to be configured with an IP address in the same subnet
with the device’s interface e0/0.
Troubleshooting and Debug Guide |TAC

After the uploading, type “6” to reboot the device, and after that, the upgrading is
finished.

Special operation in HA cluster

In HA cluster, the upgrading is more complicated. Follow this steps:


1. Remove HA preempt configuration on both master and slave device.
WEBUI
Click tab “System”  menu “HA”, in the Preempt box, choose the value “0”.
Troubleshooting and Debug Guide |TAC

CLI
Use command “no preempt” in HA configuration mode.

2. Shut down the HA interface and service interfaces in the slave device, and then
perform the normal firmware upgrading on it.
WEBUI
Click tab “Network”  menu “Interface”, choose the HA interface and edit it, check
the “shutdown” box in the tab “Advanced”. Then do the same with the services
interfaces. Then do the firmware upgrading procedures in WebUI mentioned before.
Troubleshooting and Debug Guide |TAC

CLI
Use command “shutdown” in interface configuration mode. Then do the firmware
upgrading procedures in CLI mentioned before.

3. After successfully upgrading the slave device, shut down the HA interface and
service interfaces on the master device. There would be a traffic interruption
after this action. And then enable the HA interface and service interfaces on the
slave device, so that the slave device will take over to continue the traffic
forwarding. After that, it’s time to perform a normal firmware upgrading on the
master device.
4. After successfully upgrading the master device, enable the HA interface and
service interfaces on the master device. Now everything should be OK. Enable
the preempt configuration as needed.
Troubleshooting and Debug Guide |TAC

Chapter 23 comprehensive issues

High utilization of CPU

1. Check the functions enabled on the device


1) Check that if the debug function has been enabled, disabled it if there is no needto
use it.
Show debug //show the status of enabled debug
Undebug all //disable debug
2) Show session-limit , check the session-limit configuration on the device. If
thereisno drop with the configuration, juse disable it.
2. Case by case
1) Check if there any attack on the devicxe.
Show logging security z
2) Compare the configuration used before the high ulitization of CPU
3) Use “show process” to find out which process take the most CPU resource,
especially the process besides D-Plane. If the D-Plane process take tho most CPU
resources that mean the device have reach the capacity. For other processes, please
contact Hillstone for more information.
4) Use “show logging” to check the status of the log enabled on the device, and
disable the unnecessary ones.
6) Disable the compressed file check for rar, zip, bzip2 in AV function, also run the
command “av max-decompression-length 64” to save the resources.

Packet loss

1. extend interface
Troubleshooting and Debug Guide |TAC

show interface ex/y // Check the duplex mode and speed


show controller slot 0 port x statistic// Check if bad packets existed, if so change
the cable or adjust the duplex mode and speed settings
show statistics interface-counter interface ethernet0/x second// Check if all
bandwidth bas been used

2. SWITCH status check


show controller slot 0 bri statistic, check the “Tail Dropped Packet Counter”.
Sometimes the buffer of the interface need be adjusted:
Controller slot 0
buffers egress global
3. Internal interface
show controller slot 0 port x statistic // Check if there bac packets which usually
cuased by hadware failure.
4. show cpu-cntr // Check if there any message contained “drop” or “error”. For drop
message usually caused by lack of capacity, for error message usually caused by
hardware failure.

Interruption and disconnect of direct cable

In some case, the firewall is working as a gateway, suddenly the network is down,
and the device is not able to be managed. And this condition happened over and
over again.
Some point of the failure:
1) Then network is down
2) The device can not be managed. Even using a cable to connect the device with
an admin host directly, they could not learn the ARP information from each other.
3) The access to the mgt interface is very slow too.
Diagnosis
1) Check the CPU, memory, packet-buffer, all OK.
Troubleshooting and Debug Guide |TAC

2) No ARP learned from the direct connected pot, but the ARP Table is OK. When
run “debug arp”, it turns out that there is no ARP respond sent from the device.
show controller slot 0 port 2 statistic
ethernet0/2, physical port 4:
InGoodOctets: 309385 InBadOctets: 0
InGoodPkts: 3451 InBadPkts: 0
InUnicastPkts: 0 InBroadcastPkts: 3451
InMulticastPkts: 0 InControlPkts: 0
InUndersizePkts: 0 InOversizePkts: 0
InFragments: 0 InJabbers: 0
InMACRcvErrors: 0 InCRCAlignErrors: 0
DropEvents: 3451
OutGoodOctets: 0 OutGoodPkts: 0
OutUnicastPkts: 0 OutBroadcastPkts: 0

OutMulticastPkts: 0 OutControlPkts: 0
Collisions: 0 OutDropDeferrals: 0
SingleCollisions: 0 MultipleCollisions: 0
ExcessiveCollisions: 0 LateCollisions: 0
Pkts64Octets: 1435 Pkts65to127Octets: 1957
Pkts128to255Octets: 2 Pkts256to511Octets: 2
Pkts512to1023Octets: 55 Pkts1024toMaxOctets: 0
We can find that there is only packets in the IN direction, no packets in OUT
direction.So we can tell that is something wrong with the communication between
the CPU and the SWITCH, it’s a hareware issue.
Troubleshooting and Debug Guide |TAC

It’s very slow when accessing to the Internet

Case 1. Some of the website is opened very slow, and lots of images

can not be shown

Usually this issue is cused by the network problem between the PC and Server. For
this case we are trying to analyze it from Hillstone view.
Possible reasons:
1. Affected by PBR
Some applications based on http are easily affected by the PBR in multiple SP
environment.
2. Affected by QOS
3. Affected by Session-limit
4. Exhaustion of SNAT resource

Analyzition of PC
1. DNS issue
2. Transmission of TCP segments

Case 2. Some particular websites can’t be accessed

For this case, we can eliminate the issues on the network. Check the logs of AV, IPS,
NBC. Modify the MSS value to avoid that big packets dropped druing the
transmission.
Troubleshooting and Debug Guide |TAC

Case 3. File uploading failed with firewall used

The action for file uploading in HTTP is post. Check the configurations in NBC.

Case 4. The accessing to the internal servei from Internet via DNAT is

very slow or failed

Some ISP would block the TCP 80 port for the unregsiterd public IP address.

Chapter 24 Hareware

Functions and configurations

Check the power of optical module

Show tranceiver

Troubleshooting

Case 1. The module can’t take effect after plugged to the device (for

non-SX platform)

1.       Unplug and plug the module after powering off the device.
Troubleshooting and Debug Guide |TAC

2. Check the light of the device accdrding to the list below.

3. Check if the current version of the StoneOS support the module.

4. Chech if the nessary license is installed.

5. Run “show module” to check the information and status of the module.

Indicator color/status meaning


  Green keeps on Power on
Orange blink Prepare for hot plug
  Orange keeps on Module upluged
Lights out Power off
SYS
  Green keeps on Starting up
Green blink working
  Red keeps on Failed
Lights out Failed or unpluged
 

STA
  Green keeps on Stand by
Green blink Being accessed
  Orange keeps on Stand by with fault
Lights out Uplugged
DSK(FEC-HD-
160)
  Green keeps on working
Lights out No link or failed
 

LNK
ACT Yellow blink Transmit data
Light out No Transmitting

Case 2. Harddisk card troubleshooting

1. Platform requirement
The hard disk card could only be used on G2110, G2120, G3150, G5150, and
Troubleshooting and Debug Guide |TAC

only on slot1 and slot 2 in G3150 and G5150.


2. If the indicator is red, that means the harddisk card has not been loaded, you
need to re-plug or run “exec reset slot x” to revocer it.

Chapter 25 X- series

FAQ

Powerdown by high tempreture and can not boot up even replug of

replace module card.

The high tempreture would cause CPLD lockdown in the switch board. Method of
recovery:
A: exec reset slotx

B: rewite CPLD: exec CPLD SWB WRITE 0x9 0x1

Slot number ID number


1 0x9
2 0xa
3 0xb
4 0x17
5 0x18
6 0x19
7 0x1a
8 0x1b
9 0x1c
10 0x1d
Troubleshooting and Debug Guide |TAC

Troubleshooting

Special comment for E0/0-E0/3

1. These four interfaces can not communicate with other IOM port.
2. These four interfaces can not communicate with each other either.
3. These interfaces are used for HA or management.

Minimum module install requirement for X

1. One SCM, one SSM, one IOM. And one QSM if QoS is required.
2. You can’t power on the device if you didn’t install the SCM.

Chapter 26 Wireless

FAQ
1, Which devices support WIFI and 3G?
A: E1100W,E1100WG3,E1100G3.
2, Which firmwares support WIF and 3G?
SG6000-M-3-5.0R4P2.8.bin and above.
3, Which 3G modules does the device support?
A:There are internal and external 3G mudules that StoneOS supported.
3.1Internal 3G module
So far there are CDMA2000 and WCDMA module installed with Hillstone device.
You could find the instruction in the device lable.
3.2External 3G module
Troubleshooting and Debug Guide |TAC

We recommend you to use SG6000-M-3-5.0R4P2.8.bin for external 3G modules.


CDMA 2000: HUAWEI EC177/EC156, ZET AC582;
WCDMA: HUAWEI E261/E355, ZTE MF190, ITON TU930(WC7.7);
TD-SCDMA: HUAWEI ET302/ET302S/ET306/ET127, ZTEA356

You might also like