NetXplorer and Net nforc er 7. yst em and r o u b l e s h o o t i n g Gu Gu i d e
CONFIGURA CONFIGURATION......... TION.................. ........... 5 COMMANDS..... COMMANDS............... ................... ............. .... 6 LOG FILES.......................... FILES............................ .. 12 DATABASE DATABASE ................... ........................... ........ 16 PROCESSES.......... PROCESSES.................... ............... ..... 18
NetXplorer NX7.x.x and NetEnforcer S/E7.x.x
DATA COLLECTION........... COLLECTION............. .. 18 TOOLS................ TOOLS.......................... ................... ......... 21 ISSUES..................... ISSUES............................... ............. ... 27 APPENDIX........... APPENDIX..................... .................. ........ 40
This document describes the system and troubleshooting techniques for the following products:
NetXplorer Software Version NX7.x.x
NetEnforcer Software Version S7.x.x
NetEnforcer Software Version E7.x.x
C u s t o m e r Su Su p p o r t O n l y Co n f i d e n t i a l i t y N o t i c e
Document Version: 7.3 Date: 25-JUN-07
This document contains Proprietary Trade Secrets of Allot Communications LTD and its receipt or possession does not convey any right to reproduce, disclose its contents or to manufacture, use or sell anything that it may describe. Reproduction, disclosure or use without specific authorization from Allot Communications is forbidden. Allot reserves the right to make changes, add, remove or change the schedule of any element of the plan.
NetXplorer and NetEnforcer Troubleshooting Guide
T a b l e o f Co Co n t e n t CONFIGURATION CONFIGURATION ............................ ......................................... ........................... ............................ ............................ ........................... ........................... ............................. ....................... ........ 5 PORTS........................... ......................................... ............................ ............................ ............................ ............................ ............................ ............................ ............................ ............................ ................ 5
NetXplorer Client and Server ........................... ......................................... ........................... ........................... ............................ ........................... ............................ .................. ... 5 NetXplorer Server to NetEnforcer....................... NetEnforcer..................................... ........................... ........................... ............................ ............................ ........................... ............... 5 Additional .......................... ........................................ ........................... ........................... ............................ ............................ ........................... .......................... ........................... ...................... ........ 5 ACCESSING SYBASE .......................... ........................................ ............................ ............................ ............................ ............................ ............................ ............................ ...................... ........ 5
Problems Uninstalling Sybase................................... Sybase................................................. ........................... ........................... ........................... .......................... ....................... .......... 6 COMMANDS................. COMMANDS............................... ............................ ............................ ............................ ............................ ............................ ............................ ............................ ............................ ................ 6 NETENFORCER ..................................... ................................................... ............................ ............................ ............................ ............................ ............................ ............................ ................... ..... 6 OTHER NETENFORCER TOOLS............................ .......................................... ............................ ............................ ............................ ............................ ............................ ................... ..... 7 ACSTAT
............................. .......................................... ............................ ............................ ............................ ............................ ............................ ............................ ............................ ........................ .......... 7
NICSTAT
............................ .......................................... ............................ ............................ ............................ ............................ ............................ ............................ ............................ ........................ .......... 8
ACTHRUPUT ACTHRUPUT ............................ .......................................... ............................ ............................ ............................ ............................ ............................ ............................ ............................ ................... .....
8
.......................................... ............................ ............................ ............................ ............................ ............................ ............................ ............................ ........................ .......... ACMODE ............................
9
................................ ............................ ............................ ............................ ............................ ............................ ............................ ............................ ............................ ...................... ........ ACMON ..................
9
HWADMIN ........................... ......................................... ............................ ............................ ............................ ............................ ............................ ............................ ............................. .................... ..... 10 LINKADMIN ................. .... ........................... ............................ ............................ ............................ ............................ ............................ ............................ ............................ ........................... ............. 10 GO CONFIG NIC
........................... ......................................... ............................ ............................ ............................ ............................ ............................ ............................ ........................... ............. 11
LOG FILES....................... FILES..................................... ............................ ............................ ........................... ........................... ............................ ............................. ............................. ........................ .......... 12 NETXPLORER SERVER............................ .......................................... ............................ ............................ ............................ ............................ ............................ ............................ ................ 12
C:\Allot\bin................................ C:\Allot\bin.............................................. ........................... ........................... ............................ ............................ ........................... ........................... ......................... ........... 12 C:\Allot\log................................ C:\Allot\log.............................................. ........................... ........................... ............................ ............................ ........................... ........................... ......................... ........... 12 C:\Allot\conf ............................ .......................................... ............................ ............................ ............................ ............................ ............................ .......................... ......................... ............. 13 C:\Allot\netxplorer\jboss-3.2. C:\Allot\netxplorer\jboss-3.2.6\server\allot\log 6\server\allot\log .......................... ....................................... ........................... ........................... .......................... ................. .... 13 C:\Allot\netxplorer\jboss-3.2. C:\Allot\netxplorer\jboss-3.2.6\server\allot\depl 6\server\allot\deploy oy............. .......................... ........................... ............................ ............................ ........................ .......... 14 C:\Allot\netxplorer\jboss-3.2. C:\Allot\netxplorer\jboss-3.2.6\server\allot\conf.... 6\server\allot\conf................. ........................... ........................... ........................... ........................... ........................ ........... 14 NETXPLORER CLIENT ............................ .......................................... ............................ ............................ ............................ ............................ ............................ ............................ ................ 14
C:\Documents and Settings\
.......................... name>....................................... ........................... ............................ ............................ ......................... ........... 14 NETENFORCER ........................... ......................................... ............................ ............................ ............................ ............................ ............................ ............................ ........................... ............. 14
$SWGL........................ $SWGL...................................... ............................ ............................ ............................ ............................ ............................ ............................ ........................... ........................ ........... 14 /tmp/............................... /tmp/................. ............................ ............................ ............................ ............................ ............................ ............................ ........................... ........................... ...................... ........ 15 /var/log/apache................ /var/log/apache............................. ........................... ............................ ............................ ........................... ........................... ............................ ........................... ..................... ........ 15 $SWGC ............................ .......................................... ............................ ............................ ............................ ............................ ............................ ........................... ........................... .................... ...... 16 DATABASE .......................... ........................................ ............................ ............................ ............................ ............................ ............................ ............................ ............................ .................... ...... 16 NETENFORCER ........................... ......................................... ............................ ............................ ............................ ............................ ............................ ............................ ........................... ............. 16
$SWGD..................................... $SWGD................................................... ........................... ........................... ............................ ............................ ........................... ............................. .......................... .......... 16 $SWGD/data........................... $SWGD/data......................................... ........................... ........................... ............................ ............................ ........................... ............................ ............................ ............. 17 NETXPLORER ........................... ......................................... ............................ ............................ ............................ ............................ ............................ ............................ ............................ ................ 17
C:\Allot\data\db C:\Allot\data\db............ .......................... ............................ ............................ ........................... ........................... ............................ ............................ ............................. ....................... ........ 17 Performing a Backup............................... Backup............................................. ............................ ........................... ........................... ............................ ............................ ........................ .......... 17 PROCESSES........................ PROCESSES.......... ............................ ............................ ........................... ........................... ............................ ............................ ............................ ............................ ...................... ........ 18 NETENFORCER ........................... ......................................... ............................ ............................ ............................ ............................ ............................ ............................ ........................... ............. 18 NETXPLORER ........................... ......................................... ............................ ............................ ............................ ............................ ............................ ............................ ............................ ................ 18 DATA COLLECTION................. COLLECTION............................... ............................ ............................ ........................... ........................... ............................ ............................ ........................... ............. 18 NETENFORCER ........................... ......................................... ............................ ............................ ............................ ............................ ............................ ............................ ........................... ............. 18
$SWGE/httpd/htdocs/bucket $SWGE/httpd/htdocs/bucket .......................... ........................................ ............................ ........................... ........................... ............................ ............................. .................. ... 18
www.allot.com
2
NetXplorer and NetEnforcer Troubleshooting Guide
T a b l e o f Co Co n t e n t CONFIGURATION CONFIGURATION ............................ ......................................... ........................... ............................ ............................ ........................... ........................... ............................. ....................... ........ 5 PORTS........................... ......................................... ............................ ............................ ............................ ............................ ............................ ............................ ............................ ............................ ................ 5
NetXplorer Client and Server ........................... ......................................... ........................... ........................... ............................ ........................... ............................ .................. ... 5 NetXplorer Server to NetEnforcer....................... NetEnforcer..................................... ........................... ........................... ............................ ............................ ........................... ............... 5 Additional .......................... ........................................ ........................... ........................... ............................ ............................ ........................... .......................... ........................... ...................... ........ 5 ACCESSING SYBASE .......................... ........................................ ............................ ............................ ............................ ............................ ............................ ............................ ...................... ........ 5
Problems Uninstalling Sybase................................... Sybase................................................. ........................... ........................... ........................... .......................... ....................... .......... 6 COMMANDS................. COMMANDS............................... ............................ ............................ ............................ ............................ ............................ ............................ ............................ ............................ ................ 6 NETENFORCER ..................................... ................................................... ............................ ............................ ............................ ............................ ............................ ............................ ................... ..... 6 OTHER NETENFORCER TOOLS............................ .......................................... ............................ ............................ ............................ ............................ ............................ ................... ..... 7 ACSTAT
............................. .......................................... ............................ ............................ ............................ ............................ ............................ ............................ ............................ ........................ .......... 7
NICSTAT
............................ .......................................... ............................ ............................ ............................ ............................ ............................ ............................ ............................ ........................ .......... 8
ACTHRUPUT ACTHRUPUT ............................ .......................................... ............................ ............................ ............................ ............................ ............................ ............................ ............................ ................... .....
8
.......................................... ............................ ............................ ............................ ............................ ............................ ............................ ............................ ........................ .......... ACMODE ............................
9
................................ ............................ ............................ ............................ ............................ ............................ ............................ ............................ ............................ ...................... ........ ACMON ..................
9
HWADMIN ........................... ......................................... ............................ ............................ ............................ ............................ ............................ ............................ ............................. .................... ..... 10 LINKADMIN ................. .... ........................... ............................ ............................ ............................ ............................ ............................ ............................ ............................ ........................... ............. 10 GO CONFIG NIC
........................... ......................................... ............................ ............................ ............................ ............................ ............................ ............................ ........................... ............. 11
LOG FILES....................... FILES..................................... ............................ ............................ ........................... ........................... ............................ ............................. ............................. ........................ .......... 12 NETXPLORER SERVER............................ .......................................... ............................ ............................ ............................ ............................ ............................ ............................ ................ 12
C:\Allot\bin................................ C:\Allot\bin.............................................. ........................... ........................... ............................ ............................ ........................... ........................... ......................... ........... 12 C:\Allot\log................................ C:\Allot\log.............................................. ........................... ........................... ............................ ............................ ........................... ........................... ......................... ........... 12 C:\Allot\conf ............................ .......................................... ............................ ............................ ............................ ............................ ............................ .......................... ......................... ............. 13 C:\Allot\netxplorer\jboss-3.2. C:\Allot\netxplorer\jboss-3.2.6\server\allot\log 6\server\allot\log .......................... ....................................... ........................... ........................... .......................... ................. .... 13 C:\Allot\netxplorer\jboss-3.2. C:\Allot\netxplorer\jboss-3.2.6\server\allot\depl 6\server\allot\deploy oy............. .......................... ........................... ............................ ............................ ........................ .......... 14 C:\Allot\netxplorer\jboss-3.2. C:\Allot\netxplorer\jboss-3.2.6\server\allot\conf.... 6\server\allot\conf................. ........................... ........................... ........................... ........................... ........................ ........... 14 NETXPLORER CLIENT ............................ .......................................... ............................ ............................ ............................ ............................ ............................ ............................ ................ 14
C:\Documents and Settings\.......................... name>....................................... ........................... ............................ ............................ ......................... ........... 14 NETENFORCER ........................... ......................................... ............................ ............................ ............................ ............................ ............................ ............................ ........................... ............. 14
$SWGL........................ $SWGL...................................... ............................ ............................ ............................ ............................ ............................ ............................ ........................... ........................ ........... 14 /tmp/............................... /tmp/................. ............................ ............................ ............................ ............................ ............................ ............................ ........................... ........................... ...................... ........ 15 /var/log/apache................ /var/log/apache............................. ........................... ............................ ............................ ........................... ........................... ............................ ........................... ..................... ........ 15 $SWGC ............................ .......................................... ............................ ............................ ............................ ............................ ............................ ........................... ........................... .................... ...... 16 DATABASE .......................... ........................................ ............................ ............................ ............................ ............................ ............................ ............................ ............................ .................... ...... 16 NETENFORCER ........................... ......................................... ............................ ............................ ............................ ............................ ............................ ............................ ........................... ............. 16
$SWGD..................................... $SWGD................................................... ........................... ........................... ............................ ............................ ........................... ............................. .......................... .......... 16 $SWGD/data........................... $SWGD/data......................................... ........................... ........................... ............................ ............................ ........................... ............................ ............................ ............. 17 NETXPLORER ........................... ......................................... ............................ ............................ ............................ ............................ ............................ ............................ ............................ ................ 17
C:\Allot\data\db C:\Allot\data\db............ .......................... ............................ ............................ ........................... ........................... ............................ ............................ ............................. ....................... ........ 17 Performing a Backup............................... Backup............................................. ............................ ........................... ........................... ............................ ............................ ........................ .......... 17 PROCESSES........................ PROCESSES.......... ............................ ............................ ........................... ........................... ............................ ............................ ............................ ............................ ...................... ........ 18 NETENFORCER ........................... ......................................... ............................ ............................ ............................ ............................ ............................ ............................ ........................... ............. 18 NETXPLORER ........................... ......................................... ............................ ............................ ............................ ............................ ............................ ............................ ............................ ................ 18 DATA COLLECTION................. COLLECTION............................... ............................ ............................ ........................... ........................... ............................ ............................ ........................... ............. 18 NETENFORCER ........................... ......................................... ............................ ............................ ............................ ............................ ............................ ............................ ........................... ............. 18
$SWGE/httpd/htdocs/bucket $SWGE/httpd/htdocs/bucket .......................... ........................................ ............................ ........................... ........................... ............................ ............................. .................. ... 18
www.allot.com
2
NetXplorer and NetEnforcer Troubleshooting Guide
$SWGE/httpd/htdocs/bucket/30 $SWGE/httpd/htdocs/bucket/30 (same content for 300) ........................... ........................................ ........................... ............................ ................. ... 18 Understanding the Manifest ........................... ......................................... ............................ ............................ ........................... ........................... ............................ ................. ... 19 NETXPLORER ........................... ......................................... ............................ ............................ ............................ ............................ ............................ ............................ ............................ ................ 19
C:\Allot\data\bucket\stc\ .......................... ........................................ ........................... ........................... ........................... ........................... .................. .... 19 C:\Allot\data\bucket\ltc_export\ C:\Allot\data\bucket\ltc_export\ .......................... ....................................... ........................... ........................... ........................... ........................... ........................... .............. 20 C:\Allot\data\bucket\ltc_expor C:\Allot\data\bucket\ltc_export\........................ ID>...................................... ............................ ........................... ........................... ...................... ........ 20 Allot/data/bucket/ltc/device_ID....................... Allot/data/bucke t/ltc/device_ID..................................... ............................ ............................ ............................ ............................ ............................ ................ 20 TOOLS .......................... ........................................ ............................ ........................... ........................... ............................ ............................ ............................ ............................ ............................ ................ 21
Upgrading NX Server Version............................. Version........................................... ........................... ........................... ........................... ........................... ........................... ............. 21 Enabling Compression .... ............................ ......................................... ........................... ............................ ............................ ........................... ............................. ..................... ..... 21 CHANGE ADMIN PASSWORD ........................... ......................................... ............................ ............................ ............................ ............................ ............................ .................... ...... 22 MANAGING REPORTING DATABASES ............................ .......................................... ............................ ............................ ............................ ............................ .................... ...... 22
Recreating Default (ST and LT) Databases............................. Databases........................................... ........................... ........................... ............................ .................... ...... 22 Improving Database Performance ............ .......................... ........................................ ............................ ........................... ........................... ........................ .......... 22 CHANGING REPORTING DATABASE PROFILES ........................... ......................................... ............................ ............................ ............................ ...................... ........ 23
Changing LT Reduction Profile............................. Profile........................................... ........................... ........................... ............................ ............................ ........................ .......... 23 Changing ST Profile Options .......................... .......................... ........................... ......................................... ........................... ........................... ............................ ................. ... 23 CHANGING REPORTING DATABASE PARAMETERS ............................ .......................................... ............................ ............................ ............................ ................ 24
Disabling External Hosts Reporting........................... Reporting......................................... ............................ ........................... ........................... ........................... ................... ...... 24 INCREASING THE NUMBER OF BUCKETS SENT PER TIME SLICE .......................... ........................................ ............................ ........................... ............. 24
Changing number of buckets in the NetEnforcer............................ NetEnforcer..................................... ......... Error! Bookmark not defined. Changing number of buckets in the NetXplorer .......................... ....................................... ........................... ............................ ............................ ................ .. 25 ENABLING TAP TAP MODE ........................... ......................................... ............................ ............................ ............................ ............................ ............................ ............................ ................ 25 PORT MIRROR ............................ .......................................... ............................ ............................ ............................ ............................ ............................ ............................ ........................... ............. 26
STEP 1 ............................ .......................................... ............................ ............................ ............................ ............................ ............................ ............................ ............................ ........................ .......... 26 STEP 2 ............................ .......................................... ............................ ............................ ............................ ............................ ............................ ............................ ............................ ........................ .......... 26 ISSUES ............................ .......................................... ............................ ............................ ........................... ........................... ............................ ............................ ............................ ........................... ............. 27 NTP/TIME ISSUES ............................ .......................................... ............................ ............................ ............................ ............................ ............................ ............................ ...................... ........ 27
Synchronization issues between Client and Server............................ Server......................................... ........................... ........................... ........................ ........... 27 Synchronization issues between Server and NetEnforcer.......................... NetEnforcer....................................... ........................... ........................... ................ ... 27 Problem: GUI does not start .......................... ........................................ ........................... ........................... ............................ ............................ ............................ ................. ... 29 CREATING A SNAPSHOT............................ .......................................... ............................ ............................ ............................ ............................ ............................ ........................... ............. 29
NetXplorer............. NetXplorer ........................... ............................ ........................... ........................... ............................ ............................ ........................... ........................... ............................ ................. ... 29 NetEnforcer ............................ ......................................... ........................... ............................ ............................ ........................... ........................... ............................. ............................ ............. 29 TAKING A SNAPSHOT ........................... ......................................... ............................ ............................ ............................ ............................ ............................ ............................ ................. ... 29
The Manual Snapshot ........................... ......................................... ............................ ............................ ........................... ........................... ............................ ........................... ............. 29 The Automatic Snapshot .......................... ........................................ ........................... ........................... ............................ ............................ ........................... ........................ ........... 30 Sending the Snapshot................................ Snapshot.............................................. ........................... ........................... ............................ ........................... ........................... ........................ .......... 30 HTTP SNAPSHOT .......................................... ............................ ............................ ............................ ............................ ............................ ............................ ...................... ........ 30 NAPSHOT ............................ ADD DEVICE .......................................... ............................ ............................ ............................ ............................ ............................ ............................ ............................ ................ 32 EVICE ............................ CHANGE IP...................................... IP.................................................... ............................ ............................ ............................ ............................ ............................ ............................ ...................... ........ 34
Defined Behavior............................... Behavior............................................ ........................... ............................ ............................ ........................... ........................... ............................ ................. ... 34 Current Behavior................................ Behavior.............................................. ............................ ........................... ........................... ............................ ............................ ............................ ................ 35 In-Band/Out of Band D efinitions............................. efinitions.......................................... ........................... ........................... ........................... ........................... ....................... .......... 35 PROVISIONING CHANGES ............................ .......................................... ............................ ............................ ............................ ............................ ............................ ........................ .......... 36
Add Host ............................ .......................................... ............................ ............................ ............................ ............................ ............................ ........................... ........................... .................. .... 36 CONFIGURATION CHANGES ........................... ......................................... ............................ ............................ ............................ ............................ ............................ ...................... ........ 36
Process .......................... ....................................... ........................... ............................ ............................ ........................... ........................... ............................. ............................. ...................... ........ 36 Troubleshooting.................................... Troubleshooting....................... ........................... ............................ ........................... ........................... ............................ ............................ ........................... ............. 36 DATABASES NOT SYNCHRONIZED .......................... ........................................ ............................ ............................ ............................ ............................ ........................... ............. 37
Symptoms................................ Symptoms.............................................. ............................ ........................... ........................... ............................ ............................ ............................ ........................... ............. 37 Explanation ............................ .......................................... ........................... ........................... ............................ ............................ ........................... ............................ ............................ ............. 37 Troubleshooting......................... Troubleshooting...................................... ........................... ............................ ............................ ........................... ........................... ............................. ......................... .......... 37 To Generate a Full Export........................ Export...................................... ............................ ............................ ............................ ............................ ............................ ...................... ........ 37
www.allot.com
3
NetXplorer and NetEnforcer Troubleshooting Guide RMA/BOX REPLACEMENT ............................ .......................................... ............................ ............................ ............................ ............................ ............................ ...................... ........ 38 COLLECTION PROBLEMS .......................... ........................................ ............................ ............................ ............................ ............................ ............................ ........................... ............. 38
STC Problems Related to Software.................... Software.................................. ........................... ........................... ........................... ........................... ............................ ................ 38 Data Collection Stops Due to NTP Issues .......................... ........................................ ........................... ........................... ............................ ......................... ........... 39 DEMO INSTALLATION ISSUES .......................... ........................................ ............................ ............................ ............................ ............................ ............................ .................... ...... 39
Installing NetEnforcer version 7.1.0 on a NetEnforcer AC-202/302.................... AC-202/302.................................. ........................... .................. ..... 39 Skipping installation hardware requirements.................... requirements................................. ........................... ........................... ........................... ........................... ............. 39 APPENDIX ........................... ......................................... ............................ ............................ ............................ ............................ ............................ ............................ ............................ .................... ...... 40 APPENDIX I ............................ .......................................... ............................ ............................ ............................ ............................ ........................... ........................... ............................. .................. ... 40
Host output ou tput from $SWGL/nedbg.DataSrv.log $SWGL/nedbg.DataSrv.log .......................... ........................................ ........................... ........................... ........................... ................... ...... 40 APPENDIX II .......................... ........................................ ............................ ............................ ............................ ............................ ............................ ............................ ............................. .................. ... 42
Host output from $SWGL/nedbg.AllSnmpAgent.log................. $SWGL/nedbg.AllSnmpAgent.log............................... ........................... ........................... ............................ ................... ..... 42
www.allot.com
4
NetXplorer and NetEnforcer Troubleshooting Guide
Configuration Ports N e t X p l o r e r C l i e n t a n d S e rv e r Port Number
Description
TCP:80
HTTP for initial access to Server. Once applet is downloaded, this is not required
TCP:1099
RMI (Java J2EE protocol)
TCP:4444
RMI (Java J2EE protocol)
TCP:1098
JNP (Java J2EE protocol)
N e t X p l o r e r S e r v e r t o N e t E n fo r c e r Port Number
Description
TCP:80
Data sampling
UDP:161
SNMP Configuration updates
UDP:161
SNMP Traps (Events)
UDP:123
NTP
TCP:123
NTP
Additional Port Number
Description
TCP:50000
For troubleshooting access to configuration database on NetXplorer Server
TCP:50001
For troubleshooting access to short term database on NetXplorer Server
TCP:50002
For troubleshooting access to long term database on NetXplorer Server
A c c e s s i n g Sy b a s e Database access on the Server may be required in order to troubleshoot certain issues, regarding configuration, data accuracy, data collection (and many m ore). To access the database, Sybase Central must be installed on the local PC. This can be downloaded from ftp://support:[email protected] /Sybase. To access the database open Sybase Central and perform the following: 1. Right click on ASA9 2. Select new connection
www.allot.com
5
NetXplorer and NetEnforcer Troubleshooting Guide
3. Enter user details under the Identification tab a. ID nms b. Password allot 4. Enter database (location and database) under the Database tab a. Localhost:db_port - if database resides on local server b. IP:db_port – if database resides on different server (need to ensure access to specific server, i.e. firewall issues etc.) It is possible to open all databases simultaneously.
P ro b l e m s U n i n s t a l l i n g S y b a s e At times, the uninstall procedure does not completely uninstall the Sybase application. Deleting the Allot directory and registry entries still does not complete the uninstall process. If this is the case, go to the Environmental Variables and delete the reference to the Allot folder. This will complete the uninstall process. The environmental variables can be accessed as follows: Right click on My Computer and select Properties . Click on the Advanced Tab and then click on the Environment Variables button. Under System Variables at the bottom are various entries which will show the Allot folder as the value. For additional information on uninstalling Sybase, please see KB item #6976.
Commands NetEnforcer •
acstat
•
acthruput
•
clientTest clientTest is an application used to get statistical data on the box (client) sent to the server. Usage: clientTest -s -t (30/300 seconds) -v -p -l Example (for VC statistics every 30 seconds): clienttest –s 0 –t 30 Output (for VC statistics): [Output can be found i n the nedbg.clienttest.log file] 03-14 15:27:02(201) : StatisticClient::handleNewSample, dataLen:126, sampleObject:C ollection id:270195024 StartTime:1142 342814 EndTime:114234 2822 Number of slices:24 Number of rows:1 Schema: SM_LINE_ID(1),SM_PIPE_ID(2),SM_PIPE_INST(3),SM_VC_ID(4),SM_VC_INST(5),SM_DIVIDED_BYTES _IN(19),SM_DIVIDED_BYTES_OUT(20),SM_PACKETS_IN(15),SM_PACKETS_OUT(16),SM_LIVE_CONNECTI ONS(13),SM_NEW_CONNECTIONS(12),SM_DROPPED_CONN(14),
•
swgadmin –l Output: lcd 175 DataSrv 176 SessionDispatcher 9286 coll 180 StatisticMgr 181 AllSnmpAgent 182
•
go config view (see CLI document for full list of CLI commands)
www.allot.com
6
NetXplorer and NetEnforcer Troubleshooting Guide
ther NetEnforcer Tools acstat acstat shows information about the current connections running through the NetEnforcer. Usage: acstat [ [ [ [ -l -t -u -a -n -c -r -i -s -f -F -x -m -N -B -R -I -A and dst ip
-l -t -s -I
{session/vc/pipe/h} ] / -u /-a / -n / -c / -r / -i ] ] [ -f ] [ -F ] [-x ] [ -m ] [ -N ] [ -B ] [ -R ] ,] [ -A ,]
: List session/vc/pip e/hierarchy [session] : display TCP connections : display UDP connections : display any IP connections (other than TCP and UDP) : display non IP connections : display ICMP connections : display ARP connections : display all connections : display connection allocation summary (single option, default) : display extended view : display extended view - advanced : display internal/extern al (instead of client/server) : display up to NUMBER of sessions (max 500k) : don't resolve names : dump binary data to file : read binary data from FILE (single option) , : display hierarchy all connections of pipe and vc (zero means all) , : display connections of specific src ip address address (zero means all)
acstat with no flags shows connection allocation summary Sessions are represented in the following format: Protocol Client Server State VC Client IF Protocol Client Server State
VC (Virtual Channel) Client IF TTL (Time to live) VLAN Tag ToS St (Session Status)
www.allot.com
TTL
VLAN Tag
Tos
St
Name of the protocol. If the name is unknown, the hexadecimal number of the protocol is shown. Raw TCP is shown as TCP-r. IP of the host which initiated the session (for TCP and UDP sessions also the port). IP of the host to which the client send its request (for TCP and UDP sessions - also the port). Prisma Session State. Can be one of the following: OPENED, CONNECTED, WIRED, TO BE CLOSED, CLOSED, REJECTED, DROP or NONALLOCATED (the last one should never appear; if it does, there is probably a bug). If the client-server and the server-client sides of a session are in different states, both states are shown, e.g. WI-2b for WIRED - TO BE CLOSED. VC to which the session belongs. If the client-server and the serverclient sides of a session belong to different VCs, both VCs are shown. NetEnforcer interface that the client is connected to. Time left (in seconds) until the session expires if no traffic arrives. Indicates if the connection is VLAN tagged and to which VLAN. ToS marked value. If the number displayed is 0, then there are no ToS markings on the packets. Possible options are Raw, Half, Dbl, Chng, Loop or NA. Raw indicates if the session is raw i.e. the connection was classified after it had been established. Dbl indicates a double session.
7
NetXplorer and NetEnforcer Troubleshooting Guide
nicstat Displays the mode and speed of network interfaces. It is not the speed and duplex defined in the GUI “configuration”, but the actual values. The command is used for troubleshooting access links related problems and for verification that the links are compatible with the adjacent router or switch. Certain networking related problems are coming from NICs definitions that are mis-configured. Checking the nicstat and comparing it to the router/switch definition is a useful tool in troubleshooting problems like packet loss, synchronization and network slowness issues. Command nicstat
Output
+-----------+------+-------+--------+ | Interface | Link | Speed | Duplex | +-----------+------+-------+--------+ | eth0 | up | 10 | half | | eth1 | down | n/a | n/a | | eth2 | up | 10 | half | +-----------+------+-------+--------+
acthruput Prints the amount of bits that have passed through each Interface, active pipe and active VC during one time slice (one second). The output of the command shows the bandwidth consumption of each of the active pipes/vcs and for the entire interface. It can be used also to determine if there’s a need to alter the bandwidth definition of the pipe/vc and to troubleshoot bandwidth and traffic related problems. Usage: acthruput [ -b ] [ -B ] [ -c ] [ -v ] [ -d DIR ] -b : display throughput in bits (default) -B : display throughput in bytes -c : display throughput per connection -t : display total link throughput including IgnoreQoS -d DIR : analyze data in DIR instead of / e.g. acthruput -d $W/stat/last - to analyze the last snapshot
Command acthruput
Output --------------------------------------------------------Entity Name Bits/sec --------------------------------------------------------INTERFACE Internal 0 --------------------------------------------------------INTERFACE External 2896 PIPE 1 1024 VC 8 512 VC 1 512
Note: The actrhuput command should only be used for AC-x0x devices. For AC-1000 devices, please use the acmon command (see next page).
www.allot.com
8
NetXplorer and NetEnforcer Troubleshooting Guide
acmode Switches between various NetEnforcer software modes. Shows, saves and restores modes and makes the NetEnforcer enter/exit software or hardware bypass. Examples: enable/disable QoS, TCP, UDP, etc. acmode [ [ [ [
+/-endvcs ] [ +/-srcmac ] [ +/-ignoremom ] +/-verbose ] [ +/-mtu ] [ +/-noweight ] [ +/-novc ] [ +/-wnyfast ] save ] [ restore ] [ default ] [ show ] hwbp ]
+endvcs - enable ended vcs -endvcs - disable ended vcs +srcmac - enable source mac handling -srcmac - disable source mac handling +ignoremom - enable ignore monitoring only mode on dkm -ignoremom - disable ignore monitoring only mode on dkm +verbose - enable dkm verbose -verbose - disable dkm verbose +mtu - enable Check and Fragment IP packet according to MTU size -mtu - disable Check and Fragment IP packet according to MTU size +noweight - enable counting traffic with Ignore QoS Policy for monitoring/accounting purposes -noweight - disable counting traffic with Ignore QoS Policy for monitoring/accounting purposes +novc - enable counting traffic that passes through NE prior to policy assignment -novc - disable counting traffic that passes through NE prior to policy assignment +wnyfast - enable winny fast identify method -wnyfast - disable winny fast identify method (default) save restore default show
-
save current settings restore saved settings restore default settings show current settings
hwbp
-
go into hardware bypass
Note: you can run acmode with a number of arguments, e.g. acmode +qos -tcp. The arguments are processed one by one in the order of appearance, with two exceptions: - hwbp (go into hardware bypass) is processed last.
acmon Used to get statistics ( ONLY for AC-1000 units). Usage: acmon { -p / -v / -s / -d [ -t ] -p -v -s -d -l -r -t
/ -r / -l }
: monitor specific pipe rate : monitor specific vc rate : monitor specific service rate : monitor dmu packet distribution : run acmon limited count number : monitor octet rx : time to wait between samples in seconds [1 seconds]
Example: [i ] 10:10:02 >> 0 conn ps [0] rate inbound: 0.000 bps outbound: 0.000 bps [1] rate inbound: 202.772 Kbps outbound: 0.000 bps
www.allot.com
9
NetXplorer and NetEnforcer Troubleshooting Guide
HwAdmin Controls the bypass mechanism. This command can be used to send the box to hardware bypass. Usage: HwAdmin -s -H
: displays system status : displays hardware (AC, MACH, FULL, OEM) version information.
Command
Output
HwAdmin –s
Status register = 0x3 Local machine is STAND_ALONE and in ACTIVE mode Local bypass is CONNECTED Remote machine not detected
HwAdmin -H
Hardware version - 402 Firmware version - 2 OEM version – 0
LinkAdmin Changing the NIC configuration on the AC-X02 and AC-1000 series: LinkAdmin will give you various options: LinkAdmin -[dsuc] -c [autoneg on|off] [speed 10|100|1000] [duplex hal f|full] -d - link down -u - link up -s - show link status -f - show supported link speed and duplex optional interface name eth1 eth0 nic1 nic0 etc. If we want to set the internal interface to full 100, you can use either of the commands: LinkAdmin 0 autoneg off speed 100 duplex full LinkAdmin -c 0 autoneg off speed 100 duplex full LinkAdmin -c eth0 autoneg off speed 100 duplex full
The command needs to be followed by a reboot. Please note that these commands are for the AC-X02 and AC-1000 only.
www.allot.com
10
NetXplorer and NetEnforcer Troubleshooting Guide
go config nic TheNICsettingsontheNetEnforcerAC-404,AC-804,andAC-808canbeconfiguredusingthe go config nic CLIcommand. AC:~# go config nic Command: go config nic Usage: go config nic {,...} Acceptable Labels are: INTERNAL1, EXTERNAL1, MGMNT, INTERNAL2, and EXTERNAL2 Acceptable values of Mode are: half, full, and auto Acceptable values of Speed are: 10, 100, 1000, and auto (according to box type) Acceptable values of Failure Action are: none, fail_pair, fail_all, and bypass
Example:goconfignicINTERNAL1:full:100:fail_pair Important Note:TheAC-404does not support 1000Mbps speed,althoughitispossibletorunthe go config niccommandwith1000Mbpsasaspeedvalue. Labels: FortheAC-808,theacceptablelabelsare:INTERNAL1,EXTERNAL1,INTERNAL2, EXTERNAL2,MGMNT,INTERNAL3,EXTERNAL3,INTERNAL4,andEXTERNAL4 Speed:
AcceptablevalueofSpeed:1000 -theinterfacesarecapableofworkingwith1Gbpsphysically (beconnectedto1Gbpsinterfaces). AlloftheAC-808interfacessupport1000Gbpsphysicalspeed. Values: AcceptablevaluesofFailureAction: fail_pair:ifoneinterfacewithinapair(INTERNALx-EXTERNALx)isdown,thesystem willdisableitspeer. fail_all:ifoneinterfaceisdown,thesystemwilldisableallotherinterfaces. bypass: :ifoneinterfaceisdown,thesystemwillmovetobypass. Management port Asofversion7.1.0build24,onlythemanagementportcanbeconfiguredviatheadminmenu. TheAC-80x(thenewAC-802platform,AC-804,andAC-808)managementportsupports 10/100/1000(physicalspeed).
www.allot.com
11
NetXplorer and NetEnforcer Troubleshooting Guide
Log Files NetXplorer
erver
All logs are stored under Allot\. This is usually located under C:\.
C: \Allot\bin All batch and executable files are located here, including all processes (e.g. poller, keeper). File Name
Explanation
Create_snapshot_logs.bat
Snapshot generator
Start_.bat
Batch file initializing specified database
Stop_.bat
Batch file stopping specified database
reduction_profile_upd.bat
Batch file that copies selected reduction cfg file from \allot\conf\Reduction to \allot\conf
check__db.bat
Checks if specific database (CFG, STC, LTC) alive mechanism used check_db.bat file
check_db.bat
Check database alive mechanism
conf_assist.exe
Prepare database password for \allot\conf stc_collect.cfg and \allot\conf ltc_collect.cfg files (Not in use for users)
db_install.exe
Used for Sybase install ,database create and recreate
C: \Allot\log File Name
Explanation
poller.log
Poller log
converter.log
Converter log
loader.log
Loader log
ltc_poller.log
Long Term Poller (lt_poller) log
ltc_loader.log
Long Term Loader (lt_loader) log
keeper.log
Keeper Server log file
allot_.txt
Database work process log file
allot__stop.txt
Database stop process log file
www.allot.com
12
NetXplorer and NetEnforcer Troubleshooting Guide
C: \ A l l o t \ c o n f File Name
Explanation
nedbg.conf
Configuration file for keeperServer.exe and LTreducer.exe
reduction.cfg
Configuration file for reduction process used by LTreducer.exe
stc_collect.cfg
Configuration file for stc collector processes (poller, converter, loader, manifest_manager)
ltc_collect.cfg
Configuration file for ltc collector processes (ltc_poller, ltc_loader)
hosts.cfg
Hosts list used by LTreducer.exe
Reduction
directory
Optional reduction configurations
MIB
directory
MIB files for MIB modules supported by the agent
XML
directory
XML schemas for interfacing with the agent
db
directory
Data files for static loading of certain tables
swkeeper.ini
file
Process and database initialization file including log level configuration (similar to swgrun.ini on the NetEnforcer)
static.ini
file
Database parameters and ports
C: \Allot\netxplorer\jboss-3.2.6\server\allot\log File Name
Explanation
NMS.log
Application Server log. Example messages: [EAR Deployment] Init J2EE application:…. Implication: application loading Subsequent messages: loading of each module [NamingService] Started jndi bootstat…1099… Implication: connecting to server Note: this port must be open otherwise system will not load [RARMetaData] Loading Jboss Resource Adapter… Implication: loading connection to database (will appear after above message) Subsequent messages: loading of each module, look out for [Deploy] messages. Stacked traces indicate problems
NMS.log.n
Older versions of nms.log (can be up to 40 before original one is overwritten)
boot.log
Jboss log
jsr77.log
Jboss log
server.log
Jboss log including some application server exceptions
www.allot.com
13
NetXplorer and NetEnforcer Troubleshooting Guide
C: \Allot\netxplorer\jboss-3.2.6\server\allot\deploy File Name
Explanation
NMS.ear
This is the NetXplorer software application. A software upgrade can theoretically be performed by replacing this file.
sybase-ds.xml
Contains configuration (allot_cfg) database and password
C: \A l l o t \n e t x p l o r e r \ j b o s s -3 . 2 .6 \s e r v e r \ a l l o t \c o n f File Name
Explanation
log4j.xml
Contains configuration parameters for NMS.log including debug level and number of instances of log file. o maxfilesize - log size o maxbackupindex - max number of logs
NetXplorer
lient
C: \D o c u m e n t s a n d Se t t i n g s \< u s e r n a m e > File Name
Explanation
NMS.log
Application client log. The contents of this file are not the same as NMS.log located on the Server.
NetEnforcer $SWGL File Name
Explanation
ac_reboot.log
Log of ac_reboot command
badCCBs
Not in use.
bt
Directory that contains all backtrace files.
coll_dump
Various counters from collector process that can be printed upon user request.
counters.swg
nedbg.keeper.log takes information from this file.
dbchanges.swg
Policy changes accepted by DKM.
dkmdump
Various counters from DKM process that can be printed upon user request.
errorlog.swg
DKM log
hwu.HwAdmin.log
HwAdmin utility log
hwu.lcd.log
LCD log
kpc.SessionDispatch.log
Log created by every process that uses the KPC library (IPC between user and kernel)
log.SWG
Obsolete - not used.
www.allot.com
14
NetXplorer and NetEnforcer Troubleshooting Guide
nedbg.acstat.log
Log of acstat process
nedbg.AllSnmpAgent.log
Log of SNMP agent/process (communication between Server and NetEnforcer)
nedbg.AllSnmpAgent.log.old
Old SNMP log
nedbg.Collector.log
Log of Collector process
nedbg.DataSrv.log
Log of DataSrv process. Issues with applying database changes and changes applied logged. In debug mode, this shows complete database update including XML command received from server, changed performed, counter ID updated and ok sent to Server.
nedbg.default.log
Obsolete – not used.
nedbg.go.log
CLI log
nedbg.keeper.log
Log of Keeper (hardware keeper)process
nedbg.lcd.log
Log of lcd process
nedbg.StatisticMgr.log
Log of StatisticMgr (Statistics Manager) process. Problems with buckets will be logged.
nedbg.swKeeper.log
Log of swKeeper (software keeper) process
nedbg.swKeeper.log.old
Old log of nedbg.swKeeper.log
ne-instl..log
Log of last installation process
notice.SWG
DoS attack reported by DKM
ntp.log
Log of ntp process. Can identify problems with NTP synchronization.
StatisticMgr_dump
Various counters from Stat Mgr process that can be printed upon user request.
/tmp/ File Name
Explanation
nedbg.ProvisionCli.log
check whether content was received from the Apache Server View full XML content
/var/log/apache File Name
Explanation
access_log
check whether Apache received change Look for POST to ProvisionCli.exe
www.allot.com
15
NetXplorer and NetEnforcer Troubleshooting Guide
$SWGC File Name
Type
Explanation
reduction.conf
File
Short Term reduction configuration parameters
SNMP
Directory
actype
File
NetEnforcer version and type
addnsParameters
File
DNS refreshment parameters
dataCli.conf
File
Internal config file
dkm.conf
File
dkm and prisma configuration parameters
hosts.conf
File
List of hosts referred to during the reduction of statistic data.
keeper.ini
File
HWKeeper ini file managing initialization parameters of all modules controlled by the HW Keeper
lcd_version
File
Displays lcd version
memwatch.conf
File
Memory consumption levels indicated memory issues
nedbg.conf
File
Debug level of all nedbg log files
provisioncli.conf
File
Internal config file.
reduction.conf
Link to file
Link to selected reduction configuration file
Reduction.*
File
All optional reduction configuration files
statisticmgr_boot_counter
File
Counter of restarts of statistic manager process.
swKeeper.ini
File
SWKeeper ini file managing initialization parameters of all processes controlled by Keeper
Database NetEnforcer $SWGD Name
Type
Explanation
backup
directory
Location of most recent successful policy update (schema and data directories and their content)
data
directory
Location of policy and configuration database
schema
directory
Location of policy and configuration database schema
lastSnmpUpdate
file
Maintains timestamp of last policy update received by SNMP. Used to report on synchronization status of device against the server.
www.allot.com
16
NetXplorer and NetEnforcer Troubleshooting Guide
$SWGD/data Name
Explanation
allotConfig.xml
Database of NetEnforcer configuration parameters. Including: device capabilities (modes), registration parameters, device limits (e.g. Lines, VCs, Pipes, bandwidth), data collection and reduction parameters. Network parameters are not included in this file.
allotProvision.xml
Policy and Catalog database. This is one file including all of the Catalog definitions and the Policy configuration.
lastPolicyFullExport
Maintains timestamp of the last full policy export to the device. Used to report on synchronization status of device against the server.
lastPolicyUpdate
Maintains timestamp of last policy update distributed by data server to internal clients. Used to report on synchronization status of device against the server.
NetXplorer C: \Allot\data\db Name
Type
Explanation
cfg
directory
Location of configuration database, allot_cfg.db
ltc
directory
Location of long term data database, allot_ltc.db
stc
directory
Location of short term data database, allot_stc.db
P e rf o r m i n g a B a c k u p Please note that there are two kinds of database backups for the NX server.
Cold backup – done when services can be stopped. Hot backup – done when services are running.
Cold backup 1. Stop NetXplorer Service by going to Windows Services and stopping NetXplorer Server . 2. The following lines should appear in the allot_ltc.txt and allot_stc.txt files: “Disable all events” “End of current events” 3. Backup the database by copying the following folder: c:\Allot\data\db to a different location, preferably a different disk. 4. Start the NetXplorer Service. Hot backup In order to perform a hot backup, please see KB item 6269: "NetXplorer Backup and Restore Database". Please note that this should only be given to customers in exceptional cases.
www.allot.com
17
NetXplorer and NetEnforcer Troubleshooting Guide
Processes NetEnforcer There are several processes that should always be running on the NetEnforcer. These processes can be identified using several different commands, as follows: swgadmin -l lcd DataSrv SessionDispatcher coll StatisticMgr AllSnmpAgent ps –awx|grep ntp or ntpq –p (or use ps-ax) ntp client HTTP •
•
•
NetXplorer There are several processes that should be running on the NetXplorer Server. These processes can be identified using several different tools: Windows Services (Start>Control Panel>Administrative Tools>Services) o NetXplorer Server Windows Task Manager (CTRL+ALT+DEL and select Task Manager) o Poller.exe o Converter.exe o Loader.exe o ltc_poller.exe o ltc_Loader.exe o ltreducer (runs periodically – therefore may not be seen) o manifest_manager.exe (runs periodically – therefore may not be seen) o KeeperService.exe o Dbsrv9.exe (3 instances) o ntpd.exe •
•
a t a Co l l e c t i o n NetEnforcer $SWGE/httpd/htdocs/bucket Name
Type
Explanation
30
directory
Location of 30 seconds buckets data
300
directory
Location of 300 (5 minutes) second buckets data
$SWGE/httpd/htdo c s / bu c k e t /3 0 (s a m e c o n t e n t f o r 3 0 0 ) Name
Type
Explanation
conv_stat
directory
Location of conversation buckets (binary format)
www.allot.com
18
NetXplorer and NetEnforcer Troubleshooting Guide
vc_stat
directory
Location of rules buckets (binary format)
line_burst
directory
Not in use
pipe_burst
directory
Not in use
vc_burst
directory
Not in use
manifest
Link
Link to current manifest
manifest
file
The manifest file containing a list of buckets that need to be collected by the Poller on the NetXplorer
Understanding the Manifest The manifest can be accessed through the web, by browsing to: http:///bucket//manifest Example: http://192.123.234.56/bucket/30/manifest
Format
Boot number, bucket index, bucket type (0=vc_stat, 1=conv_stat), statistic type, start time, end time, bucket duration, actual bucket duration, compression (0=no, 1-yes). Bucket duration is not always exactly 30/300 seconds. There may be a fluctuation of 1 or 2 seconds either way (for example, 299 or 301 seconds).
NetXplorer C: \ A l l o t \ d a t a \ b u c k e t \ s t c \ < d e v i c e I D > Name
Type
Explanation
conv_stat
directory
Contains conversations buckets in binary and then ascii format before import to short term database
vc_stat
directory
Contains rules buckets in binary and then ascii format before import to short term database
line_burst
directory
Not in use
pipe_burst
directory
Not in use
vc_burst
directory
Not in use
www.allot.com
19
NetXplorer and NetEnforcer Troubleshooting Guide
C: \ A l l o t \ d a t a \ b u c k e t \ l t c _ e x p o r t \ Name
Type
Explanation
directory
Multiple folders representing each device managed by the NetXplorer Server
manifest
file
Manifest file containing list of buckets that need to be imported into the long term database
C: \ A l l o t \ d a t a \ b u c k e t \ l t c _ e x p o r t \ < d e v i c e I D > Name
Type
Explanation
conv_stat
directory
Contains conversations buckets in ascii format exported from the short term database
vc_stat
directory
Contains rules buckets in ascii format exported from the short term database
line_burst
directory
Not in use
pipe_burst
directory
Not in use
vc_burst
directory
Not in use
A l l o t / d a t a / b u c k e t / l t c /d e v i c e _ I D Name
Type
Explanation
conv_stat
directory
Contains conversations buckets in ascii format before im port to long term database
vc_stat
directory
Contains rules buckets in ascii format before import to long term database
line_burst
directory
Not in use
pipe_burst
directory
Not in use
vc_burst
directory
Not in use
For details about the data collection procedure, refer to the SE training presentation.
www.allot.com
20
NetXplorer and NetEnforcer Troubleshooting Guide
Tools Upgrading NX Server Version • •
•
Stop NetXplorer Service by going to Windows Services and stopping NetXplorer Server . Open the Windows Task Manager by pressing and clicking the Task Manager button. Select the Processes tab and confirm that DbSrv9.exe does not appear in the list. Download the software version desired from the Allot ftp site by completing the following steps: 1. Log into the ftp site with your personal support login account (download\username) and password. Access will only be allowed if a valid license for NetXplorer has been 2. 3.
purchased. Type cd NetXplorer/NetXplorer_Server/Current_Versions/NetXplorer_NX7xx.zip Please note that the NetXplorer files are approximately 460MB and will take some time to download. They are compressed and must be opened with WinZip or another utility.
For complete instructions and full installation procedures, see the NetXplorer Quick Install Guide and NetXplorer Operation Guide from http://www.allot.com. •
•
•
There is no need to remove a previous installation. It will be detected automatically by the Installation Wizard. The NetXplorer Service will be stopped automatically when the upgrades starts. It will resume operation after the server is rebooted following the upgrade. At the end of the upgrade procedure you will be asked to reboot the NetXplorer Server.
Please note that if the NetXplorer Server will be down for more than 25 minutes, Real Time (Short Term) data after this period will be lost and data collection will be continued only after the server is up again. Therefore it is recommended to perform the upgrade during low traffic hours.
E n a b li n g C o m p r e s s i o n Toggling bucket compression on/off By default, compression is turned off (i.e. regular buckets). To toggle bucket compression: 1. Edit $SWGD/data/ allotConfig.xml 2. The parameter data_collection/bucket_type should be set to 1 for compression or 0 for no compression. 3. Reboot the NetEnforcer. Note: Compression is not recommended as a default configuration, but only in situations where it is absolutely necessary. Enabling compression places additional heavy load on the NetEnforcer.
www.allot.com
21
NetXplorer and NetEnforcer Troubleshooting Guide
hange Admin
assw ord
If the admin password has been lost, it is possible to replace it with the original password allot. In the SYSTEM_USERS table of the allot_cfg database, replace the admin password with: 53xXk0LYvZI=
Managing Reporting Databases R e c r e a t i n g D e f a u l t ( ST a n d L T ) D a t a b a s e It is possible to recreate empty (default) collector databases (STC and LTC). Data for the Device table will be loaded from Application Server (CFG database) as soon as the NetXplorer Server service is initialized after running the procedure. This utility replaces the current database files with clean databases (according to the configuration files c:\Allot\conf\static.ini and c:\Allot\conf\dynamic.ini created during installation process). Procedure 1. Stop the NetXplorer Server 2. Open MSDOS command window (Start>Run> type cmd). 3. c:\Allot\bin\recreate_default_db.bat . a. STC – recreate STC database; b. LTC – recreate LTC database. The following message appears in the command window - Recreate database successful or failed. 4. If the process has been successful restart the NetXplorer Server service. Note: There is more chance that the ST DB will get stuck, as it is in use approximately every 10 seconds, while the LT DB is only updated every hour. For problems with LT DB, please contact Escalation for additional assistance.
I m p r o v i n g D a t a b a s e Pe r f o r m a n c To ensure better performance for complex NetEnforcer deployments managed by a single NetXplorer server, the following post-install changes for STC and LTC databases may be considered: Change temporary file location Change transaction log location Change dbspaces location (rename DBspace) Allocate additional disk space for DBspaces • • • •
Deployment: 4 (four) files located in the directory \allot\bin: run_post_install_stc.bat; post_install_stc.vbs; - for STC database; run_post_install_ltc.bat; post_install_ltc.vbs – for LTC database; • •
Usage: First - NetXplorer Server service should be stopped. Before running, the VBscript files post_install_stc.vbs, post_install_stc.vbs should be manually edited. Carefully read all remarks,
www.allot.com
22
NetXplorer and NetEnforcer Troubleshooting Guide
comment unnecessary commands, set real paths for database files and necessary sizes for dbspaces. Recommendations for all post-install steps are available in the mentioned VBscript files. In case dbspaces file locations (paths) are changed, it is necessary to change (manually edit) the dbspaces locations in \allot\conf\dynamic.ini file. Open a command window (cmd.exe). From the command-line, run: \allot\bin\ run_post_install_stc.bat or run_post_install_ltc.bat. The following message will appear after the command has completed successfully: See post installation log in -\allot\tmp\install\post_install_stc.log
h a n g i n g R e p o r t i n g Da t a b a s e P r o f i l e s C h a ng i n g L T R e d u c t i o n
rofile
Change the reduction.cfg file for the LTreducer application. The installation copies enterprise normal profile file into directory \allot\conf. The mentioned profile then becomes active (file name is reduction.cfg ). All reduction profile files are located in the \allot\conf\Reduction directory. This utility will copy the active reduction profile file in \allot\conf from the \allot\conf\Reduction directory. The possible reduction profile types are: ent_normal ; ent_accuracy ; ent_history ; isp_normal ; isp_accuracy ; and isp_history . Please note that ent = enterprise and isp = Internet Service Provider. Usage: Open command window (cmd.exe). From the command-line, run: \allot\bin\ reduction_profile_upd.bat . Profile types are: ent_normal ; ent_accuracy ; ent_history ; isp_normal ; isp_accuracy ; and isp_history . Example: \allot\bin\ reduction_profile_upd.bat isp_accuracy For more information on profiles, see the Excel chart on profiles in the knowledge base (http://support.allot.com) (item #6423), the SE Internal Training (item #6059), and item #6836.
C h a n g i n g S T Pr o f i l e O p t i o n s Purpose: Change data aging parameters in STC database PARAM table for second, minute and hour statistical data. Server Usage: Open command window (cmd.exe). From the command-line, run: \allot\bin\ stc_profile_upd.bat .
Profile types are: ent_normal ; ent_accuracy ; ent_history ; isp_normal ; isp_accuracy ; and isp_history . NetXplorer Server service (or STC database) should be restarted.
Example: \allot\bin\ stc_profile_upd.bat isp_accuracy NetEnforcer Change collection profile: go config data_collect Acceptable values of Reduction Environment are: ent and isp Acceptable values of Reduction Profile are: normal , accuracy and history
www.allot.com
23
NetXplorer and NetEnforcer Troubleshooting Guide
hanging Reporting Database Paramet ers D is a b l i n g E x t e r n a l H o s t s R e p o r t i n g To disable external host collection, use the following CLI command: go config data_collect -no_ext_host enable The NetEnforcer will reboot after 5 seconds. Please note that by default, the AC-1000 does not include external hosts as part of the collection key and the AC-400/AC-800 does.
I n c r e a s in g t h e n u m b e r o f b u c k e t s s e n t p e r t i m e s l i c e C ha n g i n g n u m b e r o f bu c k e t s i n t h e N e t E n f or c e r Note: This should only be used in situation where the need for increasing the buckets is critical. The default number of buckets sent is 5. There is an option to increase this number, to a maximum of 48 buckets (on x0x devices). In the AC-10x0/AC-25x0 devices there is no HDD and it is not recommended to increase this number at all. Increasing the number of buckets should be followed by enabling compression on the device (see page 21 on how to enable compression). This is done as follows: 1. CD to $SWGD/data 2. Vi to allotConfig.xml 3. Modify the line marked in bold below from 5 to the new number: 30 1 0 5 5 0 4. Restart the StatisticMgr module in order to include the modification. Note: Increasing this parameter would increase the number of buckets for 30 second as well as 300 second. 48 buckets is equal to 4 hours of 5 minute resolution, and 24 minutes of 30 second resolution.
www.allot.com
24
NetXplorer and NetEnforcer Troubleshooting Guide
C h a ng i n g n u m b e r o f b u c k e t s i n t h e N e t X p l o r e r Every bucket has a time stamp. When the server receives a bucket, it checks the timestamp. If the timestamp is older than UTC time minus delta, it discards the buckets. In order to increase this delta, it is necessary to do the following: 1. Enter Sybase Central. 2. Enter the STC database. 3. Go to the PARAM(nms) table in the Table folder. 4. Choose the Data tab. 5. Go to line 66 The max time for a 30 seconds bucket time to be before of the current UTC. 6. Change the INT_VAL value from 180 to a value larger than 30sec x selected number of buckets . 7. Do the same on line 67 The max time for a 300 seconds bucket time to be before of the current UTC. 8. Change the INT_VAL value from 1800 to a value larger than 300sec x selected number of buckets .
nabling TAP Mode To enable TAP mode, right-click on a NetEnforcer and select configuration. On the Networking tab, check TAP Mode and save. TAP mode will now be enabled. Note: TAP Mode is not supported on the NetEnforcer AC-1040.
www.allot.com
25
NetXplorer and NetEnforcer Troubleshooting Guide
ort Mirror Many customers do not wish to install a NetEnforcer inline between the LAN switch and the WAN router, even in monitoring-only mode, since they need to disconnect the line when installing the NetEnforcer. Therefore they wish to install the N etEnforcer on the switch mirror port, or span port, instead and monitor the traffic in that way. The switch mirror port mirrors the traffic received and transmitted on the port to the WAN router. The NetEnforcer is used as a simple monitoring probe and the Internal or External port is connected to the switch mirror port. Therefore only one port is connected. The NetEnforcer can still monitor traffic in this case, however there are two modifications needed for the NetEnforcer to operate properly.
Procedure Step 1 Bridge learning must be disabled in order to prevent the NetEnforcer from learning and maintaining a bridge forwarding table for the port connected to the switch mirror port. 1. Connect to the NetEnforcer console via the Console port or a Telnet/SSH session. Login as user ‘root’ with password ‘bagabu’ (unless changed). 2. Open the file /usr/local/SWG/bin/init_modules for editing using the vi editor by entering the following command: vi /usr/local/SWG/bin/init_modules 3. Change the line prisma_args="stree=${STREE_MODE} to prisma_args="nolearn=1 stree=${STREE_MODE} 4. Save the changes by entering the following command :wq 5. Reboot the NetEnforcer for the change to take effect. Step 2 When the NetEnforcer has rebooted and has become active again, the handling of “double sessions” must be changed as follows: 1. Connect to the NetEnforcer console again via the serial port or a Telnet/SSH session. Login as user ‘root’ with password ‘bagabu’ (unless changed). 2. Type the following command: acmode +dbs 3. Type the following command acmode –qos 4. The QoS software will restart automatically, no need to reboot.
Conclusion Traffic between the LAN switch and the WAN router may now be monitored from the switch mirror port. All the different monitoring graphs should work with the exception of the ‘Connections’ graphs. NetAccountant and the Long Term Monitoring may also be used.
www.allot.com
26
NetXplorer and NetEnforcer Troubleshooting Guide
Issues N T P /T i m e i s s u e s S y n c h r o n i za t i o n i s s u e s b e t
e e n C li e n t a n d S e r v e r
The NetXplorer Client and NetXplorer Server have a tolerance of 10 minutes time difference. The devices may be on different time zones. For example, if the Server is set to 10:03, and the device is set to 10:05, then this is acceptable. The same goes if the time zone difference is +2:00 (12:05). Note: Daylight savings time may cause an issue with the time zones. Symptoms If the clocks are out of sync, the graphs/logs times are inconsistent. Troubleshooting After login to the client, there is always a log of the time (UTC time dump). Check c:\Documents and Settings\\NMS.log to view this time dump.
S y n c h r o n i za t i o n i s s u e s b e t w e e n Se r v e r a n d N e t E n fo r c e r Symptoms If the clocks become out of sync, then there can be many issues including data collection. When statistics are gathered by the NetEnforcer, a bucket is created with a timestamp based on the NetEnforcer clock. Periodically, these buckets are collected by the poller process (on the NetXplorer). The NetXplorer compares the time of the bucket with its internal clock. If the NetEnforcer and NetXplorer Server have a time difference larger than 180 seconds for 30 second buckets and 1800 seconds for 300 second buckets, it will discard the bucket. When the user tries to generate real time monitoring graphs, no real-time data will be displayed, and an error message will appear in red displaying: “No data for the time selected”. The following alarm/event is received if data collection is stopped (can be found in the poller.log file located in C:\Allot\log ): 'invalid bucket time on device NetEnforcer404' (id 208 - Current bucket time is older that current UTC minus delta) Cause How does the synchronization functionality work? The ntpdate command is initiated once at startup. It connects to the NTP server(s) and sets system time according to the time value received from the first server that responds. The ntpd process is initiated once the time is set by ntpdate . It is the daemon that keeps the unit time properly synchronized. If ntpdate fails to synchronize, ntpd will not be started. ntpd does not update the time at regular intervals. The update intervals are based on certain calculations to determine when synchronization is required. Typically, this is once every 30 to 60 minutes.
www.allot.com
27
NetXplorer and NetEnforcer Troubleshooting Guide
ntpdate may not initiate at startup for the following reasons: The NetXplorer Server is rebooted at the same time the NetEnforcer is booting up. The NetEnforcer does not manage to synchronize with the NTP Server because: o The server is down. o There are communication issues. • •
Troubleshooting It is important to check the NetXplorer server first, then continue to the NetEnforcer if the problem has not been solved. NTP/NetXplorer Server Verify that the NTP service is running. By default, this runs on the NetXplorer server. If this is the case, run the following command: •
C:\Allot\ntp-server\ntpq -p
ntpq:read:Connection refused This error indicates that the NTP service is not running on the NX Server. To initiate the NTP service on the NetXplorer server, do the following: 1. Go to Services in Administrative Tools on the PC, and start the Network Time Protocol Service . 2. To verify that the service is running, run Task Manager and search for the process ntpd.exe. If this process is found, run the ntpq -p command, as described above. 3. Reboot the NetEnforcer to see if the synchronization will take place after reboot. NetEnforcer Ensure that the NTP service is running on the NetXplorer server before continuing. Verify that the NTP process is running on the NetEnforcer: ps –awx|grep ntp • •
89 ?
SL
0:00 /usr/sbin/ntpd -l /usr/local/SWG /logs/ntp.log
The above line shows that the NTP process is running. If the process is not found, initiate the NTP Daemon by rebooting the NetEnforcer. •
Verify that synchronization is against the NTP server IP (NX or ext. NTP server): AC-202:~# ntpq -p remote refid delay offset jitter st t when poll reach ============================================================================== LOCAL(0) LOCAL(0) 14 l 59 64 377 0.000 0.000 0.008 *10.4.70.1 LOCAL(1) 11 u 4 64 377 0.624 -2.455 0.291
•
Status 16 indicates failure to sync against NTP server. Verify that synchronization is against the NTP server, and not the internal (local) clock of the NetEnforcer. This is marked by an asterisk (*) at the beginning of the line with the NTP server.
Verify that the Windows firewall is not enabled on the server (this is enabled by default) which could block the NTP requests.
www.allot.com
28
NetXplorer and NetEnforcer Troubleshooting Guide
For more information, the NTP manuals may be found at http://ntp.isc.org/bin/view/Main/DocumentationIndex. A document describing NTP and NTP on the NetEnforcer in general (for version 5.x) can be found at KB item 4723.
P ro b l e m : G U I d o e s n o t s t a r t To solve this issue, go to control panel on the machine that cannot access the NetXplorer and choose Java . 1. On the General tab , under Temporary Internet Files , click on delete and then OK. 2. Open browser with NX server IP address ( http://NXServer-IP) and launch the application. Note: If this does not solve the problem, run javaws.exe from the Java 1.5 environment. This may typically be located at a location similar to: C:\Program Files\Java\jre1.5.0_06\bin . Delete anything shown on this screen (this will clear the cache).
reat ing a Snapshot NetXplorer o
o
\allot\bin contains a batch file called create_snapshot_logs.bat. This file takes all the relevant logs and prepares a snapshot file that can be sent via e-mail. Please note that this file can be large at times (approx. 9MB). The snapshot will be created under \allot\tmp\snapshot_.tar.gz
NetEnforcer The snapshot procedure is the same as in previous NetEnforcer versions. To generate a snapshot run snapshot .
Taking a Snapsho The Snapshot File is a file used to help Allot Customer Support in the troubleshooting process. The file itself is a zip file that contains files which provide Allot Customer Support with a precise picture of what was happening inside the NetEnforcer when a particular event occurred. These files include log files, policy definitions, system settings, etc. The Snapshot is an essential support tool that is vital in solving any support issues. There are two ways of taking the Snapshot: Manually and Automatically.
The Manual Snapshot The Snapshot can be run manually. If an Allot Customer Support Engineer requests you take a Snapshot of the box, it is best to run the Snapshot process manually. To run the Snapshot manually, simply login into the NetEnforcer as root, and from the command prompt, run the command snapshot. This will create a Snapshot file in the directory, /usr/local/SWG/snapshots/ . The Snapshot file is created with the name snapshot.date_time.tgz. Core Snapshot While taking a regular snapshot, core files (all files under /usr/local/SWG/logs/core) will also be included in it. In some cases the core files might be very big. In cases where the size of the
www.allot.com
29
NetXplorer and NetEnforcer Troubleshooting Guide
snapshot is more than 15M, the NetEnforcer will create an additional snapshot with core files only. Example: core.snapshot.07.05.02_09.27.00.tgz
The Automat ic
napshot
There may be some specific cases where Customer Support requests that you run an Automatic Snapshot. This process configures the Snapshot to run automatically every four hours. The snapshot files are deposited in the /usr/local/SWG/snapshots/ directory. To start the Automatic Snapshot, type snap_on_cron To stop the Automatic Snapshot, type snap_off_cron Prisma Snapshot An automated snapshot that is generated after DKM or Collector restarts. The Prisma Snapshot is a “short” version of the regular snapshot and contains only /proc/prisma directory and /usr/local/SWG/logs directory.
S e n di n g t h e S n a p s h o t Note: This script does exist in the box, but there is a bug. Do not use this script for now. Normally, this takes a snapshot but currently cannot send it. Current Snapshot A utility is included on the NetEnforcer for sending the Snapshot files directly to Allot Customer Support. The utility is called send_snapshot and the syntax is send_snapshot . This utility will automatically take a snapshot of the unit’s current state and log into the Allot Customer Support FTP Server. It will then open a directory (named with box number of the NetEnforcer) and send the snapshot. The file/s are copied into the opened directory. Saved Snapshot A snapshot which has been taken previously and saved may be sent using the syntax Send_snapshot_file(s) . For example, if you have a saved Snapshot file, snapshot.01.03.00_09.54.39.tgz and you would like to send it to the Customer Support for analysis; you would type the following command from the command prompt: send_snapshot snapshot.01.03.00_09.54.39.tgz
This will contact the Allot Customer Support FTP server, log in, create a numerical directory and copy in the snapshot file selected.
HTTP Snapshot Some NetEnforcer and NetXplorer units do not have access to FTP. Therefore, it is not possible to send a snapshot directly from the box. If the unit does not have a public address or Internet access, use this workaround: 1.
Create the snapshot by typing: snapshot The snapshot file is saved to the following directory: /usr/local/SWG/snapshots/
2.
Copy the snapshot file to the /usr/local/SWG/etc/httpd/htdocs directory: cp /usr/local/SWG/snapshots/snapshot.15.03.06_16.08.33.tgz
www.allot.com
30
NetXplorer and NetEnforcer Troubleshooting Guide
/usr/local/SWG/etc/httpd/htdocs (in this example, the file is named snapshot.15.03.06_16.08.33.tgz ). 3.
Point the browser to the NetEnforcer URL: http:///snapshot name For example: http://192.1.1.2/snapshot.15.03.06_16.08.33.tgz.
4.
This will start an HTTP download of the snapshot file to the PC. It is now possible to email this snapshot, or place it on an FTP server for access to Allot personnel.
Note: If an FTP Server is available, it is also possible to connect to the NetEnforcer using the FTP, browse to where the snapshot is located, and use the mget command to get the snapshot (using bin mode).
www.allot.com
31
NetXplorer and NetEnforcer Troubleshooting Guide
A d d De v i c WhenaddingadevicetotheNetXplorerNX730,thereare10 stagesthatneedtobecompleted. Therefore,whenaddingadeviceandgettinga"failedtocreatetopologydevice"error,itis importanttoknowonwhichstageitfailed.
Stage 1: configuration : create device topology Stage 2: event : create device event counter entry Stage 3: configuration : check device software version Stage 4: import configuration : set configuration from Device to DB Stage 5: catalog : export (deviceTopology) Stage 6: policy : export default policy (deviceTopology) Stage 7: register to snmp trap : register AS To Snmp Tables Listeners Stage 8: collector : assign device to collector Stage 9: configuration : set admin and oper to 1 - ON Stage 10: get the latest topology object Todothis,gototheNMS.log,locatedunderAllot_Home:\Allot\netxplorer\jboss- 3.2.6\server\allot\log andsearchfortheword"CREATE": 2006-04-01 01:44:13 [RMI TCP Connection(57)-122.122.4.101] INFO topology.dto.TopologyDTOManager - CREATE(1/9) [admin/122.122.4.32] create device topology to DB - started 2006-04-01 01:44:13 [RMI TCP Connection(57)-122.122.4.101] INFO topology.dto.TopologyDTOManager - CREATE(1/9) [admin / 122.122.4.32 #2] create device topology to DB - finished 2006-04-01 01:44:13 [RMI TCP Connection(57)-122.122.4.101] INFO topology.dto.TopologyDTOManager - CREATE(2/9) [admin / 122.122.4.32 #2] create device event counter entry - started 2006-04-01 01:44:13 [RMI TCP Connection(57)-122.122.4.101] INFO topology.dto.TopologyDTOManager - CREATE(2/9) [admin / 122.122.4.32 #2] create device event counter entry - finished
Thefirsttwostagesalmostalwayscompletesuccessfully. KeeptrackoftheCREATE(bysearching)untilthefailedstageisfound. Fail on stage 4 - set configuration from device to database Inthisstage,theserverreadsIPconfigurationfromrc.conf .Thefollowingindicationwillprobably befound: 2006-04-01 02:07:22 [RMI TCP Connection(171)-122.122.4.101] ERROR management.ejb.C onfigurationFacad eEJB - failed to setConfigurati onFromDeviceToDB null; CausedByException is: Device 122.122.4.101/161 is unreachable when trying to send pdu
Thisindicatesthattheprobecouldnotsendtheconfigurationupdatestotheserveronport161. Inthiscase,checkthefollowing: •
Runnetstat -anontheNetEnforcerorServerandcheckwhetheraconnectiononport161is established.
www.allot.com
32
NetXplorer and NetEnforcer Troubleshooting Guide • •
CheckthatnothingisblockingSNMPtrafficalongtheway. Checkthatthedatabaseisupandavailable.
Fail on stage 5 - exporting catalogs from the Server to the NetEnforcer In this stage, the Application Server connects to the Apache Server (using CGI on port 80) on the NetEnforcer using the following link: http://122.122.4.32:80/cgi-bin/ProvisionCli.exe. IntheNMS.logthefollowingwillbeseen: 2006-04-01 02:06:48 [RMI TCP Connection(169)-122.122.4.101] INFO topology.dto.TopologyDTOManager - CREATE(5/9) [admin / 122.122.4.32 #6] export Catalogs - started 2006-04-01 02:06:52 [RMI TCP Connection(169)-122.122.4.101] DEBUG catalog.synch.Sy nchUtils - send to device= is name=
http://122.122.4.32:80/cgi-bin/ProvisionCli.exe
Potentialproblems: •
Authenticationfailuremayalsoresultfromincorrectpassword.Anotherindicationtothat wouldappearin$SWGL/nedbg.DataSrv.log ontheNetEnforcer. Makesurethecorrectadminpasswordwasentered. Trytoresettheadminpassword. • •
•
Communicationexception: Indication: 2006-04-04 11:52:16 [RMI TCP Connection(52)-10.254.48.100] DEBUG catalogs.ejb.C atalogFacadeEJB - EXCEPTION = com.allot.nms.common.net.CommunicationException • •
•
•
Checkforaccesslists(ontheNetEnforcer,routers,firewalls,etc). Checkwithnetstat -anthataconnectionfromtheNetEnforcertotheServeronport80 wasestablished. TrytoconnecttheNetEnforcertoadifferentswitch(thishasworkedinthepast).
AccordingtotheTroubleshootingGuide.Pleasenotethattheseproblemshaveneverbeen encountered: CheckthatDataSrv andProvisionCli.exe arerunning. Checkin$SWGL/nedbg.DataSrv.log whetherDataSrv receivedthechanges(checkfor fullexport). /tmp/nedbg.ProvisionCli.log -checkwhethercontentwasreceivedfromtheApache Server(viewfullXMLcontent). /var/log/apache/access_log -checkwhetherApachereceivedchange(lookforPOSTto ProvisionCli.exe). • •
•
•
Fail on stage 6 - exporting default policy from the Server to the NetEnforcer Failingonstage6maybearesultduetolargecatalogsontheserverthatneedtobeaddedto theNetEnforcer.TheNetXplorerserverhasatimeoutof1minutetocompletetheaddprocess.If theprocesstakeslonger,itmayreachstep6beforestopping. Thereisnoworkaroundtosolvethisonsite.R&Dinvolvementisneededinordertoreducethe processingtimeontheNetEnforcertolessthanthe1minutelimitation. Fail on stage 7 - Register AS to SNMP Tables
www.allot.com
33
NetXplorer and NetEnforcer Troubleshooting Guide
Failingonstage7ismostlikelytohappenwhenaddingadevicewhilemanagementtrafficgoes throughthebox.TheNetEnforcerrebootsandtheadditionfails.Theworkaroundistoswitchthe NetEnforcertobypass,andthenaddthedevice. Stage8(assigndevicetocollector),98(setadminandoperto1ON),and10(returntopology object)mayfail iftheApplicationServercannotconnecttothedatabase.Theonlyworkaroundfor thisistostopandstarttheserviceandensurethatthe3databases:CFG,STCandLTCareup andrunning. Ifoneofthedatabasesarestuck,itmustberecreatedbefore thedevicecanbeaddedagain. Indicationsthatdatabasesareupandrunning: •
•
Inallot_cfg.log ,lookforthefollowing: 02/2611:59:14.RunningonWindowsXPBuild2600ServicePack2 I.02/2611:59:14.DatabaseserverstartedatSunFeb26200611:59 I.02/2611:59:14.TryingtostartSharedMemorylink... I.02/2611:59:14.SharedMemorylinkstartedsuccessfully I.02/2611:59:14.TryingtostartTCPIPlink... I.02/2611:59:14.Startingonport50000 I.02/2611:59:19.TCPIPlinkstartedsuccessfully I.02/2611:59:19.Nowacceptingrequests Inallot_stc.log andallot_ltc.log ,lookforEnable all events: I.04/0309:15:33.RunningonWindowsXPBuild2600ServicePack2 I.04/0309:15:37.DatabaseserverstartedatMonApr03200609:15 I.04/0309:15:37.TryingtostartSharedMemorylink... I.04/0309:15:37.SharedMemorylinkstartedsuccessfully I.04/0309:15:37.TryingtostartTCPIPlink... I.04/0309:15:37.Startingonport50001 I.04/0309:15:42.TCPIPlinkstartedsuccessfully I.04/0309:15:42.Nowacceptingrequests I.04/0309:16:08.Enable all events
hange IP Defined Behavior There are three locations where the IP of the NetEnforcer can be changed: The NetEnforcer itself, using the LCD, CLI or Admin menu The IP Properties tab within the Configuration Menu of the NetXplorer Server for a specific NetEnforcer The properties window of a specific NetEnforcer within the NetXplorer Server GUI Note: If the IP address cannot be changed for any reason, manually edit the IP address in the rc.conf file, located in the /etc/rc.d directory. • •
•
The NetEnforcer Changing the IP address via the NetEnforcer does not impact the NetXplorer Server. The purpose of this is to enable a user to change the IP address of the NetEnforcer and move it to another Server, without affecting the configuration properties of the NetEnforcer within the Server. This will therefore allow another NetEnforcer to be installed in place of this NetEnforcer (using the same model and version) while maintaining the original policy configuration.
www.allot.com
34
NetXplorer and NetEnforcer Troubleshooting Guide
An event will be sent to the NetXplorer Server indicating an IP change on the NetEnforcer. An alarm may be assigned to this event within the Event Types Configuration window. To complete an IP address change, the address will also need to be configured within the device properties within the NetXplorer Server.
IP Properties tab within the Configuration Menu Changing the IP address of the NetEnforcer within the IP Properties of the Configuration Menu will change the address of the device itself and the Properties of the device within the Network tree. Properties window of a specific NetEnforcer Changing the IP address of the NetEnforcer within the Device Properties menu (accessed by right clicking on the device within the Network tree and selecting Properties) does not change any IP definitions on the NetEnforcer. This change will point the NetXplorer Server to connect to the specified IP address. To effect a change on the actual IP address of the device, the address must be defined within either the Configuration Menu or on the device itself.
urrent Behavior Please note the differences below: Changing the device IP address via the configuration menu will update the device properties (topology tree). This process will take effect approximately 30 seconds after entered.
I n -B a n d /O u t o f B a n d D e f i n i t i o n s The NetXplorer and NetEnforcer do not support In Band IP configuration. Currently, the GUI displays both in and out of band. The in-band option is be grayed out. The option will remain since it is a feature that will be available in future versions. On the NetEnforcer itself, currently the CLI enables definition of an in-band address. The LCD does not have this option.
www.allot.com
35
NetXplorer and NetEnforcer Troubleshooting Guide
rovisionin g Changes Add Host Process 1. Server sends XML command to NetEnforcer. 2. NetEnforcer performs changes and updates counters. 3. NetEnforcer sends trap to Server. Troubleshooting •
•
Server: C:\Allot\netxplorer\jboss-3.2.6\server\allot\log\NMS.log - check whether changes have been sent. a. …send to device = location… b. XML changes. c. …result from device = location… o err explanation (development not complete). o ok. Note: Asynchronous messages may not be displayed together. NetEnforcer: $SWGL/nedbg.DataSrv.log - check whether DataSrv received changes. Identify receipt, change applied and confirmation. Example successful output, see Appendix I. $SWGD/data/allotProvision.xml – check counter ID and new catalog entry $SWGL/nedbg.AllSnmpAgent.log – check for trap sent. Example successful output, see Appendix II. • •
• •
•
o n f i g u r a t i o n Ch a n g e s Process 1. SNMP config changes sent. 2. SNMP config changes applied.
Troubleshooting 1. Check NMS.log on Server. 2. $SWGL/nedbg.AllSnmpAgent.log – check for SET command.
3. $SWGL/nedbg.swKeeper.log (system changes) – check for set_conf.
www.allot.com
36
NetXplorer and NetEnforcer Troubleshooting Guide
4. $SWGL/nedbg.DataSrv.log (application changes) – look for XML.
a t a b a s e s N o t Sy n c h r o n i z e d Symptoms Full export of database from Server.
Explanation This can occur due to manual XML changes CLI changes made when SNMP agent down NetEnforcer in rescue And others • • • •
Troubleshooting 1. $SWGL/nedbg.AllSnmpAgent.log – Check for PolModifyTag=3 (bad database) 1 = good 2. $SWGL/nedbg.DataSrv.log – Check for Full Export and complete XML •
T o G e n e r a t e a F u l l Ex p o r t Touch $SWGD/data/allotProvision.xml.
www.allot.com
37
NetXplorer and NetEnforcer Troubleshooting Guide
RM
/B o x R e p l a c e m e n t
Important note: If there is no unit to replace, do not delete the unit from the server until you have another unit to replace it. Unit A is connected to the server. Unit B should replace unit A.
1. 2. 3. 4. 5. 6.
Connect unit B and add it to the server with a different IP address than unit A. After unit B is reachable, disconnect both units (A and B) through the management port. Set unit B with the original IP address that was defined in unit A. Reconnect management port to unit B. Delete the IP address that was used to define unit B. Perform touch to allotProvision.xml.
o l l e c t i o n P ro b l e m s S T C P ro b l e m s R e l a t e d t o S o f t w a r e There may be problems with the STC database due to software running on the NetXplorer PC which may be interrupting the database processes.
Symptom The short term collector is stuck. no monitoring reports NX server reports event/alarm on STC_DEF • •
Troubleshooting The following message can be found in allot_stc.txt: E. E. I. I. I. I. I.
10/28 10/28 10/28 10/28 10/28 10/28 10/28
01:19:12. 01:19:12. 01:19:12. 01:19:12. 01:19:12. 01:19:12. 01:19:12.
*** ERROR *** Assertion failed: 100909 (9.0.2.3137) Error deleting transaction log file *** ERROR *** Assertion failed: 100909 (9.0.2.3137) Error deleting transaction log file Attempting to save dump file at 'C:\WINDOWS\TEMP\sa_dump.dmp' Dump file saved
Explanation The first error, assertion failed error 100909: Error deleting transaction log file is usually caused the transaction log is locked. This indicates that there is another software application currently using the transaction log, preventing the NetXplorer databases from accessing it. Since the NetXplorer databases cannot access the log, the database is shut down. Potential software applications that may lock up the transaction log are: system backup software anti-virus software defragmentation tools or others. • • • •
www.allot.com
38
NetXplorer and NetEnforcer Troubleshooting Guide
Workaround The identified application must be configured not to access specific Sybase files (.db and .log files). Go to http://www.sybase.com/detail?id=1025501 for information on ASA, anti-virus and backup software. It is highly recommended NOT to run such programs on folders where the databases reside. After disabling such programs, it may be necessary to recreate the database. For details on this procedure, see the Recreating Default (ST and LT) Databases section on page 22.
D a t a Co l l e c t i o n S t o p s D u e t o N T P I s s u e s Symptoms The following event/alarm is received on the NetXplorer Server: invalid bucket time on device (id 208 - Current bucket time is older that current UTC minus delta).
Troubleshooting See section on Troubleshooting NTP Issues on page 27.
em o Installation Issues When installing the NetXplorer for demo or training purposes there are a couple of tricks to avoid the installation requirements. Note: These tools are internal and should only be used in exceptionally specific internal situations. This information should NEVER be distributed to anyone outside of Allot CS.
I n s t a l l i n g N e t E n f o r c e r v e r s i o n 7 . 1 .0 o n a N e t E n f o r c e r A C 2 0 2 /3 0 2 This should only be done for training purposes. The NetXplorer only supports NetEnforcer models AC-40x (AC-80x, AC-10x0 and AC-25x0 in the future). To disable the check for the NetEnforcer model, create the file /tmp/nocheck.
kipping installation hardw are requirements From NetXplorer Version 25.27 (the current version is 23.25) it is possible to avoid hardware requirements such as memory and available ports. To disable the check, create the file c:\nocheck.
www.allot.com
39
NetXplorer and NetEnforcer Troubleshooting Guide
Appendix Appendix I H o s t o u t p u t f r o m $ S WG L /n e d b g . D a t a S r v . l o g 09-12 06:36:15(163) : Message received from AS: 53xXk0LYvZI= 168427883 4 create //catalogs/*/host/parent::* 167837953 09-12 06:36:15(163) : Create element, location: //catalogs/*/host/parent::* 09-12 06:36:15(163) : Created element: 167837953 09-12 06:36:15(163) : PmChangeValidator::buildValidNewHostEntry. Validating. 09-12 06:36:15(163) : Returned ID: 2 09-12 06:36:15(163) : Set catalogs update counter to 4 09-12 06:36:15(163) : Set update owner to 168427883 09-12 06:36:15(163) : touch file data/lastPolicyUpdate : 1126506975
www.allot.com
40
NetXplorer and NetEnforcer Troubleshooting Guide
09-12 06:36:15(163) : Sending notification to clients. 09-12 06:36:15(163) : Update counter 4, number of changed catalogs 1 09-12 06:36:15(163) : Changed catalog: type host_cat, name Host 09-12 06:36:15(163) : Deleted entries. 09-12 06:36:15(163) : Number of entries: 0 09-12 06:36:15(163) : New entries. 09-12 06:36:15(163) : Number of entries: 1 09-12 06:36:15(163) : QuadID: 2. 09-12 06:36:15(163) : Entry: 167837953 09-12 06:36:15(163) : Modified entries. 09-12 06:36:15(163) : Number of entries: 0 09-12 06:36:15(163) : Tracked entries. 09-12 06:36:15(163) : Number of entries: 0 09-12 06:36:15(163) : CatSvr notify (0x83d89a0) [0] 09-12 06:36:15(163) : Sending notification to clients. 09-12 06:36:15(163) : Update counter 4, number of changed catalogs 1 09-12 06:36:15(163) : Changed catalog: type host_cat, name Host 09-12 06:36:15(163) : Deleted entries. 09-12 06:36:15(163) : Number of entries: 0 09-12 06:36:15(163) : New entries. 09-12 06:36:15(163) : Number of entries: 1 09-12 06:36:15(163) : QuadID: 2. 09-12 06:36:15(163) : Entry: 167837953 09-12 06:36:15(163) 09-12 06:36:15(163) 09-12 06:36:15(163) 09-12 06:36:15(163) 09-12 06:36:15(163) 09-12 06:36:15(163)
www.allot.com
: Modified entries. : Number of entries: 0 : Tracked entries. : Number of entries: 0 : CatSvr notify (0x84b40d8) [0] : Message returned to AS:
41