ntop FAQ...

This is an unsophisticated, automated conversion of the source file, docs/FAQ into html. Please report problems to the ntop-dev mailing list. But remember, it's not about making it look good, it's about making the content available.

Features

Q. What determines the features of ntop?

A. Whatever Luca wants

Q. Why did you do this "x" instead of feature "y"?

A. Don't know. I could guess...

Imagine you are the network manager for a large University network and have to crack down on users who are illegally exchanging copyrighted files or using University resources to run a business without paying for the resources being consumed.

You are a major vendor of infrastructure, whose customers are using networks for new features such as storage area networks and you want to give them the ability to monitor these.

You have a really cool technology that you've just donated to the community via an RFC and wish to jump-start adoption of it.

You are moving heavily into IPv6 and need to be able to monitor your 'new' network just like you monitored the IPv4 one.

or (my favorite)

You really, really, really hate that ntop generates such lousy html code and you decide to scratch that itch.

or (what should happen)

A major corporation, with out resources, time, skills and/or inclination to do it themselves sponsors the development of a feature that's critical to him/her.

Then again, it could just be because it's cool...

Q. Could ntop do "x"

A. Probably - as long as it doesn't move the tool away from it's purpose and it's strengths, almost anything is possible - especially as a plugin.

Q. Will you do "x"

A. Maybe - if it's of interest to a developer, or you provide the code such that it can be merged in, or if you're willing to sponsor the development effort (contact us through http://www.ntop.org/consultancy.html).

Documentation

Q. Why isn't there (any)(more)(better) documentation.

A. (A personal peeve from Burton...)

I get real tired of people complaining that there isn't any documentation and then being unwilling to contribute even the simplest stuff. I've said I'll edit and assemble whatever people send me... and since I started working with ntop in November 2001, I've received maybe six pages of stuff.

I'm trying to get people - who aren't coders - to contribute to ntop the project. The contribution that ANYONE can make is "documentation". A task- specific HOWTO... some sample screen shots... An FAQ entry...

I've tried being nice. I've tried asking. I've tried shaming people into it.

What have I gotten? Zip.

Nasty is all that's left... This is your fair warning. If you show up on the ntop mailing lists and complain about documentation, you will get blasted.

-----Burton

Q. Ok, where can I find what does exist.

A. http://www.ntop.org has pointers and some (very out-dated) documents.

The documentation in the docs/ directory, the Documentation files at SourceForge and some at http://www.ntopsupport.com are basically all that there is.

Search the ntop mailing lists at gmane, http://search.gmane.org. The lists are called gmane.linux.ntop.general and gmane.linux.ntop.devel.

Please contribute to the ntop community by writing things up for inclusion in this FAQ or other documents!

Q. I can't find a file at SourceForge!

A. You can reach the archives through any SourceForge mirror, or the main site:

      http://prdownloads.sourceforge.net/n/nt/ntop

It has ALL the files unless we explicitly delete them...

Problems

Q. I have a problem...

A. Make sure it's a supported release!

We support only the current versions of ntop. This is either:

       * the last release, e.g. v3.1.
       * the current cvs (cvs.ntop.org)
       * the latest development version posted at SourceForge (if one)

If you use a port/package and the latest version available for your OS is some release candidate from a year ago, sorry. Contact the packager and ask them to get current.

Luca usually asks people to try the latest cvs (development) version because problems are frequently already fixed in there.

Q. I'm running something supported and I've tried the latest & greatest. Still, I have a problem.

A. Read the ntop mailing lists.

The ntop mailing lists are at

      http://listgateway.unipi.it/mailman/listinfo/ntop
      http://listgateway.unipi.it/mailman/listinfo/ntop-dev
   and
      http://listgateway.unipi.it/mailman/listinfo/ntop-misc

If you're having non-user problems OR you are using the cvs, you should be reading and posting ntop-dev (for example, the cvs commit messages are posted there). Stuff unrelated to baseline ntop (PF_RING) belongs in ntop-misc.

You can read the lists through gmane (or other gateways) if you don't want to subscribe, but only subscribers can post.

You can download the older messages in large chunks from the mailing list subscription pages. Look for "To see the collection of prior postings to the list".

Q. I looked and I didn't find my problem.

A. Join the mailing list(s) and ask for help.

Before you post, check the "HowTo Ask for Help" at the end of this FAQ.

Please, if at all possible, use the built in PR_ form (the little 'bug' icon on the 'About' tab).

Guidelines for asking questions:

ONE and only ONE problem / issue / question per message.

With a meaningful subject.

         The goal is that if you're asking a common question, the
         subject would have allowed you to find it in the back
         traffic for the mailing list.

Post the information about your environment we ask for.

         We STRONGLY suggest you use the automatically generated "Problem
         Report" form that since it contains much of the necessary information.

Make sure you're in a supported environment (./configure --showoses).

         If it's an unsupported environment, we're interested in your efforts to
         make ntop work, but we don't have the time, resources, knowledge and/or
         interest to do it ourselves.

For software 'crashes', please run ntop under the gdb debugger and capture the full failure information.

         Brief instructions on using gdb are at the end of this file (docs/FAQ).

Q. I posted to the list and nobody answered me.

A. ntop is open source, and the lists are a community resource. If nobody answered your question, then nobody knew the answers off-hand and nobody wanted to spend THEIR time solving YOUR problem.

Q. Do you offer paid support?

A. Yes - contact us through http://www.ntop.org/consultancy.html

Configuring ntop

Q. What does ./configure do?

A. ./configure checks for the tools installed on your system - configuring ntop to compile with the ones you have and skip the ones you don't (or to tell you if you're missing something critical).

Q. Why bother - just compile the code.

A. Then you would have to have a machine configured EXACTLY like Luca's. Nothing else would work. Various OSes and Linux distributions package the files in different ways and put them in different places. Plus some packages put files into directories with release information in them, etc.

Q. OK, so ./configure

A. Is how you tell ntop where to find things. A lot of stuff it can figure out on it's own, but if things get put in 'strange' places, ntop's ./configure has switches you use to tell where to find things.

Q. And the list is?

A. ./configure --help shows the whole list. It's a bit confusing because there are standard options and ntop options mixed in there.

A. So, first let's look at the 'where are things' options. There are two types of files ntop is looking for, '.h' files and libxxxx files.

.h files are also called 'includes' and libxxxx files are called libraries or lib files.

.h files are the C source for functions provided by the OS or by libraries. They are typically in a directory named /usr/include, but they can be placed ANYWHERE.

lib files are compiled libraries of these functions (the .h tells ntop how to call something, the lib file is the actual code). Their names usually begin libxxxx (so the library gd is named libgd).

By convention, libraries end in .so or .a. A .so library is a shared library (Windows DLL), where one copy of the library is used by all programs that want it's functions. A .a library is a non-shared or static library, which must be merged (the technical term is linked) with the code.

ntop uses both - the myrrd library is a static, .a library. When it comes to things like libpcap or libgd, we use shared (.so) libraries.

Library files are typically placed in /usr/lib, where the gnu linker (ld), 'knows' automatically how to find them. However, from OS to OS and distribution to distribution, there are many other common places. Some OSes even have a file telling ld all the places to look.

Q. So ntop looks for these .h and library thingies in a couple of places. What if it doesn't find them?

A. If a basic ./configure can't find something, you'll have to tell ntop where to look.

It's complex and OS/distro dependent. For example, if you install libgd from the Sun Freeware site on to a Solaris machine, the files get put into /usr/local/include and /usr/local/lib, which are not on the lists of 'standard' places for Solaris' versions of gcc (the Gnu c compiler) or ld. So to compile ntop, you have to tell gcc to look in these additional locations.

The things ntop might be looking for are in this part of the ./configure --help output:

       +-External-source-locations:-------------------------------------------------+
       --with-pcap-root=DIR        LBNL pcap located in DIR
       --with-pcap-lib=DIR          or libpcap located in DIR
       --with-pcap-include=DIR      or pcap.h located in DIR
       --with-gdbm-root=DIR        gdbm located in DIR
       --with-gdbm-lib=DIR          or libgdbm located in DIR
       --with-gdbm-include=DIR      or gdbm.h located in DIR
       --with-zlib-root=DIR        zlib located in DIR
       --with-zlib-lib=DIR          or libz located in DIR
       --with-zlib-include=DIR      or zlib.h located in DIR
       --with-gd-root=DIR          gd located in DIR
       --with-gd-lib=DIR            or libgd located in DIR
       --with-gd-include=DIR        or gd.h located in DIR
       --with-libpng-root=DIR      libpng located in DIR
       --with-libpng-lib=DIR        or libpng located in DIR
       --with-libpng-include=DIR    or png.h located in DIR
       --with-ossl-root=DIR        openSSL located in DIR
       --with-ossl-lib=DIR          or libssl located in DIR
       --with-ossl-include=DIR      or ssl.h located in DIR
       --with-localedir=DIR        LOCALE files located in DIR (i18n)
       +-----------------------------------------------------------------------+

You can see that there is a pattern, pretty much every --with-xxxxx-root has a --with-xxxxx-include and --with-xxxxx-lib option.

So, if ntop tells you it can't find something, do this - first look for the File on your system:

     $ locate pcap.h
     /usr/include/pcap/pcap.h

(If you don't have locate, this works too:

     $ find / -type f -name "pcap.h"
   )

And then tell ./configure via --with-pcap-include=/usr/include/pcap

(Some OSes are smart enough to look in a subdirectory of the standard location, but others aren't).

Q. Ok, but why three options?

A. You use either the --with-xxxxx-root option OR either/both of the others at a time. But ntop really only looks at the --with-xxxxx-include and --with-xxxxx-lib options.

Internally, --with-xxxxx-root=/a/b/c is translated into --with-xxxxx-lib=/a/b/c/lib and --with-xxxxx-include=/a/b/c/include (that's the usual pattern).

Now sometimes libraries are installed logically - if the pcap.h file is in /usr/local/pcap/include, the library (libpcap.so) is probably in /usr/local/pcap/lib. Sometimes they are not logical and you will have to use the split options.

The --with-pcap-root=/usr/local/pcap is shorthand for the two options, --with-pcap-include=/usr/local/pcap/include and --with-pcap-lib=/usr/local/pcap/lib.

Q. Oh Ghu - aren't there any short cuts.

A. For the first time you try ./configure, there's a script on SourceForge in The user contributed area that will try to build the ./configure line for you.

Q. And the OTHER options

A. There is a set that tells ntop where to install stuff. For simplicity, the two you might want to change are:

       --prefix=PREFIX         install architecture-independent files in PREFIX
                               [/usr/local]
       --datadir=DIR           read-only architecture-independent data
                               [PREFIX/share]

--prefix tells ntop where to install the various files. The default value is /usr/local, which is where most non-OS software normal goes.

A common choice for libraries (such as pcap) is --prefix=/usr, which puts things like .h files in places easier to automatically find (/usr/include). --prefix=/usr certainly works for ntop. --prefix=/opt is another choice.

The 'proper' choice turns on which model of file organization the OS and sysadmin prefer. That's a fight I'm staying out of.

--datadir tells ntop where to put its databases and output files. The default is /usr/share/ntop, but that will give some sysadmin's agita. Another popular choice is --datadir=/var, which puts all the files in /var/ntop. That may be attractive especially if you make /var/ntop a separate partition, so the rrd files don't eat all your disk space.

Q. What's --enable-iknowbetter Override WILLFAIL

A. There are some error messages where it's possible that thing work (now) that didn't used to, or you're doing development and don't want ntop to stop you from doing something, or there's an error message that you have skipped before without getting bitten.

--enable-iknowbetter will print the message but not stop ntop from finishing ./configure.

Use it at your own risk.

Q. What are the --enable and --disable options?

A. These (and the with/without options) pretty much do what you think - they enable or disable large chunks of ntop functionality:

       +--ntop-specific:------------------------------------------------------------+
       --enable-sslv3              enable ssl v3 support [default=disabled]
       --enable-sslwatchdog        enable Watchdog for ssl hangups [default=disabled]
       --disable-plugins           disable compilation of plugins [default=enabled]
       --enable-static-plugins     Enable static linked plugins sntop, default=dynamic]
       --enable-ignoresigpipe      Ignore SIGPIPE errors [default=do not ignore]
       --disable-snmp              Disable SNMP support [default=disable]
       --enable-i18n               Enable (limited) internationalization [default=disabled]
       --enable-jumbo-frames       Enable Jumbo (9K) Ethernet frames [default=disabled]
       --disable-ipv6              use IPv6 [default=enabled]
       +-----------------------------------------------------------------------+

       +--external-packages:---------------------------------------------------+
       --without-ssl               disable HTPPS support [default=enabled]
       --without-zlib              disable zlib [default=enabled]
       --with-tcpwrap              enable use of TCP Wrapper [default=disabled]
       +-----------------------------------------------------------------------+

Q. What ELSE?

A. There are some so called 'environment variables' you can set that change things too. The common ones are:

        CC          C compiler command
        CFLAGS      C compiler flags
        LDFLAGS     linker flags, e.g. -L if you have libraries in a
                    nonstandard directory 
        CPPFLAGS    C/C++ preprocessor flags, e.g. -I if you have
                    headers in a nonstandard directory 
        CPP         C preprocessor

You would use these variables to override the choices made by ./configure or to help it to find libraries and programs with truly nonstandard names/locations.

The best place to look for examples of these environment variables are in the OS/distribution files in the configureextra directory. For example, (again picking on Solaris), we use LDFLAGS to tell ld to look in some common Solaris locations for libraries via this:

     LDFLAGS="-L/usr/local/lib -R/usr/local/lib ${LDFLAGS}"

What that does is add /usr/local/lib to the locations that ld will check.

Q. ./configure dies in some strange horrible way complaining about version conflicts, lunar phase or whatever.

A. The process of creating portable cross-platform scripts for building software is ugly and hard and prone to failure. There are tools (released by gnu) to help, named automake, autoconf and libtool. (Collectively the auto* tools).

These tools take basic files and generate more complex files based on a series of rules, conventions and macros (much of which is poorly or undocumented). Other processes (e.g. ./configure) take these files and generate more complex files based on a different series of rules, conventions and macros. This ultimately produces the 'Makefile', which a program called make uses - based on (...wait for it...) yet another series of rules, conventions and macros - to actually compile the ntop source.

There are interdependencies among the tools, partial support for older versions in some releases, but not in other releases and many more problems.

Different OSes and different Linux distros ship with wildly different versions. Some even have scripts that attempt to analyze the file and pick the correct version (which just means that trying to code a file with multiple version support confuses the tool).

So if you have a basic, total failure of ./configure, it's usually the auto* tools.

ntop used to ship with a manual script that rebuilt the generated files according to the version(s) of the tools installed on the system you were using to build ntop. Thus the standard answer was 'run ./autogen.sh -1'.

But, this meant that you had to have these 'developer' tools installed and caused much problems and gnashing of teeth.

So we spent 100s of hours rebuilding the scripts to be totally independent of having the tools installed on your system - only to run into problems because the xyz 1.6.1 tool installed by default on OS A isn't quite compatible with the 1.6.3 version on OS B.

So we put technology in place to automatically detect tool versions and rebuild the generated files if necessary. That meant you had to have these 'developer' tools installed and caused much problems.

So we rebuilt the scripts AGAIN and AGAIN, dropped support for old versions of the tools and finally reached a point where it works for most reasonably current platforms. This is a compromise:

       Systems that don't have the tools installed usually work.
       Systems that have bleeding edge versions of these tools may break.
       Systems with very old versions also may break.

The technology to detect versions and rebuild the script files if appropriate is still in there, but it's disabled from normal use.

If you have the auto* tools installed and have ./configure problems, you can activate the automatic rebuild feature via:

      $ export NTOPAUTOREBUILD=yes
      $ ./configure ...

If the rebuild fixes it, that's great. Regardless, please report the problem to the ntop mailing list. Please don't paraphrase the messages cut & paste the ACTUAL MESSAGES into your report. Also, please let us know the version(s) of the tools installed on your system:

      $ aclocal --version
      $ autoheader --version
      $ autoconf --version
      $ automake --version
      $ libtool --version

Q. I can't build ntop... (auto* tools)

A. ntop has been tested with and developed for various versions of the "GNU auto* tools" (automake, autoconf and libtool).

The basic requirements are

automake 1.6.1 or higher (automake 1.4 and 1.5 may or may not work) autoconf 2.51 or higher (autoconf 2.13 will NOT work) libtool 1.4 or higher

But that simple statement hides a lot of complexity and many potential problems. Surprisingly, this isn't solely ntop's fault, it's the combination of GNU auto* tools version to version compatiblity and because we're trying to distribute configure and make files that work on all versions. If ntop were less complex or less clever about trying to build a working program even with many components missing, things would work a lot better.

The big change between autoconf 2.13 and 2.5x generated scripts is that the ./configure and make steps are supposed to run the auto* commands automatically if they're required. As we've seen, this doesn't always work!

If you run into problems, you can ALWAYS recreate the generated files via this procedure:

      rm -f acinclude.m4 aclocal.m4 Makefile.in config.h.in configure Makefile

      find current versions of libtool, config.guess and config.sub and cp
      them into your working directory.

      cat acinclude.m4.ntop libtool.m4.in > acinclude.m4
      aclocal
      autoheader
      autoconf
      automake --gnu --copy --add-missing

and then:

      ./configure ...
      make
      make install

as usual.

Q. But how does it all hang together?

A. There's a vsd (Visio)/pdf in the docs directory - ntop-autotools.pdf.

Compiling ntop

Q. Which packages/libraries do I need to compile ntop:

    Note: In some cases the minimal header files for a tool will be in
          one "package" and the execution library in another. ntop needs
          both so that the ./configure test finds the tool. It's usually
          safest to install both the tool and development packages!

         (Some packages will have additional packages as pre-requisites)

       gdbm (We have people using 1.7.3, 1.8.0, 1.8.2 and 1.8.3)

       libpcap
          We recommend 0.8.3, but ntop works with most 0.7.2 and later versions.

          STAY AWAY from older versions, especially under Linux.

          Note: Building libpcap requires: bison/flex

       gd 
          http://www.boutell.com/gd/gd.html - note that many distros have both
          the 1.8.x and 2.x series installed for differnt packages that require
          them.  This occasionally causes problems.

       libpng (1.2.x - although ntop works with 1.0.x versions, the two libpng
              versions are not compatible and will often cause version conflicts
              and crashes).

       glibc

cpp

       gcc most versions of gcc should work, but there are no promisses.

       libtool  1.4+ (distribution is built with 1.4.2)
                     (there are successes reported with 1.3.4 or 1.3.5 -
                      use --enable-iknowbetter)

       gawk/mawk
          Note: If all you have is original awk, then ./configure will not work.

Q. Will the "xyz" compiler work instead of gcc?

A. We explicitly REQUIRE gcc.

We know that under many systems, a compiler called cc is available. On some, it's a symbolic link to gcc, BUT, when invoked as cc, it often triggers 'old' behaviors for cc compatibility. On others, cc is a retarded compiler just good enough for compiling the kernel.

The one exception is Sun's cc, which - since Luca used to do a lot of development on Solaris - was pretty well supported.

Q. The auto* tools?

A. Only if you are going to rebuild the distributed script files:

           autoconf 2.5+ (distribution with 2.57)
           automake 1.6+ (distribution with 1.8.3)

Q. Optional libraries?

A. Lots. The most common is:

       openssl (for https:// support)
          Note: For security reasons you should only have the most current
                Version of openssl installed.

Q. ntop doesn't compile.

A. Sure, it does - for me :-)

Q. Right smarty-pants...

A. First, check if you're using a supported OS.

The long and short of it is that ntop works under Linux, Mac OS X and FreeBSD.

Luca distributes a Win32 version through http://shop.ntop.org but charges a convenience fee. Or you can fire up Microsoft's Visual C++ or Visual C++ .Net and compile the Win32 version that way.

It should work under MinGW - see the BUILD file in docs.

Anything else, you are in untested waters. Some cases we know there are problems, others we just haven't tested.

Q. Ok, it's a supported environment and it still won't compile.

A. Did you run ./configure? And did it complete successfully?

Usually 'compile' problems for supported platforms are a missing (critical) library or header file, but the user ignored (didn't see) the error/warning message and tried running make anyway.

./configure checks a lot of things. When it's looking for headers and libraries, ntop will report KEY information and PROBLEMS in large, set-off, lines:

     *******************************************************************
     *
     * NOTE: Building ntop for a supported platform
     *       This means we expect ntop to work without major issues
     *
     *            'i686-pc-linux-gnu'
     *
     *    Please keep the ntop-dev mailing list updated with any
     *    successes you have or problems you encounter...
     *
     *   Support for this platform was most recently verified for
     *
     *     RedHat7.2 w/ updates           ntop 2.1.51   on 2002-10-21
     *     Suse i686, 2.4.18-4GB-SMP      ntop 2.1.51   on 2002-10-24
     *
     *******************************************************************

READ THESE BOXES. Even if you don't read the rest of the output, read the boxes. ntop can work around a lot of problems (missing libraries) by disabling features that need them. If, for example, you don't have zlib, ntop will compile a version that doesn't output compressed html pages. If you don't read the boxes, you will never know.

READ THE MESSAGE BOXES!

     *******************************************************************
     *                                                                 *
     * NOTICE:  I know you're used to ignoring output from ./configure *
     *                                                                 *
     *          ntop has a lot of complexity and interdependences.     *
     *                                                                 *
     *          Please, please AT LEAST read the stuff in these        *
     *          boxes!                                                 *
     *                                                                 *
     *>>> The ACTION taken is shown prefixed '>>>'                     *
     *                                                                 *
     *    If the ACTION is unacceptable,                               *
     *??? The REMEDIATION plan is shown prefixed with '???'            *
     *                                                                 *
     *******************************************************************

The box will tell you what's wrong, what ntop did and often how to fix it if you don't like ntop's fix.

READ THE MESSAGE BOXES!

Hint: It may sometimes be that you're missing the header files (often those are in a -devel rpm if you're running RedHat)

If you see a message box and don't understand why ("I'm sure that header file is present"), then look at a file called config.log. Search for the specific header or library reported in the message box and you will see the detailed compiler/linker error messages.

For example, ./configure reports:

        checking for linux/if_pppox.h... no

The first thing to so is check if it's on your system:

     $ locate linux/if_pppox.h
     /usr/include/linux/if_pppox.h

(If you don't have locate, this works too:

     $ find / -type f -name "ethertype.h"
   )

Open up config.log and look for if_pppox.h:

     configure:13086: checking for linux/if_pppox.h
     configure:13103: gcc -c -g -O2 -Wshadow -Wpointer-arith
      -Wmissing-prototypes -Wmissing-declarations -Wnested-externs  -fPIC
      -DLINUX conftest.c >&5
     configure:13158: parse error before '/' token
     In file included from configure:13160:
     /usr/include/linux/if_pppox.h:38: `ETH_ALEN' undeclared here (not in a
         function)
     /usr/include/linux/if_pppox.h:39: `IFNAMSIZ' undeclared here (not in a
         function)
     /usr/include/linux/if_pppox.h:40: confused by earlier errors, bailing out
     configure:13106: $? = 1
     configure: failed program was:
     | #line 13091 "configure"
     | /* confdefs.h.  */
     |
     | #define PACKAGE_NAME "ntop"
     | #define PACKAGE_TARNAME "ntop"
     | #define PACKAGE_VERSION "2.2.91"
     | #define PACKAGE_STRING "ntop 2.2.91"
     | #define PACKAGE_BUGREPORT ""
     | #define PACKAGE "ntop"
     | #define VERSION "2.2.91"
     | #define STDC_HEADERS 1
     ...
     | #define HAVE_NETINET_UDP_H 1
     | /* end confdefs.h.  */
     | net/if.h netinet/if_ether.h
     |
     | #include 
     configure:13123: result: no

You see the actual test program, the actual compile line and the error messages.

Before reporting it to us, chase down where those missing items are declared:

     $ grep -R 'ETH_ALEN' /usr/include/* | grep '#define'
     /usr/include/linux/if_ether.h:#define ETH_ALEN  6
     /usr/include/net/ethernet.h:#define     ETHER_ADDR_LEN  ETH_ALEN

And post that information with your error report. The reason is that these field definitions are often placed in very different places in different OSes and even in different distributions.

FWIW, if you look in the older configure.in:

      AC_CHECK_HEADERS([linux/if_pppox.h], [], [], [net/if.h
         netinet/if_ether.h])

should have been

      AC_CHECK_HEADERS([linux/if_pppox.h], [], [], [#include 
      #include ])

because the macro doesn't do the #include automatically.

Q. Nope, it's not ./configure...

A. If it's not the configuration, then it's usually a problem with your specific system, either:

        - A new release of a supported OS.
        - An uncommon option/configuration of a supported OS.

In other words, something is different from what we've seen or expected.

Review the output from make. The error message will usually give you a Somewhat cryptic description of what's wrong and where. Look for messages about missing files. Post as much information as you can - do locates for the missing files, etc. The more you give us the less we will have to ask you to provide.

Remember, we can't see your box - all that the people on the list see is the information you give in your message.

Q. Compile dies because it's missing depcomp

A. automake/autoconf issue. The problem should have been fixed. If not, just Copy the missing file (or make a symbolic link) into the ntop source directory.

It's in /usr/share/automake on my Linux boxes. Another user reports it is in /usr/local/share/automake in sun8.

If you have automake installed, this will do it automatically:

       $ automake --add-missing --gnu -c

Q. Make fails with a message about being unable to create a .deps file.

A. Check the permissions on the (hidden) .deps (and .libs) directories - if root owns them your non-root userid may not be able to create files in there.

Q. Make fails with a message like this:

     /bin/ld: Warning: size of symbol `pcap_open_dead' changed from 100 to 67
         in pcap.o
     collect2: ld returned 1 exit status
     make[2]: *** [libntop.la] Error 1
     make[2]: Leaving directory `/linuxadmin/ntop/ntop2.2.3/ntop-2.2.3'
     make[1]: *** [all-recursive] Error 1
     make[1]: Leaving directory `/linuxadmin/ntop/ntop2.2.3/ntop-2.2.3'
     make: *** [all] Error 2

A. Ah yes, size of symbol...changed. It means you have a conflict between the version of the shared library (be it libpcap, libgd, whatever) that was used to compile ntop with the version that was used to link ntop.

Q. 'splain please...

A. If you break down a typical link line:

      gcc -shared  address.lo ... ntop_darwin.lo
          -L/usr/local/include
          -L/usr/local/lib
          -L/linuxadmin/ntop/ntop2.2.3/ntop-2.2.3/myrrd
          -lpthread -lresolv -lnsl -lz -lc -lssl
          -lcrypto -lgd -lpng -lpcap /usr/lib/libgdbm.so
          -lmyrrd
          -Wl,-soname -Wl,libntop-2.2.3.so
          -o .libs/libntop-2.2.3.so

You see the mix of -L and -l parameters. The -L parameters ADD additional places to look for the shared libraries, which are in addition to the 'standard locations' for the system. The -l parameters tell which libraries to include.

Read the man pages (man ld, man ld.so, etc.)

The 'standard locations' are very system dependent, but usually include /usr/lib and /lib. PLUS whatever is (under Linux) in the ld.so.conf file, for example

      $ cat /etc/ld.so.conf
      /usr/kerberos/lib
      /usr/X11R6/lib
      /usr/lib/qt-3.0.5/lib
      /usr/local/lib

So, on this system, to resolve a library, ld looks in the -L values:

      1. /usr/local/include
      2. /usr/local/lib
      3. /linuxadmin/ntop/ntop2.2.3/ntop-2.2.3/myrrd

      And then the 'standard' places:

      4. /usr/kerberos/lib
      5. /usr/X11R6/lib
      6. /usr/lib/qt-3.0.5/lib
      7. /usr/local/lib
      8. /lib
      9. /usr/lib

Similarly, if you break apart the gcc COMPILE line and scrap the dups, you'll have a different set of places where gcc looks for the .h files. For those, it's the -I parameters plus whatever is 'standard' on your system, which is dependent on the specific gcc port:

/bin/sh ./libtool --mode=compile gcc -DHAVE_CONFIG_H -DLINUX

       -I. -I/linuxadmin/ntop/ntop2.2.3/ntop-2.2.3/myrrd 
       -I/usr/local/include
      -g -O2 -fPIC -g -O2 -c
      -Wshadow ... -Wnested-externs
       -o ntop_darwin.lo

Again, read the gcc man page.

A big problem is that - unlike .so files, it's not real clear what directories ARE searched for .h files, only that there is a set.

With me so far? Go back to the message... what it means is that:

      * From one set of locations, at compile time, the size of 
        the parameter list for pcap_open_dead was 100 bytes.
      * From the other set of locations, at link (ld) time, the
        library expects the parameter list to be of size 67.

Danger, Will Robinson...

The most likely cause of this problem is when you tell ntop to look at one location to resolve either the .h or .so and it finds the other, 'automatically' in a 'standard' location, but the two are actually from incompatible versions.

Say you have libpcap 0.6.2 installed by the os in /usr/lib and /usr/include, but you also install libpcap 0.7.2 in /usr/local/lib and /usr/local/include.

      $ locate pcap.h
      /usr/include/pcap.h
      /usr/local/include/pcap.h

and

      $ locate libpcap.so
      /usr/lib/libpcap.so.0
      /usr/lib/libpcap.so
      /usr/lib/libpcap.so.0.6.2
      /usr/lib/libpcap.so.0.6
      /usr/local/lib/libpcap.so.0
      /usr/local/lib/libpcap.so
      /usr/local/lib/libpcap.so.0.7.2
      /usr/local/lib/libpcap.so.0.7

If all you give ./configure is --with-pcap-header=/usr/local/lib

      * At compile time, it finds the pcap.h in /usr/local/lib (0.7.2)
      * At link time, it finds the libpcap.so in the 'default', 
        /usr/lib/libpcap.so which is actually libpcap.so.0.6.2

ntop did EXACTLY what you told it to do. The fact that it makes no sense is a problem, so you get the error message.

Q. So the real solution is?

A. Give ntop pointers to consistent sets of header and library files, and maybe don't have multiple versions of the same library installed at once.

Q. I'm done compiling and it works/doesn't work, what do I do?

A. make install (You'll typically need to be root for this to work)

Q. I'm seeing a lot of "changing search order for system directory" messages?

A. It's not a problem, just confusing. This warning, which appears if you redefine the system include directory, causes lots of confusion. Supposedly, according to this http://gcc.gnu.org/ml/gcc-bugs/2002-10/msg01080.html, it wasn't fixed in gcc 3.2.0, but should be fixed in gcc 3.2.1.

According to this: http://gcc.gnu.org/onlinedocs/cpp/Invocation.html, it's truly harmless on systems which already have /usr/local/include as a standard system library:

      -I dir 
      Add the directory dir to the list of directories to be searched for
      header files. See Search Path. Directories named by -I are searched
      before the standard system include directories. If the directory dir
      is a standard system include directory, the option is ignored to ensure
      that the default search order for system directories and the special
      treatment of system headers are not defeated (see System Headers) .

Other ./configure features

Q. How do I update the Vendor Table (MAC address prefixes)?

A. ntop has (in Makefile), a rule to automatically download the latest vendor information table from the IEEE, the oui.txt file ntop reads.

If you are seeing unknown MAC address prefixes (the 1st three units), try the full IEEE table. To rebuild it:

     # make dnvt

and then copy the new oui.txt over the one installed by ntop originally.

Also note that the table changes over time - there are almost 600 Modifications and/or new assignments between the version shipped with ntop 2.0 and the version on the IEEE site in February 2002.

Q. How do I update the passive ethernet fingerprint database?

A. ntop has (in Makefile), a rule to automatically download the latest ettercap file from SourceForge:

     # make dnetter

It will also compress the file and tell you how big the old and new files were.

Q. How do I update the IP to Country Code table?

A. Again, ntop has (in Makefile), a rule to automatically download the latest data and build the file.

     # make p2c

Note that this is actually a fairly complex script (utils/p2c) and is dependent upon data posted to supposedly well known locations by the various internet number authorities (APNIC, RIPE, etc.)

It's a LOT of data to download so don't do this on a site with a slow link to the internet. Also, read the output carefully and DO NOT apply a file if there are messages you don't understand.

Q. But the IP -> CC data is wrong.

A. Yeah. Most people don't care :-(... it's enough to see various pretty flags.

Sadly, the registries often post files with errors in them. Some Tier 1 ISPs post their own data to elaborate on the registry data. Most ISPs don't post data. Some entries in the files make 'logical' sense, but not physical sense.

If you find another set of data, let us know - the shell script is pretty easy to follow to add another data source.

The user who used to care about this (Mr. Anon E. Mouse) used to send me a file of corrections to apply to the file we posted with ntop.

If such a file exists, it would be posted @ SourceForge in the user contributed area.

Q. I have installed ntop for OSX but I cannot start it as I have installed some libraries using fink and ntop cannot find them.

A. The easiest thing to do is to do: ln -s /sw/lib/myfinklibrary /usr/local/lib Courtesy of Matthew Scholz

Running - Startup

Q. ntop won't start - I get this message:

  ** FATAL ERROR** ... open of /var/ntop/addressQueue.db failed:
       File open error

A. It's either permissions (the ntop -u userid doesn't have read/write access to that file - common if you're upgrading from 2.2 to 3.0).

The other cause is that there's still an instance of ntop running. Make sure you shutdown ntop before starting it.

Q. What is the function of the 'ntop' script in the build directory - should I call it or /usr/local/bin/ntop ?

A. (from the comments in the script):

    # ntop - temporary wrapper script for .libs/ntop
    # Generated by ltmain.sh - GNU libtool 1.4 (1.920 2001/04/24 23:26:18)
    #
    # The ntop program cannot be directly executed until all the libtool
    # libraries that it depends on are installed.
    #
    # This wrapper script should never be moved out of the build directory.
    # If it is, it will not operate correctly.

It allows you to run ntop out of the build directory before doing a "make install" by doing all the necessary linkage magic - such as forcing a relink if it didn't succeed originally - to the files in .libs.

Think of it as simulating make install, but not moving stuff to /usr/local or wherever.

Don't use it - it just causes problems...

Q. Which libraries do I need?

A. To run ntop: glibc, gdbm, libpcap

      For https://, add openssl.
      For other tools and compile options, add the appropriate libraries.

Q. ntop seems to run, but the web server isn't up.

A. Set the password - see docs/1STRUN.TXT

Q. How do you reset Admin password if we lost it?

A. Delete ntop_pw.db and follow the procedure in docs/1STRUN.txt

Q. I'm being prompted to set a password, what do I enter.

A. Anything you want, of 5 or more characters. If ntop can't find the value in it's database, it will prompt you to set on. If this happens every time you start ntop, check the permissions on the ntop_pw.db file.

Q. Characters after the 8th are being ignored in my password.

A. Probably. ntop uses the standard crypt() function provided by the OS.

While crypt() is required part of the single unix specification, the details are implementation dependent. Whatever the limitations of crypt() are, that what ntop's limits are.

Unfortunately, there's no way to tell programatically what these limits are, you have to refer to the man page for crypt(3).

There is a "GNU EXTENSION" to crypt() which implies that changing the 'salt' ntop uses would allow longer passwords. That's set via CONST_CRYPT_SALT in globals-defines.h, but AFAIK this hasn't been tested.

Q. How does the @filename option work e.g. /usr/bin/ntop ... @filename ...

A. The text of 'filename', is copied - ignoring line breaks and comments (anything following a #) - into the command line.

ntop behaves as if all of the text had simply been typed directly on the command line. Multiple @s are permitted in the command line, nesting is not. @s in the file will cause an error.

Both are displayed on the info.html report, the "Started as" shows the actual command line ntop was given and the "Resolved to" shows what ntop processed.

Started as /usr/bin/ntop -i eth0,eth1 @/root/ntop_parms -d -L

with /root/ntop_parms containing:

      -p /usr/share/ntop/protocol.list
      -P /usr/share/ntop
      -w 192.168.42.38:3000
      # -W 192.168.42.38:3001
      -u ntop
      --trace-level 3
      -m 12.239.0.0/16,10.113.0.0/16
      -E
      -K

becomes:

Resolved to /usr/bin/ntop -i eth0,eth1 -p /usr/share/ntop/protocol.list

               -P /usr/share/ntop -w 192.168.42.38 -u ntop --trace-level 3
               -m 12.239.0.0/16,10.113.0.0/16 -E -K -d -L

Remember, most ntop options are "sticky", that is they just set an internal flag. Invoking them multiple times doesn't change ntop's behavior. However, options that set a value, such as --trace-level, will use the LAST value given: --trace-level 2 --trace-level 3 will run as --trace-level 3.

It is recommended that you use FULL pathnames for @filename, since ntop may have different effective directories when run in different ways. However, you may wish to use relative pathnames to take advantage of the different effective directories (say cron vs. command line). Just know where you're starting.

Q. ntop seems to run but I don't see any traffic.

A. Make sure you aren't running against the loopback (127.0.0.1) interface. lo shouldn't see much traffic, only that originating on the host destined for it (e.g. ping 127.0.0.1).

Q. ntop is unable to open its database file. Specifically: I have following messages while running ntop

     wait please: ntop is coming up...
     24/Jul/2003 15:15:23 Initializing IP services...
     
     24/Jul/2003 15:15:23 Initializing GDBM...
     24/Jul/2003 15:15:23  ***FATAL_ERROR*** open of ...name... failed: can't be writer
     24/Jul/2003 15:15:23 Possible solution: please use '-P '

A. Multiple possible choices...

* Another ntop already running

(Most common) You forgot the -P parameter, and ntop is using a default value that doesn't exist. Just as the message says, use the -P parameter to point ntop at the directory you want it to use.

* Some sort of file system problem (non-existent directory, permissions, etc.)

1. The directory shown doesn't exist. Create it.

2. You many not have read/write rights in that directory. This can occur if you run ntop both as root and (as recommended) as a non-root user.

3. Another instance of ntop may already be running, so it has the file open and locked.

Q. What's inside the .db files and what's their format?

A. The files are stored in GDBM format. You can dump their content using tools like this: http://tboudet.free.fr/dumpgdbm/. In principle you should not be interested in the file content as they are temporary caches and contain the configuration as well as the MD5 hash of the web passwords.

I posted a trivial dumpgdbm.c to gmane - search for it.

Q. ntop starts up with this: WARNING: Discarded network 172.20.0.0/16: this is the local network.

A. No worries. The message means exactly what it says - it's a warning that you gave the local network as one of the parameter(s) to -m. Since the local networks are always local, ntop doesn't need to make them pseudo-local.

Q. What are ntop's options?

A. There are a couple of options that appear only if they're not compiled in, and a few that depend on various external libraries, e.g. openSSL.

The best way to see what is actually available is to run ntop with the -h or --help options and see. Also read man ntop.

Q. -4 and -6?

A. The default for the ntop web server is to connect to any address and any family of addresses, so if the NIC has both IPv4 and IPv6 addresses it should respond to both. Use these to restrict the web server.

Q. --trace-level?

A. The settings 0..4 control the amount of information logged or displayed. DETAIL (5) adds a [MSGIDnnnn] field and (VERYNOISY) 6 or greater adds the file:line where the message originated.

-t 5 can be useful with a log watch type package, -t 6 is mostly for debugging.

-t 5:

      [MSGID8757584] OSFP: scanFingerprintLoop() checked 1, resolved 1

-t 6:

      [MSGID8757584] [ntop:698] OSFP: scanFingerprintLoop() checked 1, resolved 1

With 3.2 we've added VERYNOISY and (7) BEYONDNOISY ... they do pretty much what they say. Have LOTS of log space available.

Q. ntop starts up find and then seems to die.

A. Are the ntop threads still there? Use ps to check, for example: # ps -C ntop -m -o "uid,pid,ppid,stat,time,command"

     UID   PID  PPID STAT     TIME COMMAND
     501 18327     1 S    00:00:00 /usr/bin/ntop -i eth1,eth2 @/etc/ntop.conf...
     501 18330 18327 S    00:00:00 /usr/bin/ntop -i eth1,eth2 @/etc/ntop.conf...
     501 18331 18330 S    00:00:05 /usr/bin/ntop -i eth1,eth2 @/etc/ntop.conf...
     501 18332 18330 S    00:00:45 /usr/bin/ntop -i eth1,eth2 @/etc/ntop.conf...
     501 18333 18330 S    00:00:00 /usr/bin/ntop -i eth1,eth2 @/etc/ntop.conf...
     501 18334 18330 S    00:00:06 /usr/bin/ntop -i eth1,eth2 @/etc/ntop.conf...
     501 18335 18330 S    00:00:00 /usr/bin/ntop -i eth1,eth2 @/etc/ntop.conf...
     501 18336 18330 R    00:00:32 /usr/bin/ntop -i eth1,eth2 @/etc/ntop.conf...
     501 18337 18330 S    00:00:41 /usr/bin/ntop -i eth1,eth2 @/etc/ntop.conf...

The 'PID' numbers match the THREADMGMT: numbers in the log:

      ntop[18326]:   INIT: Parent process is exiting (this is normal)
           ^^^^^ is the thread that issued the message

      ntop[18327]:   INIT: Bye bye: I'm becoming a daemon...
               (18327 is the 'master' thread created as ntop becomes a daemon)

      ntop[18327]:   THREADMGMT[t105061296]: SFP: Started thread for fingerprinting 
               A message is issued for each thread started.
                                t^^^^^^^^^ is the 'internal' thread #. Ignore it
      ntop[18331]: THREADMGMT[t105061296]: SFP: Fingerprint scan thread starting [18331]
                                                                                  p^^^^^ 
               is the child pid.  Each one issues a message as it starts up

So:

     UID   PID  PPID STAT     TIME is
     501 18327     1 S    00:00:00 Master
     501 18330 18327 S    00:00:00 (internal used by POSIX)
     501 18331 18330 S    00:00:05 Packet processor
     501 18332 18330 S    00:00:45 Idle Host Scan
     501 18333 18330 S    00:00:00 Address Resolution
     501 18334 18330 S    00:00:06 rrd
     501 18335 18330 S    00:00:00 Web Server
     501 18336 18330 R    00:00:32 eth1
     501 18337 18330 S    00:00:41 eth2

If you connect to the running ntop via gdb, you can see this:

$ gdb /usr/bin/ntop 18327 (gdb) info threads

         ^^^^^^^^^^^^ Shows all the threads and their current state

     9 Thread 114696 (LWP 18337)  0x40255c68 in recvfrom () from
            /lib/i686/libpthread.so.0
     8 Thread 98311 (LWP 18336)  0x40255c68 in recvfrom () from
            /lib/i686/libpthread.so.0
     7 Thread 81926 (LWP 18335)  0x420dcc31 in select () from
            /lib/i686/libc.so.6
     6 Thread 65541 (LWP 18334)  0x420b0226 in nanosleep () from
            /lib/i686/libc.so.6
     5 Thread 49156 (LWP 18333)  0x40252a35 in __pthread_sigsuspend ()
            from /lib/i686/libpthread.so.0
     4 Thread 32771 (LWP 18332)  0x420b0226 in nanosleep () from
            /lib/i686/libc.so.6
     3 Thread 16386 (LWP 18331)  0x40252a35 in __pthread_sigsuspend ()
            from /lib/i686/libpthread.so.0
     2 Thread 32769 (LWP 18330)  0x420db1a7 in poll () from
            /lib/i686/libc.so.6
     1 Thread 16384 (LWP 18327)  0x420b0226 in nanosleep () from
            /lib/i686/libc.so.6

(gdb) thread 2

         ^^^^^^^^ switch to thread 2 and see it's state

[Switching to thread 2 (Thread 32769 (LWP 18330))]#0 0x420db1a7 in

            poll () from /lib/i686/libc.so.6

(gdb) info stack

         ^^^^^^^^^^ check the call stack

#0 0x420db1a7 in poll () from /lib/i686/libc.so.6 #1 0x4024f9de in __pthread_manager () from /lib/i686/libpthread.so.0

                     ^^^^^^^^^^^^^^^^^ running __pthread_manager

Q. All the threads seem to be running ok.

A. Check if there's one that looks like this:

     4 Thread 32771 (LWP 27295)  0x420cd207 in sched_yield () from
            /lib/i686/libc.so.6

Let ntop run a few more seconds (cont command), then check again.

If it's frozen in the sched_yield, you're probably tripping a deadlock situation that we've seen but don't understand - There's a lot of stuff on the 'net, including what seems to be the same problem with other resource intensive tools but no clear answers.

This happens most often on RedHat Linux 8.0 (9 uses a different threading library and the issue hasn't been reported there).

Regardless, if you see the deadlock or just have a hung ntop, there is a run time option to try: --disable-schedyield. This option disables the calls at a slight penalty to ntop's interactivness. If you're experiencing the deadlocks, try it.

And, especially if you're running something other than RedHat 8, please let the ntop-dev list know about it.

Q. Can I set the admin password from a script?

A. Yes, you can call ntop with the option:

      ntop --set-admin-password=password

Q. What's this warning about ownership?

A. The first time you run make install and ntop creates a new directory for ntop's files, there's a problem. So you'll see this warning box:

      ************************************************************
      ************************************************************

        WARNING: This install created a directory for the ntop
                 files and databases:

                   some directory...

                 This directory MUST be owned by the user
                 which you are going to use to run ntop.

                 The command you must issue is something like:

                 chown -R ntop.ntop 
           or    chown -R ntop:users

                 man chown to check the syntax for YOUR system

      ************************************************************
      ************************************************************

Since root created the directory, root is the owner. And since we don't know - yet - what user ntop is going to be run as, we can't issue the chown (CHange OWNership) command for you.

Q. And if I don't do this?

A. ntop won't run. Typically you get a message about a bad -P file or endless prompts for the admin password.

Q. I can't merge interfaces (-M option)?

A. Check your plugins and see if either netFlow or sFlow is active. Regardless of whether you're using them, if they're active, they (silently) force the -M switch on.

If you have used the web configuration to set this, you may be unintentionally overriding your command line.

With v3.2 there are log messages telling you what sets interface merge.

        NOTE: Interface merge disabled by default
        NOTE: Interface merge enabled by default
        NOTE: Interface merge disabled from prefs file
        NOTE: Interface merge enabled from prefs file
        NOTE: Interface merge disabled due to command line switch

Q. Explain -u.

A. -u root means that you are running ntop as root. It means you don't drop root's superuser privledges, so pretty much you can read/write any file. It's not a good idea. While we're not aware of any security problems with ntop, programs that run as root are targets.

-u ntop (or -u whatever) means you run as a normal user and can be assigned only the privledges necessary to run ntop. It also means you can only read/ write files as ntop. So less is exposed.

For even better security, the -u userid should not have a valid shell, so people can't use it to login.

Q. What are the options that reduce ntop's workload?

A. These options turn off certain processing (and thus limit ntop's functionality), and may be appropriate for high volume installations.

-b | --disable-decoders

     This flag disables protocol decoders. Use it for better performance
     or if you feel ntop has problems handling these protocols in your
     environment.

     This switch disables code in a number of places throughout ntop, code
     which analyzes specific protocols, but can place additional load on the
     host.  This switch could be used to run ntop on low-end CPUs or where
     ntop is acting as a collector (netFlow or sFlow) and the GUI is not
     required.

     Disabled is the analysis of:

        DNS Sniffing - where ntop captures DNS information from other hosts'
                       requests to reduce the # of DNS requests ntop must -
                       itself - make.

        NetBIOS   \
        NetWare    \
        AppleTalk   -- resource intensive protocol analysis of less
        bootp/dhcp /   common protocols.
        OSI       /

        http (80) - Request success/failure counting on port 80 and other
                    analysis, including "Virtual Host".

        ftp passive session tracking.

        "Wrong Port" monitoring for: http, ftp and smtp (used with the
              -q | --create-suspicious-packets option to dump "suspicious"
              packets to an analysis file)  With this option, ntop checks
              the payload for each new connection, looking for text usually
              present in http, ftp or smtp requests.  If these are not on the
              "normal" ports (http's 80, ntop's 3000 or squid's 3128, ftp's
              21 or smtp's 25) (or there is a non-ftp or smtp request on the
              standard ports), the packet is logged.

-g | --track-local-hosts

     Use this flag to tell ntop that you care only about local hosts (use
     -m | -- local-subnets to specify local nets).  This flag is useful when
     ntop sees many hosts (e.g. border gateway) but only the local ones need
     to be tracked.

     This switch disables code in a number of places throughout ntop, code
     which allows ntop to track "foreign" hosts (that is ones not local
     according to the IP address(es) of ntop's interfaces or set pseudo-local
     by -m | -- local-subnets).

     Basically, ntop doesn't bother to do DNS resolution on these addresses
     and, for purposes of various counts, uses the "other" bucket instead of
     creating a unique hash table bucket for the specific host.

     This switch could be used to run ntop on low-end CPUs or where ntop is
     acting as a collector (netFlow or sFlow) and the GUI is not required.

-o | --no-mac

     Specifies that ntop should not trust MAC addresses but just IP addresses.
     This option is useful whenever ntop is started on an interface where MAC
     (Media Access Controller - the low-level Ethernet address) addresses can
     not really be trusted (e.g. port/VLAN mirror in Switched Ethernet
     environments).

     Certain processing is performed differently:

          Hash search is via IP not MAC

     Certain capabilities are disabled:

          Analysis of bootp/dhcp requests
          localRoutersList.html report
          Wrong net mask log message and flag
          Analysis of non-tcp/udp protocols like NetWare and Spanning Tree
          Router listing on Host Detailed report.
          Traffic Matrix report

     (Note that this list is subject to change as we learn more about protocols
      that do/do not depend on the MAC address)

     See also -z | --disable-sessions

-z | --disable-sessions

     This flag disables tcp session tracking. Use it for better performance or
     when you don't really need the tracking of sessions.

     Also, in situations where the MAC addresses cannot be trusted, ntop may
     - or may not - be able to accurately track tcp sessions.  There is no easy
     way to tell, so this switch puts control back into the users' hands.

     In versions after 2.0 up to & including 2.1.2, the -j | 
     --border-sniffer-mode flag (predecessor of -o | --no-mac) always turned
     this off.  Many users wanted to try turning session tracking back on, and
     did via code patches with mixed results.

     Suggested usage:  If you enable -o | --no-mac, try running ntop with
     sessions enabled.  If the data looks reasonable, congratulations - your
     network allows session tracking.  If the data does not look reasonable,
     then you will also need to disable session tracking with this switch.

Q. -s | --no-promiscuous doesn't work

A. It should work - it's passed to pcap_open_live.

But, whether the flag is supported by the OS and whether it is actually respected by the interface depends on libpcap. If it fails, you'll see a message and ntop will refuse to startup.

Understand that if it actually does work, it means that ntop sees a lot less comprehensive view of the traffic.

But from the ntop web server, you won't see anything different.

You can check (*nix) by executing ifconfig on the interface.

Note that the parameter specifies if the interface is to be put into promiscuous mode because of ntop.

Even if this parameter is false, the interface could well be in promiscuous mode for some other reason. And in some situations, the interface may be forced by the low level driver into promiscuous mode, without reporting this to the kernel!

Q. How does -m | --local-subnets work?

A. This flag allows users to specify the subnets whose traffic is considered local (called "pseudoLocal" internally).

The format is

        /<# subnet mask bits>[,/<#  subnet 
         mask bits>

For instance "131.114.21.0/24,10.0.0.0/255.0.0.0".

Q. (followup) but what does it MEAN?

A. Surprisingly, it means EXACTLY what it says. Treat traffic on the listed subnet(s) as local.

-m relates to the traffic you see on the wire at the TCP/IP level. -m tells ntop something it can't determine by itself. And that is to treat a range of addresses EXACTLY like it was local.

For example, on my Cable Modem, I see broadcasts for a number of subnets that AT&T (now Comcast) has assigned to this area (I don't see the traffic just the broadcasts).

If you have VLANs or simply network overlays (two or more networks on the same wire, but with separate address spaces). etc.

Those are the cases where you use -m. To tell ntop to treat that traffic as local.

Q. Why

A  ntop differentiates between local traffic and remote traffic.  There are
   actually four classes (although only three are routinely reported) L->L L->R
   R->L and R->R.  Some additional processing is done and some additional 
   reporting is available for L traffic.

Q. I'm confused. Explain, please!

A. Suppose your IP is 1.2.3.4 with a 255.255.255.0 netmask (a/k/a 1.2.3.4/24)

Under the TCP/IP protocol, traffic with any address 1.2.3.1 -> 1.2.3.254 does not get routed. It's "local".

Your buddy is at 1.2.3.9 and the router is 1.2.3.1, so your network looks like this:

the +--------+ world-----+ Router +--1.2.3.1-------------------------------------- +dog +--------+ | 1.2.3.4 | 1.2.3.9

                                  +--------+            +--------+
                                  |  You   |            |  Buddy |
                                  +--------+            +--------+

Say you send a packet to your buddy at 1.2.3.9. You build a packet with SRC=1.2.3.4 DST=1.2.3.9 and your data and cast it out the wire. (For purposes of this illustration, ignore the fact the your TCP stack would recognize the "local" nature of the packet and actually use another, lower level protocol, called Ethernet to deliver it.)

The router (1.2.3.1) looks at it, does the math and ignores it - it's local Your buddy (1.2.3.9) looks at it, says - gee, that's me and reads it

This is L->L traffic.

Now you send a packet to ntop.org at 131.114.21.9. Again, SRC=1.2.3.4 and Now DST=131.114.21.9.

The router (1.2.3.1) looks at it, does the math and says - oops, I have to send it out to the world. Your buddy (1.2.3.9) looks at it, says - gee that's NOT me and ignores it

This is L->R traffic.

Now it's perfectly possible to have multiple (physical) networks on the same physical wire. Say that your ISP chooses to put 1.2.4.1-1.2.4.254 (1.2.4.0/24) on the same wire. (Why would they do this? - maybe it's a big pipe and only a few users or whatever).

A packet from 1.2.4.4 -> 1.2.4.9 is seen by

The router - no, that too is local, ignore it You (1.2.3.4): (1.2.4.9) - not me - ignore it Buddy (1.2.3.9) - um... 1.2.4.9 - not me - ignore it

And that's perfectly legal.

But what if you are the ISP and you want ntop to see ALL the traffic on that wire? ntop will figure out from it's own IP address that the 1.2.3.0/24 traffic is local, but it will classify the 1.2.4.0/24 as REMOTE.

And that is what the --local-subnets switch does. It tells ntop to treat that 1.2.4.0/24 traffic as local.

If there isn't any other traffic on the wire, then telling ntop to treat it as local won't change a thing.

You can always use a packet sniffer, such as tcpdump to scan the traffic on the wire and see what's really there...

Q. And internally?

A. ntop is designed as a hybrid packet analyzer, not a pure Ethernet analyzer (layer 2) nor a pure TCP/IP analyzer (layer 3). Most of ntop's displayed counts are at the TCP/IP level, and that's what confuses people. Internally, ntop works both at the level of the Ethernet frame and the TCP/IP packet.

A single MAC address can be associated with multiple TCP/IP addresses. The MAC address -- unless something is horribly wrong on the network or with the hardware or somebody is deliberately spoofing it -- is guaranteed to be unique and refers to a physical host or network interface. For many reports, ntop displays the information using the MAC address to separate physical devices. Other data is accumulated and displayed at the TCP/IP (level 3) layer.

-m relates to the traffic you see on the wire at the TCP/IP level. -m tells ntop something it can't determine by itself. And that is to treat that range of addresses EXACTLY like it was local.

For example, on my Cable Modem, I see broadcasts for a number of subnets that AT&T has assigned to this area (I don't see the traffic, but you get the picture) in an overlay structure (two or more networks on the same wire, but with separate address spaces).

Q. What about multicast traffic?

A. Multicast traffic uses the 224.0.0.0/4 address range (that is all addresses between 224.0.0.0 and 239.255.255.255 - see RFC 1112). By default then, all multicast traffic is treated as 'Remote' by ntop.

Your actual network topology may be such that multicast traffic could all be local, all be remote or a mixture. You can use the -m | --local-subnets parameter to force some (or all) multicast groups to be counted as 'Local' traffic.

Q. What about VLANs?

A. VLAN traffic is also a special case. ntop treats all traffic within your machine's netmask definition as 'Local'. For instance, you may have a /16 network (netmask of 255.255.0.0), and you are using VLANs to distribute the traffic. In this case ntop will see the a.b.c.d/16 address and classify all traffic within this /16 network as 'Local'.

Don't design your network that way.

Q. But they did!

A. This MIGHT work:

Use an un-numbered interface, so that ntop doesn't assign any implicit 'Local' addresses. Then use the -m | --local-subnets parameter to define what is truly 'Local'. As long as there aren't too many 'Local' networks, this should work. I run something sort of like this (but using /24s not subdividing a /16) for my development network.

Q. What are the default protocols ntop monitors?

A. (These are the ones ntop monitors if the user does not supply a -p parameter) Check addDefaultProtocols() in ntop.c around line 525. The current list (December 2004) is

     Protocol   Ports
     --------   -----

     FTP        ftp ftp-data
     HTTP       http www https 3128      /* 3128 is HTTP cache */
     DNS        name domain
     Telnet     telnet login
     NBios-IP   netbios-ns netbios-dgm netbios-ssn
     Mail       pop-2 pop-3 pop3 kpop smtp imap imap2
     DHCP/BOOTP 67-68
     SNMP       snmp snmp-trap
     NNTP       nntp
     NFS/AFS    mount pcnfs bwnfs nfsd nfsd-status 7000-7009
     X11        6000-6010
     SSH        22
     Gnutella   6346 6347 6348
     Morpheus   1214
     WinMX      6699 7730
     DirectConnect 
     eDonkey    4661-4665
     BitTorrent 6881-6999 6969
     Messenger  1863 5000 5001 5190-5193

Note that the names come from /etc/services (or your system's equivalent). If you add protocols to /etc/services, you can refer to them by name on the -p parameter.

REMEMBER: You must define the list using the format illustrated in the ntop man page. Don't try to read /etc/services. It will fail.

The list changes over time as P2P protocols appear and disappear. Check the cvs and diff ntop.c (around line 550 in void addDefaultProtocols() if you want the history.

Q. What about protocol XYZZY?

A. The analysis of protocols is very limited and unsophisticated. But, theoretically, if it's there in plain text, we could report on it. The more work you can do up front in identifying the protocol (e.g. port #s, header structure, etc.), the easier it would be to add.

Q. I am using a /16 (/25 or whatever) mask and I get this message:

      Truncated network size to 1024 hosts (real netmask 255.255.255.0)

A. Yes. ntop limits each network to 1024 hosts (a /24). If you need more, alter the #define for MAX_SUBNET_HOSTS in globals-defines.h and recompile. Space has to be reserved for this many hosts for each network, so the limit exists to keep memory usage from growing to absurd levels on people with "class A" (/8) interfaces (e.g. 10. or Cable Modems, etc.).

Q. What's with the version check?

Q. Why is ntop connecting to jake.ntop.org?

A. jake.ntop.org is the cannonical name for version.ntop.org, which hosts the xml file used for the version check.

See the man page, 1st run log messages and/or privacyNotice.html page (Click on the "this" in this line: "For information on ntop and information privacy, see this page." on the 1st page ntop presents).

Running - On-going

Q. Explain L and R - why doesn't ntop double count?

A. Classification is all based on what ntop SEES in the packets. ntop sees packets and ONLY packets. Packets have a FROM and a TO address. Which packets ntop sees is determined by the interfaces it is monitoring.

Traffic (packets) is classified based on the joint classification of the FROM address (L or R) and the TO address (L or R).

Only in L->L traffic will ntop see sent=rcvd and 'double count'.

                            Host: 192.168.1.x  www.yahoo.com
                                  L->R  R->L   L->R  R->L
                                  S  R  S  R   S  R  S  R
      192.168.1.x>www.yahoo.com
        HTTP GET ...             30  .  .  .   .  .  . 30

      www.yahoo.com>192.168.1.x
        HTTP 200                  .  .  .  8   8  .  .  .

      www.yahoo.com>192.168.1.x   .  .  .200 200  .  .  .
        ...

      etc.

Q. Why does ntop use so much memory ?

A. ntop holds a lot of information about each host it has seen in an in-memory table. Periodically, it looks at all the entries in the table and flushes any which have been idle for a period of time.

You can change the sizing of the table and the flushing interval via #define statements in globals-defines.h.

But realistically, ntop needs enough memory to hold information about what's active on YOUR network.

To reduce memory, monitor fewer protocols or use the filter (-B "bpf filter") option to monitor only parts of the network.

There have been a couple of discussions on the ntop mailing list about ntop's memory usage - you might read them (search on gmane).

Q. Database.

A. No way. This data just can't be stored in a database - while each record (row) is small, we're talking 1000s of updates/second. As in Enterprise Oracle class speeds...

A. ntop's memory usage for host tables depends on the # of hosts it sees in packet traffic. This is NOT, repeat NOT controllable by ntop in ANY way. If a user kicks off a port scan, 100s of hosts appear. If somebody does a DOS attack against you, 1000s of hosts 'appear'. If a user searches Kazza for an obscure song, it can probe 4K hosts. etc.

Lots of those hosts appear, have a few bytes of traffic and then disappear. Each host has a variable amount of memory - there's a base structure, some optional counter structures and a large # of pointer fields, which may or may not be valued for any given host. It depends on the # of active sessions and a lot of other things, but 8-20K is a good guess - I usually use 12K as an guestimating size. Similarly, sessions may appear and disappear (http: opens a lot, does a small retrieval and closes them), ssh may last for days. etc. Memory is consumed tracking them too.

It's not just one table entry per host - there are a lot of ancillary tables which only get allocated if you have data for them.

Q. Got a guess?

A. Sure. Look in textinfo.html and you will see the line:

     (very) Approximate memory per host.....4.4KB

I'm not sure how much I trust this and I wrote it. This calculation just computes the (current usage - baseline) / # of hosts.

Q. So ntop's memory usage is dependent upon?

Q. So what does the purge do?

A. ntop purges idle hosts. Period.

Idle being defined as having not had packet traffic for a #define able period, 5 minutes by default.

Because of the amount of linked data, these purges take time (lots of free() calls on all those char* values), so we don't purge in 'real time'. First we build a list of idle hosts, then we purge from that list. Any idle host is eligible for purge (unless you tell ntop not to purge idle hosts, which is usable only on small networks).

Over time, all idle hosts are purged. Only to be perturbed by the next burst of activity - say a port scan or everybody logging in back in after lunch. Eventually, there's some form of steady state, but it's HIGHLY dependent upon network activity. Which, remember, is external to ntop and can't be predicted.

Q. Why?

A. Think about the overhead of sorting this huge structure by 'last packet seen time'. 1GB of ram is something like 80K+ hosts and takes a long time to sort (let alone free). During which time, on that busy network, a couple of dozen packets are processed... Meaning maybe your list of idle hosts is wrong.

Q. So how about a hard memory limit?

A. There's no way to make a hard limit without purging ACTIVE hosts (or non-IDLE given ntop's definition of idle).

Think about it ... you're at that 1GB limit and you find a new host. Do you kick off an interim purge (with it's huge overhead?) And wait 2 or 3 s for an available slot?? While 1000s of other packets appear??

Q. Soft limit?

A. It might, might, be possible as a soft limit - but it's got a lot of issues.

First off, tracking memory usage of the hosts tables is itself a huge job. There are multiple places were stuff is purged, added to the structures as pointers, etc. and there's a queue of purged entries for reuse to cut down on the malloc() calls.

Secondly, the purge is resource intensive, and has been the cause of deadlocks before - you don't dare lock the structures for too long - packets keep arriving, and FAST on the busy network that has the memory issues in the first place. Since you can't lock for long, you can only purge a small # of entries.

Q. But what about -x and -X.

A. Ah, yes, grasshopper - you have been reading the man page. Good user. The -x and -X options are pretty crude. They say "if you have allocated n (hosts|sessions), fail the allocation of (host|session) n+1".

It works in the context of preventing memory exhaustion - if you can calculate the right values based on your available memory (and remember that 4K or 12K or whatever per host value is an AVERAGE).

And, no, you can't make them memory dependent - it's all ultimately virtual and ntop doesn't get indication of whether a malloc() call causes swapping. It either works because the allocation (total) is under your limit (or the hard limit of real memory + swap space - OS usage), or it fails.

This type of limit can't make sure you save only the RIGHT hosts.

Q. These options aren't very smart, are they?

A. No. They don't know anything about the hosts. Just the # of them. So failing an allocation might hurt in that the n+1th host might be an important one and yet host #2 is basically trivial junk (the printer that sends and "I'm here" message every three minutes).

It also causes overhead, because the next packet for host n+1 will attempt to allocate a record and fail. Again and again.

But these options may keep ntop from crashing and that's a good thing.

If you use them, I would recommend setting them very high - just at the limit of what your memory permits. Use this as a last resort - say you get hit by a worm or virus.

I recommend that you use other options such as filter and track-local-hosts for the day-to-day controls on what ntop tracks.

Q. Why does ntop drop packets?

A. We used to have an extensive discussion here. But with Luca's merge of ntop-ht into baseline, we shouldn't be dropping packets beyond whatever the OS is dropping.

Until we've learned enought about ntop 3.2 and following, the whole discussion has been moved to FAQarchive.

Q. ntop stops capturing packets, except ARP and other broadcasts. Why?

A. Check if you have a daemon running that periodically checks for and resets interfaces in promiscuous mode? If that happens, all you would see were broadcast packets like ARPs...

Check back in the log and see if there is a message about the interface changing status. Determine why.

Q. How much horsepower do I need to run ntop on a network of size x?

A. Nobody really knows. ntop needs enough memory to store the active hosts and enough cpu to keep up with the average packet flow. The buffer will handle the occasional peak, but if you see frequent lost packets, you're in trouble.

Note that a few packets occasionally lost isn't a big deal for most users. After all, the network itself has losses - I've seen my AT&T Broadband connection have spurts of 30% packet loss. Ideally in a LAN environment, the packet loss should be down in the small #s... the Ethernet standard allows 1 error in 100,000,000(10^8), but most vendors beat that by a long margin (even as high as 1 in 10^12).

Of course, those are lab measurements. In the real world? Not that good. Electrical noise can be a real bugaboo. Remember, at a certain point, if the NIC doesn't understand what it's seeing, it throws it away and declares an error. The key is to keep up with the traffic.

Similarly, the OS kernel does the same thing in it's interrupt handling (throw away packets). Last resort, but better than hanging up the whole machine.

ntop drops packets when the queue gets longer than the permitted length. You can see this in the configuration page as # Queued Pkts to Process and # Max Queued Pkts.

One or two or a small number (you pick your tolerance) is ok, but constant losses isn't. What I'm saying is that as long as ntop can keep up with the NIC, then the data is as good as it gets... if ntop can't keep up, then the data isn't very good.

If you have measurements - network size, traffic flow and %CPU used (with the hardware info, of course), shoot them over to us on ntop and someday maybe we'll be able to give better #s.

Q. What about my Frobozz Model xx Magic Network card? Is it good enough?

A. Probably. Well, a lot of the cheapos just don't have the buffering and cpu offload of a 3c905c or such. If the network isn't that busy, anything will do. For a busy network, buy a decent PCI NIC.

Q. Gigabit Ethernet?

A. No clue. Remember that you can suck a lot more traffic over that network than an old PC can handle (i.e. the bandwidth limitations of 32bit PCI and PC100/133 RAM, heck even PC2700 DDR).

Q. Do "zero copy" drivers help?

A. Yeah. Maybe. Once upon a time, I read about "zero copy" - look here http://people.freebsd.org/~ken/zero_copy/ for the FreeBSD stuff. Quoting:

      "What is "zero copy"?

      Zero copy is a misnomer, or an accurate description, depending on how
      you look at things.

      In the normal case, with network I/O, buffers are copied from the user
      process into the kernel on the send side, and from the kernel into 
      the user process on the receiving side.

      That is the copy that is being eliminated in this case. The DMA or 
      copy from the kernel into the NIC, or from the NIC into the kernel is 
      not the copy that is being eliminated. In fact you can't eliminate 
      that copy without taking packet processing out of the kernel altogether.
      (i.e. the kernel has to see the packet headers in order to determine 
      what to do with the payload)

      Memory copies from userland into the kernel are one of the 
      largest bottlenecks in network performance on a BSD system, so 
      eliminating them can greatly increase network throughput, and 
      decrease system load when CPU or memory bandwidth isn't the 
      limiting factor."

What's that mean? It means that BEST CASE, you have to copy the data NIC->Kernel and without zero copy, it happens TWICE. Then, once you're in the kernel, it has to hand the data off to libpcap (another copy) and from libpcap to ntop. So we're moving the data 3 or 4 times... best case!

Let's do some off the cuff math. Looking here

      http://www6.tomshardware.com/mainboard/20020501/ddr400vsrambus-05.html
   shows a table of memory types and max bandwidth (picking a few):

      Label  Name  Effective Clock  Data Bus Bandwidth
      PC100  SDRAM       100 MHz     64 Bit  0,8 GB/s
      PC133  SDRAM       133 MHz     64 Bit  1,06 GBb/s
      PC2700 DDR333      166 MHz     64 Bit  2,7 GB/s

See the problem? A fully loaded GigE link is 1 Gb/s. 4x that is 0.5 GB/s - so if ALL that's going on is the capture of packets from the network, the CPU can keep up. Maybe. Those bandwidth numbers are theoretical, best case (nice sequential access). Throw in some other system activity, cache misses and CAS/RAS issues... and um...

The moral is that if you're going to use ntop to monitor big fat links, you need screaming fast iron.

Q. Can I disable logging? Totally?

A. No.

Q. I'm seeing weird "hosts" on my network with names like "Bridge Sp. Tree/OSI Route". What are they?

A. There is a list of "special" MAC address prefixes in vendor.c, specialMacInfo[]. There are blocks of MAC addresses reserved (sometimes not formally) for special uses, such as sharing information about Spanning Tree for bridges. These do not have an IP address - they operate at a lower level - so nothing gets displayed in some of ntop's fields.

A reference about protocols at the wire level is here:

      http://www.oreillynet.com/pub/a/network/2001/03/02/net_2nd_lang.html

If you only want to see TCP/IP, then I suggest you use -B "ip" to filter only TCP/IP protocol on your ntop line...

Q. How do I see fully qualified names for all my hosts? Some are netbios names!

A. ntop doesn't SEND NetBIOS queries, it sniffs them off the traffic already on the network.

There is only ONE case where ntop uses the NetBIOS names, which is if it can't resolve them via DNS (both it's own queries and from sniffing responses to other's queries off the network).

So, if you have a properly functioning DNS, you'll see DNS names. If these are (for example) internal names, unknown to the DNS server, you'll see NetBIOS names if they are available. Lastly, you'll get IP addresses...

If you do have a DNS, and the name is resolved as part of the default domain, you won't see a fully qualified name back from the DNS, so ntop won't have that information.

So, on a real network you'll often get a mix of name resolution types:

    Host                            IP Address      MAC Address
                                                                   Other Name(s)
    netnews.attbi.com               63.240.76.16
    tigger.homeportal.2wire.net     192.168.0.xx   00:D0:09:xx:xx:xx
    homeportal.homeportal.2wire.net 192.168.0.1    00:D0:9E:xx:xx:xx
    swallowtail                     192.168.0.XX   00:A0:CC:xx:xx:xx SWTL [DMN]
    12-xxx-xxx-xxx.client.attbi.com 12.xxx.xxx.xxx 00:D0:9E:xx:xx:xx
    12-xxx-xxx-yyy.client.attbi.com 12.xxx.xxx.yyy

Q. What does this log message (and others like it) mean? **WARNING** releaseMutex() call with an UN-LOCKED mutex [hash.c:559] last

               unlock [pid 22176, pbuf.c:2598]

A. Those messages are part of an error check in our mutex handling routines.

If things are working properly, then these messages tell us things are really, really, really wrong.

releaseMutex() call with an UN-LOCKED mutex means that the corresponding accessMutex() call never occured.

releaseMutex() call with an UN-INITIALIZED mutex means that the access/release occured before the createMutex().

The file/line comments, e.g. [hash:559], tell you where the access/release was called from (e.g. hash.c @ 559) and where the last recorded release was.

There are others, look in leaks.c for the entire set.

Q. How serious are they?

A. None of these stop ntop from processing, but they're indications of unprotected accesses to shared data areas, which could lead to lost counts.

In the development cycle after 3.1 was released, I was able to track down and fix the remaining cause of spurious messages (but I've thought this was fixed before).

For most users, these warning messages shouldn't be ignored.

The root of the problem is that POSIX mutexes are stupid and don't record a lot of state information. To debug things like deadlock, it's really, really, nice to know where the mutex was locked from.

So ntop has added 'extra' information to the mutex data for recording this. This added data is not protected by the mutex and could get out of sync. So we lock a state change mutex whenever we are changing the state data.

BUT: You can't hold the state change mutex while waiting for the main mutex or you deadlock.

So the 'access' call becomes this pattern (see utils.c):

lock(statechangemutex)

     set state data

     tryLock(mutex)
     if(fails) {
       unlock(statechangemutex)
       lock(mutex)  <-- which may block, but does not hold the statechangemutex!
       lock(statechangemutex)
     }
     set state data
     unlock(statechangemutex)

Q. Threads. ugh...

A. There's a good set of articles on POSIX threads at

      http://www-106.ibm.com/developerworks/library/l-posix1/,
      http://www-106.ibm.com/developerworks/library/l-posix2/ and
      http://www-106.ibm.com/developerworks/library/l-posix3/.

Q. What does the message "URL security(1): ERROR: Found percent in URL...DANGER... rejecting request" mean?

A. It means that ntop received a request with a percent sign (%) in it, often used as part of Unicode exploits against various web servers. Since there is no situation where ntop should process this, we reject it. URLsecurity in http.c is the place where these tests occur.

Q. What does the message "Rejected request from address x.y.z.t (it previously sent ntop a bad request)" mean?

A. Once you send ntop a request that URLsecurity rejects, the sending address goes into a ring buffer on a 5 minute timeout where we simply drop subsequent requests... rather than waste cycles ignoring an attack...

Q. What are the other URL security(#) codes?

A. 1. Found a % in the request (Unicode problems) 2. Found a parameter type code (//, &&, ??) 3. Found a directory transversal code (..) 4. Found a prohibited (RFC1945) character 5. Found a bad extension

Q. ntop doesn't report any traffic at all.

A. Understand how ntop works: It simply listens on the interface(s) for packets, then counts and interprets them. If there aren't any packets, ntop doesn't count things.

ntop does not sample. It processes every packet it sees and counts them. Only if there is more traffic than ntop can handle for a long period of time will the packet queue hit it's limit and packets be lost. But this is still not sampling.

Make sure that there's traffic on the interface(s) you are using. You can use tcpdump or a similar network sniffer tool to check.

If you are on a segmented network (i.e. switched), you may not see traffic that isn't destined for the ntop machine unless you configure the switch to set the port for the ntop host into "mirror" or "management" mode (different vendors call it different things, but it's a mode where ALL traffic is copied to a specific port, regardless of which port the destination host is on).

If there is more than one interface in the ntop host, perhaps you aren't listening on the one that has traffic? Check using ifconfig:

eth0 Link encap:Ethernet HWaddr 00:D0:09:77:85:B9

             inet addr:192.168.0.34  Bcast:192.168.0.255  Mask:255.255.255.0
             UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
             RX packets:1105906 errors:0 dropped:0 overruns:0 frame:0
             TX packets:601935 errors:0 dropped:0 overruns:0 carrier:0
             collisions:0 txqueuelen:100
             RX bytes:119869887 (114.3 Mb)  TX bytes:112203781 (107.0 Mb)
             Interrupt:11 Base address:0xc000

If the RX and TX numbers are increasing, this shows that traffic IS flowing...

If you have an unnumbered interface (listening only), remember you need to use -m to tell ntop what is local and what isn't:

eth1 Link encap:Ethernet HWaddr 00:30:F1:54:55:00

             UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
             RX packets:1596612 errors:0 dropped:0 overruns:0 frame:0
             TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
             collisions:0 txqueuelen:100
             RX bytes:566953031 (540.6 Mb)  TX bytes:0 (0.0 b)

You can select an interface using the '-i' flag, e.g. -i eth1 or -i eth0,eth1.

Q. ntop doesn't understand xxxxx?

A. True. IP Packets have a source address & port and a destination address & port... you MUST get your head out of the application layers and revert to that simple concept.

How does Apache handle virtual hosts? It analyzes the flow at the application level (layer 4) not the wire/packet/protocol (layers 1, 2 and 3). It does this by re-assembling packets into a layer 4 message (e.g. GET http://virtual.host.name.com/page.html)...

Now there are some layer 4 analysis routines - virtual hosts was added in 2.2 (and the folks who have virtual hosts have been pretty pleased), ftp, http, and some others - mostly looking for traffic on non-standard ports, etc.

So, since ntop works at the packet level, it doesn't understand virtual hosts. Unless it's SPECIFICALLY coded for. ntop is a NETWORK analyzer, not an application level one.

Q. tcpwrappers doesn't work

A. Oh yes it does... for http: connections

1) You have to configure it this way before compiling ntop:

        ./configure --enable-tcpwrap

2) You must have the headers and libraries installed on the build machine

      (and on the execution machine if they aren't the same).

Remember to make the appropriate entries in hosts.allow (e.g. ntop:192.168.0.) and hosts.deny (e.g. ntop:ALL)

However, tcpwrappers and https:// is known not to work - see docs/KNOWN_BUGS

Q. My filter doesn't work! I'm running ntop like this:

      /usr/local/bin/ntop -u nobody -L -d -E -w 3000 \
                      -m 192.168.10.0/24,xxx.xxx.xxx.xxx/32 \
                      -M -i eth0,eth1 \
                      (src net 192.168.10.0/24 or src host xxx.xxx.xxx.xxx ) \
                       and not dst net 192.168.10.0/24

A. Yup, it doesn't work. Use the -B option and put the filter in quotes:

      -B "(src net 192.168.10.0/24 or src host xxx.xxx.xxx.xxx ) and 
           not dst net 192.168.10.0/24"

ntop used to assume anything it didn't recognize was a filter. But not since 2.1.3. If you try this now, you should see a log warning that says maybe you forgot the quotes

Q. I have experienced problems defining multiple filters: ntop reports 'syntax error'

A. If you believe the filter is syntactically correct then it's likely that the Libpcap you have used has been compiled using an old non-reentrant version of flex. Please make sure you're using version 2.5.4 or above.

Q. Can you give some additional examples of filters?

A. man tcpdump -- see "expression"

A couple of simple examples, courtesy of B. Loic:

       -B "host not 192.168.1.100 and not 192.168.1.101"

to exclude hosts 192.168.1.100 and 192.168.1.101 from tracking (FQDN such as www.yahoo.com will work too).

If you need to exclude a full IP range, you will want to use something like

        -B "net not 192.168.1.0/24"

Q. What about the backtrace?

A. Sadly, probably useless. Why? Too much information isn't available:

  Here's one (and a pretty good, obvious one at that):

      ntop[23720]:   **FATAL_ERROR** RRD: caught signal 11 SIGSEGV
      ntop[23720]:   **FATAL_ERROR** RRD: BACKTRACE: backtrace is:
      ntop[23720]:   **FATAL_ERROR** RRD: BACKTRACE:  1. /usr/bin/ntop [0x42028c48]
      ntop[23720]:   **FATAL_ERROR** RRD: BACKTRACE:  2. /usr/bin/ntop(getopt_long+0x43) [0x420c4e83]
      ntop[23720]:   **FATAL_ERROR** RRD: BACKTRACE:  3. /usr/lib/ntop/plugins/rrdPlugin.so(rrd_update+0x61) [0x44157f7d]
      ntop[23720]:   **FATAL_ERROR** RRD: BACKTRACE:  4. /usr/lib/ntop/plugins/rrdPlugin.so [0x441481a2]
      ntop[23720]:   **FATAL_ERROR** RRD: BACKTRACE:  5. /usr/lib/ntop/plugins/rrdPlugin.so(updateCounter+0x19) [0x44148a15]
      ntop[23720]:   **FATAL_ERROR** RRD: BACKTRACE:  6. /usr/lib/ntop/plugins/rrdPlugin.so [0x4414ba49]
      ntop[23720]:   **FATAL_ERROR** RRD: BACKTRACE:  7. /lib/i686/libpthread.so.0 [0x41138941]
      ntop[23720]:   **FATAL_ERROR** RRD: BACKTRACE:  8. /usr/bin/ntop(__clone+0x3a) [0x420da1ca]

    See: It doesn't give the routines, just the hex offsets from some points in the executable.
    It seems to say that rrdPlugin's updateCounter called rrd_update which called getopt_long. But
    then getopt_long called something else and that's what died...  plus we don't see the parameters.

    Sometimes it works great for me on my box, because I can use the tools to figure out precisely
    where it means.  But often I use gdb too.  Move off my box and the odds of your compile and my 
    compile (and our ./configure and all the tool versions) matching so that the offsets mean something
    are pretty small.

Q. What do these log messages mean?

     **ERROR** mVLAN: Host (identical IP/MAC) found on multiple VLANs
     mVLAN: ntop continues but will consolidate and thus probably overcount this traffic
     mVLAN: Up to 10 examples will be printed
     mVLAN: Host 192.168.aaa.bbb (00:12:17:xx:yy:zz) VLANs 3 and 4

A. Once again, they mean what they say. ntop found something pretty weird. It found packets for the same 'host' (that is the same IP and MAC address) on two different VLANs. In this case, IP 192.168.aaa.bbb, MAC 00:12:17:xx:yy:zz on VLAN 3 and VLAN 4.

It's not WRONG, but it's definitely odd. If we see a packet twice, we count it twice, so - via this message - we're warning you that we might be double counting some traffic. We warn you ONCE per host (per ntop run) and for only up to 10 hosts, so as not to innundate you with meaningless warnings.

On the about page is a count (if you've even seen this junk), in the "Host/Session counts - global" section. If it's just a couple, ignore it.

If you see a lot of this stuff, you may want to rethink your ntop sensor placement or at least understand what you are doing. On my home LAN, I use a 802.1q VLAN trunk to move traffic between two switches and have a passive tap on the trunk. Traffic to/from the firewall moves over this trunk, so I see the same packet on the YELLOW(DMZ) VLAN and the GREEN(TRUSTED) VLAN all the time.

Why even bother trapping and warning? Well, there is one bad thing from ntop's perspective with duplicated hosts, which would occur if we didn't trap this, and that is that you would also see a lot of RRD errors when ntop tries to update the same host twice:

**WARNING** RRD: rrd_update(/usr/.../ipBytesSent.rrd) error: illegal attempt to update using time

                    1109523297 when last update time is 1109523297 (minimum one second step)

Running - Web Server

Q. Why does ntop display bits of my web site, instead of its own pages?

A. ntop is designed to search the current working directory for data files, such as the html subdirectory, before it searches the default directories.

This is a feature.

You are one of those rare souls who happen to have had an unrelated subdirectory 'html' with a file named 'index.html' as a subdirectory of the current working directory at the time that you launched ntop. cd to an acceptable directory, such as /usr/share/ntop, before launching ntop.

Q. What are High/Medium/Low risk flags

A. They are set in reportUtils.c based on fairly self-obvious functions Well documented in the help.html page, reachable by clicking on the problem descriptions or via http://127.0.0.1:3000/help.html

Often seen if you are monitoring a backbone or common network (high) or if you have cloned MAC addresses for, say, a home Firewall box.

Q. What does the "Users" flag mean on a host?

A. If you go to the "Info about host xxxx" page, there will be data in the "Known Users" section, if it's acting as a server for certain protocols.

In sessions.c, the function updateHostUsers() is used to maintain the list of "users" of a host. In handleSession(), as part of the protocol level analysis, the "user" information for various protocols is pulled out of the packets. Stuff like the "X-Kazaa-Username" header, the "MAIL FROM:" header, etc.

We tag users as one or more of the following types:

        P2P_USER, SMTP_USER, FTP_USER, POP_USER, IMAP_USER

Note that for P2P, we also record - where possible - whether this user is in P2P_UPLOAD_MODE and/or P2P_DOWNLOAD_MODE.

Q. Why are some of the host names in different colors?

A. Colors are used on several of the ntop pages to convey extra information to the user. (in particular the ACTIVE TCP SESSIONS and the LOCAL HOST STATS pages). There are five colors used to depict how long ago the host was first seen by ntop.

The pages which display these colors use a html style sheet called style.css located in the normal html subdirectory (where ntop is installed). This happens by setting the 'class=' parameter of the html 'A' (Anchor or hyper-link) tag. The style sheet defines the following:

      Age of host  'class' name  Color code         Color description
      (minutes)
      -------------------------------------------------------------------
      0-5          A.age0min    { color:#FF0000 }   Red
      5-15         A.age5min    { color:#FF00FF }   Fuchsia/Magenta
      15-30        A.age15min   { color:#FF7F00 }   Coral (lt orange)
      30-60        A.age30min   { color:#007FFF }   Slate blue
      60+          A.age60min   { color:#0000FF }   Blue

The color legend is displayed on the About | Configuration page (info.html).

Q. What does the P2P flag mean on a host?

A. If ntop knows enough to tag you as a P2P user, it's also looking at the other headers to see if it can track what files you're exchanging. If a host (i.e. a workstation) downloads a file from another host ("server"), the file name is recorded in the list ntop maintains for both of them.

If a host has at least one file name recorded, it's tagged with the "P2P" flag.

Q. What does a "Virtual Host" mean.

A. If a single instance of a web server handles many web sites, all of the references resolve to the same name. The web server uses the "Host:" header to determine which "index.html" page to serve up.

ntop monitors port 80 (http:) exchanges and looks for the Host: which allows it to build a list of virtual hosts being handled by the web server.

Running - Web Server (https:)

Q. SSL is not working! I have the following error in the log/terminal:

     10/Jun/2002 22:58:17 Started thread (6151) for network packet sniffing on
          eth0.1700:error:140EC0AF:SSL routines:SSL2_READ_INTERNAL:non sslv2
          initial packet:s2_pkt.c:187:

A. You forgot to put https:// instead of http:// in the URL you put in your browser!

Q. Unable to find SSL certificate 'ntop-cert.pem'

A. ntop looks such file under the current working directory, then /etc or in Whatever directory you configured with ./configure.

If you want a personal certificate, you need to create it by:

      >make ntop-cert.pem

It should be installed as part of "make install". If you have a special Certificate or it's not present, do it (one-time) manually:

For example to install it under /usr/local/etc, do:

mkdir /usr/local/etc cp /usr/local/bin/ntop-cert.pem /usr/local/etc/ntop

See docs/README.SSL

Q: What is the ssl watchdog?
A: Short answer: There are reported problems w/ the ntop web server hanging when
                 accessed via ssl (https://) from Netscape 6.2.2 (Win2K) (and
                 others).

                 The ssl "watchdog" keeps an eye on the web server - it waits
                 for 3 seconds and then if the SSL_accept call (openSSL) hasn't
                 finished, it aborts it. This leaves the user with nothing on
                 their web browser, but at least ntop's web server continues on.

                 There is no known way to send something back to the user. DON'T
                 EVEN ASK. It's not in ntop, it's the browser-server handshake
                 that's hung.  So, it looks - to the user - like a failed 
                 connection.  S'be'it...

If you are using https:// and seem to have the problem, run ntop with the --ssl-watchdog command line parameter... The item to look for on the configuration page (info.html) is:

       # HTTPS Request Timeouts

Or messages in the log:

...: SSLWDERROR: Watchdog timer has expired. Aborting request, but ntop

        processing continues!

You can also enable it via a ./configure parameter (./configure --help | less) if it's something you're going to always require.

Q. Tell me more

A. The problem is that ntop's web server is single threaded until we determine that the request is simply one that will be reading data. At that point we fork to generate the page. But the basic "accept a request" code is single threaded. This happens all but instantaneously and hasn't been a problem previously.

The code is pretty basic and pretty common:

       select() to wait for a connection, then
       ssl_accept() to fire up a "server", meaning the ssl handshake.

       Then process the http request (i.e. the GET and associated headers).

With Netscape 6.2.2 (and others), there seems to be a bug in the Netscape code (ntop's is identical to other projects like sshd).

According to something I read - but now can't find again - Netscape doesn't accept a legal combination of options on the handshake back from openSSL and hangs in a deadly embrace. Supposedly openSSL 0.9.6c (or was it d - it's not in the changelog) built in a patch. However, I didn't find the new version changed the behavior.

There is stuff about a bug w/ Netscape 4.x on the openSSL website, but I'm not having trouble with Netscape 4.x.

I don't understand the details and really don't care to find out. It boils down to a hang in a call, SSL_accept() that doesn't have a timeout parameter. Argh

Because the code is invasive, I built it (like the SIGPIPE stuff) so you can turn it on at ./configure time:

      --enable-sslwatchdog    Watchdog for ssl hangups (Netscape 6.2.2)
                              [default=disabled]

or via a command line option:

      --ssl-watchdog          Use ssl watchdog (NS6 problem)

With the "fix", ntop's web server hangs for at most 3 seconds, then continues on. The user gets nada - and I don't know a way to send them anything, because we haven't retrieved the request yet nor done the handshake (so there isn't a TCP connection!)

It only affects https:// requests and I've coded the watchdog so it doesn't activate unless we have openSSL and either the compile or runtime parameter set. If you don't get https:// requests, it's just another idle thread.

The fix is working for me... What I've tested (and the results with and without the watchdog):

    Win2k
      MS Internet Explorer 5.5 - ok
      Netscape 4.61 - ok
      Netscape 4.79 - ok
      Netscape 6.2.2 - user gets no response
                     - old: ntop webserver hung and must restart ntop!!
      Opera 6.03 - user gets a partial response
                 - old: browser says "setting up secure connection" and
                        never continues, but ntop's webserver is ok
                        (SOMETIMES you get SSL errors in log, esp.
                        if you cancel the browser)

    Linux
      Konqueror 2.2.2 - ok
      Mozilla - 1.0 - ok
      Netscape 4.78 - ok
      Galeon 1.2.5 - almost complete response, browser session is toast
                     (must restart)
                   - old: user gets nothing, but the ntop webserver is ok
      Opera 6.0B1 - user gets a partial response, but browser session is ok
                  - old: browser says "setting up secure connection" and never
                         continues, but ntop's webserver is ok.

Running - Web Server - Security

Q. I'm being prompted for a userid/password, what do I enter.

A. The default admin userid is 'Admin' (without the quotes) and the password is whatever you set on the 1st run of ntop (look in this FAQ for --set-admin-password).

Q. Why create Userids (beyond the Admin id created by --set-admin-password)

A. Multiple users allow you to control who can alter ntop's performance and/or view specific information. If you look on the "Admin" tab, you will see that you can create additional users and also control which URLs can be executed by whom.

Userids could allow, for example, an ISP to allow users to access SOME network performance statistics, but not the proprietary stuff...

Suppose you want to restrict who accesses the Multicast statistics page, multicastStats.html.

ntop uses terminal wildcards matching the names, so multicast is treated as multicast* and matches multicastStats.html plus any other name beginning multicast...

howto:

1st add a new user 2nd add "multicast" to the list of controlled screens and allow admin

       and the new user to access it (note the * wildcard is automatically
       added)

Try and access the screen and you are prompted for a userid/password...

Look in http.c for all the names and #defines used...

Q. So, How do I restrict access to the main http or https ntop web page?

A. To stop everyone from logging into ntop, do the following: Select ADMIN tab Select URL's then "Add Url" Don't fill in anything (the wildcard * is implied) Select only the users who you want to authorize (hold control key and click on user to add more users if you added users)

and click on "Add Url"

You will see URL '*' is added, e.g.

     'showU*'
     '*'
     'shutdown*'
     'deleteU*'
     'modifyU*'

Then only users who know the user id and password (remember to keep the .db file secure!) will have access.

Q. How good is the default security ntop provides through the web server.

A. Good question...

The default ntop configuration is not appropriate if you put ntop in a publicly reachable location. We assume that ntop is either running in a small, trusted, LAN or that you've used other tools such as firewalls to protect the ntop web server.

The userid/password scheme is good enough to prevent you from accidentally shutting down ntop or getting into 'dangerous' places. But that's really all it does.

Also, the security of the ntop web server is only as good as the security of the passwords file, ntop_pw.db - only the ntop -u userid should be able to read from or write to it.

Read man crypt for information on the security of the encrypted password.

Q. The plugins aren't very secure.

A. True.

Q. How do I prevent users from turning plugins on/off?

A. The default configuration of ntop does not protect the plugin pages - no password is required to access showPlugins.html.

This allows any user who can connect to the ntop web server to view reports FROM the plugins, but also allows them to make plugin configuration changes.

This may not be desirable. You may wish to add additional URLs to the Default list of those which require entry of a userid/password.

You can prevent unauthorized individuals from turning plugins on/off by adding this URL: "showPlugins.html?" to the list via Add URL.

Q. Ok, but they can still get into the configuration pages and change things.

A. Yes. Add the following URLs to the controlled list:

        plugins/sFlow*
        plugins/netFlow*
        plugins/rrdPlugin*

    This will keep everybody who doesn't know the userid/password out of the
    configurable plugins.  Unfortunately, it will also prevent them from seeing
    the rrd graphs, because those are created out of the rrd plugin.

A. Instead of plugins/rrdPlugin*, create these:

        plugins/rrdPlugin?d*
        plugins/rrdPlugin?h*
        plugins/rrdPlugin?i*
        plugins/rrdPlugin?r*

It's still not perfect, the reasons why are left as an exercise for the user.

Q. I created a new user, adminp for administering the plugins and ntop is also accepting the admin userid/password.

A. Yes, the matching of userid's isn't too swift. It's better to make sure that there are NO common initial substrings among ANY of the userids.

Q: How can I use Apache (with it's security, etc.) to serve up ntop pages?
A: (Toby Johnson [public@tobiasly.com], Sun 10Nov2002)

A while back, I had written about the possibility of configuring ntop to use only relative URL's, in order to facilitate proxying ntop's web interface through Apache. I have decided it's easier to simply use Apache's ability to rewrite ntop's URL's when necessary. So, based on my experience, here is a mini-HOWTO on how to proxy ntop through Apache.

------

Proxying ntop's web interface through a secure Apache virtual host is a convenient way to make use of any existing security measures you may already have. In my case, I wanted to be able to access ntop from anywhere outside my LAN, but opening another port on my server for ntop's dedicated web server wasn't an option.

I already had a password-protected, secure web server that I use for admin purposes -- I'll call it https://admin.tobiasly.com. I wanted ntop's web interface to appear as a subdirectory under this host: https://admin.tobiasly.com/ntop/ .

Here's how to configure such a setup. Change the server names and ports to match your own. I'm assuming that you already have a working, secure Apache virtual host (using HTTPS).

First, pick a port for ntop's HTTP server. I'll use 15123. You won't need ntop's built-in HTTPS server, since you're proxying its content through a pre-existing Apache HTTPS server. Configure ntop to start with the correct HTTP port, and with HTTPS disabled. Something like "ntop -d -w 15123 -W 0". (See the ntop man page for more startup options.)

Now, you need to tell Apache that anything under the /ntop/ URL should be proxied to the ntop web server. In my case, the Apache server is running on the same machine as ntop, so it's just a proxy to a different port on localhost. In your Apache secure host configuration, add a line like this:

      ProxyPass /ntop/ http://localhost:15123/

Now, whenever Apache receives a request for something like "https://secure.tobiasly.com/ntop/home.html", it will proxy this request to the location "http://localhost:15123/home.html". Ntop will take it from there, generate the web content, and pass the result back to Apache. Then Apache passes that result back to the original client.

It's important to note that you don't need to open port 15123 to the outside, since the connection actually goes through your existing Apache port, and then is transparently proxied by Apache on the server itself. Of course, you don't even have to run ntop on the same machine; as long as the Apache server can connect to ntop's port, it'll work.

This is not the same as URL redirection. As far as your web browser knows, everything is going through https://secure.tobiasly.com/ntop/. The Apache server does all the proxy work behind the scenes, and simply serves up the results to the requesting client. And since the "outward-facing" server is Apache instead of ntop, you'll be using your existing Apache secure server certificate, instead of ntop's ntop-cert.pem.

Everything appears to work OK at first, but we quickly run into a problem: some of the URL's that ntop generates are absolute. For example, to draw bar graphs, ntop's web pages will request the image "/gauge.jpg". This would translate into "https://secure.tobiasly.com/gauge.jpg". Also, host info pages are absolute. If I click on the host "10.1.2.3", it tries to take me to the page "https://secure.tobiasly.com/10.1.2.3.html".

This is a big problem, because unless the URL is underneath the /ntop/ directory, Apache doesn't know that it needs to proxy the request to ntop, and you get broken links. Luckily, Apache has the Rewrite module that lets us fool with requested URL's. In order to get the required URL's rewritten, add the following to your Apache secure virtual host configuration:

      RewriteEngine On
      RewriteCond %{HTTP_REFERER} tobiasly.com/ntop
      RewriteCond %{REQUEST_URI} !^/ntop
      RewriteCond %{REQUEST_URI} !^/error
      RewriteRule ^/(.*)$ http://secure.tobiasly.com/ntop/$1 [L,P]

In English, this basically says "If I get a URL request that comes from a page that has tobiasly.com/ntop in it, and that request doesn't begin with /ntop, rewrute the URL to begin with http://secure.tobiasly.com/ntop/, and pass this rewritten URL to the Proxy engine." At this point, the Proxy engine will see that it is getting a URL that begins with /ntop/, and correctly pass it to the ntop web server. Rewriting the request to begin with HTTP instead of HTTPS may seem incorrect, but since that URL will be handed directly to the Proxy engine, it can't be HTTPS or ntop's web server will not recognize it.

Now, you should be able to simply connect to https://secure.tobiasly.com/ntop/ , and you're ready to go!

NOTE courtesy of Bruno Lebayle

I had many problems with the "Rewrite" stuff described in the FAQ. After some Google search, it appears that HTTP_REFERER is sometimes not reliable, and the browser I am using (Firefox/Solaris) does not seem to present this header properly. So I've found an other way to do it, and this proves to work with many different browsers:

within mod_proxy.c:

    ProxyPass /ntop/ http://localhost:3000/
    ProxyPassReverse /ntop/ http://localhost:3000/

    RewriteEngine On
    RewriteLogLevel 0
    RewriteLog logs/rewrite_log
    RewriteCond %{REQUEST_URI} !^/$
    RewriteCond %{REQUEST_URI} !^/home.html$
    RewriteCond %{REQUEST_URI} !^/ntop/
    RewriteRule ^/([^/]+\.[a-z]+)$ http://nms.my.domain:my_port/ntop/$1[L,P]

Where http://nms.my.domain:my_port is: - the host where the network management tools are located with the proper authentication - the host where the above Apache config has to be set - the host where ntop is installed on port 3000 in this example Where the Web site on this host has no URL in the form of /xxx.xxx aprt from / and /home.html (additions can be made in the RewriteCond)

Especially, the trailing "/" after /ntop in the RewriteCond is mandatory, since e.g. the Ntop logo is /ntop_logo.gif and would match /ntop

# Rewrite help summary: # HTTP_REFERER = pattern in the name of the page from which the request comes # THIS IS NOT RELIABLE AND DIFFERS AMONG BROWSERS !!! # REQUEST_URI = contents of the request (^ = start of line, $ = end of line, # ! = not the text which follows # All conditions preceding a rule are evaluated # All rules are processed in sequence # Rewrite: () = group of text used for the substitution, # . = any char, * = repeated 0-N times, # + = repeated 1-N times, ^ = not this char, [chars] = one of chars # special characters escaped by \ e.g. the dot as \. # $1 means the 1st group of text between () # Flags L = last (i.e. exit from the rewriting process) # P = proxy (i.e. use the proxy module for this URL) # # Full doc in http://httpd.apache.org/docs/1.3/mod/mod_rewrite.html # Useful help in http://rewrite.drbacchus.com/rewritewiki/

Running - Web Server - i18n

Q. Is ntop localized for language x? (i18n)

A. No. ntop wasn't really written with i18n in mind.

Most of the text is generated in-line, on the fly. Plus ntop must dynamically support multiple locales simultaneously.

However, beginning with v2.1.56 (2.2 development release), there is limited, optional, i18n support in ntop.

Q. So, what internationalization (i18n) support does ntop provide.

A. The key word is LIMITED

This only applies to the pages that are pulled from .html files, NOT those created internally. This includes the menus and the few static text pages, but none of the pages with interesting data on them.

The localized pages must be placed in parallel directories to the existing html ones.

For example, if ntop is installed in /usr/share/ntop, the html files are in /usr/share/ntop/html.

To support them Canadians, then, you would need to create a /usr/share/ntop/html_en_CA AND that locale would need to be installed on the ntop host system.

Note that there are NO i18n files distributed with ntop (yet!)

At ./configure time, you enable support via --enable-i18n. ntop MUST be told how to find the locale files. In ./configure, a "standard" location is defined per OS.

(Initially only the value for FreeBSD is populated). All others assume the "default", /usr/lib/locale. If that isn't right for your OS, then you MUST use the optional parameter --with-localedir= to tell ntop where to find the files.

At run time, ntop scans the host for the installed locales (locale -a should - on most systems give you a list) and checks if a comparable html_cc_XX directory exists.

This builds a list of supported languages, which (along with i18n status) is shown on the configuration pages, info.html and textinfo.html.

When an http request is made, your browser sends a list of languages it is willing to accept in the http Accept-Language: header.

(check View | Internet Options | Languages in IE to see what you're sending)

For example,

       Accept-Languages: en_US, en

Means that you prefer US English, but will accept any English dialect if US English isn't available.

Be aware that the locale settings and Accept-Language settings are not well standardized, nor common and may not necessarily map very cleanly. You should see what's defined (perhaps it's locale 'german' instead of 'gr') and make or link directories as necessary. You can always create the directory you tell ntop to use via --with-localedir= in the /usr/share/ntop structure and create links from there to the real locale directories!

Limits in the per-request and total # of languages to support are in globals-defines.h

Because of directory structure limits, a lack of interest in multiple character sets, etc. the locale and accept-language headers are coerced into a common format:

locales are ll[_XX][.char][@modifier]

       ll - language, usually the 2 character ISO abrev., such as us, it.
       XX - dialect (often a country), such as CA or US (en_US != en_CA)
       char - character set (we sort of assume UTF-8)
       modifier - euro

Accept-Language: values are ll-XX or ll or ll-*

Once the user makes a request, each page pulled is checked:

       1. For each of the Accept-Language values.
       2. For the ntop host locale value.
       3. In the ntop default (English) set.

These checks are performed for each of the libraries specified in the config value (CFG_DATAFILE_DIR).

Q. What about the country flags.

A. There are other sets available on the web, of different quality and size. Rather than chase down permissions and rights, we'll stick with what we have but let you know here of other options.

If you find a set you like, just download them and replace the xx.gif files in .../html/statsicons/flags

Much better, but about 4x larger:

      http://users.skynet.be/hermandw/fl/smalgifs.html

Same height, but wider (so they look better):

      http://www.kidlink.org/www/miniflags.html

Q. What pages can be customized?

A. ls /usr/share/ntop/html/*.html (or wherever the ntop pages are installed):

Also, remember that a file overrides ntop's internal page generation, so you can also use this facility to override ANY of ntop's pages and return a customized page (perhaps you don't want users seeing them?).

Running - Web Server - p3p

Q. What's up with P3P?

A. P3P is a W3C recommendation - http://www.w3.org/TR/P3P/ - for specifying how an application(typically a web site) handles personally identifiably information. What information the site collects and what it does with the information.

p3p is pretty complex! There are basically two ways to enable an application for p3p. One is to add another HTTP header, P3P:. The second is to support a well-known file location, /w3c/p3p.xml (like robots.txt).

Browser support is pretty spotty, as is web site adoption.

Some 3rd party browsers have some support... up to CrazyBrowser which claims "full support", whatever that means...

Q. So why put P3P into ntop?

A. It's coming. P3P is gradually making it's way into the top web sites - right now (Dec2002), for example dell.com supports it and yahoo.com doesn't.

Q. Ok, but what's that got to do with ntop?

A. Since ntop collects personally identifiable data in it's access log (-a option) and it's various reports and makes those available to pretty much anyone in the default configuration, it's probably not a bad idea to OFFER some support. Especially if you're running ntop at a site that has started to support P3P, if you don't have a mechanism for your own policies you'll have to adhere to corporate ones. And that could require massive changes to ntop.

Q. IE6?

A. Since ntop doesn't send the P3P: header, IE6 ignores ntop wrt p3p. Besides, IE6 uses p3p to block 3rd party cookies. If you want to see the p3p stuff, it's view | privacy report in the menus. If the site's policies don't match your settings, there will be a red "do not enter" icon in the third box on the bottom right of the IE6 window - double click on it to see the report. See http://support.microsoft.com/default.aspx?scid=KB;en-us;q293513

Q. Mozilla

A. Unknown if it's enabled by default. Mozilla had support, ripped it out in Feb 2002 and put a new version back in.

Q. Other browsers

A. See their home pages or search the web. One that I know that claims "full support" (whatever that means) is at http://www.crazybrowser.com/

Q. Privacy Bird?

A. A browser-add-on, AT&T's privacy bird (http://www.privacybird.com/), that I'm playing with is a lot more aggressive in supporting p3p. If Privacy bird doesn't see the P3P: header, it then requests the "well known" file, /w3c/p3p.xml file and gets nailed by ntop as a hostile application, since we don't have support for returning .xml files (yet).

Q. So when & how does ntop support p3p?

A. A patch in the cvs on 4Dec2002 adds minimal support for p3p -- specifically:

      1) ntop will respond to queries for /w3c/p3p.xml and ntop.p3p -- returning
         the ntop.p3p file, IF ONE EXISTS.

         If the file does not exist, a 404 error is generated (vs. pre 4Dec2002
         behavior of adding the address to the myGlobals.weDontWantToTalkWithYou
         list).

      2) New parameters, --p3p-cp and --p3p-uri allow you to return the P3P:
         header with either or both of the parameters (cp="" or policyref="")
         set.

         ntop doesn't validate the text in any way other than the usual
         stringSanityCheck().

This allows me to run the Privacy Bird and still talk to ntop. I'll admit that option #2 is speculative, since I really don't have much of a way to test it.

Q. But there isn't a sample .p3p file provided.

A. Right.

Please note that there is no sample file provided. This is not an oversight.

After careful consideration, I am not providing one. The reason is that a .p3p file is intended to be a legal contract between your site and your users. While I could provide a default file that has the right tags - as I understand p3p - for the data ntop collects and stores, I don't want the responsibility and/or liability.

If anyone wants this "sample p3p file", I will make it available for a fee, Provided your organization - through an appropriate officer, in writing:

      1) Acknowledges that Luca Deri, Burton Strauss and other developers of
         ntop have no liability for any use(s) you make of the sample p3p file
         or anything you derive from it.

      2) You will defend us - at your expense - from any lawsuit, arbitration
         proceeding, etc. filed in conjunction with your use of the sample p3p
         file.

      3) You will pay any judgments, legal expenses, etc. related to any
         lawsuit, arbitration proceeding, etc. in conjunction with your use of
         the sample p3p file.

Since your legal department would be nuts to agree to that I doubt it will come up.

Q. So How do I create a .p3p file?

A. There are tools available to create p3p policy files - search the web for 'p3p editor'. One that I've used is a zero cost albeit beta tool, p3peditor from IBM (http://www.alphaworks.ibm.com/tech/p3peditor).

Networks, Network cards and Networking

Q. My security people won't let me run in promiscuous mode.

A. Tough...

Or, use the -s option and accept the limitations...

Ask them "honestly, what is the problem" - other than having an interface in promiscuous mode is a signature of a sniffer and security folks look for unauthorized sniffers?

ntop needs promiscuous mode so that it sees the full range of traffic. Any similar product will do the same thing.

If the security people think traffic on the wire is secure, they're wrong! Face facts - just about every Windows user, except for 2K/XP Pro (and then only if TBTP have especially locked them down) can install the windows version of tcpdump...

If it's a checklist item, just gen up a form to "authorize" it, have the boss and VP/CIO sign it and give it to them.

Q. What is Ethernet and TCP/IP and how do they differ?

A. Both are protocols - that is the definition of how to interpret bits on wires (or in packets) into meaningful conversations.

Ethernet is the lower level, wire (or wireless) protocol, concerned with moving the physical bits of data.

TCP/IP is the higher-level protocol, which explains how to interpret the block of bits (frame).

TCP/IP uses a familiar 32-bit "IP" address, e.g. 192.168.0.1.

Ethernet uses a less familiar, 48 bit unique to the NIC (some times called "burned in") address, e.g. 00:40:05:DE:AD:00. This is called the MAC (Media Access Control) address.

FYI: The official IEEE MAC address lookup is at

       http://standards.ieee.org/regauth/oui/index.shtml
   (Look up the first six digits, separated by -s, e.g. 00-40-05)

Q. OK, but how is stuff sent from my computer to, say, Yahoo!?

A. First off, your computer does a lookup - using a service called DNS (Domain Name Service) to convert www.yahoo.com to a numeric value, such as 66.218.71.80.

Then it builds a collection of characters that says send this data from me, 192.168.0.1 to Yahoo at 66.218.71.80. This is called a packet. That gets wrapped in an Ethernet frame (addressed from 00:40:05:DE:AD:00 to the MAC address of the local gateway router, 0:d0:9e:6:38:00 and squirts it out the router.

Packets are forwarded step by step along a path from you to Yahoo by computers called routers. This is done based on the 32 bit IP address and the router's knowledge of the network.

Each router sees a Ethernet frame addressed to it (by MAC address), checks the TCP/IP address to figure out where to send it next, re-wraps the TCP/IP packet in a new Ethernet frame (with the from MAC as it's own and the to MAC as the next hop).

This happens until the TCP/IP packet reaches the final segment (the last router). Once it reaches a router that knows it has addresses 66.218.71.0-66.218.71.255 on one of it's interface, the routing stops using the TCP/IP address.

The last hop is done (like each intermediate hop - at the lowest level) based on the MAC address! Specifically, the last router does an "ARP" (Address Resolution Protocol") query, to find out "Who Has" address 66.218.71.80. The NIC responds with it's MAC address:

      arp who-has www.yahoo.com tell router
      arp reply www.yahoo.com is-at 0:d0:9e:6:38:00

And the packet is routed to that address.

Alright, that's a bit simplified, but see Douglas Comer, "Internetworking with TCP/IP, volume I", page 25 and 73ff.

Q. Tell me more.

A. OK, gang time to teach Ethernet & TCP/IP basics one more time. With pictures...

Suppose you have a network that looks like this (we'll use impossible addresses 288 - just pretend it's ok):

              (ext) 288.1.1.1     (int) 288.2.2.1
                            +-----+
     World+Dog ------------ + ISA + ------- LAN ----- WS 288.2.2.2
         |                  +-----+
         |                     |(dmz) 277.1.1.1
        me 299.0.0.1           \----------- DMZ ----- MAIL 277.1.1.2
                                                  \-- WEB  277.1.1.3

ISA can be acting solely as a router or it can be acting as a NAT device. That's irrelevant, so we assume it's not.

I send you a packet. It travels the Internet and arrives at your 288.1.1.1 the ISA(router) with src=299.0.0.1, dst=288.2.2.2.

Like every router along the way, ISA(router) looks at the destination address and realizes it has to route the packet on to 288.2.2.2. So the ISA(router) sends the packet on, out the best interface to reach 288.2.2.2.

Remember, however, the TCP/IP packet is wrapped in a lower level (Ethernet) packet at the wire level. Read your TCP/IP and Ethernet standards - the actual delivery of packets over links is this Ethernet level and it uses the 48 bit MAC address.

This "Ethernet" packet is actually what travels hop to hop to hop (you can even see these headers if you have visibility to the traffic - it's called the link level header by tcpdump and you'll see the 48 bit MAC addresses if you use the -e parameter).

In order to be able to handle the Ethernet level signaling, each router rewrites the packet so that the 48 bit source MAC address it it's own (from-router that is) and the destination MAC address is the one that from-router has in it's tables for to-router (the next hop).

So the packet looks like this, where the srcMAC and dstMAC get rewritten each hop, so that the routers on both ends of the link know whom it's addressed to:

Hop1 (srcMAC=00:00:00:aa:aa:aa dstMAC=00:00:00:bbbbb:bb frame=IP data=(src=299.1.1.1 dst=288.2.2.2 data="Hi!") Hop2 (srcMAC=00:00:00:bb:bb:bb dstMAC=00:00:00:cc:cc:cc frame=IP data=(src=299.1.1.1 dst=288.2.2.2 data="Hi!") Hop3 (srcMAC=00:00:00:cc:cc:cc dstMAC=00:00:00:dd:dd:dd frame=IP data=(src=299.1.1.1 dst=288.2.2.2 data="Hi!")

Notice how the TCP/IP stuff isn't changed. But the MAC address is.

At each hop, the NIC card, operating at the Ethernet level, sees its own MAC address and knows to accept the packet. It passes it up the protocol stack, where the next layer (TCP/IP) realizes it needs to be routed further on...

Ultimately, the packet gets delivered to some service listening on your WS.

Here's a packet capture to show you:

# tcpdump -Xx -c1 -i eth0 -e tcpdump: listening on eth0 11:49:10.809890 0:3:47:b1:xx:xx 0:e0:18:b4:yy:yy ip 118: tigger.ssh > zebra.2714: P 1824243567:1824243631(64) ack 2328789523 win 11792 (DF) [tos 0x10] 0x0000 4510 0068 b305 4000 4006 b1e4 c0a8 2a24 E..h..@.@.....*$ 0x0010 c0a8 2a21 0016 0a9a 6cbb bf6f 8ace 8213 ..*!....l..o.... 0x0020 5018 2e10 ee1a 0000 469a 3e34 eda7 549e P.......F.>4..T. 0x0030 0ec4 4847 8983 fb4f 65ea 5c3e 0bbe c325 ..HG...Oe.\>...% 0x0040 7db8 9954 dae1 55b6 54f9 cdfd ac07 a2b5 }..T..U.T....... 0x0050 ce4f

So this says, the packet came from tigger (MAC address 0:3:47:b1:xx:xx) -> zebra (MAC address 0:e0:18:b4:yy:yy)

     Within that is the tcp/ip packet, from c0a82a24 -> c0ae2a21
   (192.168.42.36 -> 192.168.42.33)

Here's another one, from tigger -> router.

# tcpdump -Xx -c1 -i eth0 -e host 192.168.42.1 tcpdump: listening on eth0 11:52:48.712750 0:3:47:b1:aa:aa 0:d0:9e:6:bb:bb ip 72: tigger.32782 > homeportal.gateway.2wire.net.domain: 41356+ A? cvs.ntop.org. (30) (DF) 0x0000 4500 003a bf7e 4000 4011 a5be c0a8 2a24 E..:.~@.@.....*$ 0x0010 c0a8 2a01 800e 0035 0026 ce2e a18c 0100 ..*....5.&...... 0x0020 0001 0000 0000 0000 0363 7673 046e 746f .........cvs.nto 0x0030 7003 6f72 6700 0001 0001 p.org.....

(0035 = port 53, so it's a dns query)

And one more, from cvs.ntop.org -> tigger (which has to have passed through the router)

# tcpdump -Xx -c1 -i eth0 -e "not src net 192.168.42.0/24" tcpdump: listening on eth0 11:53:39.688806 0:d0:9e:6:bb:bb 0:3:47:b1:aa:aa ip 69: 195.31.151.66.cvspserver > tigger.42964: P 2566885448:2566885451(3) ack 2903504154 win 24616 (DF) 0x0000 4500 0037 5f3c 4000 2906 ad56 c31f 9742 E..7_<@.)..V...B 0x0010 c0a8 2a24 0961 a7d4 98ff 9048 ad0f f51a ..*$.a.....H.... 0x0020 8018 6028 279a 0000 0101 080a 173c 72b5 ..`('........

See the MAC address? It's the routers, not cvs.ntop.org's

Q. Why do I care?

A. Because that's sometimes a problem you'll have with ntop. See ntop is nosy, so it puts the interface into promiscuous mode, where every packet - addressed to the ntop host or not - is processed. Now the next layer up, the tcp/ip layer will throw away any 'junk' (You can just hear it going tisk, tisk, tisk). But libpcap can intercept them at the Ethernet level and feed them to ntop...

For REMOTE hosts, ntop uses the IP address as it's the only valid data. But for LOCAL hosts, ntop prefers to use the MAC address as a way of resolving multi-homed or multiply addressed hosts.

See if you have two IP addresses assigned to the same host on your local network, say 192.168.42.42 and 192.168.42.43, how is ntop to tell they're the same host? Well, the MAC addresses are the same... (for some manufacturers, e.g. Sun, ALL of the interfaces on a host use the same MAC address).

read getHostInfo() in hash.c.

      /* This is a local address hence this is a potential
         multihomed host. */

      if((el->hostIpAddress.s_addr != 0x0)
         && (el->hostIpAddress.s_addr != hostIpAddress->s_addr)) {
         isMultihomed = 1;
         FD_SET(FLAG_HOST_TYPE_MULTIHOMED, &el->flags);
      }

i.e. if the address we've stored for this host doesn't match this one, it's multihomed.

      } else if((hostIpAddress != NULL)
                && (el->hostIpAddress.s_addr == hostIpAddress->s_addr)) {
         /* Spoofing or duplicated MAC address:
            two hosts with the same IP address and different MAC
            addresses
          */

         if(!hasDuplicatedMac(el)) {
            FD_SET(FLAG_HOST_DUPLICATED_MAC, &el->flags);
            ...
         }

         setSpoofingFlag = 1;
         hostFound = 1;
         break;
      }

If the addresses DO match, we've had two MAC addresses, so this is being spoofed.

etc.

Q. So why do I get bad output?

A. If, somehow, you've confused ntop - for example telling it that 277.1.1.0/24 in the ascii art example (above) is local, then ntop is going to believe you. And it will see a packet with the 277.1.1.1 IP and a MAC address. And use that. Only it's not the MAC address of the MAIL host, it's really the MAC address of ISA.

No matter, ntop doesn't know this -- all it sees is the packets and the data you gave it. So later on, when it sees a packet with the same MAC address, but a different IP, well, it will assume that it's the same host... and the data will all be lumped together.

Q. Does that explain why I'm seeing xxxx as multihomed?

A. Maybe. Remember - it only takes one packet, not even an ack, for ntop to create a host record. If that's wrong, it will carry forward - and you'll probably see the host tagged as 'Multihomed' when correct packets show up. Say:

      Host 1: IP 192.168.1.1 MAC 00:00:00:aa:aa:aa
      Host 3: IP 192.168.1.3 MAC 00:00:00:cc:cc:cc

If somebody has the incorrect hosts table, dns, cached, whatever but has the info that Host 1 is 192.168.1.3 and is on the same subnet, then it will send a packet where the Ethernet layer and the ip are nonsense. But because it's on the same wire, the ip is ignored:

(Ethernet from:00:00:00:dd:dd:dd to:00:00:00:aa:aa:aa)

       (tcp s=192.168.1.4 d=192.168.1.3)

ntop will read both out of the packet and create the association

      192.168.1.3=00:00:00:aa:aa:aa

Since it doesn't know better.

Then when it sees

(Ethernet from:00:00:00:ee:ee:ee to:00:00:00:cc:cc:cc)

      (tcp s=192.168.1.5 d=192.168.1.3)

It will create the multihomed association...

Q. So what's a hub vs. a Switch

A. A hub is a device that links a bunch of computers together at the wire (Ethernet) level. Logically, Ethernet is a bus, that is everybody sees all the traffic, just like cars crossing under a highway bridge. Physically, Ethernet is wired like a star - with all the wires coming back to a central "hub". The hub is just the device that makes the electric star look like a shared bus.

Switches and Hubs operate at the Ethernet level, not TCP/IP.

A. Watch out for 'Switched hubs', which are hubs that include an internal switch between 2 or more segments (for example, BUT NOT LIMITED TO a 10BaseT and 100BaseT) segment. These are hubs within a segment, but switches across segments. ntop may not see the traffic you expect if you have a 'switched hubs' and manufacturers are pretty bad about marking them. See http://article.gmane.org/gmane.linux.ntop.general/5081

A. A switch is a smart hub.

Switches improve performance by creating a virtual Ethernet bus for the duration of the packet that joins JUST the source and destination ports.

A switch operates via an internal table of MAC addresses. It learns (or is programmed) that 0:d0:9e:6:38:00 is on port 1, while 00:40:05:DE:AD:00 is on port 3.

A packet coming in port 1, destined for 00:40:05:DE:AD:00 is sent out ONLY port 3.

If the switch doesn't know (or the packet is a broadcast), it gets sent out all ports.

This doesn't make for MORE bandwidth, but it does use it more efficiently. That is in addition to the session between ports 1 and 3 at 100Mbps, a second, simultaneous 100Mbps session can occur between ports 2 and 4.

Q. How do I use ntop in a switched network?

A. First off, you need to be or have the support of your network administrator. (Yes, you can do something called "ARP poisoning" to - maybe - get the switch to send you all the traffic, but that's beyond this FAQ... STFW)

Many switches (although not the USD$50 cheap "workgroup" units) have a special port or mode, where by all the traffic for the entire network gets copied out that port, in addition to the normal switch action.

When you invoke the monitoring mode (called span, mirror, monitor, analysis, etc.), you are forcing the entire switch bandwidth out one port. This may exceed the bandwidth of the port. 100Mbps+100Mbps >> 100Mbps!

Traffic that is being sent to the monitoring port in excess of the capacity of that port is usually dropped. It should NOT slow down the switch on other ports.

Some switches have some buffering capability and it *may* be able to keep up with an occasional burst of traffic, as long as the average is below the port capacity and the buffer isn't exceeded.

See, for example, http://www.cisco.com/warp/public/473/41.html#archXL.

One list of switch manufacturers is the document is titled "REFERENCE: Configuring a Switch to Monitor All Traffic" from Elron Software. (The URL is long, do a Google search for "site:elronsoftware.com wi6038").

Dumping data

Q. Can I use ntop from php/perl?

A. Yes you can. Please see the www directory under the ntop source tree.

A. Look at the Admin | Dump Data page.

rrd (myrrd)

Q. How do I save data between runs?

A. Use rrd.

Q. What's rrd?

A. There's a 12 page writeup on what rrd is and what ntop does with it. A pdf version is posted at SourceForge in the "Documentation" release. Download it and read it.

Q. Do I gotta?

A. Yup - the pretty pictures won't work in this FAQ.

Q. I'm lazy - What is rrd?

A. RRD stands for "round-robin database". It is a special type of database designed for holding sequences of information over periods of time, without growing in size.

Rrdtool is a tool for manipulating RRDs. The home page for rrdtool is at http://people.ee.ethz.ch/~oetiker/webtools/rrdtool/

What do they do? Well, suppose you want to compute the average of the traffic to your web site for the last fifteen minutes. If you record the data each minute and save it in a traditional database it looks like this:

                10:45   1.00 MB
                10:46   1.02 MB
                10:47   0.27 MB
                ...
                11:00   0.54 MB
                ...

While you have the data to compute the total, the database grows in size forever. And all you really need are the last 15 values. Certainly you can create a purge routine and periodically remove the old data. But this type of constantly growing SQL database - even with a prune process - will require reorganization and rebuilds over time.

Suppose you had some kind of data structure where the last value was thrown away each time you added a new one. When it comes time to store the 11:01 value, you overlay the 10:46 value. At any time, you still have the last 15 values. That - slightly simplified - is an RRD. The benefit is that your database never grows in size. The down side is that everything else in your history is gone - if your needs ever change, tough.

The ring buffers are called round-robin archives or RRAs. The RRA actually stores the RATE (bits per second), so the 10:47 value of 0.27 Megabytes is 0.27 * 1024*1024 * 8 / 60 or 37748.736 (bits per second). But it functions just like the rings described above.

ntop uses RRDs to store data over long periods of time. Separate files are created for each counter, in a structure that reflects the interfaces and hosts ntop sees. The specifics of what's recorded - interfaces or not, hosts or not, etc. is controlled by switches on the ntop rrd plugin.

(Read the paper - it goes on from here, with specific details on how ntop uses RRDs).

Q. What's myrrd?

A. ntop includes a frozen and slightly patched version of rrd 1.0.49 in the ntop source tree. This is called myrrd.

rrdtool 1.0.49 was released 08-Aug-2004.

           1.0.50 was released 25-Apr-2005.

rrdtool 1.2.x began to be released in Apr 2005.

Q. What's rrdtool?

A. rrdtool is the packaged program to access the various rrd routines (export, dump, graph) from the command line.

Q. How do I get it?

A. The myrrd version included with ntop doesn't have rrdtool. Assuming you haven't installed an rrdtools package (which ntop ignores), here is how to get rrdtool:

The myrrd version is a frozen copy of rrdtool 1.0.49, so your best bet is to go to the rrdtool home page, http://www.rrdtool.com/ or the download page, http://people.ee.ethz.ch/~oetiker/webtools/rrdtool/pub/ and download rrdtool-1.0.49.tar.gz.

Once you have it, it's pretty standard.

$ tar xfvz rrdtool-1.0.49.tar.gz $ cd rrdtool-1.0.49 $ ./configure

(You can add --prefix=/usr if you want)

$ make

This runs a few minutes.

Now you can do

$ make install

but remember, ntop just ignores this, so why bother? All you really need are the rrdtool executable program (src/rrdtool) and maybe a few of the files in the doc/ directory. Just copy them where you want them and then delete the whold build directory.

Q. Where do I get rrd?

A. Unless you need rrdtool, you should not need to do anything to get rrd (see myrrd, above).

The home page for rrd is http://people.ee.ethz.ch/~oetiker/webtools/rrdtool/

rpm's are available at http://people.ee.ethz.ch/~oetiker/webtools/rrdtool/pub

Q. What about the multi-threaded development version?

A. Stay away.

(UPDATED) I was able to patch ntop to work with 1.2.7+ and experimented with it a little bit.

The binary .rrd file formats are different, so if you try 1.2.x, any new .rrd files which are created are incompatible.

The new version doesn't use freetype, it uses libart, which introduces a different (not better, nor worse) chain of dependencies.

Q: I've enabled the rrd plugin and there's no data ... there are messages in the log:

        RRD call stack:
        argv[0]: rrd_update
        argv[1]: ...rrd/matrix/12.239.98.199/12.239.181.175/pkts.rrd
        argv[2]: 1037289548:1
        rrd_create(...) error: creating '...': No such file or directory
        rrd_update(...) error: opening '...': No such file or directory

A: Create the rrd directory and make sure that the -u userid has read/write
   access to it (typically /usr/share/ntop/rrd).

Q: Still nothing...
A: Remember to activate the plugin.  You will need to configure it, and remember that
   the default configuration does not include per-host data.

Q. What's the difference in the Host Detail Level for RRDs?

A. Low is just the bare counts, pktSent/pktRcvd and bytesSent/bytesRcvd.

Medium adds:

     pktDuplicatedAckSent/pktDuplicatedAckRcvd,
     pktBroadcastSent, bytesBroadcastSent,
     pktMulticastSent, bytesMulticastSent,
     pktMulticastRcvd, bytesMulticastRcvd, 
     bytesSentLoc, bytesSentRem, bytesRcvdLoc, bytesRcvdFromRem, 
     ipBytesSent, ipBytesRcvd, 
     tcpSentLoc, tcpSentRem, tcpRcvdLoc, tcpRcvdFromRem, tcpFragmentsSent, tcpFragmentsRcvd, 
     udpSentLoc, udpSentRem, udpRcvdLoc, udpRcvdFromRem, udpFragmentsSent, udpFragmentsRcvd, 
     icmpSent, icmpRcvd, icmpFragmentsSent, icmpFragmentsRcvd, 
     ipv6Sent, ipv6Rcvd

    NonIP:

     stpSent, stpRcvd,
     ipxSent, ipxRcvd,
     osiSent, osiRcvd,
     dlcSent, dlcRcvd,
     arp_rarpSent, arp_rarpRcvd, arpReqPktsSent, arpReplyPktsSent, arpReplyPktsRcvd,
     decnetSent, decnetRcvd,
     appletalkSent, appletalkRcvd,
     netbiosSent, netbiosRcvd,
     otherSent, otherRcvd

    And the per-protocol Sent/Rcvd

High adds:

     totContactedSentPeers, totContactedRcvdPeers

    And the per-IP-protocol Sent/Rcvd, e.g. IP_HTTP...

Q. rrdPlugin - problem with rrd/myrrd?

A. By default, ntop's Makefile binds the static libmyrrd library to create ntop's rrdPlugin shared library. THIS IS DELIBERATE so that you use the myrrd library and not some other version of rrd that's installed somewhere else on your system.

Sometimes this causes problems, where there are special tricks required to tell (a non gnu ld) loader about static (.a) libraries, which ntop doesn't have in the Makefile nor the configureextra files.

As a SHORT TERM WORK-AROUND, you can TRY this:

      $ cd myrrd

Edit Makefile --

       all: $(LIBRRD) libmyrrd.so
                      ^^^^^^^^^^^ add this

Add these lines:

       libmyrrd.so: $(OBJECTS)
       ld -shared -o libmyrrd.so $(OBJECTS)

Now do make. You should see a libmyrrd.so file.

The main 'make' should now complete. Copy that libmyrrd.so file where the other ntop library files are, and it MIGHT work.

However, the whole idea behind having a static libmyrrd.a is to prevent version conflicts and use a stable version of rrd.

The right fix is to get the configureextra/ file changed.

netFlow and sFlow

Q. How do I access netFlow or sFlow data from ntop?

A. You need to configure ntop as a listener.

First, use the appropriate plugin to set the parameters - basically the port you want ntop to listen on. Then, using the Admin | Set Interface menu item, switch ntop to report on the sFlow/netFlow pseudo-device (NetFlow-device or sFlow-device).

Q. Can I use ntop as a netflow collector.

A. Not in the current versions - you used to be able, but that code was removed.

netFlow

Q. Which versions of netFlow.

A. v5 And v1/v7/v9 - in that internally a v1/v7/v9 flow is copied to a v5 buffer and then processed. We default/ignore fields that are different. And nFlow - similar conversion.

Q. netFlow doesn't work.

A. You MUST make sure the ntop plugin is ACTIVE. With the change to allow setting parameters while inactive, it's easy to miss that last step. If you don't activate the plugin, you'll still have the netflow-device, but no data on it...

Q. What's Virtual NetFlow Interface?

A. Be sure and set it. It's important for pseudo-local classification, which affects L R reporting. You need to set it to the (network) and mask for the netFlow collector. So ntop knows 'where' the data is coming from.

Q. 'splain some more, Lucy...

A. OK.

It's best to think of netFlow like this:

The physical interface which is monitoring the packets is like a temperature probe you stick into a roast.

Even though the display of the data can be right there at the probe, or the other end of a (long) wire, or somewhere entirely elsewhere via a wireless connection, the probe is monitoring at the tip. If it says 145F, that's the temperature of the meat - not the oven and not the kitchen.

Similarly, the netFlow data ntop is receiving is based on the probe location.

So, if you have a router and are monitoring a single interface to collect netFlow data, then the ip address you want to give to ntop is that of the router interface.

If you are monitoring a router with more than one interface, you will need to give ntop ONE of those addresses and use the -m | --local-subnets option to tell it that the other addresses are also local.

Q. Where is info about netflow?

A. Dale Reed pointed out a good tech doc (no flak, just the formats) for netflow

    V1/5/7:

http://www.cisco.com/univercd/cc/td/doc/product/rtrmgmt/nfc/nfc_2_0/nfc_ug/nfcform.htm

(As of Oct2003, it includes v8 and is here:)

http://www.cisco.com/univercd/cc/td/doc/product/rtrmgmt/nfc/nfc_3_0/nfc_ug/nfcform.htm

Q. How Do I Enable NetFlow Data Export on a Cisco Device?

A. To enable netFlow Data Export (NDE) from a Cisco device to an ntop netFlow receiver on port 2055 (default) at address 10.1.1.1:

     ip flow-export destination 10.1.1.1 2055
     ip flow-export version 5

You may want to designate the source interface, e.g.:

     ip flow-export source Ethernet0

Enable netFlow on each interface to be monitored. netFlow normally only captures data from each incoming packet, so to see traffic in both directions netFlow must be enabled on both the incoming and outgoing interfaces. As an example, for an Internet access router this would mean enabling netFlow on both the internal (e.g. Ethernet) and the external (e.g. ISDN / Frame Relay etc) interfaces:

     interface Ethernet0
     ip route-cache flow

     interface Dialer1
     ip route-cache flow

By default netFlow will only export flow statistics shortly after the flow Terminates or when 30 minutes have elapsed. In many environments, you want ntop to be a bit more up to date. To change the timeout to five minutes:

     ip flow-cache timeout active 5

The following 'show' commands are useful for examining netFlow statistics directly on the Cisco box and may assist when setting up ntop:

     show ip flow export
     show ip cache flow
     show ip cache verbose flow

Obviously, there is a lot more to it than this, for more information, see the Cisco web site: http://www.cisco.com/go/netflow

                                     (Created by sholmes at snapshot, 02Feb2003)

Q. Can I run ONLY w/ netFlow (Running ntop as a Collector for Net Flow only)

A. Sure.

ntop is usually configured to capture all traffic from local interfaces. If no interface is given (e.g. option -i|--interface is missing), the "first" interface is taken. This value, typically eth0 on linux boxes, may not be what you wanted.

If you want to use ntop as a collector for Net Flow traffic only, you may want to supress all local traffic.

In this case use -i none or --interface none.

Remember to activate Net Flow plugin from Menu Admin->Plugin and to configure the plugin by setting "Local Collector UDP Port".

sFlow

Q. What is sFlow

A. The core component of the sFlow toolkit is the sflowtool command line utility. sflowtool interfaces to utilities such as tcpdump, ntop and Snort for detailed packet tracing and analysis, NetFlow compatible collectors for IP flow accounting, and provides text based output that can be used in scripts to provide customized analysis and reporting and for integrating with other tools such as MRTG or rrdtool.

Some info:

http://www.inmon.com/sflowTools.htm http://www.faqs.org/rfcs/rfc3176.html

Q. I have activated the sFlow plugin in ntop. But it doesn't seem to generate any output based on the collected sflow datagrams.

A. sFlow can be a collector or a receiver or both, depending on the settings configured via the plugin.

If you configure ntop as an sFlow collector, it will use sFlow data for generating reports, treating the remote collector(s) as another network interface - see Admin | Switch NIC.

Q. sFlow doesn't work.

A. Check this out:

     This talks about a bad experience I had setting up sFlow reception. For the longest
     time, I could see that ntop was getting sflow packets, but no data would show up. 
     It turns out the switch I was exporting from didn't see any real traffic, and it was 
     just sending COUNTERSAMPLE packets.....

     - - - - - - - - I figured out that it was indeed "invalid" sflow packets.

     Apparently, sflow sends COUNTERSAMPLE and FLOWSAMPLE packets. COUNTERSAMPLE packets 
     give a quick look at interface counters on the machine, whereas FLOWSAMPLE packets 
     are actual packet fragments from IP connections. Ntop seems to simply parse, 
     debug_print, and discard COUNTERSAMPLE packets...which made it confusing to look at 
     the debug output and say "wow, lots of sflow coming in!" when in fact it was just for
     show, as Burton suggested. I added more switches (with active connections) to the
     switches sending sflow packets and I now have hosts with pretty graphs.

Solaris

Q. How do I install the ntop package on Solaris?

A. For instance do 'pkgadd -d ntop-2.2-solaris.i386'

Q. What c compiler do I need?

A. Sun's cc or gnu's gcc.

Q. What about /usr/ucb/cc is that the one?

A. No. cc is software you pay for. /usr/ucb/cc is a stub compiler and good for, recompiling the kernel and absolutely nothing else.

The cc I mean is Sun's Commercial Compiler.

BSD - FreeBSD

                                        During the ntop 3.2 development cycle,
                                        we did development/testing under:
                                           4.10 and 4.11
                                           5.3 and 5.4

Q. When I type 'make' it complains about a makefile error.

A. Always remember to use gnu make.

On many *BSD systems which have other 'make's, gnu make is called gmake. Try make --version -- if it shows a Gnu version stamp you're ok, otherwise try gmake.

Q. I get "ntop: /dev/bpf0: Device not configured", what's wrong?

A. This is because bfpX has not been configured inside the generic bsd-kernel config file.

If you use generic kernel config file put "pseudo-device bpfilter 16" in kernel config file and rebuild the kernel.

Q. I remember rumors about something not being right under FreeBSD with threads?

A. Yes. See FreeBSD bug bin/17437 at http://www.FreeBSD.org/cgi/query-pr.cgi?pr=17437

Basically, due to limits in FreeBSD, there is no pthread_atfork() function. So, when ntop does it's fork() call to create http pages, it can't fixup the Mutexes. It wrong and could conceivably cause problems.

However - ntop ran for years without the pthread_atfork() code, so we're no worse off in 3.0 under FreeBSD than in 2.2 or 2.1...

Q. The web server problem?

A. There was a flurry of problems late in the 3.0 development cycle having to do with a seeming deadlock of the ntop web server (it's actually not dead, just walking at about 0.001KPH).

Thanks to Yeoman efforts by Stanley Hopcroft, Michal Meloun and, well, me, we have a work-around.

With 3.1 we tried to automate this workaround, but fell a foul of FreeBSD's fixes. So in 3.2 we've reverted to requiring the command line flag.

If you're running under FreeBSD and have problems, use the flag, --set-pcap-nonblocking.

For more on this, read the threads at gmane - look for "FreeBSD and pthreads" - that's probably the best summary. But there's stuff on this back at least to October 2003 - look for Stanley's problems with CPU usage.

A. Also, understand that --set-pcap-nonblocking is going to increase ntop's cpu usage. It will probably come close to pegging the CPU at 100%. Yet strangely other processes won't seem to be impacted. (Of course, you really should be running ntop on it's own host, anyway, right?).

This is because of how the work-around is coded. ntop should step aside briefly and let them run.

Just be aware of it and don't ask on the mailing lists.

It's not giant lock vs. fine grained, etc., it's really the handling of signals (user interrupts) that is different.

Say you have a process, A running.

In linux, POSIX thread x is kernel process x' and POSIX thread y is kernel process y'. Now the POSIX standard doesn't prevent thread y from receiving a signal 'intended' for x, so you have to code for that. But when a thread 'sleeps', the process sleeps, which is something the kernel understands and so the kernel can do other work.

In a userland threads model (i.e. xBSD), POSIX thread x and POSIX thread y are both parts of kernel process A. To implement the POSIX threading, the thread library code creates a thread manager, thread z'. That thread receives all signals, interupts etc. and parcels the work back out.

This too is perfectly within the POSIX model.

BUT: To implement the thread manager implies that the user task (A+x+y+z) is active whenever it's waiting for ANYTHING. How well this works depends upon the sophistication of kernel processes that recognize when the user task is 'idle', meaning it is spinning waiting for somethings, vs. doing useful work.

In FreeBSD 4.x, we see that this recognition is not very sophisticated when it comes to the bpf pseudo driver and so ntop will use 100% of the available CPU. Even so, the scheduler bounces between tasks of equal priority, so productive work does get done, but you see the high CPU usage charged to your task.

Linux - all

                                        During the ntop 3.2 development cycle,
                                        we did development and/or testing under:
                                           Gentoo 2005.0
                                           Fedora Core 2 and 3

                                        The cvs version was run by users on many
                                        other versions.

                                        Others should certainly work and there
                                        are many user reports of success.

Linux - RedHat

Q. "application bug: ntop(...) has SIGCHLD set to SIG_IGN but calls wait(). (See the NOTES section of 'man 2 wait'). Workaround activated."

This message and the NOTES section of the man page lead me to believe that the problem is handled but the kernel feels the need to report it from time to time.

A. Read the NOTES section. The problem has been handled, but the kernel feels the need to report it from time to time. See the article at http://article.gmane.org/gmane.linux.ntop.general/5304

Q. libpng version conflicts?

A. See libpng, below.

Win32 - Common

Q. Where can I find GDBM for Windows?

A. GDBM for windows can be found at http://www.roth.net/libs/gdbm/

Q. ntop -i1 ... doesn't work

A. ntop has special parameters under Win32

         Under win32 there are TWO COMPLETELY SEPARATE TYPES OF PARAMETERS.

     There are the parameters to the win32 stub AND there are parameters to ntop
     itself.

     AFTER THE win32 parameters are the ntop parameters in the standard (Unix)
     -xxx format.

         ntop /c   runs ntop INTERACTIVELY with the specified ntop
         parameters

         ntop /i  installs ntop as a service to run with the specified parameters

         ntop /r removes the ntop service

Remember, ntop /i and ntop /d don't actually run the service - you need to start it.

REMEMBER: there are TWO ways to run ntop, one as a service, one 'interactively' (/c).

They are totally separate. Just because you ran ntop w/ interactively with some parameter set, does not affect the stored parameter set for the service.

To change the stored parameters, reinstall the service vi /i.

Q. How do I figure out what my network interface numbers are for the -I parameter?

A. (Thanks to jac engel [jacengel@home.nl] for the example)

If you only have ONE network interface, it doesn't matter as the default is fine. However, that's the RARE case. Most people have multiple network interfaces (NICs), with virtual ones for VPNs, Dialup Networking, etc.

The Windows tools ipconfig, winipcfg and the Device Manager (depending on which version of Windows you have) will probably show you them. However, it's easier and better to use ntop to show you how ntop sees the network interfaces.

If you start ntop /c (interactive mode, with only the default parameters) it Will display all your network interfaces (NICs), like this:

    Running ntop for Win32.
    Wait please: ntop is coming up...
    23/Aug/2002 20:43:55 Initializing IP services...
    23/Aug/2002 20:43:55 Initializing GDBM...
    23/Aug/2002 20:43:55 Initializing network devices...
    23/Aug/2002 20:43:55 Found interface [index=0] '\Device\Packet_{14...1C}'
    23/Aug/2002 20:43:55 Found interface [index=1] '\Device\Packet_{86...B4}'
    23/Aug/2002 20:43:55 Found interface [index=2] '\Device\Packet_NdisWanIp'
    23/Aug/2002 20:43:57 ntop v.2.1 MT [WinNT/2K/XP] (11/07/2002 build)
    23/Aug/2002 20:43:57 Listening on [3F1C}\Device\Packet_{14...1C}

  By default, ntop will use the lowest numbered interface.  Because #s are
  assigned based on the sequence cards are discovered, and this is altered if
  cards are removed and added, this is often not what you want.

  After you figure out which NIC you want, start ntop /c -i1 or -i2 or
  whatever...

Q. OK, but how to I translate \Device\Packet_xxxxx to my Froboz ModelT network card and not the Fubar27 that's on the motherboard.

A. ntop should report both the index and the human readable information.

A. A Google search on script "CurrentVersion\NetworkCards" finds a couple of scripts/utilities that might work in various environments.

A. Otherwise...

You're going to need to view the registry. All the usual warnings - back up your pc, etc.

If you damage the registry, you may not be able to reboot the computer.

You're not going to CHANGE anything, but an inadvertent keystroke could be disaster ... BE CAREFUL!

Under WinNT/2K, to find the interface name of your NIC look in the registry at the keys in

     HKLM\Software\Microsoft\Windows NT\Currentversion\NetworkCards\
   The two subkeys, Servicename and the Description tells you which id maps to
   which NIC.

Q. Where does ntop look for html (and gif) files under Win32?

A. ntop looks in two places. The first is the current directory and the second is configurable through a constant in ntop_win32.h, #define DATAFILE_DIR "."

Note that the current directory, or ".", may not be what you expect.

When running ntop as a Win32 service, "." is %SystemRoot%\system32, meaning that ntop looks in %SystemRoot%\system32\html for the .html and .gif files.

When running ntop from the command line,

       ntop /c parameters...

"." is whatever directory is current. This means that if you run ntop with a full, explicit path (c:\ntopnew\ntop /c ...) there may be an unexpected difference between what ntop finds for "." and what you THINK "." is! This will lead to missing .html and .gif files.

If you wish to have ntop look in a specific place for the files, the best choices are:

     1) Create a .bat file to run ntop which does a cd to the expected directory
        first.
     2) Edit ntop_win32.c and then recompile.

Note that the settings for DATAFILE_DIR (and other constants) are reported on the text version of the configuration page, textinfo.html.

Win32 (MinGW) (Windows)

Q. What's the scoop with ntop on Windows?

A. Semi-officially,

ntop for Windows 95/98/ME/NT/2K is also provided as a binary application with limited capture capability (1000 packets). This is intended to allow demonstration of ntop for people without access to a Unix system.

We call this version Win32 after the old official name of the Windows library.

If you want to use the full version with unlimited packet capture you can either:

       * Recompile ntop from the source by yourself (Luca says just open the
         files in MS Visual C++ 6.0 and press compile)
       * Register your ntop for Windows 95/98/NT/2K copy by paying a 
         convenience fee to receive the prebuilt executable.

If you decide to register your copy, Luca will send you an URL from which you can download the full version periodically.

Q. So where did MinGW come in?

A. All of the necessary open software tools have been ported to run under windows (sort of), so it is theoretically possible to build and run ntop under Windows.

However, Windows and *nix are very, very different internally. So we have to use special versions of the packet capture library (winpcap instead of libpcap). And we needed our standard tools (gcc et al), plus some 'glue' so we could make *nix calls and have Windows things happen.

For the tools and glue, there are three choices: native, Cygwin and MinGW.

Native means you put lots of #ifdefs in to make Windows calls where you had *nix calls.

Cygwin is a shim - it's a Windows dll (Dynamic Link Library) that pretends to support the *nix calls and then does the right Windows things for you.

MinGW is a project to create native Windows tools and executables from *nix code.

Each of these have limits, pluses and minuses. For example, Cygwin's dll has caused all sorts of problems (dlls and their *nix equivalent, shared libraries, usually do). MinGW has some limits. As MinGW grew, ntop for Win32(MinGW) got closer to ntop under *nix. Native means supporting two 'separate' code bases.

Luca actually picked a hybrid - he uses Microsoft Visual C++ 6.0 - which has it's own (albeit incomplete) shim layer. As long as you stay within the limits of the shim, the same code works across platforms. In other places you have to make thinks like Windows wants them. How close are we? grep for WIN32.

The advantage of the hybrid was that is was also pretty close to working under MinGW - creating native executables from the base code using free tools. So people sent in patches and it pretty much worked. Up until 2.1.3, that is. But it was never officially more than an 'it also runs'.

After 2.1.3, Luca embarked on a process of bringing the WIN32 and *nix code closer together. Surprisingly, this actually broke MinGW!

Q. Where do I get MinGW?

A. The MinGW home page is http://www.mingw.org. Quoting:

      "MinGW is a collection of header files and import libraries that 
      allow one to use GCC and produce native Windows32 programs that do
      not rely on any 3rd-party DLLs. The current set of tools include 
      GNU Compiler Collection (GCC), GNU Binary Utilities (Binutils), 
      GNU debugger (Gdb) , GNU make, and a assorted other utilities. We
      are currently working on creating a complete set of Mingw-hosted
      GNU toolchain, and looking for volunteers."

Q. Does ntop work under Cygwin?

A. The .exe distributed through ntop.org is built with Visual C++ 6.0. It proved just barely possible to use the same code under MinGW. Forget about cygwin.

Q. Does ntop (v3.2) work under MinGW?

A. Maybe (I think) - During the 3.1 development cycle changes were made to support MinGW and it should again work - see docs/BUILD-MinGW.txt

Win32 (MS Visual C++)

Libpng

Q. Bad things - I see the following messages:

        libpng warning: Application was compiled with png.h from libpng-1.0.x
        libpng warning: Application is running with png.c from libpng-1.2.x 
        gd-png: fatal libpng error: Incompatible libpng version in application
              and library

A. You have a version problem with libpng.

First off, following the instructions in BUILD-NTOP.txt should work just fine. These problems come about when you have libpng installed (i.e. using shared libraries).

1. If you are compiling from source, you may have png.h left over from the

      earlier version of libpng. Remove it.

2. (Most common under RedHat). RedHat 7.2 installs a libgd.so.1.8.4 library,

       which was compiled against 1.0.x series of libpng (which is fine, because
       RedHat 7.2 includes libpng-1.0.12).

Updating RedHat to newer (RawHide) packages for libpng, http://www.rpmfind.net//linux/RPM/rawhide/1.0/i386/RedHat/RPMS/libpng-1.2.2-5.i386.html, should work. However, there are reports of version conflicts and required updates to multiple packages. Proceed with caution (especially if you decide to uninstall 1.2.2-5). Also, do not use --nodeps or --force, as this can leave you with two partially installed versions (see item #1, above).

3. (Slackware) Users have reported this error from an older header file in

      /usr/include.
   Make sure to run "make install" in the libpng directory so that the latest
   files are in the common library locations. You can do this with buildAll.sh,
   just navigate back down to the libpng-1.2.x directory first.

4. If you are building ntop on one machine and running on another, they may

      have different libpng.so versions.  Even if you think you are using the
      static linked version (buildAll.sh), be careful - see the entry (above) on
      "make install" for libpng.

Q. When I run ./configure, it finds png.h but not libpng:

      *******************************************************************
      *
      * ERROR: libpng header or library routines are missing
      *           (yes means it was found, no means it was not found)
      *
      *              png.h...yes
      *              png_read_info() in -lpng...
      *
      *>>> No way to proceed.
      *
      *???        Install libpng (and/or libpng-devel), check www.libpng.org
      *???    and Rerun ./configure
      *
      *******************************************************************

A. You're missing libpng.so. Look for it (locate libpng) and tell ntop where it is via the --with-png-lib= parameter.

Q. I seem to be missing libpng.so - when I do locate libpng, it finds:

      /usr/lib/libpng.so.3
      /usr/lib/libpng.so.3.1.2.2
      /usr/lib/libpng12.so.0
      /usr/lib/libpng12.so.0.1.2.2
      /usr/lib/libpng.so.2.1.0.13
   but no libpng.so...

A. Yeah. Send RedHat a nasty gram.

Q. I don't understand...

A. In a normal libpng install, say from the source, you would have - in addition to the .so.n.n.n.n files - a symlink named libpng.so, like this:

      lrwxrwxrwx 1 root  root   19 Apr 23 15:46 libpng.so -> libpng12.so.0.1.2.2

But, that link seems to be missing. Without it, -lpng doesn't properly resolve and you get the ./configure error.

Q. Why?

A. RedHat ships Linux with both versions of libpng, a 1.0.x and a 1.2.x version.

Do this:

      $ rpm -qa | grep libpng

And you'll see the libpng 1.0.x run time:

      libpng-1.2.2-8
      libpng-devel-1.2.2-8
      libpng10-1.0.13-6

Dig into the files installed by them and you'll not find libpng.so.

Since they're incompatible RedHat doesn't create the libpng.so link. Instead they patch the makefiles to point the various packages at one or the other .so file they install. This allows them to ship packages that require one or the other.

It works fine unless you try and install something other than via an rpm. Then you're missing the libpng.so file that normal packages look for...

Best bet is to create a symbolic link from the libpng.so.xxxx installed by the package which matches the -devel (because that's where png.h is found), e.g.:

ln -s /usr/lib/libpng.so.3.1.2.2 /usr/lib/libpng.so

And remember this in case you update the libpng rpm's in the future.

Silly Season

Q. Who is Pixel

A. My cat.

HowTo Ask For Help (ntop mailing lists)

WTO ask for help on the ntop or ntop-dev mailing lists:

WHERE TO POST

ntop is for user questions - "How to I install", "data isn't being recorded",
etc.

ntop-dev is for code and development questions.  The ntop-dev list goes to fewer
people, those who have self-selected themselves to be interested in ntop at the
code level.

ntop-misc is for other products - nProbe, nBox, PF_RING, etc.

Discussions about the current cvs version belong in ntop-dev.

If a discussion gets too technical, you may be asked to "move it to ntop-dev".
Please honor that request (even if you have to subscribe for a while - ntop-dev
is fairly low traffic).

There used to be mailing lists and trackers at SourceForge, which were rarely
looked at and have been discontinued. Use the ntop and ntop-dev lists (go to
http://www.ntop.org to signup for them).

OFFICIAL vs UNOFFICIAL

A response from Luca Deri should be considered official.  He is the author of
ntop and controls the project and it's destiny.

***Please understand that the mailing lists are a community support effort***

Besides Luca Deri, a number of people answer questions to the best of our
ability. None of the rest of the people who may respond to your question on ntop
or ntop-dev are able to respond "officially".

Likewise, this HOWTO is unofficial.

Everyone is welcome to help with the evolution of ntop - that is to find
problems, create and test patches and send them in to patches@ntop.org for
inclusion.  There are a small number of people with write access to the cvs,
but anything we commit is subject to being ripped out by Luca for any reason...
or no reason at all...

QUESTION FORMATS

ONE QUESTION per message, and you MUST use meaningful message subjects - one's
that would have helped YOU find the prior discussion of this or a similar
problem in the archives.  Titles such as "urgent" or "ntop problem" will often
not get a response - it may be urgent to you but it's probably not an issue
for others.

WE STRONGLY SUGGEST YOU USE THE BUILT IN AUTOMATICALLY GENERATED PROBLEM
REPORT (About | "bug icon").  This includes most of the internal configuration
data we ask for (and more) and has blank spots for you to fill it.

Generate one, cut & paste into your mail client, edit the data and sent it.

Beyond that, don't worry -- it's about information, not format.

RESPONSES

Despite any individual's frequent postings, nobody is "responsible" for
answering your question. It's all on a "best efforts" basis. Our responses may 
be incomplete, inaccurate, even dead wrong. Caveat Emptor! The only "guarantee" 
is that free support will be worth what you've paid for it.  It may be worth MORE, 
it won't be worth LESS.

Just because you post a question does NOT mean that you are OWED an answer.

If nobody answers, then maybe it's because:

* Nobody knows. * People are busy. * You've asked the same question multiple times and it's already been

     answered.
   * You have been asked for additional information and are unable/unwilling
     to supply it.

or, well, any one of a dozen other reasons.

Asking the same question multiple times - or asking it again because you don't
like the answer you received - is a slap in the face of the person who took the
time to answer you in the first place and will more than likely not get a
different response.  If you're not sure that your message posted, check the
archives to see if your message is there -- please don't just keep reposting it.

You can always use gmane (http://www.gmane.org) to see the last 600 or so
postings to the lists.

Please direct all original postings and subsequent replies to the list, not to
someone privately.  Most of us will reply solely to the mailing list, unless
you specifically request otherwise.  If you do request otherwise, the individual
you sent it to may choose not to respond.  Our posting here is NOT a public
invitation to invade our e-mail boxes for your free private support.

THE BACK AND FORTH PROCESS

"Why don't you just fix my problem instead of asking for more information?"

Understand that we can't see your machine (and wouldn't want the
responsibility of sshing into somebody else's box as root).  The only
information we have is what you post and the responses to our questions.  Few
failures in ntop are related to the core processing routines - so if you're
having a problem, it's most likely because of some combination of your network
and your ntop configuration.  It may be unique to you -- and only with YOUR help
can it be resolved.

WHAT IS SUPPORTED?

Releases are hosted at SourceForge.

At the time of this writing the stable version is 3.0. What support is available
is for the development version ("the cvs").  All support is in the form of
fixing things in the cvs.

However we also attempt to support the current "stable" release (3.0).

Older versions are not supported -- especially 1.3 and the 2.0.99 series of
2.1 release candidates.  If you have a problem with them, please obtain the
current cvs version and see if it's still a problem.  Unlike certain much larger
projects, we don't fix things in older versions - there simply aren't enough
resources available.

intop is not supported.  It's gone and no longer in the cvs.  For a look at
Rocco's new, work-in-progress, download ntcsh, the ntop enabled tcsh shell.

Please understand that the only way to fix your problem may be a source patch,
which you will have to apply, compile, install and test against the cvs version
prior to it's inclusion in the cvs.

If you aren't capable of or willing to do these steps -- for whatever reason --
then you should not be compiling from the cvs.

CVS

The cvs is at http://cvs.ntop.org, userid is anonymous, password ntop.

The cvs is a DEVELOPMENT version.  The code in the cvs is subject to rapid
change.  At any point in time, it may not compile.  It may not compile with
certain options or on some platforms. s'be'it -- it's a DEVELOPMENT version.

3.1

The actual flow of ntop development was 3.0.50.. ->
                                    -> 3.0.50.. -> 3.1rc1 -> 3.1

3.2

The actual flow of ntop development was 3.1.1.. -> etc (I think .4 was the last Win32 version),
                                    -> 3.1.50.. -> 3.1.51 -> 3.2rc1 -> 3.2rc2 -> 3.2

FEE-BASED SUPPORT

If you want better than "best-efforts" support, contact the individual you
Desire support from off-list to make financial arrangements.  Please understand
that people are doing development in areas that are of personal interest to
them, to improve ntop.
If you want to discuss payment for support or a specific change that is of
Interest to you, feel free to email the individual off-list - some of us are
Computer consultants and can be bought, with the understanding that the work
product is offered back to the community in the spirit of the open source
movement and the strictures of the GPL.

SO WHAT INFORMATION SHOULD I POST?

BEFORE POSTING:

1. Please review the output from ./configure.

    We all have the bad habit of skipping over this, but there are often
    warnings which explain why things don't work.  ntop tries to build itself by
    turning off features where the required libraries and/or headers aren't
    available.
    The minimum required set is just that - minimal.  This is often the source
    of "feature x or switch y doesn't work" reports.

2. Please review the docs/FAQ file.

3. Please review back message traffic from the mailing lists.

     Yes, we know that there isn't a search function at ntop.org.  Did you know
     that the lists are spidered every couple of months or so and can be
     searched through Google??  For example, "site:lists.ntop.org rpm" will find
     mail list messages with the word "rpm" in them.

     Do you know about gmane (http://www.gmane.org) has archives (searchable) of
     the ntop lists going back into late 2001.  The lists are called

          gmane.linux.ntop.devel
          gmane.linux.ntop.general

     You can read these online (the last 600 messages or so) or through the nntp
     server.

POSTING:

Do not worry about posting TOO much information - we're pretty good at filtering
out the noise.

WE STRONGLY SUGGEST YOU USE THE BUILT IN AUTOMATICALLY GENERATED PROBLEM
REPORT (About | "bug icon").  This includes most of the internal configuration
data we ask for (and more) and has blank spots for you to fill it.

Generate one, cut & paste into your mail client, edit the data and sent it.

If you can't or won't use the automated problem report (say, for example you
can't get ntop up to generate it) - don't worry -- it's about information, not
format.  Send us what you can, organized this way:

1. A brief summary of the problem.

2. Operations
     The EXACT command line you use to invoke ntop.
          If it's in a script, cut & paste it and
          resolve all the variables!

     Error Messages:  Cut & paste the exact text.
          If it's in the log, give us 15 or 20 lines before.

     The exact URL you used from the browser.

3.  Software
     ntop version, source and any applied patches

            If you've compiled from the source, say so!

            If you're using a package (such as an .rpm), where
                did you get it from and what is the EXACT name,
                version information and date? (for example, post
                the output from rpm -q ntop -i)

     OS vendor & version

     gcc version (e.g. gcc --version)
       (For ./configure problems, the versions for autoconf, automake and
        libtool too)

     glibc version

     Any major upgrades (kernel, networking, etc.)

     What else is running

4. Hardware
     Type & # of processors

     Amount of memory

     # network interfaces and types (vendor, bus, etc.)

5.  Network
     Roughly where are the interface(s) you're monitoring
          (Public Internet, Private LAN, what?)

     What's the bandwidth (e.g. 10 Mbps University internet,
           1.5 Mbps T1, Cable Modem capped at 1.5Mbps, 56K dialup)

     How many machines (traffic sources/destinations) and users

(If you're uncomfortable giving specifics, then leave it generic, but the
information is necessary to allow efficient use of the community's time helping
YOU with YOUR problem)

AFTER POSTING:

Please let us know if our help fixed the problem, didn't solve it or enabled you
to solve it yourself and what the result was.  The historical record of the ntop
and ntop-dev archives is the complete chain from problem to resolution.

(Originally posted on 07Jan2002 to ntop and ntop-dev, updated.  This is version 12April2003)

GDB ultraMini-tutorial - Running ntop under gdb (debugger)

or, 'capturing the failure point'

The very best way to debug a segmentation fault in ntop is to use gdb. The
Standard ntop compile already has the flags necessary to do this set.

(Note - if you don't have gdb, or aren't compiling yourself, this won't work)

> gdb /usr/bin/ntop (or wherever ntop is installed)
...
(gdb) set args (your usual arg string) -K

[That is, add the -K argument. While you are at it, don't give it the -d
argument and add -u root (replace any existing -u value) - yes, it's insecure
running as root, but you're not planning on doing this in production nor as a
routine situation!]

it will run... when it bombs...

"bt full" does a decent job of printing the stack and the back trace and the
local variables at each level.

Just make sure you are in the thread you are interested in (i.e. DO THIS FIRST)

(gdb) bt full
#0 0x40592557 in __libc_pause () from /lib/i686/libc.so.6
No locals.
#1 0x4046b5a3 in pause () at wrapsyscall.c:123
result = -1073743680
oldtype = 0
#2 0x0804ac1b in main (argc=22, argv=0xbffffa44) at main.c:928
argc = -1073743680
argv = (char **) 0x0
i = 0
userSpecified = 1
ifStr = "eth0,eth1", '\000' 
lastTime = 1025633918
#3 0x404f3647 in __libc_start_main (main=0x804a74c , argc=22, ubp_av=0xbffffa44,
init=0x8049600 <_init>, fini=0x804d000 <_fini>, rtld_fini=0x4000dcd4 <_dl_fini>,
stack_end=0xbffffa3c) at ../sysdeps/generic/libc-start.c:129
ubp_av = (char **) 0xbffffa44
fini = (void (*)()) 0x40016b4c <_dl_debug_mask>
rtld_fini = (void (*)()) 0xbffff87c
ubp_ev = (char **) 0xbffffaa0
(gdb)

Now, continue:

if there are any seemingly interesting variables involved, you can print them:

(gdb) print deviceId

[gdb can handle pretty complex arguments in the print command, so you can say
"print myGlobals.device[0].hash_hostTraffic[myGlobals.broadcastEntryIdx]"
if that's what it bombed on.]

Sometimes gdb won't find a variable, that's because C used a register. S'be'it.

Other interesting things:

(gdb) list [this shows where in the code it died]

(gdb) info threads [this shows the status of the multiple threads]

(gdb) info stack [this shows the call stack]

If you want, you can check each of the threads and THEIR status via:

(gdb) thread 1 [or any valid number]
(gdb) info stack

gdb has all kinds of included help, and frankly that's all I know...
Here are some extracts and examples:

(gdb) help info thread
IDs of currently known threads.

(gdb) info thread
  13 Thread 11276 (LWP 8967)  0x405f651e in __select () from
/lib/i686/libc.so.6
  12 Thread 10251 (LWP 8966)  0x405f651e in __select () from
/lib/i686/libc.so.6
  11 Thread 9226 (LWP 8965)  0x405f651e in __select () from
/lib/i686/libc.so.6
  10 Thread 8201 (LWP 8964)  0x405f651e in __select () from
/lib/i686/libc.so.6
  9 Thread 7176 (LWP 8963)  0x405f651e in __select () from
/lib/i686/libc.so.6
  8 Thread 6151 (LWP 8962)  0x405ca071 in __libc_nanosleep () from
/lib/i686/libc.so.6
  7 Thread 5126 (LWP 8961)  0x4053db85 in __sigsuspend (set=0x4300998c)
    at ../sysdeps/unix/sysv/linux/sigsuspend.c:45
  6 Thread 4101 (LWP 8960)  0x405ca071 in __libc_nanosleep () from
/lib/i686/libc.so.6
  5 Thread 3076 (LWP 8959)  0x405ca071 in __libc_nanosleep () from
/lib/i686/libc.so.6
  4 Thread 2051 (LWP 8958)  0x405ca071 in __libc_nanosleep () from
/lib/i686/libc.so.6
  3 Thread 1026 (LWP 8957)  0x4053db85 in __sigsuspend (set=0x4100967c)
    at ../sysdeps/unix/sysv/linux/sigsuspend.c:45
  2 Thread 2049 (LWP 8956)  0x405f4e17 in __poll (fds=0x823dbec, nfds=1,
timeout=2000)
    at ../sysdeps/unix/sysv/linux/poll.c:63
* 1 Thread 1024 (LWP 8952)  0x405ca027 in __libc_pause () from
/lib/i686/libc.so.6

The * indicates the current thread.  Switch with this:

(gdb) thread 7

Once you have a thread selected, you can look at the call stack:

(gdb) help stack
Examining the stack.
The stack is made up of stack frames.  Gdb assigns numbers to stack frames
counting from zero for the innermost (currently executing) frame.

At any time gdb identifies one frame as the "selected" frame.
Variable lookups are done with respect to the selected frame.
When the program being debugged stops, gdb selects the innermost frame.
The commands below can be used to select other frames by number or address.

List of commands:

backtrace -- Print backtrace of all stack frames
bt -- Print backtrace of all stack frames
down -- Select and print stack frame called by this one
frame -- Select and print a stack frame
return -- Make selected stack frame return to its caller
select-frame -- Select a stack frame without printing anything
up -- Select and print stack frame that called this one

Type "help" followed by command name for full documentation.
Command name abbreviations are allowed if unambiguous.
(gdb) info stack
#0  0x4053db85 in __sigsuspend (set=0x4300998c) at
../sysdeps/unix/sysv/linux/sigsuspend.c:45
#1  0x404db1c9 in __pthread_wait_for_restart_signal (self=0x43009be0) at
pthread.c:969
#2  0x404dc1ec in __new_sem_wait (sem=0x804d5c8) at restart.h:34
#3  0x402960e0 in waitSem (semId=0x804d5c8) at util.c:1126
#4  0x4027ae0f in dequeueAddress (notUsed=0x0) at address.c:425
#5  0x404d8c6f in pthread_start_thread (arg=0x43009be0) at manager.c:284

And select a specific frame via:

(gdb) select-frame 4

(gdb) help list
List specified function or line.
With no argument, lists ten more lines after or around previous listing.
"list -" lists the ten lines before a previous ten-line listing.
One argument specifies a line, and ten lines are listed around that line.
Two arguments with comma between specify starting and ending lines to list.
Lines can be specified in these ways:
  LINENUM, to list around that line in current file,
  FILE:LINENUM, to list around that line in that file,
  FUNCTION, to list around beginning of that function,
  FILE:FUNCTION, to distinguish among like-named static functions.
  *ADDRESS, to list around the line containing that address.
With two args if one is empty it stands for ten lines away from the other
arg.
(gdb) list address.c:425
420
421         while((myGlobals.addressQueueLen == 0)
422               && (myGlobals.capturePackets) /* Courtesy of Wies-Software
 */
423               ) {
424     #ifdef USE_SEMAPHORES
425           waitSem(&myGlobals.queueAddressSem);
426     #else
427           waitCondvar(&myGlobals.queueAddressCondvar);
428     #endif
429           key_data.dptr = data_data.dptr = NULL;

(gdb) help print
Print value of expression EXP.
Variables accessible are those of the lexical environment of the selected
stack frame, plus all those whose scope is global or an entire file.

$NUM gets previous value number NUM.  $ and $$ are the last two values.
$$NUM refers to NUM'th value back from the last one.
Names starting with $ refer to registers (with the values they would have
if the program were to return to the stack frame now selected, restoring
all registers saved by frames farther in) or else to debugger
"convenience" variables (any such name not a known register).
Use assignment expressions to give values to convenience variables.

{TYPE}ADREXP refers to a datum of data type TYPE, located at address ADREXP.
@ is a binary operator for treating consecutive data objects
anywhere in memory as an array.  FOO@NUM gives an array whose first
element is FOO, whose second element is stored in the space following
where FOO is stored, etc.  FOO must be an expression whose value
resides in memory.

EXP may be preceded with /FMT, where FMT is a format letter
but no count or size letter (see "x" command).

(gdb) print key_data
$3 = {dptr = 0x0, dsize = 1076452514}
(gdb) print data_data
$4 = {dptr = 0x0, dsize = 4}

Original version Luca Deri, 1999-2001
Updated Burton M. Strauss III 2002, 2003, 2004, 2005