Apr. 5, 2007 - SurfControl 6.0 Memory Usage / Notes on New Version (6.1) Update - As
of April 5, 2007, my SurfControl database (sys:etc\cpfilter\SurfControl Categories.cdb) files was at
version 1714, and the file size had grown to 790,331kb in size. In order to get this to
load in my BorderManager 3.8 (NW 6.5SP6, BM38SP5) server, I had to preallocate 900mb of RAM in the
CSPCONFIG.INI file AND I had to load NetWare with a -u665000000 parameter in autoexec.bat. I also
had to reduce my NSS cache balance percent to free up RAM, which I did by setting cache balance
percent to 15. My server, which only has to support 1-2 users in my office at any one time, has
just 1.5GB of RAM. This amount of RAM is barely enough to support SurfControl v6.0. Monitor
reports 392,082 original cache buffers, but only 28,000 total cache buffers once SurfControl loads.
So - in order to use the currently-shipping SurfControl, you need:
a. a ton of RAM for it,
b. a -u memory allocation parameter in autoexec.bat, (this value is suggested in NoRM)
c. a memory preallocation value in the 900-1000MB range in CSPCONFIG.INI
Now let's take a look at a new version of SurfControl (v6.1) that I am just starting to beta test.
I downloaded a file from SurfControl (you can probably get it too by emailing them, but I think it
may be released by the end of April). The file is a Windows executable, and running it just goes
through the usual SurfControl installation process. It took my old sys:\etc\cpfilter directory,
and renamed it to sys:\etc\SCV6_1.bak. It then created a new sys:\etc\cpfilter directory, copied
my old registration and cspconfig.ini files into it, and created a new
sys:\etc\SurfControl Categories.cbd file. The new file was only 377,198kb in size, and was at
version 1705. (A word to the wise here - be sure you have enough room on the SYS volume to add
another cpfilter directory and do the updates). I have heard that the database
was completely rewritten to take less memory, and I can believe it, since the new version is at
about the same category list revision as my 6.0 SurfControl at half the database size. I ran
CSP_LIST to update SurfControl to the latest category list, and the update worked, bringing it up to
revision 1715 in about an hour and a half.
Next, I tested to see if I could load the database with a lot less RAM. I removed the -u665000000
parameter completely from my autoexec.bat file. I remarked out the preallocation parameter in the
cspconfig.ini file, and I rebooted the server.
The 6.1 version of SurfControl came up just fine - allocated 524MB of RAM to itself, and did not need
the -u parameter or a cspconfig preallocation. This may change eventually, but it looks very good
for now.
What about new SurfControl features? I was mistakenly thinking additional categories were to be
added with version 6.1, but SurfControl tells me that is not the case. The only change is in
memory management and reduction of the size of the database file.
Nov. 11, 2006 - SurfControl Updates no longing working - As was noted in the March 30,
2006 update, SurfControl has been taking HUGE amounts of RAM to cache the database as it grows
with each update. What started out (in NBM 3.7) as a 250MB database that would never grow up to
512MB has turned into a RAM-gobbling monster, requiring (currently) upwards of 650MB!. If you have
not manually updated the CSPCONFIG.INI file with enough RAM preallocation, at some point your
updates will fail because you do not have enough RAM to hold the updated database in memory.
You can see the last time your SurfControl database was updated by looking at the date of the big
.CDB file in sys:\etc\cpfilter\cpfilter. You can see when the database failed to update by
looking at the end of the cpfilter.log file in the same directory.
I am currently allocating 700mb of RAM for caching SurfControl, and hoping this will be enough for
a while. This effectively means you can no longer run a NW 6.5, BM 3.8 server on only 1GB of RAM -
you will want to have at least 1.25GB, and preferably 2GB (or more). You can see the syntax for
allocating the RAM below.
[SurfControlGeneralList]
CDB_PREALLOC_SIZE=700
March 30, 2006 - SurfControl Updates no longing working - For some time now, the
default size for SurfControl updates has not been enough to allow the updates to complete. Lately,
I see increasing numbers of people who have been setting the limit WAY up there, in the hopes that
we will finally not have to keep creeping the number up. (What happens is that the database gets
so large that one update pushes it past the limit of allocated RAM, and the updates stop at that
point. This is clearly seen in the cpfilter.log file, if you look).
So, if you have plenty of RAM, you might want to just allocate 512MB of RAM for the SurfControl
database in the SYS:ETC\CPFILTER\CSPCONFIG.INI file as shown below.
[SurfControlGeneralList]
CDB_PREALLOC_SIZE=512
July 2, 2005 - SurfControl Updates no longing working - In the past couple of weeks,
the downloaded updates for SurfControl have grown the SurfControl database so large that it no
longer fits into the default memory allocation, and so new updates are not applied. You need to
allocate additional memory in the SYS:ETC\CPFILTER\CSPCONFIG.INI file as shown below. I bumped
mine from 350MB (default) to 400MB. (Go up to at least 360).
[SurfControlGeneralList]
CDB_PREALLOC_SIZE=400
July 29, 2004 - Just made some corrections to verbiage below. The latest version of SurfControl is Ver. 6, not SP3, and it added 10 categories from version 5 (SP2) not 8.
May 15, 2004: SurfControl has released Version 6. Version 6 has 10 additional categories (40 total), now matching the Windows version of SurfControl. The other settings are essentially the
same as Service Pack 2. You should be able to download the latest version, install it one the BorderManager server over the top of the version 2, and just have new categories show up in the access
rules menu. If updating previous versions of SurfControl, you may want to delete the old access rule and replace it, and be careful that you are using the latest NWADMN32 snapins and BorderManager
patches, especially if you run NWADMN32 from another server.
Sept. 24, 2003: SurfControl has released Service Pack 2.
You need to go to www.surfcontrol.com/downloads, and log in to their system. You can then download Service Pack 2 for BorderManager. The download is 63MB in size.
Service Pack 2 Updates:
Applies to BorderManager 3.7, using the SurfControl version of CPFILTER.NLM, not the older CyberPatrol version.
The shipping version of CPFILTER.NLM (installed from CP_SETUP.EXE) has a problem when used in NetWare 6.0. (This does not seem to be a problem in NetWare 5.1). If you run the CSP_LIST.NLM to update the SurfControl categories, huge chunks of RAM disappear for an extended period. I have seen servers with 1GB of RAM run out of memory, getting 'allocator out of memory' messages. The update process can take quite a while, but if / when it completes, the 'lost' memory is eventually recovered. SurfControl has a patched version of CPFILTER.NLM dated April 12, 2002 (ver. 5.00d) which they call Service Pack 1. In order to run the update process you need to get at least an evaluation serial number from SurfControl's web site. If you enter a correct email address, you should immediately receive an email from SurfControl telling you about a patch to download, and instructions for installing the patch. SurfControl does not tell you WHY there is a patch, but this is the reason.
In some circumstances, you can see a problem where the SurfControl database update process fails, with the following entry in the CPFILTER.LOG file:
[Thu May 30 18:05:16 2002] Starting SurfControl Content Database
[Thu May 30 18:05:30 2002] Beginning LiveUpdate procedure
[Thu May 30 18:05:30 2002] LiveUpdate: Current Database Version is 378
[Thu May 30 18:05:37 2002] LiveUpdate: Incorrect syntax near the keyword 'ORDER'.
Statement(s) could not be prepared.
The problem is rather odd, and is related to the version of CPFILTER.NLM running, and a DNS PTR record for your BorderManager server public IP address. If the public IP address has a
PTR record pointing to a name that is more than 36 bytes long, the above error can occur, and the update fails. (This can also happen if you have a long name for the server's public IP address in the
HOSTS file on the server).
SurfControl is aware of the issue as of early June 2002, and a patch will be coming out. I have tested a beta patch which fixes the problem. I do not know at this time how SurfControl will version,
advertise or supply the patch.
For several months, CSP_LIST updates worked OK, gradually updating the SurfControl database file and making it a little larger on each update. However, the database finally became so large that the default memory allocation setting for CPFILTER was too small. CSP_LIST updates would then fail. Thankfully, the CPFILTER.LOG file gives you the answer. You must add an entry in the SYS:ETC\CPFILTER\CSPCONFIG.INI file. I guessed at a value to use, and the following has worked for me (as of Apr 24, 2003)
[SurfControlGeneralList]
CDB_PREALLOC_SIZE=270
Note: For the first part of January 2003, I had set that size to 195 instead of 205. That worked up to a point, and then 196 was required. By Early April, I had to increase the value to
260, then 270.
While at BrainShare, I talked to SurfControl at length on the issues listed on this page. Apparently, some extra, and unnecessary data got into the database update in the March / April timeframe, and
it swelled the database file size considerably. This size increase (my .CDB file went from about 150MB to over 183MB) caused a RAM increase requirement. By the time I got back from BrainShare 2003,
the problem had been corrected, and the database was pruned. On April 24, my database was at version 656, and was down to 151MB in size. I have left my RAM preallocation at 270, to cover further
increases in the future, but if you are tight on RAM, you can reduce the allocation value somewhat. As of April 26, 2003, my .CDB file is now down to 147MB in size, at version 667.
There is some circumstance whereby the SurfControl categories refuse to display in the access rule for it. Even if the categories were there initially, they may disappear, making
control of the rule impossible. In my server, as of June 12, 2002, I have been unable to get the categories to show up, even when I reinstalled SurfControl from scratch.
I noticed this at first when I suddenly could not browse to web sites that I used to. When I checked the SurfControl access rule, there were no categories showing up at all.
The SYS:ETC\BORDER\ENGLISH\SCONTROL.ACL file was suspiciously truncated. (This file is rewritten when CPFILTER loads). Under the [Content Database] section, there were no entries at all.
The problem appears to have been corruption of the SYS:ETC\PROXY\SURFINFO.DAT file. When I deleted that file, along with the SYS:ETC\BORDER\ENGLISH\SCONTROL.ACL file, SurfControl categories suddenly
appeared again in my access rule.
If the above doesn't help you, try reinstalling the snapins from the SYS:\PUBLIC\BRDRMGR\SNAPINS\SETUP.EXE file. Or reinstall SurfControl, as noted below.
April 26, 2003: Note that the BM37FP3.EXE file has a RESTRICT.DLL snapin that you have to manually copy to the snapins directory, and would be overwritten with an older version if you ran the snapin
setup program
If you cannot get SurfControl to load properly, and you see the following error messages on the server:
You may have experienced a failed update attempt leaving problem files behind in the SYS:ETC\CPFILTER directory. Delete any .TMP files found there and try to reload CPFILTER, or reboot the server.
If you experience problems loading SurfControl CPFILTER where it loads but uses only a few K of RAM (normally usess 183MB-205MB of RAM), you may need to change the load order in
AUTOEXEC.NCF. CPFILTER should be loaded before BorderManager proxy. That is, load CPFILTER before STARTBRD / BRDSRV.
If you still can't get it to load properly, try loading CSP_LIST instead, as it will try to autoload CPFILTER. This will kick off an update process, but generally works.
The CPFILTER process may not load the actual database into RAM unless it is 'registered' with an Access Rule, or perhaps with CSP_LIST.
I don't have an answer for this issue yet, but I have seen it at a client with BM37SP1 installed. You select a SurfControl category, save the changes, and then look and find that all,
or almost all categories are selected. You cannot get the rule to select only the desired categories. In my case, a backrev of RESTRICT.DLL did not help, nor did reinstalling SurfControl. When the
answer to this issue come out, I will post it here.
Apr. 25, 2004 Update:
There seem to be two schools of thought on this one. One is a complete reinstall, while the other (which makes the most sense to me) is to be sure the snapins are correct. Here are two things to
try:
1. Be sure you have the correct snapins. At least you should have (after BM37SP1 or later patches) the choices for None, SurfControl, N2H2 and Connectotel at the bottom of the access rules menu. If
you don't see that, you definitely have the wrong snapins. In any case, try running the snapin copy program SYS:PUBLIC\BRDRMGR\SNAPINS\SETUP.EXE. April 26, 2003: Note that the BM37FP3.EXE file has a
RESTRICT.DLL snapin that you have to manually copy to the snapins directory, and would be overwritten with an older version if you ran the snapin setup program.
2. Complete reinstall of SurfControl (and do #1 as well). See the following issue for some detail on a complete reinstall.
This one happened on my main Internet server immediately after applying NW51SP6.EXE. The server was at BM37SP2 (with BMMACSSL1 proxy version) and was at NW51SP5 before I applied the new
patch on Apr. 23, 2003. I think in retrospect that the SP6 patch probably was not a root cause, but instead some file may have gotten corrupted.
Symptom: BorderManager loaded, then a <1> ABEND occurred shortly afterward. The abend.log file showed
Server BORDER1 halted Wednesday, April 23, 2003 7:52:25.829 pm
Abend 1 on P00: Server-5.00k: Page Fault Processor Exception (Error code 00000000)
and later in that log...
It didn't take long to find out the the ABEND only happened when I had a SurfControl access rule in place. The ABEND would occur shortly after ACLCHECK read the rules at BorderManager
startup, or after I added back in a SurfControl rule. Prior to this, the server had been quite stable, and SurfControl had been working well.
I tried a variety of fixes, but ultimately did a complete reinstall of SurfControl to solve the issue. (Note: Or so I thought! The abend came back several hours after I posted this the first time.
I may now have an answer - Apr. 25, 2003 - It appears my .CDB database was corrupted. I downloaded a fresh one from SurfControl, and my server has been up 6 hours without an abend since then. Apr.
26, 2003 - It's been about 24 hours since I put in the fresh .CDB file, and still no abend. Last night, the .CDB file was updated by CSP_LIST from version 654 to 667. During that process, it dropped
in size from 180MB to 147MB.)
Here are the steps I took to Reinstall SurfControl:
I don't really know what happened here, but I suspect the procedure above will work for a variety of situations. It may be that the SURFINFO.DAT file was corrupted, or a snapin issue
occurred, or the .CDB database file was corrupted, or all of the above. The reinstall and .CDB file replacement has me going again though.
Note: If you do not have a good backup of the Surfcontrol Database.cdb file, you can call SurfControl technical support, and they should be able to give you a temporary URL that you can use to
download the latest revision of the database. (This will be a large download, on the order of 64MB or so...) SurfControl prefers not to publish a URL for anyone to use to download the file, so you
will have to call to get it. The alternative is to run the CSP_LIST process and wait for the updating to complete. Be advised now that it is a fine idea to periodically back up that file so you don't
have to start at the original version after a reinstall. Just copy the newer version to the CPFILTER directory, and reload CPFILTER.NLM to use the new data. CSP_LIST will pick up its update from the
version in that file and go on from there.
If you get errors on the system console about being out of memory, and you actually have enough RAM in the server, it may be related to a problem with a SurfControl update. Check the CPFILTER.LOG file in the SYS:ETC\CPFILTER directory for error messages. If you have any 0 byte .TMP files in the CPFILTER directory, you will definitely need to delete those. I have not found the cause of this problem yet, but it occurs now and again on my server, even though I have plenty of free RAM for the CSP_LIST update to finish. Usually, I find and delete several 0 byte files, and reboot the server, and it is fine again for days or weeks. I have a feeling that the memory issue may come up only after CSP_LIST fails to update and leaves several 0 byte .TMP files in the CPFILTER directory, causing some other issue.
After installing BM 37FP3, some sites may not be blocked by SurfControl (Same issue with LinkWall). I found that www.sex.com was not blocked by the adult / sexually explicit category in
my SurfControl rule. I'm trying to track this issue down.
There have been reports that you can fix the issue by back-revving ACLCHECK.NLM to the version from BM37SP2, but I just tested that and it did not work here. I tried back-revving ACLCHECK to BM37SP1,
and that DID work, but I assume brings back whatever bugs were present in that version of ACLCHECK. (Keep your eyes open for BM37FP4 or some other new patch).
A forum user reported that back-revving ACLCHECK.NLM, PROXY.NLM and RESTRICT.DLL to the BM37SP2 versions fixed his problems.