Description of the changes for release 1.8b of LISTSERV(TM) ----------------------------------------------------------- Copyright 1995 L-Soft international, Inc. May 8th, 1995 ************************************************************************* ************************** List owner's notes *************************** ************************************************************************* The release notes for version 1.8b of LISTSERV(TM) have been split into three documents: executive notes, list owner's notes, and LISTSERV maintainer's notes. The present list owner's notes describe all the changes that list owners need to be aware of, although in some cases list owners may be impacted by changes described in the LISTSERV maintainer's section. For instance, the availability of a new systemwide option may prompt the LISTSERV maintainer to make a change affecting all the lists on his machine. Thus, list owners are advised to read the maintainer's notes as well. Note to non-VM customers: the present release notes describe all the changes from the date of release of version 1.8a (7 Dec 93). Because the first VMS(TM) and unix(R) versions of LISTSERV were released in June 94, the improvements made during the first half of 1994 were already included in the code released as "1.8a" for non-VM systems. Similarly, due to the release date of the Windows NT(TM) version, very few of the changes in these release notes will be new to Windows NT(TM) customers. ************** * Highlights * ************** - Comprehensive, 95-page list owner's manual now available online. - (VM) Numerous changes to facilitate Internet migration. BITNET addresses are only used when there is no other alternative, or when explicitly requested. - New SCAN command and automatic delivery error monitoring system simplify processing of delivery errors. - "Spam" detector shields LISTSERV lists from unsolicited advertisements. - New message templates for better configurability, including support for "infobot" style lists, automatic acknowledgement of xxx-request messages, top and bottom message banners for disclaimers or short SIGNOFF instructions, and so on. - New QUERY format with short description of individual options; new QUERY ... WITH command to identify all subscribers with a particular option activated. - New subscription options, support for multiple moderators, new message approval procedures, poster authentication options for secure lists... ****************************************** * Documentation: New list owner's manual * ****************************************** A comprehensive, 95-page list owner's manual is now available online. A plain text version is included with the 1.8b update, replacing the old "INFO KEYWORDS" and "INFO OWNERS" documents. You can also FTP a copy of the new manual from FTP.LSOFT.COM, CD DOCUMENTS. The file is available in a variety of word processing formats - check the FTP server for more information. **************************************************************** * Usability: Changes to default options for Internet migration * **************************************************************** In order to facilitate the transition from BITNET to the Internet, a number of changes have been made to default keyword and system configuration variables in version 1.8b. Some of these changes were made during the development of LISTSERV-TCP/IP (1-2Q94) and were thus present in the first VMS/unix builds released in June 94. The rest were made in 3-4Q94. All changes are present in the January builds for VMS/unix, and in the Windows builds. In other words, if you were running the latest 1.8a builds for VMS/unix/Windows, these changes are actually not new. They are new to VM sites, and some may be new to VMS/unix sites running an older build. Many of the changes have little relevance to non-VM sites and are marked "(VM)". Changes to list header keywords ------------------------------- - (VM) The "Files=" keyword now defaults to "Files= No", reflecting the fact that the distribution of NJE files to a LISTSERV list is now a historical feature. - (VM) The "List-Address=" keyword now defaults to the long list name and Internet address (formerly, it defaulted to the 8-character list name and the BITNET address). Non-VM servers are not affected because they have no BITNET address and thus always use their Internet address, and because they can support lists with longer names. - The "Newsgroups=" keyword now defaults to "Newsgroups= None", causing LISTSERV not to generate any "Newsgroups:" field. This tag was required by old list<->news gatewaying software and is now considered obsolete, whereas the presence of a "Newsgroups:" field in mail headers is misleading subscribers into thinking the list is gatewayed to usenet. The "Newsgroups=" keyword is still supported of course, and lists which have an explicit "Newsgroups= a.b.c" in the list header will continue to generate a "Newsgroups:" tag. - The "X-Tags=" keyword now defaults to "X-Tags= Comments" rather than "X-Tags= Yes". While this change is not specifically related to the Internet migration, it was scheduled to be made at the first opportunity. - The TOCOUNT/NOTOCOUNT suboption of the "Loopcheck=" keyword is no longer supported. This ancient loop checking mechanism is now totally obsolete and has not been carried over when the loop detector was rewritten for portability. This option was already turned off by default, and in practice the only difference is that list headers containing this option will have to be edited the next time they are stored. Changes to server configuration defaults (set by maintainer) ------------------------------------------------------------ - (VM) The MAXGET and MAXGETK variables were raised from 50 messages totalling 2 megabytes to 200 messages totalling 16 megabytes. These variables are used by the default LSV$GETQ exit shipped with the VM version of LISTSERV and the change may or may not have any effect depending on local customization. The temporary file server functions currently supported by the non-VM versions do not use the LSV$GETQ exit and are thus not affected by this change. - The default value for the PRIMETIME variable was changed to the empty set, that is, LISTSERV now processes "overnight" jobs immediately. This change reflects the fact that, with today's networking technology, holding jobs until the night shift is no longer appropriate except in a few specific cases, and thus should not be the default. List owners should note that most sites made this change manually over the last 5 years or so, ie in most cases the policy is already in effect. - The default values for FILEMAXL and MAILMAXL have been increased to 50,000 and 10,000 lines, respectively. Mail sent to the LISTSERV address (as opposed to mail sent to a list) is no longer check against the MAILMAXL threshold. Other miscellaneous changes --------------------------- - Header default: the default header format was changed from SHORT to FULL headers, for the convenience of MIME users. Note that some mail software packages still have maximum header size limits dating back to the early 80s. Because FULL headers are much larger, it is possible that some of your subscribers will complain about not receiving their mail any longer. In that case, you may want to try switching them to SHORT headers to see if the problem disappears. L-Soft recommends configuring mailers to accept at least 50 header lines. L-Soft's products accept up to 100 kilobytes of header data, and the only purpose of this restriction is to reduce the impact of large misformatted mail messages (such as misdirected VM batch console logs, which can be millions of lines long) on the system. - (non-VM) Null record padding removed: because the VM file system does not support null (zero-length) records, mail software on VM systems has traditionally been unable to generate zero-length lines. While LISTSERV does internally support such null lines even on VM, it is not possible to pass them on to the VM SMTP software. Similarly, VM LISTSERVs will always receive one-blank records from the VM SMTP server, because the (non L-Soft) SMTP software stages the data through an unformatted disk file. Thus, the VM version of LISTSERV is currently restricted in its ability to properly receive and generate null lines. To guarantee the compatibility of server-to-server requests, the original non-VM versions had code to convert all null records to one-blank records. The concern was not that the new VMS/unix code would not be able to handle null records properly, but that existing, older versions of the VM LISTSERV would not be able to process such requests, in which case the customers in question should at least be warned of the impending incompatible change. Fortunately, extensive testing has revealed that both versions 1.7f and 1.8a are able to process null-record requests from non-VM servers, and the null record padding code has simply been removed from the non-VM versions. *************************************** * Usability: New subscription options * *************************************** Three new subscription options have been added to facilitate the management of large lists. These options can be set by the list owner only, using the SET command. The EDITOR option allows a subscriber to post messages to a moderated list, without having to go through the moderator. It is virtually identical to adding the subscriber's address to the "Editor=" keyword, but easier to manage. The only difference between the EDITOR option and the "Editor=" keyword, other than not being visible in the list header, is that the "Editor=" keyword also defines a (seldom used) access level class which can then be used in keywords such as "Review=". Thus, one could have a list with "Review= Editor", indicating that only the users listed in the "Editor=" keyword are allowed to review the list. The EDITOR option does not confer this privilege. Note that the EDITOR option is only meaningful on moderated lists. The REVIEW option indicates that all postings from the subscriber in question should go to the list owner for approval. By default, the messages are forwarded to the primary list owner (the first address in the "Owner=" list). An alternate moderator (or list of moderators) can be designated using the "Moderator=" keyword. Note that adding a "Moderator=" keyword will not make the list a moderated one; it is the "Send=" keyword that controls whether the list is moderated or not. The "Moderator=" and "Editor=" keywords simply define moderator/editor addresses to be used when and if moderation is called for, such as when a user with the REVIEW option posts a message or when "Send= Editor" is in effect. For lists discussing sensitive topics, it might be appropriate to set "Default-Options= REVIEW" so that all postings from new subscribers go through the moderator during a "probation period". Once the subscribers have shown that they are able to participate in the discussion in a productive way, the REVIEW option can be selectively removed by the list owner. The NOPOST option can be used to prevent a particular subscriber from posting to the list. LISTSERV returns the postings to the user with the following message: "You are not authorized to post to the XYZ-L list. For more information, please contact the list owners at XYZ-L-Request@XYZ.COM". Note that this option is only effective on private lists. With "Send= Public", the user could still post from a different account; with "Subscription= Open", the user could leave the list and re-subscribe. The antonyms of EDITOR, REVIEW and NOPOST are NOEDITOR, NOREVIEW and POST, respectively. Because these options are stored in the subscriber's entry in the list, and not in the list header, they may not be effective with peered lists unless the user is subscribed to all the peers. For instance, a subscriber with the EDITOR option will only be granted editor privileges if he submits his posting to the peer on which he is subscribed (which in practice should not be a problem). For NOPOST and REVIEW, it may be necessary to add the user to the other peers and set these extra subscriptions to NOMAIL. IMPORTANT: In order to add these new options, the format of columns 81-100 of the list file returned by the GET command had to be changed. LISTSERV will continue to accept the old format as input and will not reformat old entries until they are modified (to minimize user impact in case the LISTSERV maintainer should need to fall back to an old version). However, you will no longer be able to locate (say) subscribers with the CONCEAL option by looking for a 'C' in column 92. Use the new QUERY ... WITH CONCEAL command for this purpose. ******************************************************* * Usability: Change in MAIL, DIGEST and INDEX options * ******************************************************* Originally, the MAIL, DIGEST, INDEX and NOMAIL options corresponded to the four possible settings of a single option. That is, switching from DIGEST to NOMAIL would stop the delivery of new digests, as expected, but would also make LISTSERV forget that the subscription should be in digest mode. A subsequent switch to MAIL would restore delivery, but in normal (non-digest) mode. This was confusing to novice users. In version 1.8b, the MAIL/NOMAIL option has been isolated from DIGEST/INDEX. The MAIL/NOMAIL option controls whether messages should be delivered, and the DIGEST/INDEX/NODIGEST/NOINDEX option controls the format in which messages should be delivered. Thus, switching to NOMAIL and back to MAIL now preserves the digest/index/normal delivery setting. To provide as much compatibility with the old syntax as possible, the four options now operate as follows: - DIGEST: enable digest delivery mode (which negates INDEX), enable mail delivery. No change from version 1.8a. - INDEX: enable index delivery mode (which negates DIGEST), enable mail delivery. No change from version 1.8a. - NOMAIL: disable mail delivery. No change from version 1.8a. - MAIL: restore mail delivery, without altering the digest/index/normal delivery setting (new behaviour). For compatibility with 1.8a, if mail delivery was already active, the MAIL option negates INDEX/DIGEST. Thus, a user going from NOMAIL to MAIL will keep his previous delivery options, whereas a user going from DIGEST or INDEX to MAIL will in fact deactivate index/digest mode. To revert from digest/index subscription mode to normal delivery, you can use either the MAIL option as before, or the NODIGEST/NOINDEX option. The NODIGEST and NOINDEX options were actually present in versions 1.7f and 1.8a, as synonyms for the MAIL option. In other words, you can update your instructions to indicate that the DIGEST/INDEX options are negated by the NODIGEST/NOINDEX options, even if your server is not yet running version 1.8b. ******************************* * Usability: New SCAN command * ******************************* A new command has been introduced to make it easier to locate a subscriber's entry in the list given the subscriber's name or approximate address. The syntax of the new command is: SCAN listname string1 > This will search the address and name fields of all the subscribers in the list for matching entries. Case is ignored unless you enclose the search string in double quotes. For instance, 'SCAN XYZ-L Joe' will match 'JOE', 'Joe' and 'joe'; 'SCAN XYZ-L "Joe"' will only match 'Joe'. If multiple search strings are specified, the entry must contain ALL of them in order to be selected. For instance, if you receive a delivery error indicating that one "Smith, Joe" has left XYZ Corporation and that his account has been deactivated, you can locate all the Joe Smiths in your list with 'SCAN XYZ-L Joe Smith'. This will find 'Joe Smith', 'Smith, Joe', 'Joe H. Smith', etc. A limited version of the SCAN command is available to non-privileged users that are authorized to review the list. The syntax is the same, but concealed subscribers are skipped during the search. That is, concealed entries are totally ignored during the search (as opposed to being searched, but without being displayed at the end). Since concealed entries do not participate in the search, it is impossible for the user to gain any information about concealed participants through the SCAN command. Note that you can, of course, continue to use the QUERY command to make searches on the address field. For instance, 'QUERY XYZ-L FOR *smith*@*' will display all the subscribers whose username contains the string 'smith' in either case. The QUERY command, however, cannot search the name field, and returns a lot of unnecessary information. The SCAN command only shows you what you are really interested in: the user's name and e-mail address. ************************************* * Usability: Enhanced QUERY command * ************************************* The QUERY command has been enhanced to make it easier to identify users with a particular subscription option. For instance, you may wish to check for NOMAIL users regularly, or for users with the CONCEAL option. To do this, simply issue the following command: QUERY listname WITH option1 > FOR pattern For instance, QUERY XYZ-L WITH CONCEAL FOR *@* will return a list of all the subscribers which have the CONCEAL option. This new WITH option is only available to list owners. ********************************************************* * Usability: Enhanced reporting of subscription options * ********************************************************* The QUERY command has been enhanced to provide a more meaningful description of the various available subscription options. Previously, the subscription options were reported as follows: ---------------------------------------------------------------------- Subscription options for Eric Thomas , list TEST-L: Ack=No, Mail=Yes, Files=Yes, Repro=No, Header=Full, Conceal=No ---------------------------------------------------------------------- While compact, this format was not very helpful to inexperienced users. Here is an example of the new format: ---------------------------------------------------------------------- Subscription options for Eric Thomas , list TEST-L: MAIL You are sent individual postings as they are received FULLHDR Full (normal) mail headers (formerly "FULLBSMTP") NOREPRO You do not receive a copy of your own postings NOACK No acknowledgement of successfully processed postings ---------------------------------------------------------------------- Options which are considered "important" are shown regardless of their setting. This makes the user aware of the existence of the option, and the possibility to change it if the current setting is not satisfactory. The more exotic options are listed only if they are in effect, to avoid confusing the user with a long list of available options. ************************************************** * Usability: Automatic delivery error monitoring * ************************************************** In order to give list owners more flexibility in meeting their particular policies for mail "bounces" while allowing the tedious delivery error processing tasks to be delegated to LISTSERV, delivery error monitoring and reporting functions have been added in version 1.8b. Whereas version 1.8a would immediately remove subscribers from the list upon the receipt of a delivery error indicating a permanent error ("No such local user", "No such host", etc), the new error monitor will keep track of the kind and amount of errors received, and take action based on a number of configurable parameters. The new monitor can also be run in "manual" mode, in which case it merely generates daily condensed reports for the list owner, without automatically removing users from the list. Naturally, the old 1.8a behaviour remains available, although it is no longer the default option. By default, version 1.8b will now wait 4 days before removing a user from the list as the result of permanent delivery errors. This value strikes a balance between the desire to tolerate human or computer error over a "long weekend", and the need to keep the number if delivery errors processed by LISTSERV to a reasonable value. To avoid overwhelming the LISTSERV host with large numbers of delivery errors for active lists, the monitor will also delete the user upon the receipt of 100 permanent delivery errors (this value can be increased). Finally, the monitor will notify the user of the deletion, so that, if for any reason the problem had actually been solved, the user will know what happened. The delivery error monitor is configured through the "Auto-Delete=" keyword. The basic syntax remains unchanged: Auto-Delete= No or: Auto-Delete= Yes,option1,option2,... The SEMI-AUTO and FULL-AUTO options remain unchanged from version 1.8a. Here is a list of the new options for the delivery error monitor: - The MANUAL option configures the monitor in passive mode. Every day, you receive a report with a list of bouncing addresses, statistics for each individual address, and a copy of the last error message (one-line abstract). It is then up to you to decide how to handle these errors. - The DELAY(nn) option lets you change the "grace period" during which the monitor will tolerate permanent errors without taking any action. DELAY(0) means that the users should be deleted immediately upon the receipt of a permanent delivery error (1.8a behaviour). The default value is DELAY(4), which should be adequate for most purposes. A shorter value might be appropriate for very active lists. A larger value is not recommended, and in particular values close to 7 should be avoided, because most "false bounces" occur over the weekend. In other words, a value of 7 does not provide much additional tolerance compared to a value of 5, because it is during the weekends that errors are likely to occur. If 5 is not lenient enough, the next logical choice would have to be 10 or 11. - The MAX(nnn) option defines the maximum number of errors that the monitor will tolerate for any given user. The default is 100. After receiving this many errors, the monitor will delete the user regardless of the DELAY(nn) option. While this value can be increased, you should check with your LISTSERV maintainer before doing so. The threshold is unlikely to be exceeded unless your list is very active, in which case the number of bouncing addresses is likely to be in the 20-100 range. 100 bouncing addresses times 100 tolerated bounces per address is alreay 10,000 messages for the LISTSERV host to process. Depending on the hardware configuration, this may or may not be an issue. In order to understand how the monitor works, it is important to realize that it only receives notifications of NON delivery. The monitor has no way to know that the problem has been corrected, other than to note that no error has been received in a certain time frame. Generally speaking, the monitor counts errors, keeps track of their severity, and remembers when they occurred. When certain criteria are met (as explained above), the monitor deletes the user. After a certain period of time without new errors, the monitor assumes that the problem has been solved and stops monitoring the user in question. The following tips may prove helpful: - Remember to increase the MAX threshold if you make a large increase to the DELAY threshold. Otherwise the users may be deleted anyway. - When decreasing the DELAY threshold, expect a large number of deletions the first day. This is because the monitor probably has a number of users on record that have been bouncing for the amount of days that you specified, but for less than the old (larger) threshold. It does not necessarily mean that the monitor will continue to delete users at that rate. Because most errors occur over the weekend, it takes about a week to get an idea of how many users will be deleted with a particular threshold value. - Making a big increase to the DELAY threshold to provide more leniency during a holiday may not be a good idea. While it will indeed disable the monitor for the duration of the holiday, switching back to the normal threshold when you return will cause the monitor to delete all the users that had been bouncing during the holidays. In general, you should avoid making temporary changes to the DELAY threshold, because it takes the monitor a while to adapt to the new settings. - The best way to relax the rules during a long holiday is to leave the DELAY threshold unchanged but switch the monitor to passive mode ("Auto-Delete= Yes,Manual"). Noone will be deleted over the holidays, but the monitor's cycle will not be perturbed. When you return, you should wait about a week before switching back to automatic mode. This is because, after a long holiday such as Christmas, it usually takes about 2 working days for system administrators to solve all problems. In some cases, the problems will have caused bounces to remain undelivered. So, by fixing the problems, the system administrators may actually send a flood of new bounces corresponding to problems that have now been solved. Unfortunately, since the monitor only receives NON-delivery reports, it has no way to know that these problems have in fact been solved. As a rule of thumb, you will note that your daily delivery error reports are much longer than usual over the vacation. When you return, you should wait until they are back to their normal size before switching back to automatic mode. IMPORTANT: Regardless of the way you instruct LISTSERV to handle them, delivery error reports indicate that a message has been lost. One cannot overstress the fact that false permanent errors ("No such user" when the user exists, "No such host" when the host does exist) are a VERY SERIOUS PROBLEM which should be addressed by the corresponding system administrators, and not simply ignored. Make sure to tell your subscribers that this problem affects ALL messages destined to them, and not just the messages from your list. Because their mail system was rejecting any and all mail to their mailbox over a certain period of time, they may have lost very important private mail along with the messages from your list. ****************************************************************** * Usability: New import procedure for non-LISTSERV mailing lists * ****************************************************************** A new list "import" procedure has been added in version 1.8b to facilitate the migration from non-LISTSERV mailing list managers. For best portability, this procedure has been implemented as a LISTSERV command option rather than a system script. This also makes it easier for list owners to use the import facility on their own, for instance when merging two lists. The first step in migrating a non-LISTSERV list to LISTSERV is to prepare a list header with the new list. L-Soft intends to develop a GUI interface for this purpose, but currently this must be done with a regular text editor. Because of the large number of non-LISTSERV list managers and the greater flexibility of LISTSERV, and after analysing feedback from customers who had successfully migrated to LISTSERV, it was felt that an automatic conversion procedure for the list's configuration options would not be very useful. Indeed, most of the sites that migrated to LISTSERV immediately took advantage of the new configuration options, and found it simpler to prepare a new configuration based on their business needs than to attempt to convert the configuration information from their old list manager. Once the list is created (with an empty subscriber list), the subscriber data from the old list manager can be merged in using the new ADD IMPORT command. The first step is to prepare a text file containing the following information: ADD listname DD=SUB IMPORT //SUB DD * ... /* where '...' is replaced with the subscriber data from the old list manager, in its native format, with one subscriber address and/or name per line. You do not need to indicate the old list manager's brand as LISTSERV supports most major formats and will recognize them automatically. Experienced list owners will have noted that this is the same as a regular ADD job, except for the IMPORT option. This option tells LISTSERV to suppress all notifications (as per a QUIET ADD), to use the new "import" parser when analysing the various addresses, to suppress all successful responses (only errors are reported), and to speed up processing by considering the whole job as a single transaction, whereas in a regular ADD job there is one transaction per subscriber. This "transaction" difference relates to the way LISTSERV manipulates the list data and to the implications in case a hardware or software failure should occur during the execution of the job. With a normal ADD job, users are notified as each entry is processed. To guarantee that the users have actually been added when they receive their notification, and to prevent duplicate notifications from being issued in case a failure should occur and the job should have to be retried, the subscriber entries are checkpointed as they are added to the list. If the job fails after processing 500 entries, the 500 entries that were added to the list at that point are guaranteed to be correct. With the IMPORT option, the transaction works in an "all or nothing" mode, and has to be restarted from scratch in the event of a hardware or software failure. NOTE: Due to the multitude of possible syntaxes and options to check, an import job for a large list (over 1,000 subscribers) can take several minutes to complete, depending on the hardware being used and on whether you are using the High Performance or regular version of LISTSERV. The server prints progress messages to its log at regular intervals to confirm that it has not gone into a loop. ***************************************** * Usability: Automatic "spam" detection * ***************************************** Ever since the media started talking about the "Information Super-Highway", the Internet has drawn a lot of attention from the "man in the street". What a few years ago was mostly an academic and research network is turning into a general purpose resource; some think that in another few years, every household will be connected. This is a profound cultural change and, like most cultural changes, it comes with its share of good things and bad things. One of the bad things is that some unscrupulous people have realized that the charging model used by the Internet service providers makes it possible to use other people's resources and money to deliver advertisements to millions of people, for a one time cost of a few dollars. Taking advantage of the chronical lack of legislation in the networking area, they are actively distributing "spams" - unsolicited advertisements sent to large numbers of mailing lists, with no consideration for whether or not the product being offered is relevant to the topic of the list. Thus, we have seen advertisements for thigh creams posted to all the known mailing lists, causing an outburst of traffic for which many people have to pay by the byte. Each of these spams is estimated to cost over $100,000 to the victims, collectively, in the form of higher connection charges. This is real money that actually shows up on people's bills and has to be paid. In addition, the spams and the resulting flood of complaints to the service providers (often directed to the lists by mistake) waste hundreds of thousands of dollars in manpower, machine time, people's time wasted due to slow/unavailable service, etc. These spams have to be stopped. The root of the problem is that, most of the time, it is the recipient who pays for the message, not the sender, because to a large extent this is where the costs are incurred, and there is no mechanism (to date) for the Internet service providers to charge back other providers' customers in the way that phone operators do. Imagine what your mailbox would look like if any company could send you junk mail at YOUR expense! However, there is no consensus that such a change in accounting principles would be generally beneficial; for instance, if people had to pay for thousands of deliveries in order to answer a question on a large mailing list, there would be many questions and very few answers, making the mailing list totally useless. At any rate, this is not a change which can be done overnight. Similarly, "anti-spam" legislation is at best a long term prospect, whereas a solution is needed immediately. There are already books with simple, step-by-step instructions for preparing a spam and reaching as many people as possible, and they are selling very well. Fortunately, LISTSERV's nature as a distributed network of interconnected servers makes it possible to identify and eliminate these spams before they do much harm. While it is virtually impossible for a small isolated server to detect a spam (unless the sender listed the thousands of lists he was targeting in the "To:" field), for the simple reason that it will only ever receive a few copies for its own public lists, the LISTSERV network as a whole receives thousands of copies of the spam. By comparing notes with each other, the servers can quickly determine that a spam is occurring and raise a network-wide "spamming alert", stopping the message before it does much damage at all. When LISTSERV determines that a spam is occurring, it blocks all further copies of the message and forwards them to the list owner for manual verification. The poster is informed once, by the server that first raises the alert (due to the distributed nature of LISTSERV, the poster may actually be informed more than once if several servers identify the spam at the same time). The poster is NOT informed when subsequent messages are rejected. That is, if the poster is the innocent victim of a computer error, he will know that something went wrong. If, however, he is indeed a spammer, the poster will not know which lists were missed and may have to be retried. If fact, spammers seldom read the replies they receive from LISTSERV servers, simply because there are over 6,600 public lists, and most of them will return a positive or negative acknowledgement. The message is sent from a "junk account" whose mail is not read, or then filters are used to automatically discard unwanted computer replies. As always with automatic procedures, there is a risk of false alerts. Ultimately, what determines whether a message is a "spam" or "cross-posted material" is its relevance to the topic of the list. Unfortunately, LISTSERV is not capable of determining whether a particular message is relevant to a particular list. All it can do is note that many copies of a particular message have been sent in a short time frame, and assume that the message is a spam. To take a concrete example, a conference announcement, complete with registration form and conference fees, could be a highly relevant item or a spam depending on which lists it is posted to. The poster may just be trying to reach as many people as possible in the hope of increasing attendance (this has actually happened with conferences on multiple occasions), or he could be posting the announcement to germane lists, with the list owner's prior approval. There is simply no way for a computer to tell. When a genuine message needs to be posted to large numbers of mailing lists, L-Soft recommends sending the message to the list owners, rather than posting it directly (this is already required for closed lists anyway). The list owner can then determine whether the message is relevant, and forward it to the list as appropriate. Another advantage of this method is that the subscribers know for a fact that the owner authorized the use of the list for this purpose, and will not accuse the poster of having abused their list. List owners who wish to disable spam detection for their lists can do so by adding: Loopcheck= NoSpam to the list header. When spam detection is disabled, postings are always accepted. Conversely, the spam detector does not guarantee that spams will never make it to the lists. In particular, the LISTSERVs cannot tell that a message is a spam until they have received a certain number of copies. In other words, when it receives the first copy of the spam, the LISTSERV network has no way to know that 5,999 other copies are on their way, and the message will be processed normally. Due to the distributed nature of LISTSERV, with all the servers working in parallel on the "investigation", dozens of messages will actually be treated as "first copies" and distributed to the lists. The bulk of the messages, however, will be stopped, and this is what really matters. Until legislation is in place to provide a more satifactory solution, we are likely to see a typical "armour and shield" race. For this reason, L-Soft will not disclose any information about the methods use by the spam detector, and will keep improving the algorithms constantly. If the algorithms were disclosed, it would take about 3 to 6 months for a new edition of the "Make money fast" books to hit the bookstores, and we would all be back to square one. Luckily, the odds are in the list users' favour, because the spammers are not really interested in a technological race. Their only goal is to reach millions of people quickly and cheaply and, by definition, they have absolutely no interest in the nature of the audience that they are reaching; the only thing that matters to them is numbers. Thus, a LISTSERV subscriber is worth as much to them as a Majordomo subscriber or a usenet reader, and there is no reason for the spammers to try to break LISTSERV's defences at all costs when usenet remains totally undefended. ************************************ * Usability: New message templates * ************************************ Version 1.8b introduces many new configurable message templates, and, in particular, two new types of message templates for "linear" and optional messages. Traditionally, message templates have contained the text of "long" administrative messages, such as messages informing subscribers that they have been removed from a mailing list. These notices were sent unconditionally, as a separate message. The template processor now supports "linear" messages, which are sent as a normal command reply and allow the list owner to modify the replies from selected commands, and "optional" messages, which are only sent if a template for this action has been specifically provided by the list owner. Here is a list of the new template messages: - SUB_CLOSED (linear): this is the message that is sent to a subscriber attempting to join a list with "Subscription= Closed". The default is "Sorry, the &LISTNAME list is closed. Contact the list owner (&OWNER) for more information." - SUB_OWNER (linear): this message is sent to a subscriber attempting to join a list with "Subscription= By owner". The default is "Your request to join the &LISTNAME list has been forwarded to the list owner for approval. If you have any question about the list, you can reach the list owner at &OWNER." Because this is a linear template (see below), it is not the best place to put long questionnaires, application forms, terms and conditions, or other material that the subscriber should be required to review prior to joining the list. See the "Tips" section below. - POST_EDITOR (linear): this is the message LISTSERV sends to people attempting to post to the list, if it is moderated. The default is "Your &MESSAGE has been submitted to the moderator of the &LISTNAME list: &MBX(&MODERATOR)." - TOP_BANNER, BOTTOM_BANNER (optional): when these templates are present, their contents are automatically inserted at the top (respectively bottom) of each and every message posted to the list. Typically, the top banner would be used for a copyright or short legal warning which absolutely has to be seen by each and every reader. The bottom banner could contain instructions for signing off the list, a disclaimer, an acknowledgement of a sponsor's contribution, a "tip of the week", etc. - REQACK1: this message is sent automatically in reply to any message sent to the xxx-request address. The message acknowledges receipt, explains the difference between the LISTSERV and xxx-request addresses, and contains instructions for joining and leaving the list. To suppress this message for your list, simply redefine it in the 'listname.MAILTPL' and use the .QQ instruction: >>> REQACK1 This message is not wanted for our list .QQ - AUTODEL1: this is the message that is sent to users who are deleted by the delivery error monitor. You can customize it to fit your needs, or suppress it using the same procedure as for REQACK1. - POSTACK1 (optional): when present, this message is sent in reply to any message posted to the list. This is very useful for creating "infobots", or just for returning a standard acknowledgement to contributors. The &SUBJECT variable contains the subject of the original message, and naturally the usual substitutions (&LISTNAME, &DATE, &TIME) are available. - ADDREQ1 (changed): this message, which was already present in version 1.8a, is sent to the list owner when a user requests to join a list with "Subscription= By owner". In version 1.8a, a copy of the message was sent to the subscriber, to confirm that the request had indeed been forwarded to the list owner. Unfortunately this was confusing to the many novice users who do not understand the difference between primary and secondary message recipients ('To:' vs 'cc:'). In version 1.8b, only the list owner is sent a copy of the ADDREQ1 template. If you were using this template to send new subscribers a questionnaire, application form or similar material, you will need to add a '.TO &WHOM' instruction to your modified template, as by default the user will no longer receive a copy. In a linear message, most special instructions are ignored. This is because the contents of the template are just a few lines out of a larger message that is being prepared by LISTSERV to contain the reply to the user's command(s). For instance, you do not have any control over the "Reply-To:" field of the message, because the message in question is shared with other commands and, in fact, may not be a mail message at all but an interactive message to the user's terminal, a GUI request, etc. Generally speaking, with a linear message you are providing the TEXT of the reply to be shown to the user, but you do not have any control over the methods used for delivering this information. Tips: - Many list owners require prospective subscribers to fill in a little questionnaire before being added to the list, or to explicitly state that they have read the list charter and agree to follow all rules or be removed from the list. The most convenient method, for both list owner and subscriber, is to have the SUBSCRIBE command return a copy of the questionnaire (or list charter, etc), and not forward the request to the owner. The user answers the questions and returns them directly to the list owner, who then adds the subscriber manually. Naturally, it is more convenient for the user if this information arrives in a separate message, with a 'Reply-To:' field pointing to the list owner's address. Thus, you should not use the SUB_OWNER template for this purpose, because it is a linear template and does not give you any control over the 'Reply-To:' field. The SUB_OWNER template could be modified to read "A copy of the list charter is being sent to you, please read it carefully and follow the instructions to confirm you acceptance of our terms and conditions." The list charter would then be sent separately, through the ADDREQ1 template. You would use the .RE OWNERS command to instruct LISTSERV to point the 'Reply-To:' field to the list owners, and .TO &WHOM to change the destination from list owner to subscriber. If you want to receive a copy of the message, you can use .TO &WHOM cc: xxx@yyy. - When writing templates, it is a good idea to use substitutions (&XXXX) for information which may change in the future. In particular, it is not uncommon for lists to have to be moved from one host to another, and this will be a lot easier if the template uses substitutions for the list address and list host. The &LISTADDR substitution translates the full address of the list (XYZ-L@XYZ.COM), whereas &LISTNAME is just the name (XYZ-L). For references to the server and host, use &MYHOST for the Internet hostname, &MYSELF for the server address (normally LISTSERV@&MYHOST), and &OWNER for the xxx-request mailbox address. These substitutions are "universal" and can be used in all templates. For instance, if you decide to make a bottom banner with instructions for leaving the list, the text could read: "To leave the list, send a SIGNOFF &LISTNAME command to &MYSELF or, if you experience difficulties, write to &OWNER." ********************************************** * Usability: Enhancements to list moderation * ********************************************** Support for multiple moderators ------------------------------- Traditionally, moderated LISTSERV lists have allowed multiple "editors" (ie multiple authorized contributors who may submit messages without going through the moderator), but all contributions from regular members were sent to a single address, often called the "first" or "primary" editor because of the convention that the first person listed in the "Editor=" keyword was the moderator, and that the second and following ones were editors. While the moderator's address could of course be a mail alias pointing to several people, there was no simple mechanism for coordinating the work of multiple moderators. The moderators had to inform each other of which messages they had processed and which they had left for the other moderators to process. Usually, teams of moderators would adopt a simple convention, deciding for instance that messages from people whose name starts with A-L go to a particular moderator. However, each had to read all the submission, and this does not scale up to large teams of moderators. With version 1.8b, you can make LISTSERV assign messages to multiple moderators automatically through the use of the new "Moderator=" keyword. Simply list the addresses of all the moderators in the group, and LISTSERV will send each of them a fair share of the traffic (ie if you have 4 moderators, each of them will get 1/4th of the messages). If you want a particular moderator to receive more than a fair share of the messages, simply repeat his addresses until the desired proportion is reached. Each address in the list corresponds to a fair share. Thus, if you have 3 moderators (Joe, Jean and Julie) and you want to send half of the traffic to Joe, use "Moderator= Joe,Joe,Jean,Julie". Addresses listed in the "Moderator=" keyword are automatically editors. Thus, if you have a simple setup with just one moderator/editor, you do not need to provide an "Editor=" keyword. For compatibility with older versions, if no "Moderator=" keyword is present, LISTSERV uses the first address from the "Editor=" keyword. New message approval method --------------------------- By default, when LISTSERV receives a message from a regular subscriber to a moderated list, it simply forwards it to the moderator. On some lists, where the moderator actually edits the messages (for instance by creating a manual digest), this is entirely satisfactory. On other lists, the moderator merely screens incoming messages for compliance with the charter, relevance to the topic, etc. In that case, the moderator essentially acts as a simple accept/reject filter. To accept the message, the moderator forwards it back to the list, and LISTSERV distributes it. A special mail header convention allows the moderator to tell LISTSERV that this is a forwarded copy of an accepted message, and LISTSERV updates the mail headers accordingly, making the message appear to come from the original poster, and indicating which of the moderators approved the message. Unfortunately, with some mail packages it can be very difficult to obtain the desired combination of mail headers. While it has the advantage of allowing minor updates "on the fly", this procedure may not be the most convenient for all audiences. In version 1.8b, an alternate procedure can be activated by changing the "Send=" keyword from "Send= Editor" to "Send= Editor,Hold". The "Hold" option instructs LISTSERV to keep the messages on hold after they have been forwarded to the moderator. The moderator can then approve the message through LISTSERV's "OK" procedure, ie by replying to the message and typing just "OK". Naturally, the moderator is free to make changes to the message, but in that case the modified copy will need to be posted to the list since LISTSERV does not have it on file. In the interest of expediency, the moderator need not do anything to discard a message; LISTSERV will quietly and automatically delete unapproved messages after a week. Note again that the messages are also forwarded to the moderator, and thus expiring the messages merely removes LISTSERV's copy. The expiration delay can be increased (but not decreased) using the "Confirm-Delay=" keyword. Make sure to inform the LISTSERV maintainer before any significant increase. ********************************************************* * Security: New authentication option for list postings * ********************************************************* While LISTSERV offers a number of security mechanisms (configured through the "Validate=" keyword) to protect mailing lists, these mechanisms had so far applied only to the execution of LISTSERV commands, and not to list postings. That is, while one could prevent hackers from making unauthorized changes to the list configuration or membership, there was unfortunately no mechanism to protect the material being posted to the list. Because Internet mail can be forged by non-privileged users, malicious users could submit postings to a list in other people's name and, in particular bypass "Send=" access control by making their message looks like it came from someone who is authorized to post to the list. With version 1.8b, it is now possible to use of the "OK" mechanism to confirm the authenticity of list postings, by modifying the "Send=" keyword to read "Send= ...,Confirm" (where the ellipsis represents the previous value of the "Send=" keyword). When this option is activated, LISTSERV puts all incoming messages on hold and sends a confirmation request to the original poster. The message is not processed until the confirmation is received. Because this procedure is tedious and potentially confusing to non-technical users, it should only be activated on lists which need this level of security. For instance, send confirmation would not be appropriate on a large open public list. It is ideal, however, for announcement/newsletter lists and other low-volume lists with a well defined number of authorized contributors. ******************************************************* * Usability: New option to facilitate list relocation * ******************************************************* When relocating a LISTSERV list from one system to another, you can now instruct the LISTSERV running the old list to forward all commands and postings to the new address, and return an explanatory message to the sender with instructions for using the new address. This is done through the "New-List=" keyword. Simply add "New-List= XYZ-L@XYZ.COM" to the old list header, and store the list. Once you insert the "New-List=" keyword, all commands except PUT will be forwarded to the new host. LISTSERV will also stop advertising the old list (as if you had added "Confidential= Yes") so that all subscription requests are sent to the new host. Finally, you will be required to strip the list of most of its configuration keywords before you can store it back with a "New-List=" keyword, to reflect the fact that the list no longer exists and that these keywords have no effect. ************************************************* * Usability/Security: Changes to the PW command * ************************************************* The syntax of the PW command for general users has been simplified in version 1.8b. In addition, the authentication procedures have been strengthened through the use of the "OK" mechanism. In version 1.8a, a customer-provided exit was responsible for filtering and authenticating PW commands. In most cases, the exit either did no authentication at all, rejected all requests not issued through a secure channel (such as CP SMSG on VM or LCMD on VMS/unix), or simply forwarded them to a human being for verification. While the exit remains available in 1.8b (and does not need to be refitted), LISTSERV now authenticates commands received from insecure channels through the "OK" mechanism, in addition to any authentication that may be done by the exit. This higher degree of security makes it possible to eliminate the LSV$PW exit in most cases. The new syntax of the PW command is as follows: - 'PW ADD password' will register the specified password, as before. - 'PW CHANGE newpw ' can be used to change your password. As before, you have the option of validating the change with your old password. In addition, you may also omit the old password and go through the "OK" procedure. - 'PW RESET' will, after authentication through the "OK" procedure, reset your password and allow you to use the PW ADD command to select a new password. You no longer need to rely on the LISTSERV maintainer to assist you when you forget your password. - The PW DELETE command, while still supported, is now obsolete. IMPORTANT: LISTSERV passwords only provide "weak" authentication, because the users transmit the password in clear text over the Internet. Once the password has been compromised, it can be reused at will. The "OK" mechanism provides a stronger degree of authentication because it cannot be reused. That is, while a malicious user installing a "sniffer" on an ethernet would be able to intercept the information necessary to confirm an "OK" request on behalf of an innocent third party, this would only be possible for as long as the sniffer is active. With passwords, on the other hand, the knowledge gained while the sniffer is in place can be reused multiple times. ******************************** * Miscellaneous: Minor changes * ******************************** This section contains a list of minor changes which, while noteworthy, do not warrant a long description. - (VM) Long list names now used consistently: when a "long" list name has been defined through the "List-ID=" keyword and the "List-Address=" keyword is set (or defaults) to the "long" ID, LISTSERV will now use the long name consistently in all messages. Previously, LISTSERV accepted the long name in user commands, but always used the short name when replying. - New LIST GLOBAL format: the format of the LIST GLOBAL response had to be changed to accomodate the much longer Internet addresses. Each list is now split across two lines, one with the name and address of the list, and one with its description. Internet addresses are also used for LISTSERV-NJE servers whenever possible. - Generalized use of Internet addresses: generally speaking, LISTSERV now uses Internet addresses in messages wherever possible. BITNET addresses are only used when there is no alternative. - New "Notebook-Header=" keyword: this new keyword lets you select the amount of header information you want LISTSERV to store in the list archives. The default, "Notebook-Header= Short", is compatible with version 1.8a and corresponds to the 'SHORT' header set. Setting "Notebook-Header= Full" will use the 'FULL' header set for the archives. Note that enabling this option can double the amount of disk space used up by your archives! - (VMS/unix) The "Renewal=" keyword is now supported in the non-VM versions. - Administrative messages shortened to 73 columns: because many PC mail interfaces display less than 80 columns of text, administrative messages have been reformatted to fit in 73 columns. - Changes in header option names: to reflect the fact that the so-called "BSMTP" headers have been the default header type for all users since version 1.8a (and for Internet users since version 1.6a), whereas the so-called "RFC822" headers are only retained for compatibility with historical software implementations, the header option names have been renamed as follows: the old SHORTHDR and FULLHDR options were renamed to SHORT822 and FULL822, respectively, and the SHORTBSMTP and FULLBSMTP options were renamed to SHORTHDR and FULLHDR. The xxxBSMTP options are still accepted and produce the desired result; however, users who still absolutely need the "RFC822" headers will have to use the new SHORT822/FULL822 option names. Note that "RFC822" vs "BSMTP" headers actually refer to the name of delivery "exits" for the so-called "Crosswell Mailer", the first implementation of a RFC822 mail transfer agent for VM. Both sets of headers are RFC822 compliant, but the "BSMTP" headers could only be submitted to the Crosswell Mailer's through its "BSMTP interface", which was not available in the first few versions. Again, this distinction is purely historical and totally irrelevant to Internet users. For all practical purposes, the "BSMTP" headers are the normal RFC822 headers and the "RFC822" headers are an old type of header required by very old versions of the Crosswell Mailer. ------------------------------------------------------------------------- L-SOFT and LISTSERV are trademarks of L-Soft international. Unix is a registered trademark of UNIX Systems Laboratories, Inc. VMS is a trademark of Digital Equipment Corporation. Windows and Windows NT are trademarks of Microsoft corporation. All other trademarks, both marked and not marked, are the property of their respective owners. -------------------------------------------------------------------------