Mailing List SIMS@mail.stalker.com Message #6676
From: Howard Shere <hshere@greendragon.com>
Subject: Re: Stall Problem
Date: Sun, 13 Aug 2000 21:31:24 -0500
To: (SIMS Discussions) <SIMS@mail.stalker.com>
X-Mailer: Microsoft Outlook Express Macintosh Edition - 4.5 (0410)

>>We have turned on additional logging and we believe we have found the
>>problem. Below are log fragments around the time of the stall. See anything
>>unusual?
>>
>>15:19:47 4 SMTP-131(scyther.omnigroup.com) Input Stream ended
>>15:21:02 4 POP-787([216.220.143.142]) Sending +OK 20746 messages in
>>queue\r\n
>>
>>15:13:08 4 POP-533([216.220.143.100]) Input Stream ended
>>15:14:13 4 POP-633([216.220.143.142]) Sending +OK 20746 messages in
>>queue\r\n
>>
>>This user seems to get a lot of mail.
>
> Um, yeah. And he is checking it every 7 minutes without deleting any messages.
>
> See how it is taking exactly 1:05 both times to do anything while it
> counts those messages? That is a sure sign that the machine is doing
> exactly the same thing and NOTHING else during that time.
>

Yup.

>
> Welcome to MacOS's synchronous I/O.
>
> Without knowing the details of the SIMS code, my guess is that in this
> time SIMS is reading the entire account file and counting up messages. It
> is even doing a pretty darn efficient job of that. It may even be using
> asynch calls, but doing so in a fashion that with this size of a total
> read may end up looking like a synchronous read.

That is (more or less) the reason for my post. It should be possible to read
the data async in small chunks and keep everything else working.

>
>>The account file seems to be 1.3 GB (no, that is not a typo).
>>
>>What can be done to make this work better?
>
> Get the user to actually delete their mail after reading. POP3 is not
> designed to provide IMAP-like service where most messages reside on the
> server until the user is permanently done with them. POP3 is designed to
> be primarily used to retrieve and delete massages, not retrieve and leave
> on the server.

In this case it is a user account for someone who disabled their account
with us a few months ago. The mail piled up. I am not certain why they
suddenly had an interest in popping their mail 'cause they couldn't send
mail out through our system from another ISP.

>
> SIMS *COULD* be more polite about this but at the cost of speed for that
> action (i.e. it could do small reads asynchronously and count the messages
> in a less direct way, but the result would be slower response to that
> client.) SIMS could also stash meta-information for the mailbox in a
> resource so that it doesn't have to directly count all the messages on
> every attempt to access the mailbox, but that would open a path to account
> file corruption that doesn't currently exist. What you are seeing is the
> Mac reading 1.3GB in 65 seconds, which equates to 20MB/s, probably very
> near the practical limits of your system. Short of a radical change in how
> SIMS works or a serious beefing up of your disk subsystems (like RAID
> striping the mailboxes across multiple LVD cards) you won't be able to
> speed this up much. SIMS isn't made to handle this sort of mailbox because
> the protocol isn't intended for such use, and I doubt that Stalker would
> revise SIMS in deep and potentially negative ways just to accommodate such
> a pathological case.

I would rather have the response be slightly slower overall than have any
case where a single operation can stop the server from serving anyone else.

>
> This user is very likely having serious trouble reading mail. The fact
> that it takes 65 seconds to get the initial response likely means that
> their client is timing out. The fact that they have not brought this to
> your attention makes me very suspicious that they may know a reason for
> their mail problems that they'd rather not discuss with you. Such events
> include things like spam runs that feed large numbers of bounces and/or
> complaints back at them. Since this is clearly causing problems with your
> system, I would think that this is sufficient justification to dive into
> that mailbox and see what all the volume is, and if you don't want to do
> this, you really need to talk to the customer via some means other than
> email, which is likely completely useless to them at this point.
>

The user is unlikely to be able to get their mail at all at this point since
the account file has been moved out of the accounts folder. We have a policy
of not looking or filtering the content of our customers mailboxes, but they
are not a customer anymore so they don't need a mailbox.

Some mailing lists are going to be very unhappy for a little while.

_________________________________________________________________________
 Gridz 1.3 ---- www.gridz.com ---- NetSpace will never be the same again
_________________________________________________________________________
Howard Shere       |  Green Dragon Creations  |  Water Valley Interchange
President          |  301 N. Main St.         |  P.O. Box 70
Software Sculptor  |  Water Valley, MS 38965  |  Water Valley, MS 38965
                   |  hshere@greendragon.com  |  hshere@watervalley.net
                   |  www.greendragon.com     |  www.watervalley.net
                   |  1-662-473-4225          |  1-662-473-9209

Subscribe (FEED) Subscribe (DIGEST) Subscribe (INDEX) Unsubscribe Mail to Listmaster