[Angus_D]

Looks like it should be fixed RSN. Now all bow down and kiss my feet.

From: Godfrey van der Linden
Date: Mon Feb 18, 2002 08:50:52 PM Europe/London
To: Finlay Dobbie, [email protected], Christophe Allie , Ron Dumont
Subject: Found: PPP Hang

THANK YOU, THANK YOU, THANK YOU FInlay,

'IOMallocContiguous' ahh all is explained.

Do you have any idea how much SPAM I've been getting and I have JUST not been able to reproduce this problem.

We should have a quick fix for SccSetupReceptionChannel soon, we can just allocate a single page directly, we should audit the use of IOMallocContiguous in all of your drivers.

IOMallocContiguous has a long standing bug where it tries to allocate 2 pages contiguously for 1/2 page to one page allocations, this is unnecessary. This bug was never considered urgent but now it much more important.

As you can see if you need to allocate > 1 page contiguously then you MUST be a boot driver and do it at boot time.

Finally thanks again this is really good info. I personally have only seen this bug once and we really, honestly have not been able to reproduce this bug.

We have had a team of 4 engineers trying to reproduce this in house. I think we don't see it because we reboot our systems for more often than most users so we tend not to frag up the contiguous memory space.

Godfrey

At 20:24 +0000 02-2-18, Finlay Dobbie wrote:
Hi all,

I've been working on tracking down this issue for a while now, and I've narrowed it down quite considerably from the previous "it's happening in acquirePort". Ron, could you add this in comments to the right radar, I forget the number offhand. Also posting to darwin-development because it's happening in a generic IOMallocContiguous call. WHY DOES THIS TAKE 7 SECONDS (and longer for some people, up to 10 or 20 minutes)? :-)
Perhaps dmaInfo->dmaNumberOfDescriptors randomly becomes exorbitantly large, or something.

Edit: Remove e-mail addresses just in case any spam harvesters stumble across this

[ 02-18-2002: Message edited by: Angus_D ]

[Lew]

Woo-hoo!

[sungwoo]

Woo.... In the mailing list, someone said that this PPP hang is due to Mach VM bug. Sounds like it will take a lot of time to fix it!!

But, well~ anyhow, now we know what's wrong.

[ 02-18-2002: Message edited by: sungwoo ]

[normyzo]

Great news!!

I'm not one to knock the software development process, I know how bugs go, been in Godfrey's place a couple times myself. But then again, how could Apple ignore the 100 or so posts that say:

"The longer the machine is up, the longer the hang..."

So many people posted this and he still says he couldn't reproduce it because Apple engineers reboot their machines more often? Its as simple (ok, maybe) as just leaving your damn machine on long enough, as so many people said...

[edddeduck]

Thank god for that

Hey they could hold 10.1.3 for the fix...

Or we with have a modem patch update....

Anyway now they know the problem...

Cheers Edd

[GnOm]

Originally posted by Angus_D:
<STRONG>Now all bow down and kiss my feet.</STRONG>

*bows down and kisses his feet*




bye.

[edddeduck]

Originally posted by Angus_D:
<STRONG>Looks like it should be fixed RSN. Now all bow down and kiss my feet.
</STRONG>

Don't you mean Finlay Dobbie?



Cheers Edd

[Angus_D]

Originally posted by edddeduck:
<STRONG>Don't you mean Finlay Dobbie?</STRONG>

That would be me (duh ).

Oh, and I'd just like to say a great big CONGRATU-FSCKING-LATIONS to all the idiots who spammed Godfrey, Christophe, Avie, Steve etc: your contributions have done NOTHING WHATSOEVER TO HELP THE PROCESS, and they have annoyed the engineers in question. You have made them, and likely others, to not want to work on bugs earmarked opensource through fear of becoming scapegoats for bugs. They are worried that their careers have been impaired because their names are associated with major bugs, even if it was not their fault.

Well done. I'm sure your "help" was greatly appreciated. People like you make me

[Angus_D]

Originally posted by sungwoo:
<STRONG>Woo.... In the mailing list, someone said that this PPP hang is due to Mach VM bug. Sounds like it will take a lot of time to fix it!!</STRONG>

Correction: J.M. was referring the more serious underlying problem in the VM architecture which was manifesting itself. This doesn't actually have to be fixed for the hang to be fixed, since the serial driver probably doesn't need contiguous memory in the first place.

[ 02-18-2002: Message edited by: Angus_D ]

[edddeduck]

Originally posted by Angus_D:
<STRONG>

That would be me (duh ).

Oh, and I'd just like to say a great big CONGRATU-FSCKING-LATIONS to all the idiots who spammed Godfrey, Christophe, Avie, Steve etc: your contributions have done NOTHING WHATSOEVER TO HELP THE PROCESS, and they have annoyed the engineers in question. You have made them, and likely others, to not want to work on bugs earmarked opensource through fear of becoming scapegoats for bugs. They are worried that their careers have been impaired because their names are associated with major bugs, even if it was not their fault.

Well done. I'm sure your "help" was greatly appreciated. People like you make me </STRONG>

Yes but there are loads who didn't I had the right email addresses (I Think) and I poked around with the drivers I didn't find anything which had not already been found by others so I didn't mail.

Get you point though just there are quite a few level headed people who don't spam addresses with meaningless posts. I least I would hope that's true.

Cheers Edd

[edddeduck]

Originally posted by Angus_D:
<STRONG>
That would be me (duh ).
</STRONG>

Well then I will bow down and kiss you feet (After you wash them)

What's with the fake name then? I always thought Angus was your name.

Cheers Edd

[The DJ]

Freaking beautiful work man. I don't use modem (LAN), but saw the bug a couple of times, and i can see it would really piss all the users. Good work on tracking it down. I'll have to remember that little trick if i ever get into trouble.

DJ,
Kneeling, screaming halleluyah and praising Angus D almighty

[Xeo]

It totally shocks me that they couldn't have taken the same initiative you did. You added in logging code and then asked people for system.log lines for confirmation. Oh well. As long as the bug goes away then I'm all for it.

Thanks for the work you put into it. Now we have to sit back and wait for the big boys to do their thing.

[sungwoo]

Originally posted by Angus_D:
<STRONG>Looks like it should be fixed RSN. Now all bow down and kiss my feet.
[ 02-18-2002: Message edited by: Angus_D ]</STRONG>

I do.

[SYN]

I wasn't concerned, but congratulations nevertheless, this had to be one of the most annoying bug in OSX.

Well done. They should hire you or something

[BTP]

Originally posted by edddeduck:
<STRONG>
What's with the fake name then? I always thought Angus was your name.</STRONG>

Maybe he likes steaks? And your real name is edddeduck? ; )

Angus, I have not been affected by this bug, but I have read the many post of those that were and I must say, nice work. I hope you have stomped this bug for good and that it will now be fixed.

[benh57]

Subject: Re: PPP Hang
From: Jim Magee
X-BeenThere: [email protected]
Date: Mon, 18 Feb 2002 22:32:06 -0500

On Monday, February 18, 2002, at 06:51 PM, David A. Gatwood wrote:
On Mon, 18 Feb 2002 [email protected] wrote:

as well ... if this has to do with contiguous memory allocation,
wouldn't it happen less often the more free memory is available (as
reported by top, for instance)?

You'd think that, wouldn't you.... But alas, no. The way that Mach VM
freelist management works (Jim, correct me if I'm wrong here), is that it
keeps a list of blocks of memory of various sizes. When it needs a single
page of memory, it steals a page away from the first block, then returns
that block (in pieces) back to the free list.

Actually there isn't any "block" notion. Just a random list of pages. They start out sorted by happenstance, but quickly get out of order as we are more interested in keeping the right content in memory than keep free memory contiguous.


This means that when you need to find a block of two pages in length, that
entire list has to be sorted. (Well, not really. You'd ideally do a
partial ordering until you found a block that was long enough, then stop,
but that's not what happens currently.) This means that the worst case is
when you have lots of memory and none of it is used (but when a lot of it
has been used in the past).

We sort the whole thing. With the algorithm used, you do get a boost if the list is already partly sorted (it becomes O(N) instead of O(N**2) only for the pages that are already sorted). But as you say, we do not stop when we sorted enough to satisfy the request. So in your 2GB example, the worst case sort would be O(N**2) where N=524288 pages.

That is where the ~7sec delay is coming from in Finlay's trace. The code does not wait for contiguous memory to become available. If, after sorting, we still don't have enough contiguous free memory, we simply return a failure condition. So those who have postured that the (significantly) longer delays are coming from this same area waiting for contiguous memory would be mistaken. It's bad, but not likely the only problem.

And of course, during boot, allocating a contiguous block of memory is
trivial because memory allocated in that fashion early in boot can
generally never be freed. That means that you never need to sort and
merge in order to find a contiguous block of a given length during boot
time, and thus you merely have to search a relatively short free list
linearly and see if there's a block that's big enough.

Sadly, we still do the sort during boot-time contiguous allocations as well. Because the free memory is already contiguous, it becomes an O(N) sort, but it still happens. So obviously, much time could be saved by cleaning this up.

--Jim
_______________________________________________
darwin-development mailing list | [email protected]
Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/l...in-development
Do not post admin requests to the list. They will be ignored.

Emphasis mine.

Again, unless they say "IT'S FIXED", don't get all riled up, folks. Software development is very complex.

[JKT]

Originally posted by Angus_D:
<STRONG>

That would be me (duh ).

Oh, and I'd just like to say a great big CONGRATU-FSCKING-LATIONS to all the idiots who spammed Godfrey, Christophe, Avie, Steve etc: your contributions have done NOTHING WHATSOEVER TO HELP THE PROCESS, and they have annoyed the engineers in question. You have made them, and likely others, to not want to work on bugs earmarked opensource through fear of becoming scapegoats for bugs. They are worried that their careers have been impaired because their names are associated with major bugs, even if it was not their fault.

Well done. I'm sure your "help" was greatly appreciated. People like you make me </STRONG>

No offence, but Apple have known that this issue has been affecting people for nearly a year now and not once have they formally acknowledged this issue even exists. I have gone through all the normal channels (OS X feedback, Apple Discussion boards etc) and received not one single reply or seen a single KB article posted to say that this problem occurs. Yes, I e-mailed Avie Tevanian and Steve Jobs (assuming they even read the e-mails at the "obvious" mac.com, apple.com, etc addresses) and I was polite in my criticism (which considering how long this bug has existed for me was damned hard), but demonstrative of how frustrating this bug has been. When no other form of acknowledgement exists, what else can you expect people to do? ALL IT WOULD HAVE TAKEN is for someone to post at the Apple Discussion boards that (a) they were aware of the problem and, as it was taking them so long to find out what was happening, (b) keep us informed of the progress (or lack of due to an inability to reproduce it). We as end users, sorry sufferers, have been left in an information vacuum... This entire issue has made me

Regardless of all that, thank you for finally kicking them up the arse and getting them to do something about it. Jesus, just how much work and effort didn't that take... a whole f**king year and you practically find the problem in less than a few weeks - JUST WHAT THE F**K WERE THEY DOING ALL THAT TIME?

FWIW, the so-called SPAMing didn't begin until after the 10.1.2 update had arrived and yet again failed to cure this bug. I think all of us were pretty damned patient with Apple over this, don't you?

[ 02-19-2002: Message edited by: JKT ]

[Krypton]

Well done. I'm sure your "help" was greatly appreciated. People like you make me

Well, I sent emails asking why it was taking so long, and could I test any stuff for them. I think may other people sent similar stuff and were ignored - they didn't flame them as they knew this would get them nowhere.

But if the engineers don't communicate to the outside world, it is very difficult for users to grasp what sort of progress is being made - one of the reasons I tried to email them - I don't consider that "Spam".

Anyhow, thak you Finlay - and there I was thinking "Angus" and Finaly were working together on this....

[boots]

Angus

Thanks for the time and effort you spent. I hope this does fix it. It certainly seems to fit the symptoms.

I have a feeling, though, that this is only one of several bugs that are causing the extremely long hangs.


As to the engineers career options being blackened by this bug...they've known about it for over a year now, and had the same opportunity (probably more) that you did, yet they couldn't figure it out. Doesn't sound like they earned their keep. I have little sympathy that their names are associated with the bug. Or that it will (hopefully) be your's that is associated with the fix.

[sr]

I wouldn't be so harsh on the Apple folk working on this. It appears that they were unable to reproduce the problem in-house, which I know (as a developer) is a nightmare to deal with. On top of that they probably receive endless spams complaining about problems which may or may not be: 1) real, 2) important, 3) their responsibility. How do they determine the difference between a non-existent bug reported 1000 times to them because someone on a list has posted saying "SPAM them about this and it'll be fixed", and a real one for which they receive 100 emails distributed randomly around the company and describing the problem in different ways, some of which turn out to be bogus?

If you want a problem fixed the best thing is to report it through the Mac OS X feedback page (http://www.apple.com/macosx/feedback/) or the developer bug-reporter (http://developer.apple.com/bugreporter/index.html), giving as much info as you can. At least that way they get reasonably good information and it all comes in to the same place.

Apple's development team are doing lots of tremendous work and I'd guess they're extremely busy most of the time. Sometime I think folk expect them to be super-human and unfortunately they aren't! (well not quite ).

All that said, I think they could do with having more people allocated to collaborating with external developers and users. Apple users are a generous and talented bunch, and their willingness to help out is something that I reckon is probably a major win-win opportunity waiting to be grasped.

[boots]

Originally posted by sr:
<STRONG>I wouldn't be so harsh on the Apple folk working on this. It appears that they were unable to reproduce the problem in-house, which I know (as a developer) is a nightmare to deal with. On top of that they probably receive endless spams complaining about problems which may or may not be: 1) real, 2) important, 3) their responsibility. How do they determine the difference between a non-existent bug reported 1000 times to them because someone on a list has posted saying "SPAM them about this and it'll be fixed", and a real one for which they receive 100 emails distributed randomly around the company and describing the problem in different ways, some of which turn out to be bogus?

If you want a problem fixed the best thing is to report it through the Mac OS X feedback page (http://www.apple.com/macosx/feedback/) or the developer bug-reporter (http://developer.apple.com/bugreporter/index.html), giving as much info as you can. At least that way they get reasonably good information and it all comes in to the same place.

Apple's development team are doing lots of tremendous work and I'd guess they're extremely busy most of the time. Sometime I think folk expect them to be super-human and unfortunately they aren't! (well not quite ).

All that said, I think they could do with having more people allocated to collaborating with external developers and users. Apple users are a generous and talented bunch, and their willingness to help out is something that I reckon is probably a major win-win opportunity waiting to be grasped.</STRONG>

I wouldn't be so harsh if it wasn't for the fact that

a) I and a number of other people DID send feedback to apple (via the feedback page),

b) the apple discussion forum has a topic that is closing on 500 posts about this problem (and developers were informed of it),

c) the developers were told that it happened after a machine was up for a couple of days...the developers even admit they were having problems reproducing it because they don't usually have machines running that long without a restart (Try leaving one up, morons) and

d) this has been a problem for over a year, and it takes an outsider to crack it.


Item "c" makes me the angriest. How moronic are you to not listen to the people giving you feedback THROUGH PROPER CHANNELS! I never spammed, though I was given the email addresses.

If apple really wanted OS X and Macs to be the center of the digital hub, this is an obvious stumbling block, yet the engineers either didn't care about the problem or were not competent enough to fix it. Either way, it is not good for their resum�s.

And we still don't KNOW if the problem will be fixed any time soon. I started out trying to give the developers the benefit-of-the-doubt, but the more we learn about the story, the more irate I get.


Again Mr. Dobbie, thank you for your work.

[ 02-19-2002: Message edited by: boots ]

[BlackGriffen]

I think that part of the problem may be Apple's culture of secrecy (enforced by SJ). My guess is that they are not only not used to dealing with outsiders, but are expected not to in most cases. For Apple to be successful at being open source, it's going to take a transition of corporate culture, at least at the core OS level.

Having said that, I have to agree with boots: the engineers responsible for fixing this bug really need to take a long hard look at themselves, and learn some humility. It's really simple: following instructions 101. I don't care if the user claims that the problem is only reproducible by doing the hokey pokey while givving an effigy of SJ the bird, you start by doing everything the user did, and then eliminate the unnecessary via trial and error. You don't assume that something will have no effect until it can be demonstrated that it won't.

Last, here's a tip from Unix land for the Apple users: you may not need to restart every day, but it is a good idea to log out from time to time (once a day should be more than sufficient, every two or three days is probably enough). At work I use a Sun Sparc running some flavor of Unix or another with CDE. Even there, my boss always tells me to remember to log out from time to time. I suspect that this should clear up the memory fragmentation problem (it may not be possible for Apple to fix that one, unless they have the kernel spontaneously start rearranging memory from time to time, which at the least means a big performance hit).

I wonder if the hang problem is reproducible right after a login (after the system has been on a couple of days, of course)?

BlackGriffen

[JKT]

Originally posted by BlackGriffen:
<STRONG>
I wonder if the hang problem is reproducible right after a login (after the system has been on a couple of days, of course)?

BlackGriffen</STRONG>

Yes, absolutely it is reproducible right after a login... one of the first things I tried way back in April last year when I first encountered the hang was to login under a different account and try connecting (to see if it was a corrupt pref file in the other account) and the problem was exactly the same. That wasn't the only attempt at this either and the result was equally as bad each time. Hang.

[ 02-19-2002: Message edited by: JKT ]

[boots]

Originally posted by BlackGriffen:
<STRONG>

Last, here's a tip from Unix land for the Apple users: you may not need to restart every day, but it is a good idea to log out from time to time (once a day should be more than sufficient, every two or three days is probably enough). At work I use a Sun Sparc running some flavor of Unix or another with CDE. Even there, my boss always tells me to remember to log out from time to time. I suspect that this should clear up the memory fragmentation problem (it may not be possible for Apple to fix that one, unless they have the kernel spontaneously start rearranging memory from time to time, which at the least means a big performance hit).

I wonder if the hang problem is reproducible right after a login (after the system has been on a couple of days, of course)?

BlackGriffen</STRONG>

If the vm pages are fragmented, does a logout/login really affect it? Certainly a physical memmory fragmentation would be helped...

How does the vm paging system differ from the swap partitions I'm used to dealing with in the unix world (SGI/Irix)?

[theolein]

I have cable, thank God, but I agree with BlackGriffen. I do think it is a rather obvious thing to do in the face of a showstopping bug to block it in and write feedback code to see what's happening. I think it says something about Apple's QA that there wasn't enough stress testing of the system. i.e. Writing a script that connected and disconnected every few minutes while leaving the Mac logged in and up for a month at a time.

All the same I can apreciate that Apple has basically created a Unix variant almost from scratch here and has an enormous amount of work to do. This bug contrasts sharply with the HFS apache security bug that Apple fixed in a couple of days at most.

What bothers me most at the moment though, as I'm learning ObjC is that fact that fairly large parts of the documentation are still missing. I'm going to post a letter to the dev list about this one, because I think that after one year apple has surely had enough time to complete the documentation. Given that Apple has probably been working on Carbon for most of the time (new carbon SDK's every month or so) I assume they have simply not gotten around to it, but ObjC is really the heart and soul of OSX's dynamic features and i makes it very hard to learn it when important aspects of the Docs are missing.

[The DJ]

Originally posted by boots:
<STRONG>

If the vm pages are fragmented, does a logout/login really affect it? Certainly a physical memmory fragmentation would be helped...

How does the vm paging system differ from the swap partitions I'm used to dealing with in the unix world (SGI/Irix)?</STRONG>

This is a post from the darwin-dev list explaining some things about the vm. It is not complete, and i believe there were some additional statements from ppl about some lowlevel stuff that he didn't describe correct, but this will probably do just fine.

&gt; as well ... if this has to do with contiguous memory allocation,
&gt; wouldn't it happen less often the more free memory is available (as
&gt; reported by top, for instance)?

You'd think that, wouldn't you.... But alas, no. The way that Mach VM
freelist management works (Jim, correct me if I'm wrong here), is that it
keeps a list of blocks of memory of various sizes. When it needs a single
page of memory, it steals a page away from the first block, then returns
that block (in pieces) back to the free list.

Say the first block had 4 pages. It steals the first one. But since
those three pages don't start on a 2 or four page boundary and aren't an
even power of two, the block has to be broken up into a 1 page block and a
two page block.

Later, when a page is freed, no attempt is made to unify it with the pages
that are physically near it. As a result, you end up with random,
out-of-order blocks of memory, most of them only a page in length, and not
remotely ordered by size, address, or in any other way.

This means that when you need to find a block of two pages in length, that
entire list has to be sorted. (Well, not really. You'd ideally do a
partial ordering until you found a block that was long enough, then stop,
but that's not what happens currently.) This means that the worst case is
when you have lots of memory and none of it is used (but when a lot of it
has been used in the past).

In a really bad case, with a machine with 2 gigs of RAM (theory, here)
divided into 4k pages, in which no memory is in use (not possible, but
again, theory here), the kernel would have to sort a list of 524288
entries, each one block in length, by address, then merge those blocks
(also probably not stopping until it has merged all of them), then either
sort the resulting list by length (which is a no-op in this case, since
there's only one resulting entry in the list) or go through the entire
list linearly (which is also a no-op in this case).

Technically, the worst case is probably where every other block is used,
since it has to sort by address, then either sort that list by size or go
through the entire list linearly, either of which is really bad....

Of course, the nice thing about the free list is that it is trivial to get
a single page. You just grab the first thing on the free list. Likewise,
it is trivial to discard a page. You just add it onto the free list.

And of course, during boot, allocating a contiguous block of memory is
trivial because memory allocated in that fashion early in boot can
generally never be freed. That means that you never need to sort and
merge in order to find a contiguous block of a given length during boot
time, and thus you merely have to search a relatively short free list
linearly and see if there's a block that's big enough.

It basically boils down to the data structure being heavily optimized for
the most common operations, but whose worst case is horribly bad for
certain operations that are particularly rare. In order to get better
performance for the contiguous requests after booting, you end up
spreading that load out over the other operations.

Whether this results in a net boost or loss depends on the algorithm
chosen, on the frequency of the uncommon operations relative to the common
ones, and on whether you require correctness or just a "best guess"
heuristic (i.e. occasionally saying no large enough block exists even if
one could exist if you did additional merge work or didn't require
alignment on a 2^k byte boundary or whatever).

I know that's probably a lot more than you wanted to know, but.... :-)

[ 02-19-2002: Message edited by: The DJ ]

[The DJ]

Originally posted by theolein:
<STRONG>What bothers me most at the moment though, as I'm learning ObjC is that fact that fairly large parts of the documentation are still missing. I'm going to post a letter to the dev list about this one, because I think that after one year apple has surely had enough time to complete the documentation. Given that Apple has probably been working on Carbon for most of the time (new carbon SDK's every month or so) I assume they have simply not gotten around to it, but ObjC is really the heart and soul of OSX's dynamic features and i makes it very hard to learn it when important aspects of the Docs are missing.</STRONG>

I totally agree. This really is a tough deal for newcomers. I've learned to first look at the docu and immediatly after that at the header of the class. But it can be worse. Any one ever checked out the JAVA docu version of Cocoa? Way worse.

BTW. Cocoa is in no way superior to Carbon or vice versa. There are just a few things one can do and the other can not. Prefpane's, Drawers, Services, drag and drop selected text etc.

DJ

[boots]

The DJ:

Thanks...I thought that was just refering to physical memmory because it referred to 2 gig of RAM. I didn't realize vm was treated in the same way.

[Graham Nelson]

Credit where it's due. I have a lot of respect for anyone who can port Unix to a Mac (i.e. Darwin) and then a Mac to Unix (i.e. Classic) and still have time for a new front end with easy configuration. OS X is one of the great technical achievements of the decade. And I didn't email Apple people directly, I confined myself to the proper forums: the feedback page and the discussions boards. A couple of my comments were picked up and put on the bug report web page, so I wasn't totally ignored. As I say, credit where it's due.

But discredit where it's due as well. Apple got into this position by largely ignoring copious patiently-worded bug reports, back to the Public Beta. The bug status was "analyze", with notes that it could not be reproduced, and little else was done. Apple did not ever say "we know there's a problem, we're working on it", or ask for help to "analyze" the bug. Instead it concluded, apparently, that there wasn't really any problem. Then it pulled the "open source" bug report page so that the URL returned only a 404 error, so that the only remaining acknowledgement of the issue vanished. People phoning the support lines were frequently told to wipe their machines and reinstall OS X and all its updates in sequence, at a time when other parts of Apple knew perfectly well that this would make absolutely no difference. This was a failure of public relations.

Next, the bug reports were misread. The freeze is worse if machines have run a long time since restart; and yet a team of four engineers failed ever to reproduce the bug because, apparently, they never thought to leave a machine working. (Personally, I reproduced it for 4 minutes 40 seconds just now in logging on to post this message; it wasn't hard.) If it really is something like an O(n^2) sorting of free memory pages, then I'll bet the bug reports will tend to be from machines with more RAM. (My own machine has 640M in it.) Had bug reports been thoroughly studied, then, a finger might have been pointed at memory management routines over a year ago. And the fact that freeze delays reduce if one reconnects soon after an earlier freeze was a clue, pointing to some internal change of state, with something deteriorating over time and then being fixed in some way. Memory defragmentation is just such an activity; at one time it was suggested that the modem being sluggish after long periods powered down might be the problem, and that would also qualify. But it was a clue which narrowed the field, and it was certainly to be found on the discussion forums. I don't believe Apple ever logged it on the bug report web page. This was a failure of analysis.

Next, the bug wasn't accorded any priority until, because of a pestering developer, senior Apple staff got to hear about it, despite the huge volume of feedback and bug reports known by lower-level engineers. I don't condone people spamming Apple executives, which is simply rude and counterproductive, but it _is_ true that a user's email to Mr Tevanian changed the priority of the bug, according to the bug page. This should not have happened. It should already have been a high priority issue, especially with the no-reset-button new iMac hitting the streets, a machine whose hardware design encourages indefinite up-times using sleep rather than shutdown. This was a failure of the feedback systems at Apple.

Software development is complex and demanding. The fact remains that, whether the exact cause has been found or not, one person outside of Apple with spare time and competence has made a breakthrough which eluded all Apple's own full-time engineers, by doing (forgive me, Mr Dobbie) the fairly obvious - tracing the code. If the reputation of the relevant Apple engineers has fallen among their peers, I don't think they have any grounds for complaint.

We really do owe a debt to Finlay, though. Bloody well done.

[bluetooth]

Originally posted by BlackGriffen:
<STRONG>
I wonder if the hang problem is reproducible right after a login (after the system has been on a couple of days, of course)?
</STRONG>

indeed it does all the time in fact, usualy when i get home at night i login in, and usually check my mail before anything else, it it freezes almost every single time . although the time frozen changes, some days it will only be a few seconds and others it will be minutes.

but if they can fix this bug properly, then i will really be happy

[Angus_D]

Originally posted by edddeduck:
<STRONG>What's with the fake name then? I always thought Angus was your name.</STRONG>

Angus after Angus Deayton, random person on TV over here in the UK that someone once said I bore a resemblance to. It stuck as my nick on Hotline, so...

Originally posted by Xeo:
<STRONG>It totally shocks me that they couldn't have taken the same initiative you did. You added in logging code and then asked people for system.log lines for confirmation. Oh well. As long as the bug goes away then I'm all for it.</STRONG>

Actually, the PPP guy did this with the help of JM Zorko, but they only managed to get so far. I have no idea why they stopped. I hear that Apple was working on doing this with a select few (probably not enough), I just beat them to the punch. Technically what I did wasn't terribly wonderful, in fact I ripped some of the time delay comparing code from an e-mail someone posted to darwin-development.

I just wish I'd got to it earlier

[squiggy]

Originally posted by The DJ:
<STRONG>
I totally agree. This really is a tough deal for newcomers. I've learned to first look at the docu and immediatly after that at the header of the class. But it can be worse. Any one ever checked out the JAVA docu version of Cocoa? Way worse.
</STRONG>

Don't make the mistake of thinking the Carbon documentation is any better, often the only recourse is to look at the headers as well.

The frequent Carbon releases are mostly bug fixes directed at compatibility with older versions of the Mac OS - this shouldn't be construed as Apple putting large amounts of their resources behind it. I'd be willing to bet the Carbon staff is much smaller than any other OS X group.

Someone posted to one of the development lists that they could port their code to Windows easier than Carbon. While that's probably just the caffeine and frustration talking, it illustrates that things aren't idyllic over in Carbon land either.

Back on topic, I think Graham nailed it pretty well, so what he said .

[Brazuca]

An-Gus!!! An-Gus!! Well done.... Though it was just in time for me to switch to dsl, but I remember those days....

About the Spam, I'm sure that it was done only out of frustration. Imagine if you didn't take the initiative that you did: Apple engineers probably would still be shrugging their shoulders saying: "no problems here".

All's well that ends well....

[apple4ever]

THANK YOU ANGUS! It is about FREAKING time! Almost a year of this, with MOUNTAINS of information and TONS of people reporting this, it took a non-Apple person to finally fix this!
From reading the bug reports, and reading Apple's feedback, it seemed to me that Apple and its developers didn't want to fix this. They kept passing in on to each other. After patiently waiting, users got fed up of this annoying bug, and started taking action. They finally got Apple to pay attention, but it still didn't do anything. Took somebody on his own initiative, who is unpaid and doesn't work for Apple, to figure it out. The developers said they couldn't reproduce it, but Geoff admits why: they failed to listen to the users. They didn't keep their machines up that long, and yet that was the central trigger cited for when the bug would occur. All they had to do was LISTEN! Yet they ignored.

My hat is off to you Angus, excellent work.

[theolein]

Angus, excuse me if this is a stupid idea: To fix this issue, wouldn't it be possible to create some sort of idle handler i.e. a low priority process that automatically starts sorting out the memory when the user is doing nothing. Since a good deal of the time a lot of computer cycles are not used when the user is reading or just looking at the screen I would assume that it would be possible and not force a rewrite of important system functions.

[philzilla]

Originally posted by JKT:
<STRONG>Yes, I e-mailed Avie Tevanian and Steve Jobs (assuming they even read the e-mails at the "obvious" mac.com, apple.com, etc addresses)</STRONG>

i find it amusing that you sent them mail, yet you hide your own email address on these fora. why's that? so you don't get mail from people you don't know about things that aren't your problem?

[DannyVTim]

I believe the point has been made or asserted again and again that Apple simply didn't try to fix the problem. This assertion seems unfounded and unfair to the individuals at Apple. Moreover, it seems to lack the understanding of how hard it can be to find bugs. Once found, the answer seems obvious, but it’s only obvious in hindsight. I’m sure the individuals at Apple were trying to find the bug because of the profession of relief expressed in the e-mail. If Apple didn’t care about the problem, they wouldn’t be so relieved that it was fixed.

[apple4ever]

Originally posted by DannyVTim:
<STRONG>I believe the point has been made or asserted again and again that Apple simply didn't try to fix the problem. This assertion seems unfounded and unfair to the individuals at Apple. Moreover, it seems to lack the understanding of how hard it can be to find bugs. Once found, the answer seems obvious, but it�s only obvious in hindsight. I�m sure the individuals at Apple were trying to find the bug because of the profession of relief expressed in the e-mail. If Apple didn�t care about the problem, they wouldn�t be so relieved that it was fixed.</STRONG>

Did you see the bug report over at opensource.apple.com? Because it was obvious that they didn't care about the bug. They kept passing it to one team or another saying it wasn't there responsibility or they couldn't reproduce it. Only after 10.1 was released, and then mostly after 10.1.2, that they outcry really began. People were submitted tons of bug reports to the OSX Feedback page on this. In fact the one Apple Developer said it was the biggest hitter on the feedback page! People were also email the head PPP guy(Geoff in the letter above), and many other high level Apple people. They were fed up of being ignored. The Apple Discussion Thread on this has reached 500 or so replies! That thread was also referenced on the bug report. And the "relief" that Geoff expressed was because he was getting massive amounts of mail on this! Are you now so sure Apple cared about getting this fixed because of the professed relief, which was based on angry emails and not caring about the bug?

[Brazuca]

Originally posted by Brazuca:
<STRONG>An-Gus!!! An-Gus!! Well done.... Though it was just in time for me to switch to dsl, but I remember those days....

About the Spam, I'm sure that it was done only out of frustration. Imagine if you didn't take the initiative that you did: Apple engineers probably would still be shrugging their shoulders saying: "no problems here".

All's well that ends well....</STRONG>

Well, I jinxed myself. Earthlink just disconnected me because of a silly mistake ON THEIR PART! So I'm back to dial-up (and the bug) until, get this, the cancellation order goes through, Verizon releases my line, then I place a new order, and the reconnect me.

For some reason they can't simply STOP the cancellation. Geniuses...

Now Apple, when are you going to release the fix???

[electropura]

Originally posted by Brazuca:
<STRONG>

Well, I jinxed myself. Earthlink just disconnected me because of a silly mistake ON THEIR PART! So I'm back to dial-up (and the bug) until, get this, the cancellation order goes through, Verizon releases my line, then I place a new order, and the reconnect me.

For some reason they can't simply STOP the cancellation. Geniuses...

Now Apple, when are you going to release the fix???</STRONG>

Uh oh...I went thru this cancellation fiasco myself.....it took 6 months to get my service up and running again...the only way to do it was to change my phone number....Earthlink would swear up and down that they had released my line, Verizon would claim they didnt. God help you....

[Frank Restly]

What the fuc!&lt; ever happened to TESTING AN OPERATING SYSTEM? Apple in its early days had a program that was specifically designed to generate random system events to test the system stability, and Apple would let that program run for days / weeks on end.

That kind of program would reveal all sorts of memory related problems if allowed to run continuously (Memory leaks, virtual memory allocation problems, etc.)

The "Internet Connect Bug" will be just the beginning, unless Apple is willing to spend the a little bit of resources for a 24 hours per day / 7 days a week "proof test" of their newest operating system.

More beta testers is NOT the answer for long term stability. There is no way for Apple to "force" a beta tester to leave his / her computer on for an indefinite period of time. Instead Apple needs to have a dedicated group of G3, G4, G5? computers running Mac OS X 24 / 7.

[michaelb]

Originally posted by Frank Restly:
<STRONG>What the fuc!&lt; ever happened to TESTING AN OPERATING SYSTEM?</STRONG>

What the fuc!&lt; ever happened to FITTING AN OPERATING SYSTEM on a 400K 3.5 in floppy disk, with enough room left over for a dozen apps...?!

In short, the complexity of any modern OS has increased by several orders of magnitude from the early 80s. With complexity comes potential for bugs.

I can't remember it, but I'm sure there is a law for it like Metcalfe's Law for Networking which states that the usefulness of a network increases by the square of the number of users.

For programming, it would be something like: the potential for bugs increases by the number of lines of code raised to e^3, tripled by the square of the number of 3rd-party devices it interacts with, and raised to a googleplex times the number of users.

You do the math, but you'll need Mathematica and a 64-bit processor.

If you think bug-testing can be completely taken care of with a couple of dedicated machines and a random keystroke generator, I've got this 17th century wristwatch to sell you, keeps perfect time...

I heard an interesting statistic the other day: there is more computing power in the graphics processor of the nVidia GeForce 4 Titanium, than existed on the whole of the planet in 1985...

[Angus_D]

Originally posted by apple4ever:
<STRONG>Did you see the bug report over at opensource.apple.com? Because it was obvious that they didn't care about the bug. They kept passing it to one team or another saying it wasn't there responsibility or they couldn't reproduce it.</STRONG>

This is a mis-interpretation and I can see why the engineers were getting upset: people with no understanding whatsoever of the Apple bug-fixing process or any of the issues involved are reading bug reports and jumping to conclusions.

[Krypton]

This is a mis-interpretation and I can see why the engineers were getting upset: people with no understanding whatsoever of the Apple bug-fixing process or any of the issues involved are reading bug reports and jumping to conclusions.

True, but you can't exactly people to be ecstatic when the same pesky bug is still with us after over a year. When people are seriously ticked off they start blaming the people responsible: Apple engineers.

This would happen with any product. The only difference this time is people couldn't quite resist the temptation to email Apple direct.

New driver Finlay?
If the engineers are altering your driver, will you be able to pass it along if Apple decides to hold it back till 10.2?

[KellyHogan]

Originally posted by michaelb:
<STRONG>

I heard an interesting statistic the other day: there is more computing power in the graphics processor of the nVidia GeForce 4 Titanium, than existed on the whole of the planet in 1985...</STRONG>

And yet the proportional usefulness is far lower.

[AKcrab]

Originally posted by KellyHogan:
<STRONG>

And yet the proportional usefulness is far lower.</STRONG>

HAHAHAHA!!! Good one!

[slboett]

Originally posted by Frank Restly:
<STRONG>What the fuc!&lt; ever happened to TESTING AN OPERATING SYSTEM? Apple in its early days had a program that was specifically designed to generate random system events to test the system stability, and Apple would let that program run for days / weeks on end.

That kind of program would reveal all sorts of memory related problems if allowed to run continuously (Memory leaks, virtual memory allocation problems, etc.)

The "Internet Connect Bug" will be just the beginning, unless Apple is willing to spend the a little bit of resources for a 24 hours per day / 7 days a week "proof test" of their newest operating system.

More beta testers is NOT the answer for long term stability. There is no way for Apple to "force" a beta tester to leave his / her computer on for an indefinite period of time. Instead Apple needs to have a dedicated group of G3, G4, G5? computers running Mac OS X 24 / 7.</STRONG>

Yes, it's that easy.

[Jablabla]

A non expert user I know had "problems". They use a modem and ppp to connect to earth link. In addition, one thing I remember was they said it would connect to the modem when they were not using the computer. ( when it was idle) They switched back to os 9 a week after I set 10.1 up for them. I was always perplexed because I seemed not to have many bad issues. I could not really update them that much because some of those update files are to big for modem users.

Well, it sounds like a QA problem to me. Are the users really being emulated? Most soccer moms probably use modems still. Maybe the QA dept. needs a computer that is always hooked up to a modem. ( and nothing else ) What is the user experience?

I can sympathize with some of the engineers though. You see the same code everyday. If you do not see the problem within a few days you can become blind. You think your assumptions are correct. So, we get a new pair of eyes and bam. I guess that is a real plus for open source. (assuming this is the fix)

Great job!

[Macnerd]

There seems to be some rough stuff being said here and on other MacNN forums rgarding the PPP hang and Apple's attitude to it; Some of it is unpleasant but true, some of it is just plain stupid. I'm hoping that my post falls into the first category.

I have had this problem since I started using OSX (i.e. from the day it was released as 10.0 here in the UK). I leave my machine up and running 24/7 (as advised by an Apple support agent). The bug is an embarrasing thing to happen to an OS which is touted as being the most advanced in the world. The problem I have is that it seems so many people reported this bug in the correct manner (sending feedback to Apple via their specified e-mail address) and yet it took this long for it to finally be "recognised". The bigger issue is that Apple still have not officially fixed it and it has taken someone totally uninvolved with Apple Computer to find a way around the problem.

OSX is a magnificent piece of work IMHO - I adore using it and could never go back to 9.x. It still has some rough edges but these can be ironed out as the OS matures. Having said that, I see this bug as one of the most basic and rudimentary which was EASY to replicate. I've been following posts on it around the web for ages now and I would bet that almost everyone who reported it could get it to happen on demand. It's simple - Leave the Mac on overnight (asleep), try to connect in the mnorning and then find something to do while the entire machine stutters to an unusable crawl (it especially bad if you start listening to something with iTunes before you connect). I just restart now since I can't be arsed to wait for it to free up.

There are those who argue that the Engineers should somehow be spared from being criticised by us lot (the people who purchased and use the OS); I don't agree. In my job (I often get involved with customer service) I am accountable for any failures and I am at a loss to see why Apple's engineers (and/or those around them), who only needed to do some minor searches of forums and their feedback e-mails to find out more about this problem, should not be equally accountable. Insulting them etc. is NOT trhe way to go - They're human beings and deserve some respect for what must be a difficult and often thankless job, however it is evident that somewhere along the way the message has not got through, despite the efforts of everyone affected by this bug, that it is unacceptable for a simple dial up attempt to disable the entire machine.
As a layperson I cannot claim to know what causes this problem (I only understand some of the postings I have seen on the reason for it and it appears to be to do with memory allocation) but I do know that it frustrates me day in, day out. I can't be the only one. When OSX was initially released, there were swarms of people who, despite being told it didn't support DVD playback (and a ton of other stuff) whined terribly about it. I chided them for it, since in my opinion DVD playback is not a crucial part of the machine, however communication with the outside World is, and this bug should have been found and sorted before all of the frills were added to the OS.
I hope that Apple can get a fix out soon for this, since I'm a little uncomfortable with using 3rd party mods to my system (no offence to the person who found the problem intended, but I find UNIX pretty scary). As an aside, it would be nice to be able to not have to suffer the jibes of my PC owning friends every time I try and open a PPP connection in their presence.


Macnerd, UK.