Welcome to the MacNN Forums.

mikkyo · Apr 20, 2003, 04:43 PM

Is it just me?

reader50 · Apr 20, 2003, 05:12 PM

The root page loaded right away for me. Checking the box revealed loads of 8-12 and 2GB of swap used. Looking into it.

reader50 · Apr 20, 2003, 06:15 PM

box has been restarted. It may take a while to catch back up.

Shaktai · Apr 20, 2003, 06:33 PM

They're working fine now, and are catching up.

proux · Apr 20, 2003, 09:08 PM

Thanks for the quick repair...

OneMacGuy · Apr 21, 2003, 02:02 PM

The team stats for the SETI team have not updated in probably around 24 hours or so, my guess?

reader50 · Apr 21, 2003, 05:15 PM

Working on that now. There was some database corruption yesterday. Corruption is deleted, and it's been parsing fresh data all along. With a little luck, it will be caught up soon.

reader50 · Apr 21, 2003, 09:18 PM

Removed an earlier corrupted block from 2.4 days ago. This sent the stats back further, but it will catch up faster now. Courtesy of the bad block, it was processing us as a 2,000-member team.

OneMacGuy · Apr 25, 2003, 12:16 PM

Originally posted by reader50:
Removed an earlier corrupted block from 2.4 days ago. This sent the stats back further, but it will catch up faster now. Courtesy of the bad block, it was processing us as a 2,000-member team.

It appears that something has gone wrong again. The actual number of Work Units done is out of synch with the SETI page.

This new system is really cool, but it is certainly slower and seems more prone to problems.

enola · Apr 25, 2003, 12:23 PM

The FAH stats don't look like they are refreshing either.

reader50 · 01:29 PM

You can see how old the data is by checking the bottom of the QuickStats bar for the Time ref. The time referenced is how long ago the data was fetched from the project servers.

In order to speed up processing and page loads, we have been spreading out the stats over two extra boxes during the last few weeks. It now looks something like this:

Main server (AXP 2000, 1 GB)
Serves all web pages and hosts the MySQL database server.
DB is currently over 5GB.

Server #2 (G3 300, 176 MB)
Handles RC5-72, Ubero, D2OL.

Server #3 (AXP 2100, 256 MB)
Handles SETI, Fold, dFold.

The last few days' problems have come since we move all three of the heavy-processing projects to Server #3. It needs more RAM, and assorted tweaks were unable to work around that. I've moved dFold back to Server #1 and made a data retention change that will cut perhaps 20% off the DB size. The DB size affects many parts of the stats, especially RAM demands.

Server #3 should recover and start catching up SETI and Folding, dFold will start catching up again on Server #1. I'll check on them further throughout the day.

Apologies guys and gals, we'll upgrade hardware further as Scott's finances allow. And I'll keep working on code improvements.

OneMacGuy · Apr 25, 2003, 01:53 PM

Originally posted by reader50:
You can see how old the data is by checking the bottom of the QuickStats bar for the Time ref. The time referenced is how long ago the data was fetched from the project servers.

In order to speed up processing and page loads, we have been spreading out the stats over two extra boxes during the last few weeks. It now looks something like this:

Main server (AXP 2000, 1 GB)
Serves all web pages and hosts the MySQL database server.
DB is currently over 5GB.

Server #2 (G3 300, 176 MB)
Handles RC5-72, Ubero, D2OL.

Server #3 (AXP 2100, 256 MB)
Handles SETI, Fold, dFold.

The last few days' problems have come since we move all three of the heavy-processing projects to Server #3. It needs more RAM, and assorted tweaks were unable to work around that. I've moved dFold back to Server #1 and made a data retention change that will cut perhaps 20% off the DB size. The DB size affects many parts of the stats, especially RAM demands.

Server #3 should recover and start catching up SETI and Folding, dFold will start catching up again on Server #1. I'll check on them further throughout the day.

Apologies guys and gals, we'll upgrade hardware further as Scott's finances allow. And I'll keep working on code improvements.

What type of memory does Server #3 use? I may have some lying around gathering dust.

mikkyo · Apr 25, 2003, 03:37 PM

ACk!
176MB??? I really really hope that isn't running OS X, but it must be.
You guys make a $2 paypal donation link.
I'm sure many users would donate $2 towards upgrading the servers.
It looks like you have old servers too.
There may even be some nice folks willing to donate like OneMacGuy.
Should be able to get 512MB DIMMs for $50 or less.
And no, Scott can't use the new ram in his farm

The stats are TeamMacNN's biggest draw and asset, those servers should be RAM happy.

enola · Apr 25, 2003, 04:56 PM

Originally posted by mikkyo:
The stats are TeamMacNN's biggest draw and asset, those servers should be RAM happy.

I'm with mikkyo. Our stats is one of the main reasons I'm with the team. Set up a paypal account and I'd be glad to donate some bucks.

DAlex · Apr 25, 2003, 05:37 PM

Originally posted by enola:
I'm with mikkyo. Our stats is one of the main reasons I'm with the team. Set up a paypal account and I'd be glad to donate some bucks.

Count me in! Ditto for mikkyo.

DAlex

reader50 · Apr 25, 2003, 06:57 PM

Ok, to post a few more details...

Server #1 started out as the only box. It served pages, ran stats, and hosted the database. It had too little swap, and was having a bad time running my earlier less-optimized code. Scott added swap space and my code has improved.

Still, we now have 6 projects and hits are way up - we track 59 teams all told and had over 3K hits yesterday. The trend has been steadily upwards since the beginning of the year. Every foreign visitor gets a good look at our logo and name every time. It's not exactly a banner ad ... well, maybe it is. We have had a small but steady stream of joiners, hopefully some of that is from the stats services.

Anyway, to keep pages loading acceptably fast, we have been offloading box#1. SETI was added on box#3, which worked well. Then Ubero was added, also on box#3. A few days ago, I moved the other four projects off box#1 and onto the others, but we've had to fix a few problems along the way. dFold just got moved back to box#1, and dFold/Fold/SETI are all catching up now at 2x realtime speed.

Box#1 is an AXP 2000 (1666MHz) running Linux.
typical load is 1-2 but can vary wildly depending on page loads.
1GB currently, takes PC2100 or PC2700.
Serves all pages, hosts the database, and processes dFold stats. A Folding@home client is running at nice 20 on Scott's account. So just remember that you can cut Scott's production by checking your own stats more often.

Box#2 is a PM8500 upgraded to a G3 300MHz running Linux.
typical load is 1-2
176MB currently, takes 5v 168 pin dimms. Either FPM or EDO, but not intermixed. I don't know which is in it now.
Processes Ubero, RC5-72, and D2OL stats. A dFold client is running at nice 20. If we add any other "small" projects, this box could take more.

Box#3 is an AXP 2100 (at 1950MHz) running Linux.
typical load is 1-3, but was sometimes over 10 with heavy paging when trying to run the big 3 projects concurrently.
256MB currently, takes PC2700.
Processes SETI, Folding. A Fold client is running under Wine at nice 20.

We have fresh pictures, I need to get the updated pics onto the Pictures page so people can see the servers. Scott may give us a box#4 before too long by moving the SetiQ to an older dedicated box.

As for RAM or paypal offers, I'll leave those to Scott and stick with the code side. Still, thanks to everyone. This is a good team.

OneMacGuy · Apr 29, 2003, 09:47 AM

The SETI stats page is not updating again. Maybe we need to pursue the necessary means to get some more memory into that box.

reader50 · Apr 29, 2003, 12:18 PM

Looks like Apache didn't start up on that box after some routine work last night. I've sent a message Scott's way.

reader50 · Apr 29, 2003, 04:57 PM

Apache got reinstalled and restarted. Fold stats will catch up in a few hours. SETI will catch up more slowly, I'll accelerate it tonight after work.

OneMacGuy · Apr 29, 2003, 10:35 PM

Originally posted by reader50:
Apache got reinstalled and restarted. Fold stats will catch up in a few hours. SETI will catch up more slowly, I'll accelerate it tonight after work.

I REALLY like whatever you did to the server! I picked up about 5-6 work units per day and finally managed to go back ahead of lasvegasgamer in my average per day!!!

Look out Cobra, I will be passing you by in about 24 days at this rate. As soon as I pass Cobra, I am going to start transitioning to dFold. It may be time for some new challenges.

OneMacGuy · May 7, 2003, 08:14 AM

Originally posted by OneMacGuy:
I REALLY like whatever you did to the server! I picked up about 5-6 work units per day and finally managed to go back ahead of lasvegasgamer in my average per day!!!

Look out Cobra, I will be passing you by in about 24 days at this rate. As soon as I pass Cobra, I am going to start transitioning to dFold. It may be time for some new challenges.

I think that the stats server is getting behind again? I know that the SETI server that we get WU's from was dropping connections last night, I had several CPUs setting idle for hours.

OneMacGuy · May 7, 2003, 11:19 AM

Originally posted by OneMacGuy:
I think that the stats server is getting behind again? I know that the SETI server that we get WU's from was dropping connections last night, I had several CPUs setting idle for hours.

I see someone fixed it! It was down for a short while and is now back up to speed. Thanx!!

bousozoku · May 7, 2003, 12:20 PM

It's probably not a good idea to talk to yourself too much. People might start to wonder.

reader50 · May 7, 2003, 03:29 PM

I didn't do it!

If Berkeley is dropping web connections, teamstats will abort on incomplete pages, rather than accept bogus data. Sounds like when Berkeley settled down, teamstats resumed normal updates.

OneMacGuy · May 7, 2003, 05:23 PM

Originally posted by bousozoku:
It's probably not a good idea to talk to yourself too much. People might start to wonder.

No need to wonder, I am definitely crazy!

That's what all of the Windows developers say about me anyway, as I sit over here in the corner and code away on my trusty old Mac that never crashes!!

Scotttheking · May 15, 2003, 04:06 PM

If someone wants to donate a shiny new powermac to me, I'll be happy to get a dual athlon mp 2400 system with 2GB of ram for doing stats. Otherwise, my G4 400 keeps reminding me that WC3 and UT2K3 require slightly faster processors

The main bottleneck that I can see, other then not enough ram (you can never have too much ram in a database server), is that the database is very large (currently at 5.2GB), and it's living on an IBM 60GXP 40GB HD, which is not the fastest drive around.

OneMacGuy · May 21, 2003, 11:05 AM

It appears that the stats are not updating. I hate to keep bothering you guys, do you want us to tell you when we spot a potential problem?

reader50 · May 21, 2003, 01:00 PM

Originally posted by OneMacGuy:
It appears that the stats are not updating. I hate to keep bothering you guys, do you want us to tell you when we spot a potential problem?

Yes. If no one says anything, it may not get noticed for some time.

Scott, please check Box#3. It is not responding, not even from the LAN.

Scotttheking · May 21, 2003, 02:50 PM

Originally posted by reader50:
Yes. If no one says anything, it may not get noticed for some time.

Scott, please check Box#3. It is not responding, not even from the LAN.

Looks like my lunch break will be spent driving home and back.

reader50, if stuff like that happens, send me a message on my phone, since that requires a trip home, and dashing home during lunch takes me a little while. I usually don't check the web until after I eat lunch.

Scotttheking · May 21, 2003, 03:36 PM

It is back up.
It looks like it went down at about 4:45am

I'd like to thank the good folks at folding at home, who's project I will not be running any more, for making a client that is capable of hard locking a system. If it really is processor errata at AMD, I'm asking for new processors.

Scotttheking · May 21, 2003, 07:26 PM

it's almost caught back up, give it another 30 minutes and it will be.

--Scott

OneMacGuy · May 23, 2003, 07:56 AM

Originally posted by Scotttheking:
it's almost caught back up, give it another 30 minutes and it will be.

--Scott

The SETI stats are not responding again.

Scotttheking · May 23, 2003, 10:11 AM

Originally posted by OneMacGuy:
The SETI stats are not responding again.

But luckily not down.
box1: load average: 7.13, 6.66, 5.85
box3: load average: 1.83, 2.40, 2.08

siliconman · May 23, 2003, 10:29 AM

Originally posted by reader50:
Yes. If no one says anything, it may not get noticed for some time.

Is this why Rc5 Stats are saying:

Reference
Time: 2.6 days ago
Active: 2.6 days ago

reader50 · May 23, 2003, 10:35 AM

Box3 is healthy this morning. Box1 is finishing off the night's repair scripts, and mostly waiting on mysql HD access. The raw data has been parsed and will presently catch up.

I need to work on the script scheduling this weekend.

reader50 · May 23, 2003, 11:05 AM

Originally posted by siliconman:
Is this why Rc5 Stats are saying:

Reference
Time: 2.6 days ago
Active: 2.6 days ago

Good question, my local copy says 16 hours. Will look into it this afternoon or evening, depending on available time.

reader50 · May 23, 2003, 04:30 PM

RC5 unstuck.

OneMacGuy · May 27, 2003, 06:59 AM

Originally posted by reader50:
RC5 unstuck.

DF and SETI are responding so slow that the page never finishes loading. These new stats pages are awesome but they sure seem to be significantly slower and more prone to crashing.

reader50 · May 27, 2003, 12:07 PM

Sorry guys, I'd meant to spread the heavy scripts more evenly around the clock this weekend. Got working on other stuff instead.

teamstats does heavy repair scripts after midnight PST, and lately the database has gotten large enough that it's taking until after 9 AM PST to finish. I really need to spread the heavy stuff around more evenly. Tonight maybe.

OneMacGuy · May 27, 2003, 01:26 PM

Originally posted by reader50:
Sorry guys, I'd meant to spread the heavy scripts more evenly around the clock this weekend. Got working on other stuff instead.

teamstats does heavy repair scripts after midnight PST, and lately the database has gotten large enough that it's taking until after 9 AM PST to finish. I really need to spread the heavy stuff around more evenly. Tonight maybe.

As usual, your efforts are greatly appreciated! If I could only invent a device that would allow one to be able to squeeze 30 hours into a day now!

Scotttheking · 02:13 AM

Just to give you an idea, this is my server load.
All times are Arizona time.

Daily

Weekly

Monthly

I can fix it very quickly

PM me to get the address to send the dual opteron server with SCSI raid and 16GB of ram to.

Edit: Changed the links to live ones

mikkyo · May 28, 2003, 01:59 AM

oh that's the problem..
Running those metric gathering programs on the load of the machine, which is reflecting the load on the databases just sucks even more cpu time

It's like upping the overall load with the load-montioring load on the loading of the overloaded database.

No wonder it is slow.

Scotttheking · 02:20 AM

Originally posted by mikkyo:
oh that's the problem..
Running those metric gathering programs on the load of the machine, which is reflecting the load on the databases just sucks even more cpu time

It's like upping the overall load with the load-montioring load on the loading of the overloaded database.

No wonder it is slow.

It samples every 5 mins. I also get some nice bandwidth graphs. They tell me fun things like my internal NIC on my server is moving an average of 100KB/sec of data out, and getting an average of 10KB/sec of data back in.
I also know that my external NIC is averaging 14KB/sec in, and 1.7KB/sec out.

reader50 · 03:42 AM

SETI, Fold, and dFold now have distributed repair times spread around the clock. The other processing scripts on those projects had been spread around all along.

The remaining projects are much smaller data-wise and do not appear to be a problem.

We'll see how the boxes do tomorrow (today) and perhaps persue additional modifications.

Scotttheking · May 29, 2003, 01:14 AM

I just changed everything around. No idea if it'll work or not. Only time will tell.

OneMacGuy · Jun 6, 2003, 09:35 AM

The SETI stats server is not responding again and the dFold stats server is extremely slow. Maybe running some kinda update this morning?

Scotttheking · Jun 6, 2003, 07:54 PM

The graphs at the top of page 2 are my server load graphs.
If it's at 3 or above, it's going to be very slow to unresponsive.
We are working on reducing the load from those scripts.

Scotttheking · 03:30 AM

Preemptive I know that seti and f@h are out of date.

It's too late to deal with tonight, I'll let them catch up on their own, they should be back up to date by tomorrow midday.

Edit: now only seti is out of date, it might be caught up in a few hours.

Scotttheking · Jun 13, 2003, 05:18 PM

To everyone who contributed, I'd like to thank you all for the load average of 28 that I had two hours ago.
My machine was SWAMPED.
The load average hit 7 at abokut 8:40am, and at 2pm I'm watching it fall down again (I killed apache and it's 75 active threads)

reader50 · Jun 13, 2003, 05:32 PM

Must be some new email 'bots around.