[DAlex]

Statsmans data has not been updated in 24 hours and Densman has closed down. Is there anyone else doing 3 or 6 hour dfold stats?

Our stats page is cool but I'm needing a smaller breakdown to asses what difference new machines are making as they are added by my mates.

(Ok, I need to know how much MORE work I have to do to keep RIP from driving right over me!)

DAlex

[Shaktai]

Me too! No statsman since 3:00 AM US eastern time (the 13th). Oh well, by morning readers stats will be updated. My bedtime now though, can't wait for them. (Yeah, I know it is already morning for most folks, but I work a swing shift).

[Scotttheking]

We do accept stats feature requests

[DAlex]

Originally posted by Scotttheking:
<STRONG>We do accept stats feature requests </STRONG>

DOH! I never thought of that! oops.

(sound of tape rewinding) I need my stats fix! Statsmans data has not been updated in 24 hours and Densman has closed down. Is there any chance of our stats page carrying 3 or 6 hour updates?

DAlex

[ 05-14-2002: Message edited by: DAlex ]

[Scotttheking]

Originally posted by DAlex:
<STRONG>


DOH! I never thought of that! oops.

(sound of tape rewinding) I need my stats fix! Statsmans data has not been updated in 24 hours and Densman has closed down. Is there any chance of our stats page carrying 3 or 6 hour updates?

DAlex

[ 05-14-2002: Message edited by: DAlex ]</STRONG>

What stats exactly?

[enola]

Originally posted by DAlex:
<STRONG>Is there any chance of our stats page carrying 3 or 6 hour updates?
</STRONG>

That would be really cool. I've got a bag of candy bars here if it would help.

[reader50]

Multiple stats updates per day are a long-term goal. Multiple member stats updates would be possible on the current server, but a lot of code would have to be rewritten, much of our code currently assumes the once-per-day update.

When making our code assumptions, I had figured the "Current" listings would give live stats, and the other lists would not need to update more often than daily. At the time, we only did SETI (which offered no production or rate listings) and RC5 (which offered Yesterday rates, once per day). I extended the Yesterday tracking to SETI, then offered longer term comparisons to all projects. This matched the best available on RC5, then exceeded all of the projects. Statsman was nowhere to be seen at the time, or we were just unaware of him.

Personal Stats are intended to be upgraded with personal production tables and graphs (as live as your data in the Current listing), along with Ranks Graphs showing you against nearby members (as live as the history data, currently once-per-day).

Scott has a dedicated box online again, which is what we would need for Ranks to update more than once per day. Ranks are the hardest of all stats because we basically have to track member stats on lots of other teams, 30-50 other teams per project (varies by project). Although our tracking code for them does not have to be as complete as for our own members, it still runs ~20x as much CPU / bandwidth as for our own stats. That many stats runs also produce a much higher chance of glitches, which need to be monitored.

Today, Ranks run once daily on my box for each project. It takes a total of ~90 minutes of CPU time per day to execute all code for the day, along with any needed fixes. Code rewrites and the dedicated server would allow more updates of stats checks on nearby teams, which would in turn allow more frequent Team Ranks updates.

I would love to outdo Statsman in every category related to our Team. I'm not interested in providing stats tracking for other teams though, our stats are part of our competitive advantage, as well as a recruiting aid. If they want cool stats, they can always join our Team.

The main holdup now is my free time, gotta maintain the real job in the real world that pays real money. My site upgrade plans go something like this, with the order not fully fixed, and no particular dates for each goal:

• Keep adding missing resource pages. Eight missing pages have been added so far, others are still to go. Incidentally, we are over 140 pages now. Note to those counting pages, Personals count as two pages. A dynamic page you reach with a valid user number, and the generic listing if the user number isn't valid. We also have a few error pages that are not listed, but may eventually get tied into the Apache config. Example
• Add Genome stats as needed. This isn't too hard since our code now clones readily into new projects.
• Once we get a Mac client, Genome resource pages will be needed also. Those are not so easy.
• Rewrite Personals module for personal production tables, personal production graphs, and personal ranks graphs showing nearby members. Also allow Current data for your own totals (outside the personal ranks) if feasable, duplicate membername resolution may not allow that part to be accurate, in which case live data there would have to wait on livlier stats across the board.
• Rewrite our member matching code for better duplicate resolution. We use a better method that Ars or other folks, and much better than Statsman (who just lumps duplicates together). But we could be better still.
• Consolidate almost-identical code sections into shared functions. We are ~30,000 lines of code today, much of it near-duplicate modules across projects, and across comparison periods. Pulling a ton of that code into functions would make upgrades easier, along with general code maintenance. Currently, the tables and graphs are produced by shared functions, and those never seem to glitch up. This should be extended to a ton of other code sections.
• Look into multiple Team Ranks runs per day on Scott's server, using minimal code modifications of current code. This would be less efficient, and would eat lots of Scott's disk space, but it would get us livelier data sooner.
• Convert our code to database-driven with SQL, we currently use a custom files system, with over 10,000 data files and about 300 MB of disk space on my box. The MacNN server copy omits the Ranks stuff, that cuts their figures down to ~2,500 files and 70-80 MB. Note, separate files are much more resistant to data corruption, that is why Apple pulled preferences out of the resource fork and into separate files in the Preferences folder. This conversion will require all-new backup and glitch-maintenance routines, but it will also make more-frequent updates a lot easier across the board.
• Add more projects to our backyard, as members request. There are Mac clients for GIMPS and Evo@home for example. I did not find GIMPS too interesting, and we have no Altivec advantage like we do in RC5. In GIMPS, the P4 rules. Evo@home looked to have a dead-ish website, I have yet to investigate further. There may be another project or two with Mac clients out there, several existing projects plan to release such clients in the future.
• Move our stats to a 1-hour update schedule to beat the pants off Statsman. Allow for even faster updates if he tries to one-up us on live stats. Being second place in stats coverage for our own team is not acceptable, nor even our team's analysis of our prospects against nearby teams.
• Assorted code maintenance as needed. Every once in a while, one or more projects get the "mess with our stats listings" bug. When they do, everyone else's stats break. Historically, SETI has done this the most frequently, but all projects have created issues from time to time. The team stats guys for every team with their own stats have to scramble to fix code before we lose history data, or worse, get miss-parsed data corruption into our history data. None of the projects provide "safe" stats pages for data tracking by us, that is a common assumption which is incorrect. We have to check the same public pages that others do, and just apply detailed analysis on the data. Some teams are not able to keep up, MacAddict did not on SETI. With multiple page changes not corrected for, and weeks of downtime (weeks of missing data in their history, or not in them depending on your viewpoint), they were unable to get their scripts to work again. They gave up, and the largest Mac Seti team currently has no stats. They will have to do a complete rewrite from scratch, which will require hundreds of man-hours of volunteer time. So far, after a good many months, nothing visible has happened. We managed to keep up with the changes, and never lost daily data. Note: this will be harder with faster updates, I'm likely to check on them once per day. With hourly updates, that would mean up to 23 hours of glitched data in the stats before I even start the diagnostic and code / data repair process. My data fixing utilities had better be good, and the stats code had best allow for missing hourly data.
• Add SkiBikeSki's CPU benchmark chart in there somewhere, it's a good idea, and something we could use to be more competitive. It actually would be easy to add, the hard part would be updating it.
• Write a bunch of code for our News pages so trusted people could web-post news. Currently, I have to edit the news files and upload, that means server access, which cannot be extended beyond me and Scott. The server in question is shared with most of the other MacNN Network sites -- that is why when they go down, we go down. Web-posting would allow news to be posted more often, hopefully real news and not the filler some teams post. "Team member #3 is gaining on Team member #2" is realy of interest to only those two team members, I've long felt we should not post fluff. Better to have fewer real updates, without fake news taking up space between the real stuff. Note, web-posting code would allow SkiBikeSki (and others) to submit entries to the CPU page, which would solve that update problem.

I know this is a rather long answer, but we've never really posted a state-of-our-site rundown. At one time, we would not even have discussed how our stats worked in general, some other teams had asked for copies (!). Today, the general idea (parse stats pages, enter data into history file, analyze and post tracking data) is common knowledge, and often gets discussed on other teams' boards. We'll still keep our specific code secret though.

A lot of work is involved in a serious DC team site. Ars solved the manpower problem by having different people do the site and stats for each project. That produced completely different sites, and most members familiar with one site would be lost on any other. At Team MacNN, it's kept consistent. If you know your way around the SETI section, you could probably type the exact URL to the equivalent page in the dFold section. And you might even be able to do it blindfolded. This means we are much more able to have our members reinforce across projects than other teams can, it's all "familiar home turf". The stats work the same way, and get upgraded identically. The resource pages you need to jump projects are presently (mostly) and work the same way.

All in all, I think we have a class act here, and the best team site around. It's going to stay that way too. I'll put in as much time as it takes to stay ahead of all other teams, and whatever it takes to bury Statsman. But it's hard to say when that time will happen for each new feature.

Not fishing for sympathy here, and it's been great. If I didn't have to work, the hours would flow to the site. It really is fun, and more fun when members of other teams start bookmarking our pages because ours are better.

Later, guys and gals. The updates will come.

[DAlex]

SOUND OF STUNNED SILENCE......

IMAGE, A LONE MAN SITS AT HIS COMPUTER, MOUTH OPEN.

Wow, Reader50 that is some answer! I'm gonna keep my gob shut and appreciate the work you do!

heh, believe me tho' I was in no way being unappreciative of the excellent stats you provide, I'm just a bit of a closet stats junky

DAlex

[DAlex]

Originally posted by Scotttheking:
<STRONG>

What stats exactly?</STRONG>

Sorry Scott, I was refering to dfold, but Reader50's workload has blown me away with respect

I was more curious as to whether there were any other source of dfold stats data.

DAlex

[reader50]

A few notes, missed a few things. Yep, really. heh heh

• It is by no means a one-man show. We have the best members around, including tons who seldom post to the Forum. We hear from many on AIM or email, it's a great group. That would apply if we had no stats or site of our own. And it's a large part of how much fun DC is here. Thanks everyone.
• RC5 proxy data is expected to someday be worked into the Personal stats (and maybe main stats?) so we will have live data on that project too. It sure would be nice. Seti proxy data too maybe, though it's less needed (mostly for agent breakdown).
• We fight behind the scenes for the team also, and Scott handles more of that than I do. We have often been asked to make our site look more like the other tabbed MacNN sites for example, that is how we lost our real logo (which is in the sigs) and got the current orange one. A main tab is good, it brings us traffic and thus more members, but our logo needed to look more like the others if we shared tab space. That's just good GUI.
  
  The other MacNN sites look fine, but I think ours looks better. And if we adopted that right sidebar from the other sites .... well, it would work on our news pages. But for the 130 resource and stats pages, it would be a killer. We have either wide pics (resource pages) or wide tables and graphs (most stats pages). It's a long-term goal to make the site work on a 640 x 480 monitor, and most of it does work today at 800 x 600.
  
  Anyway, I don't have the time to rebuild our page code so a new layout would work right in all non-IE browsers, has a sidebar we really don't need, loses our distinctive page looks, and get another replacement logo. We may lose our spot on the tabs after the next MacNN redesign (we would instead get a related-sites link like DealNN, MacGeneration, etc), but on the plus side, we would eventually get our own logo back, once I had the free time to rebuild just our header code.
• Scott has a few things up his sleeves too, neat things planned. He can post his ideas if he likes, our current proxy servers came from his end of the scale.

By all means, make suggestions people. At least half our current features came from suggestions and requests. Just don't hold your breath on implementation.

[Shaktai]

Stunned!!!! Wow!!!

     

[Scotttheking]

Hmm, what did reader50 forget...

• Stats at the top bar of the forum. I can post HTML up there, so we are going to put some lite mini stats there.
• Personal stats, extreme Users will be able to log in and put there user names for each project, and thereafter be able to get all personal stats in one location
• Power tracking - Tied in with the user logon, you will be able to list your power and where it's at, giving us a breakdown of what we have, and where
• Some other fun stuff that I probably can't remember right now

All of these fun stats will be running off my server

Did I miss anything?

[DAlex]

Ok, Ok, Ok enough!

I'm sorry I dared mention some 'other' stats person/supplier, it won't happen again, in fact I've forgotten who they were (who who was?)

Just the mention of all those impending stats options courtesy of you two has got the sad stats geek in me all a quiver :o

Man, I need a computer break - I'm off for five days from Thursday, by the end of which my craving for info should be thoroughly under lock and key! (gonna leave my machines running dfold tho'! )

DAlex

[ 05-14-2002: Message edited by: DAlex ]

[reader50]

DAlex, it was not meant in that fashion. The thread just seemed like a good place to post an update on where the site is going, apologies if it sounded otherwise. Get stats where you can, I browse Statsman myself, and browsed Den's also, despite the horrid color scheme. The reason? They had more up-to-date stats, so I could check and see if OCN had dumped a surprise on us, etc.

Until our future upgrades actually arrive, they are all vaporware. It could take months before most of the current list shows up.

Besides, I'm a stats junkie myself. For several months, I did little but stats code. We got our current goodies, on Seti, RC5, and three new projects. But besides News, we got nothing but stats. If I'm not careful, I'll dive into the new stuff and nothing else will be done for a long long time.

The To-Do list does look a bit long ( I hadn't added it up for a while). But the web site will never be "finished", there will always be upgrades or revisions planned so long as the Team goes on. No biggie.

Have a great time, I certainly am. And keep asking for new features. If our members want them, then we want them too.

[Scotttheking]

Oh yeah, we'll also be making a web based benchmark page, which users can update themselves

[slipjack]

Jeez, just checked the page... never would have guessed you cared about stats reader. That's quite the response.

Right now the Team is #7 in DF!! Until MDSP catches us that is...

[SkiBikeSki]

Until the offcial benchmark page is up, you can use the very simple benchmark pages I set up. I put links to all of them in the becnhmark topic. Just post a reply with your data and I'll add it to the table soon.

[enola]

Originally posted by slipjack:
<STRONG>Right now the Team is #7 in DF!! Until MDSP catches us that is...</STRONG>

I think that's because Overclockers Collective, the team that was in front of us, is trying to regroup under a new team - The Genome Collective. If they can get the word out to the rest of their team to switch they will be back in the Top 10 again.

I've noticed a lot of shifts in movement over the last week so I've been over at several different teams' forums checking things out. It's been an interesting read.

[DAlex]

Originally posted by reader50:
<STRONG>DAlex, it was not meant in that fashion. The thread just seemed like a good place to post an update on where the site is going, apologies if it sounded otherwise. </STRONG>

Heh, believe me, I am well Well impressed with the stats and running of team.macnn.com (my 'Ok enough' posting was firmly tongue in cheek )

If I've caused any offence it most certainly was not meant. I am genuinly amazed by the plans you and Scott have and it only reinforces why this team is so good to be a part of - our HQ is built on a firm base.

Keep up the good work, I know I couldn't do it (I don't know perl from Pearl, my dotty aunty!).

DAlex

[ianxwin]

I'm a newcomer to team macnn. It was the apparent organisation, the provision of stats and the general air of enthusiasm and activity which attracted me to join in and contribute my puny cpu cycles. I even stayed whe Skibikeski rocketed past me

Dealer amazes me with what he has put together, and perhaps more importantly, the way he has maintained the code and expanded it, even to producing custom code for races.

I'd just like to say thanks to all you guys.

[rkadowns]

Holy cow-patties Batman!

Reader50, you do a whole hell of a lot more than I thought. I could imagine that running these stats pages took some effort, and I see little things here and there get added, so I knew development time was being used, but I had no idea so much thought was going into this stuff.


Honestly, where to you and your invisibe friends find the time? Can you email some of that time my way?


Great job, it's really appreciated.

RIP

[rkadowns]

BTW

AIM was mentioned. I really don't use IM'ing as I have no life and nobody to chat with, but I may actually install an use something like that if there was a official Team MacNN chat room somwhere out there in IM Land that I could use to chat with the fine people associated with this team.

Any thoughs? Am I simply out of the know?

RIP

[Grozni Majmun]

Originally posted by rkadowns:
<STRONG>BTW

AIM was mentioned. I really don't use IM'ing as I have no life and nobody to chat with, but I may actually install an use something like that if there was a official Team MacNN chat room somwhere out there in IM Land that I could use to chat with the fine people associated with this team.

Any thoughs? Am I simply out of the know?

RIP</STRONG>

You are indeed out of the know, but you can get in there!!!
Actually I don't use it much either, but there is an IRC chat room for Team MacNN on irc.macnn.com

it is the channel called #distributed. You need to get an IRC client to use it. For example "snak". (Danish for "chat", I think the developer is Danish, but that is entirely OT). There are always people there. I think I saw scott there a few times even during his self imposed exile.

AIM is more for direct person-to-person chat, whereas IRC is more for public chat rooms, I think.

[ 05-15-2002: Message edited by: Grozni Majmun ]

[Scotttheking]

Originally posted by rkadowns:
<STRONG>BTW

AIM was mentioned. I really don't use IM'ing as I have no life and nobody to chat with, but I may actually install an use something like that if there was a official Team MacNN chat room somwhere out there in IM Land that I could use to chat with the fine people associated with this team.

Any thoughs? Am I simply out of the know?

RIP</STRONG>

We've got a IRC channel.
server is irc.macnn.com, channel is #distributed

Edit: heh, another post while I was wandering my house. Yes, I live in IRC.
Random question: what part of Phoenix you at? I dunno what my location says, but I'm in east mesa right now.

[ 05-15-2002: Message edited by: Scotttheking ]

[slipjack]

I'd like to see some comparitive stats on the same page...like they had on statsman. It's nice to be able to see increments of 3 6 9 12 etc hours of the stats. Every 6 hours would be a good enough start, but it provides a better sense of how we are keeping up the crunching, both versus other members and versus other teams.

[Grozni Majmun]

Well, statsman.org is back online, but with a somewhat erratic update frequency.

Doesn't look like our stats are updating tonight. They are two hours late now. Usually that means somebody turned them off, right?

Darn, then I have to stay up till 3 am, when statsman updates....

[reader50]

Ours have updated. Seti and Fold went normally, but I had to redo some code on dFold, the scripts were failing to complete. Sorry about that, the specific problem should not recurr.

Edit: Ranks are different from the other stats because I run them all manually. If they are too old, it is only one of two things: I forgot to upload (happened once), or I'm working on a glitch (much more likely). Get on AIM or ICQ and nagg mercilessly until they are fixed and/or uploaded.

[ 05-18-2002: Message edited by: reader50 ]

[Grozni Majmun]

Thanks reader! For me 'stats' is almost synonymous with the advanced ranks table. That is the one I always check. Especially because it has the cool charts.