Welcome to the MacNN Forums.

reader50 · Jul 9, 2002, 10:06 PM

<a href="http://teamstats.macnn.com/dfold/stats.php" target="_blank">http://teamstats.macnn.com/dfold/stats.php</a>

It should update every 15 minutes. Note that Personals and Last 24 Hours (fixed) do not work yet.

Also, our existing stats history has not been imported, so the data begins a few days ago. That is why everyone is shown joining the team at that time.

Comments are welcome.

[ 07-24-2002, 03:49 AM: Message edited by: reader50 ]

The_Equivocator · Jul 10, 2002, 12:47 PM

Sweet. It just keeps getting better. I'm really looking forward to the personals.

How much stats coding can one man do?

discstickers · Jul 11, 2002, 08:12 AM

=)
Ya, nice job reader!

SkiBikeSki · Jul 14, 2002, 12:46 AM

Looks great, thanks for all the hard work.

BTW Remember when I said I was going to learn PHP...
... That hasn't happened yet. But there's still time left in the summer.

reader50 · Jul 14, 2002, 05:27 AM

Personals are minimally working. The pages now map to Scott's server where the database is, but the non-Personal links over there do not send you back to the MacNN server. This will get fixed.

Tactical Report and Personal Ranks work. The other stuff isn't done yet, and I'm investigating assorted server errors that keep wanting to turn up.

reader50 · 04:00 PM

More stuff has been fixed. Personal Ranks, the Tactical Report, and personal QuickStats work, personal graphs and production table(s) are still pending.

Note, our recent production graph is very rough because we update every 15 minutes, while dFold updates every 10, this gives us alternating high and low values. This will get fixed when the crons are done properly for all projects.

Also, you can check out the stats teaser feature. We track several other teams at the moment, you can look over their internal stats by adding a TID (teamnumber) to the end of the URL.

Team MacNN:
http://teamstats.macnn.com/dfold/stats.php

Ars Technica (dFold):
http://teamstats.macnn.com/dfold/stats.php?TID=78

Free-DC (dFold):
http://teamstats.macnn.com/dfold/stats.php?TID=5

Normally, these will not be accessible except during stats teasers for other teams, but they are on open access while the new code remains in beta. Also, alternate team stats do not work on the team.macnn.com URL, only on Scott's server.

Scotttheking · Jul 16, 2002, 02:29 AM

Just as a note, my server is a bit slower then macnn's, so please don't be impatient while waiting for stats to load.
It might take a few seconds, or, if 20 people hit it at once, a little longer then a few seconds.
I'm looking into ways to make it run a little faster, although I don't notice any speed issues, but then, that might be my direct 10 or 100mbit (depending on what comp I'm on) line to the server

reader50 · Jul 24, 2002, 02:34 AM

In order to get more feedback, and to show off the new stats, we posted links to the Ars and FreeDC stats on their respective boards.

<a href="http://www.free-dc.org/forum/showthread.php3?threadid=1227" target="_blank">FreeDC thread</a> ... <a href="http://arstechnica.infopop.net/OpenTopic/page?a=tpc&f=122097561&m=2650981915" target="_blank">Ars thread</a>

This can be considered our first stats teaser for other teams, and it looks good. Guess they are not used to serious stats.

It has brought useful feedback, resulting in numerous bugs getting fixed, along with another bug I still need to track down.

Anyway, once we are out of beta, we won't be serving other teams' stats needs, except for teasers now and then. Though FreeDC has found enough bugs to have their stats left on longer.

Next steps: I need to build the new Ranks code, and rebuild the graph function so Personal Graphs can be turned on. Along with fixing any bugs that crop up along the way. Feedback is welcome, we've been fiddling with the page and stats details since they first went up.

Note, to reach the new stats, use the "Stats" button in the dFold section of the team site, or use <a href="http://teamstats.macnn.com/dfold/stats.php" target="_blank">http://teamstats.macnn.com/dfold/stats.php</a> (we just got this URL). The beta stats copy that was in our dFold section has been removed. It had been set up to save Scott's bandwidth for just the live-rendered pages that had to be on his server, but it turned out the data relay used more bandwidth to keep the Team copy updated than just serving all the pages directly from his server.

[ 07-24-2002, 03:47 AM: Message edited by: reader50 ]

reader50 · Jul 26, 2002, 05:47 PM

The new stats may report data that is up to several weeks old over the next day or two. I'm working on several bugs, and having to recompile our data, which takes 8-12 hours to catch back up to the present.

Ranks code conversion has started, but is on hold while I chase bugs.

jamesa · Jul 28, 2002, 05:01 AM

<blockquote>quote:<hr />Originally posted by reader50:
The new stats may report data that is up to several weeks old over the next day or two. I'm working on several bugs, and having to recompile our data, which takes 8-12 hours to catch back up to the present.

Ranks code conversion has started, but is on hold while I chase bugs.<hr /></blockquote>just a question with regards the stats - how are the averages calculated? I mean, obviously I know how to calculate an average, but for average/day, how many days do you use?

Maybe it would be worth looking into a long term average, medium term average (a week?) and a short term (daily, or maybe even hourly) average.

Either way, it would be good to know what the average currently uses.

Thanks

-- james

reader50 · Jul 28, 2002, 11:27 AM

On our current stats, the avg per day uses one week. If a full week isn't available, it still divides by 7 - this is the reason why current Personals are inaccurate until a member has been here for at least a week. This bug does not affect Team Ranks calculations.

With the new stats, the avg per day uses one week. If a full week isn't available, it divides by the time that is available to give the correct rate regardless. There is also a "Spot Rate" listed in the Quickstats (for both Team and Personal pages) - this gives the rate per day since the last stats pull 10 minutes ago.

The units a day are also shown, this is since yesterday's stats pull (current stats) or since 24 hours before (new stats). Not a true average since it is an exact amount.

In both old and new stats, the (Team) Quickstats offer averages for Monthly and Yearly, which assume the full time period. I'll fix that presently in the new stats. Personal pages do not offer averages longer than one week, not sure if that will change or not. Opinions?

[ 07-28-2002, 11:30 AM: Message edited by: reader50 ]

The_Equivocator · Jul 29, 2002, 04:24 PM

Here's a suggestion:

About the whole active users graph.... How often does it update? Does a member have to be inactive for a week before they become "inactive"? A day? 6 hours?

Ideally, I think a 6 or 12 hour period would be a good way to do it. The graph definitely has room for this, and it would be nice to see the "peak times" of activity. How many people do we lose at night? etc.

[EDIT: clarity]

[ 07-29-2002, 04:25 PM: Message edited by: The_Equivocator ]

reader50 · Jul 29, 2002, 06:08 PM

Active users graph shows the Actives during the past 7 days. It updates once per day (current stats) and every 10 minutes (new stats). Note that the Inactive table shows those who have not returned a unit during the past 30 days.

Sounds like you want an "active daily" line added to the Active graph, or even an "Active Hourly"? A spot activity line would give a series of spikes. The shorter the time period, the more the Active would look like the Rate graph in the new stats.

The_Equivocator · Jul 30, 2002, 10:24 AM

<blockquote>quote:<hr />Originally posted by reader50:
Active users graph shows the Actives during the past 7 days. It updates once per day (current stats) and every 10 minutes (new stats). Note that the Inactive table shows those who have not returned a unit during the past 30 days.

Sounds like you want an "active daily" line added to the Active graph, or even an "Active Hourly"? A spot activity line would give a series of spikes. The shorter the time period, the more the Active would look like the Rate graph in the new stats.<hr /></blockquote>Yeah, I was kinda thinking along those lines. Instead of adding a whole new graph, having the thing update itself every 6-12 hours instead of 7 days would be nice. Anything shorter than that would have too many spikes, as you said. It just seems wierd to have a graph that spans less than a week but only updates inactivity on a weekly basis. For any given day, there are not going to be as many active users as that graph suggests.

reader50 · Jul 31, 2002, 11:41 AM

The graphing function got rewritten yesterday, which needed to be done to correctly match it to the new codebase. This tends to smooth out the long-term graphs, and it makes Personal Graphs possible.

I need to do a little clipping on the long term graphs now, look at long-term Rate and you will see why. After that, it looks like Personal Graphs are next.

Active Daily line has been added to the Active Members graphs.

reader50 · Aug 4, 2002, 04:53 PM

I'm optimizing the backend code and fixing a few missing features before moving on. Among other things, this will produce another 15-20x speedup on the server side, which we will need for tracking all the projects.

If anyone thinks of some new stats features, this is the time to ask. If they require a change to the database definition, well, I am going to fiddle the definition anyway. Doing so may cost us the data already built up, so it would be best to do this only once.

Note, team Ranks and Personal graphs are coming, along with some sort of unified Personals across projects. Also in the planning is stats sigs.

Anyone want some more stats features?

The_Equivocator · Aug 6, 2002, 05:48 PM

Here's a request with regards to the dFold personals:

How about listing each person's overall ranking, as well as how it has changed within the past day or so? It would be nice to see how we are doing in relation to everyone else and not just our teammates.

reader50 · Aug 7, 2002, 12:29 AM

The database is already built to allow this, there are fields for Allrank and Allrank Change. The problem is obtaining the data, which I have not tried to implement so far. The "Top 1000" users on dFold may allow that data to be obtained for some users, and some other projects place that data within easier reach.

I'll look into this feature later on. Any other requests or suggestions?

The_Equivocator · Aug 7, 2002, 11:15 AM

We have those nice near-term and long-term graphs... how about the option of overlaying the 3 teams above and below us on the same graphs a la Statsman?

reader50 · Aug 7, 2002, 01:01 PM

Team Ranks.

( code not translated to the new stats yet )

The_Equivocator · Aug 9, 2002, 02:09 PM

Hmmm... when you go to personal stats, there is usually a 3 to 10 structure difference between the 24 Hour field at the top and the Units Yesterday field in the table.

Why does that happen?

reader50 · Aug 10, 2002, 12:16 AM

On the beta Personals page, your Units Yesterday (last 24 hours) appear in three places: Quickstats, Recent Production, and the Ranks table. I just checked about ten people, and all numbers agreed.

It is possible for the number in the Recent Production table to disagree with the other two slightly, it uses a different method to figure up the numbers (a simpler method for selecting a data block 24 hours ago). I may change that the next time I can play with the code.

IronBits · Aug 10, 2002, 01:53 AM

Scott made me do it!

I'd like to see a 30 day graph of the dailies per individual.
With flexability to change that to look back 90 days or more, so I can look at trends... Choose multiple 'players' and look backwards for trends and comparisons etc.
Nice stats to!

reader50 · Aug 11, 2002, 01:14 AM

You will be getting Personal graphs (placeholders are already present under the Personal ranks table), showing data vs nearby members. It will probably be the 5 above / 5 below and 30 days, much as we do for other graphs. At least at first.

Customizable comparisons are possible under the new stats. It would be a cool feature, but will likely not turn up until after all our standard stats features are working. Interteam Ranks such as our old stats have are the main feature not ported to the new stats yet.

Note, nothing but emergency stats repairs have been happening for the past couple of weeks because of critical work in the real world. I'm not sure how long that will continue before I can return to the coding fun.

IronBits · Aug 18, 2002, 12:07 AM

Originally posted by reader50:
You will be getting Personal graphs (placeholders are already present under the Personal ranks table), showing data vs nearby members. It will probably be the 5 above / 5 below and 30 days, much as we do for other graphs. At least at first.

Customizable comparisons are possible under the new stats. It would be a cool feature, but will likely not turn up until after all our standard stats features are working. Interteam Ranks such as our old stats have are the main feature not ported to the new stats yet.

Note, nothing but emergency stats repairs have been happening for the past couple of weeks because of critical work in the real world. I'm not sure how long that will continue before I can return to the coding fun.

That sounds real good to me. I just wanted to throw out a bone to give you time to chew on it while doing all the other stuff

Thanks! and very nice stats!

unregistered · Sep 19, 2002, 10:03 PM

Stats oddity:

Right now on the MacNN dfold stats, looking at the Team Stir Fry section (1) roughly between the users ranked 35 and 110, there's an unusual number of people showing 'Units Ystrdy' totals ending with 999, 998, 997, 996, 995, etc. It's as if it's displaying (structs - 1) , and most people upload in multiples of 1000. There seem to be too many instances for this to be just coincidence.

Oddly I didn't see this behavior on any other teams stats sections I checked (FreeDC, KWSN, MacNN). Also, the zerothelement stats (2) do not show the 999 phenomenon for those users, so I don't think it's a problem with the official stats.

Just wanted to let reader50 know about this, because if it is a bug, it seems like an elusive one.

Linkage:
(1)http://teamstats.macnn.com/dfold/sta...?TeamNumber=78
(2)http://www.zerothelement.com/tsf-stats/index-2.shtml

-NovaCane, of Team Stir Fry

unregistered · Sep 19, 2002, 10:15 PM

Well, I just looked again and the problem seems to have fixed itself. It's more elusive than I thought

-NovaCane, of Team Stir Fry

reader50 · Sep 20, 2002, 12:50 AM

NovaCane, thanks for the bug report. Bugs need to be eliminated, so I do like to hear about them asap. It's also nice to get feedback in general, it means people actually use the stats.

This "oddity" is not a true bug. Units in the last 24 hours means exactly what it says, it checks for a comparison data block exactly 24 hours ago, and corrects for the rate if a block cannot be found exactly 24 hours back. Stats are pulled every 10 minutes, but server load and bandwidth can move the parser's exact time by up to a minute.

You and I both know that dFold project stats are constant within a 10-minute block, but that is not true on many projects which process stats continuously (Seti, Ubero). Our beta stats code is project-independent wherever possible, and timestamps a block by the exact second the data was parsed. This is so when the bugs are worked out, the new code will be extended to all projects MacNN participates in. Instant bug-free stats on a half dozen more projects, using a consistent interface.

It looks like the code corrected for a data block being parsed 23 hours, 59 minutes ago instead of a perfect 24. This would most likely correct itself within 10 minutes, on the next processing pass. This correction error will only show on projects that don't update continuously, and it should always be very small. On projects with continuous stats, this will provide extreme accuracy, even if our server updated at erratic times, such as when Berkeley is having "issues".

Once again, thanks for stopping by.

Scotttheking · Sep 20, 2002, 04:55 PM

I'm going to sticky this so that it's easy to post changes and updates and issues.

reader50 · Oct 30, 2002, 12:49 AM

It's been a while. My free time dried up for several months.

The enforced delay did not leave enough time for coding, all I could do was tinker with the final database definition. The final definition benefitted a great deal from months of attention, we will be recording far more data than the beta stats keep track of.

Currently, I'm working on a local set of updated scripts, which will eventually replace the beta versions on the server. The critical backend modules have been heavily rewritten from the beta versions, to the tune of thousands of lines of new/updated code to support new features. Not all features are implemented yet, but I'm about a day away from being able to test the new code. A minimum number of modules have to be functional before the system can run, and I'm one feature away from having enough modules completed.

No doubt Bugs Bunny will live in my computer for a few days. After that, some more modules get written and the final website front end can be built.

I'm afraid there will be little to see right away. Even with the backend code running, I'll have to view the data straight from MySQL in the Terminal. No one will be able to see the pretty stats until after the website code is updated.

For the moment, the beta stats on the website are going to be left running as-is. When the Final version is ready, it will be easy to import the existing data from the beta stats. It will be a bit harder to import our existing stats history, but we will do that too.

Oh yes, the new features. We've done our bragging about new features in previous threads, quite a lot of vaporware. I think it's time to play the strong silent type, and do some delivering instead. It's likely to take days (this means weeks), but the ball is rolling again. Stay tuned.

reader50 · Nov 1, 2002, 03:53 AM

Been chasing wabbits. Got a LOT of wabbit stew now.

The new code runs without obvious errors anymore. Data is entered, matched, and processed from hourly through yearly stats. The database has stopped showing bogus entries, this is another good sign.

Assuming the new modules are now wabbit-free (hah!), the next step is to write a few more modules, which will implement more of the planned features.

An upgrade has been made to the beta stats running on the server. "Units done for the current protein" are being tracked for members, and shown in the All Members pages. This had been asked for in the project forum. Please post if anyone is using them, I'm undecided about adding their display to the other pages.

SkiBikeSki · Nov 5, 2002, 03:15 AM

In my CIS courses I'm now making C++ programs that output HTML code, which is a lot like php, right? So I now realize how much work goes into all the pages. It took me 5 hours just to make this glossary. I was provided a small list of words and definitions, it would be more impressive if I was given all of Websters!

So here's a well done and a (pat on the back).

reader50 · 04:28 AM

The data backend is now feature-complete. It runs with no error messages.

An overview of where the new stats are:
Data Backend This parses data from the project server, does all manner of member/team processing, and maintains an orderly MySQL database with the results. This block is required before any other block can run. Status: Feature complete for member/team tracking. Ranks code modules pending. Probably still has some bugs.

Web Frontend This puts up the nice stats pages. It pulls processed data from the stats database as needed, and does some analysis for Personals. Mostly it generates appropriate output tables for each page. Status: Beta version running, but Final will require many modifications to run because of changes to the data backend. High end features such as custom member / team comparisons will require a lot of new code.

User Database This stores accounts across projects, and allows users to enter some data, such as their farm size and Personals viewing preferences. This block is required for Unified Personals. Data backend and web frontend must be running first, preferably on multiple projects. Status: In planning.

---------------
I need to get the web frontend working to easily find any remaining bugs. After that, there will no longer be a reason to stick to one project. The code can be applied to every project we are involved in, including D2OL.

edit: caught a couple more wabbits, changed descriptions to match.

jarling · Nov 9, 2002, 12:38 PM

I knew there was a reason I found this to be the best DC team around. Thanks for all the hard work.

reader50 · Nov 12, 2002, 04:32 AM

Progress report. The general structure of the web frontend is laid out, and I got the first member tables to print. Had to get past all the wabbits in ambush first.

Many of the new functions are working, others seem lousy with bugs. Probably just a few typos somewhere, but it messess up the printed data. Something about the database is odd too, need to look into that.

It's late. To be continued...

reader50 · Nov 15, 2002, 02:55 AM

The database has been fixed. Basic web page controls are complete, so I can move back to data formatting and additional pages. Not to mention graphs, Personals, Ranks ...

reader50 · Nov 21, 2002, 04:05 AM

Member and Active pages are finished. Inactive needs a little more work. Changes and Milestones are pending, but I hope to finish those in the next week.

That leaves:
1. the root stats page with the team's current graphs
2. Personals
3. Ranks

Once #1 and #2 are done, Final code can replace the Beta stats on the server. Once #3 is done, Final code can replace our current stats - this would free up a modest amount of time for coding. Current stats require about an hour a day of attention, mostly regarding the Ranks.

reader50 · Nov 23, 2002, 03:50 AM

Member, Active, Inactive, Changes, Milestones.

All are done, except for a feature I forgot to finish adding to Member pages. It's hacked in now, but I'll do a more complete job. Er, the QuickStats bars are missing too. And some graphs...

reader50 · Nov 28, 2002, 05:51 AM

Missing feature done. The QuickStats are done also, subject to further tinkering. A few more wabbits were caught along the way.

The teamroot stats page has been plotted out with more graphs, I just need to code it now. The graph function has been reworked somewhat to bring it on par with current code, though another feature still needs to be coded in. It's needed for the teamroot page, so I'll code it while doing the page.

reader50 · Dec 1, 2002, 04:44 PM

QuickStats got another rewrite. I'd forgotten they had to be safe for teams other than MacNN - total ranges have to adjust for various team sizes, or the numbers will overflow the sidebar.

Teamroot stats page is finished, including the graphs. And still more wabbits were removed from the graph function. I really should have gone easier on the good stuff while coding that function...

Some comparison graphs are needed elsewhere, along with the Personals page. The Personals page will finally get graphs. A little backend tinkering needs to be done too, after which Final code can replace the Beta code on the server.

Shaktai · Dec 1, 2002, 05:42 PM

An awesome job as always. Thanks Reader50!

reader50 · Dec 8, 2002, 05:41 AM

More progress. Compare stuff is done, and the Personal page is loading as far as the QuickStats and Personal Ranks table. I'm still tinkering and fixing small things, after which the other good stuff gets added to the page.

reader50 · Dec 8, 2002, 10:45 PM

Personal Graphs, they are a-working. Polished up many other little problems and details, I've got the wabbits on the run now.

Some graphs still need to be added, along with the Tactical Report, and the recent production table.

reader50 · Dec 11, 2002, 03:35 AM

The remaining graphs, production table, and Tactical Report have been added to the Personals page. Final code now has every feature offered by the Beta stats, plus a lot more. Including those long-lost Personal Graphs.

Some more backend tinkering is still needed, along with polishing up many small details. Also needed are data importers for the Beta stats data, and our regular stats data. I might even think up another feature or three to add.

Scotttheking · 05:40 PM

http://teamstats.macnn.com/dfold/sta...mber=5&page=c1

Click on any user's # to go to their rank page

Goes back to the main page, and also this error:

Warning: Supplied argument is not a valid MySQL result resource in /distcomp/webserver/teamstats/dfold/stats.php on line 31

Could you see about fixing it now, apparently it's quite annoying.

AND

data is 10.0 days old

reader50 · Dec 12, 2002, 03:18 AM

Seems to be fixed now. The table datafile for FreeDC was damaged. After table repairing, cleaning out 10-15 incomplete/corrupted blocks, and running update scripts ... it seems to be online again. We'll see if it stays clean, or needs more attention.

About the new stats, polishing and tinkering going on.

reader50 · Dec 20, 2002, 04:13 AM

Polishing and tinkering close to done. The last new feature that was scheduled to be done before it goes on the server ... has been completed.

Four items are left on the to-do list before the code can go up. Only two of those will take much time, the others are easy. The harder ones are a final change to the DB definition, and coding the data importers for our existing stats history.

reader50 · Dec 22, 2002, 03:24 AM

Progress report.

The last Database definition change went without a hitch. I've since imported all the Beta stats data (from 18 teams) into the Final codebase and have abandoned the Beta stats on my local box. Final is a going concern.

The importer for our current stats data is also complete, and I'll integrate the data shortly. Some tinkering is still needed here and there in the page code, but it's basically ready to go on the server. At this time, it looks like it will go up sometime tomorrow.

It will (probably) be just dFold at first, with other projects following over the week ahead. I'd like to get them all running before the end of the year.

This is the new Member Stats. Team Ranks, the User Database stuff, and the News Publisher will have to come later.

reader50 · Dec 22, 2002, 07:05 PM

The site files have been changed over. We are importing the Beta data now, which is proving to be a major pain time-wise.

After the importing is done, it will take more time to reprocess each team back to the present.

reader50 · Dec 22, 2002, 08:13 PM

All dFold data has been imported. Data is being reprocessed by the Final code now. It will be a while before they return to the present, and browser access can be turned back on.