== Current Tasks ==
#Hammer out the specifics of the proposed rating system
#Request software modifications
#Hope and pray it gets implemented
#Start rating
#Decide on a verification method
#Wikipedia 1.0!
== A Message ==
I would just like to make it known that I am deeply devoted to this project. It has a LOT of potential, and I am willing to see it through to the end. However, that does not mean I intend to hijack it! Rather, we need to do this together. Let's git 'er done!the1physicist 03:51, 24 October 2005 (UTC)
== Guidelines ==
We will need a systematic way of developing the rating system. Post everything that is agreed upon to the article page. Discrete areas of discussion will focus our efforts, so any areas of disagreement should be started in (or moved to) their own section on this talk page.
== Rating Scale ==
I know this is going to come up so I thought it best to pre-emptively create this. What range of numbers will this rating system cover? Will it be 0 to 3, 1 to 10, or something in between? Personally, I strongly favor a scale of 1 to 10. Among other things, this would make it much easier on the software. More importantly, I think people are used to rating things on a scale of 1 to 10, which jives with the goal of making rating as easy as possible.the1physicist 03:13, 21 October 2005 (UTC)
== Online vs. CD/DVD vs. Paper ==
This will also come up, so we might as well address it now. A disk version of Wikipedia will naturally contain less information than the online version, and a paper version will contain even less. The wonderful thing about a rating system is by varying the selection criteria, it automagically allows for both. For example, a disk version could include all articles with a rating of say 7 and up, whereas a paper version could limit itself to articles rated 9 or 10. In contrast to our sister projects, a rating system will allow for any conceivable distribution method, ''without'' pulling from the editor pool.the1physicist 03:13, 21 October 2005 (UTC)
== Categorized Rating ==
While reading other folks' past suggestions, I saw that a few had the idea of additional rating categories, so I have added this to the project page. I am definitely in favor of any and all additional categories (suggestions, please!), but feel we should limit them to the top 5 or 10. I also think the scope and comprehensiveness ratings are most important, so they should be separate from the others.the1physicist 03:13, 21 October 2005 (UTC)
== Rating Accuracy ==
An issue that may or may not happen is that people may (unintentionally) not reliably rate articles. This will primarily be a problem with casual viewers who may not be well versed enough in wikipedia. I personally don't think this will be a problem (due to averaging, etc), but for some articles it very well could be. At the very least, an article with wildly varying ratings suggests an issue with the article itself. As User:Stirling_Newberry put it, "even noise is data".the1physicist 20:21, 22 October 2005 (UTC)
=== Scope Rating ===
This is particularly true of the scope rating, where only an expert in that topic will truly know how much an article fulfills it's scope. The solution to this problem would be to weight experts' scope ratings (or possibly suggest an admin to override it), which of course requires a way to identify experts.the1physicist 20:21, 22 October 2005 (UTC)
=== Expert Determination ===
==== Expert Rating ====
An editor could select areas where he or she is knowledgeable. Then other editors could rate that editor's level of expertise after reveiwing edit histories, etc. It may even be possible for the software to determine the expert rating. The software could compare an article's rating before and after an edit (assuming an improvement), and using it's category determine that the editor is knowledgeable in that subject.the1physicist 20:21, 22 October 2005 (UTC)
==== Wikiversity ====
Alternatively, if the (proposed) Wikiversity ever decides to include testing (which I believe they should), those scores could be used to determine an expert rating.the1physicist 20:21, 22 October 2005 (UTC)
== Last Minute Fork vs. Continuous Validation ==
Vandalism is going to be an issue. We have two options. The first would be to fork the 'pedia at the last minute and have editors do a final check for vandalism/copyvio. The other option would be to verify each version of an article as free from vandalism, and have the software choose the latest vandalism-free version for release. This option would require much more effort than a last minute fork. More importantly, a continuous validation scheme would probably undergo final validation anyway, so we might as well do it all at once.the1physicist 02:10, 24 October 2005 (UTC)
:My vote is for continuous validation. Academic studies show that the average act of vandalism is fixed with in minutes. We can mine the database to get up-to-date and detailed values for x as a function of the number of users who list the article on their watchlist, the number of those users who are admins, etc. Once we set a statistical confidence level (such as 95%), we can choose a length of time , tailored to the watchlist status of the article in question, and simply use the newest version which is older than that time interval. In our example, . The figure from 2004 is , but that's the aggregate, and I'm sure it's a bit smaller now, especially for well-watched articles. An extremely simple version could be based on the old aggregate, though.--Joel 15:40, 4 November 2005 (UTC)
::Have you seen the latest version on the project page? Note this sentence: "Perhaps the best solution would be to verify revisions of articles, fork at the last minute, and simply present to editors articles whose most recent revision isn't already verified." I think it combines the best of both worlds. Let me know what you think.the1physicist 16:31, 4 November 2005 (UTC)
== General Discussion ==
On the specifics of the rating system: I think that users should be prompted to rate articles, and that a system should be devised which uses previous ratings and the size of the diff file to choose which artilce is brought to the users' attention. I'm quite fond of the randomization trick that Slashdot uses to prevent the rating of things one already cares about; I think we should steal it. Perhaps the developers involved would even donate their services to the Wikimedia team; failing this, their software is open source. See Coase's_Penguin (the paper itself, not the Wikipedia article) if you aren't familiar with the finer points of their rating system; a quick treatment (which omits my favorite part of it) can be found here.
On the implementation of software modifications: see Slashdot comment. Also: the Unification church ("Moonies") is working on sorting out the best of Wikipedia, as well. Perhaps they'd help with financing the software development. Of course, this won't affect whether or not it gets accepted into MediaWiki...
On the last-minute fork: If the article rating applies to the version itself (with, perhaps, a spillover if the edits are small), there is no need to fork or skim. Only the articles that have been automatically rated would be included.--Joel 06:43, 20 October 2005 (UTC)
:On second thought, ratings of the scope of the topic should be kept through all versions, but ratings on quality and thoroughness within that scope should apply to particular versions.--Joel 21:43, 20 October 2005 (UTC)
As the semi-official project leader, I thought it would be a wise to read all of the previous discussion concering WikiSort and the 1.0 project in general. Between your links and User:Walkerma's, this may take a while. I'll post sometime soon with a fresh batch of ideas.the1physicist 22:01, 20 October 2005 (UTC)
Some random thoughts: If the article rating is to be used to rate the editors it would be logical to only allow logged in users to do the rating. Articles should also be rated on structure and readability. Users taking a stub and adding significant content would change a rating significantly, and this would not be vandalism. Users adding links, correcting grammar, etc. are just as valuable as content adders, and additions that improve navigation (e.g. disambiguation) are possibly more significant. User rating should take account (somehow, but I don't know how) of their participation in edit wars or vandalism. Joe1011010 19:03, 25 October 2005 (UTC)
*Why exactly should we limit ratings to logged in users? Are you worried about rating vandalism? Perhaps I should have included all the anti-vandalism ideas on the project page after all. True, it would be much more accurate if we could rate editors' edits directly (rather than a before and after snapshot of the article), but that would be VERY tedious, which goes against the idea of making this as easy as possible. More importantly, it wouldn't be _that_ much more accurate. Alternatively, we could allow users to rate other editors directly (perhaps on categorized criteria). Anyhow, check the project page in a little bit, and post back if it doesn't answer your concerns.the1physicist 02:01, 26 October 2005 (UTC)
----
Hi,
When I started reading your "Wiki Sort" page and stumbled on the phrase "Catagegorized ratings" and saw your examples on comprehensiveness rating and importance rating, I thought to myself - what a wonderful idea! But then I started to read more about your implementation suggestion, and got a bit disappointed. You are suggesting massive software changes to add the new concept of "rating", and new ways to work with these ratings. I'm thinking to myself - why not do something much simpler, and have actual '''category'''-based ratings?
What I mean is, imagine we create Wikipedia categories named Comprehensiveness-1 through Comprehensiveness-10, and Importance-1 through Importance-10, and so on, and let people add these categories to articles. Disputes in these ratings will be handled just like any other dispute (including perhaps a discussion in the talk page), they are protected from vandals just like any other change (history, etc.), and most importantly, this requires absolutely no change to the software. When building Wikipedia 1.0, these categories, as well as existing categories like "stub", "NPOV", and so on, will be used to decide which articles should be included.
What do you think?
Nyh 11:57, 8 November 2005 (UTC)
:I think that's a great idea. I was thinking an external system like de.lirio.us, but your way sounds even better. Go ahead and set up the categories yourself. However, I would suggest using the pipe trick: 9.3, PageName will produce a synoptic "category" page ranked by rating and then sorted alphabetically. I'd also suggest creating a template for the talk page that allows users to vote, and invites people to check that the average/consensus of votes is what the page is listed under.--Joel 17:22, 8 November 2005 (UTC)
::That would be a very good idea, except it wouldn't allow for all the nifty data-mining techniques discussed on the project page. More importantly, I recently talked with User:Magnus_Manske, and according to him, most of the software changes have '''already''' been implemented. I I understood him correctly, the only thing we're waiting on is the appropriate data mining algorithms (some of which I have outlined).the1physicist 17:51, 8 November 2005 (UTC)
== Observations off the top of my head ==
Hi, I had been thinking about this topic, having seen some of the editorial team assessment stuff somewhere. Also I was reading comment following a (fairly) recent Guardian article about Wiki, which was critical in some respects, and discussion I was having with a librarian. Wiki pages need rating for the benefit of people using them, to get an idea of how good we think a particular article is. The fact that some pages are acknowledged to be virtually worthless is a real liability in a source to be recomended to someone.
I didn't see how you plan to display this information, but it needs to be available on the article page so someone can quickly see it. Perhaps a tab at the top of the page could bring up a 'rate' page with more statistics and tick boxes to make a rating? A history of past ratings (somewhere, I suppose not necessarily visible, though if it has a dedicated page this could be quite extensive). A brief read suggests you are branching out into editor rating. What will you do with the ratings. is Kate's tool out of date?
But this does need to work automatically with minimum special intervention by teams of assessors.Sandpiper 00:50, 26 November 2005 (UTC)
:My plan would be to display a disclaimer at the top of each page (perhaps below the "From Wikpedia, etc") that would say something like "This page has been rated X by users of wikipedia. This means that it may have Y or Z issues." or something like that. As far as the rating goes, the powers that be have determined a tab at the top of the page would be best. You can see the demo here (click the validate tab). Keep in mind that those are just test categories. You may also want to check out the metawiki. Thanks for commenting!the1physicist 01:58, 26 November 2005 (UTC)
::That was quick work. Good of you to to set that up overnight: just the sort of thing I had in mind.