Github reviews as a way to improve code quality?
Thursday, 17 March 11
Today I tweeted quite a bit about Ruby software quality.
The trigger was simply a Ruby/Sinatra application I run for personal use that crashed from time to time due to a GC bug in the Ruby interpreter I was using (the default of a recent Ubuntu install, sadly).
This is an interpreter bug, and I hope it is a rare one, but this episode made me thinking about how many bad stories I accumulated since my preferred high level programming language is Ruby: bad documented gems, not working gems, dependencies hell, hard to install things, bad performances, and so forth.
The test culture of Ruby helps a bit, but if you are an experienced software developer you know how a few tests can't guarantee software quality. Honestly I saw testing as part of the problem from time to time, with the attitude "if it passes tests it can be merged". Testing is useful but not so powerful, unfortunately.
There is also a lot of great and documented code, but in general it seems to me that there is a real problem about code quality, and maybe there is something that github.com can do to improve the state of things: to make users aware that some code may not be perfect, and to make the developers aware, too.
It is as simple as this: github, please make us able to rate projects. If project-wide reviews are too bold or may appear too rude, just do it the old way, with "stars". If users could rate:
All this anonymously (you could be surprised about how much people in a given community are connected, coworkers, and in general not willing to say the truth about your project).
with a vote between 1 and 5 starts, this may help a lot of people to make an initial idea about the quality of a project. The developer can gain very important info about what could be improved, that is a very valuable information, as there are a lot of people that are ready to say your project is cool, but constructive criticism about what is the weak side of the project is hard to obtain.
This could be done as a separated web service using the github API, but what's the point in doing this? Having it integrated into github is much better of course. But if github will think that it's not a great match for the site, it can be an interesting week end project to work on.
Please leave comments on HN instead of using the blog comments.
Edit: Arguments from discussions on twitter / comments:
This is an interpreter bug, and I hope it is a rare one, but this episode made me thinking about how many bad stories I accumulated since my preferred high level programming language is Ruby: bad documented gems, not working gems, dependencies hell, hard to install things, bad performances, and so forth.
The test culture of Ruby helps a bit, but if you are an experienced software developer you know how a few tests can't guarantee software quality. Honestly I saw testing as part of the problem from time to time, with the attitude "if it passes tests it can be merged". Testing is useful but not so powerful, unfortunately.
There is also a lot of great and documented code, but in general it seems to me that there is a real problem about code quality, and maybe there is something that github.com can do to improve the state of things: to make users aware that some code may not be perfect, and to make the developers aware, too.
It is as simple as this: github, please make us able to rate projects. If project-wide reviews are too bold or may appear too rude, just do it the old way, with "stars". If users could rate:
- usability / installation
- documentation
- code quality
- stability
- performances
All this anonymously (you could be surprised about how much people in a given community are connected, coworkers, and in general not willing to say the truth about your project).
with a vote between 1 and 5 starts, this may help a lot of people to make an initial idea about the quality of a project. The developer can gain very important info about what could be improved, that is a very valuable information, as there are a lot of people that are ready to say your project is cool, but constructive criticism about what is the weak side of the project is hard to obtain.
This could be done as a separated web service using the github API, but what's the point in doing this? Having it integrated into github is much better of course. But if github will think that it's not a great match for the site, it can be an interesting week end project to work on.
Please leave comments on HN instead of using the blog comments.
Edit: Arguments from discussions on twitter / comments:
- How to prevent spam? The only hope I've is that the OSS programming community is less interested into producing some spam vote. But you can restrict the vote to followers of a project or to use other tricks, like allowing to vote only if the user is trusted enough by a given number of parameters.
- Using number of followers of a project as a meter? I think does not work, I'll follow a project that's not very good if I need it, and there are little or not better alternatives. Also I may love a project that is badly documented, so I'll provide five stars for everything but for doc.
- What about the project getting better with time, or the the other way around? Just provide a lot more weight in the average computation to recent reviews. Also put the old ones into the mix but with some math rule so that, the older, the less relevant in the weighted sum.
Do you like this article?
Subscribe to the RSS feed of this blog or use the newsletter service in order to receive a notification every time there is something of new to read here.
Note: you'll not see this box again if you are a usual reader.
Subscribe to the RSS feed of this blog or use the newsletter service in order to receive a notification every time there is something of new to read here.
Note: you'll not see this box again if you are a usual reader.
Comments
1
17 Mar 11, 18:06:40
+1
17 Mar 11, 18:09:35
If they ever allow anonymous voting i will make sure to spam Redis with 1-Star ratings into oblivion to punish you for suggesting this atrocity. ;)
17 Mar 11, 18:11:30
@Honest Guy: what I hope is that in general the programmer's community is able to resist to spam a lot more than other communities. At least I hope so, but indeed this can be a real problem. You can still allow voting only if you have an account older than that, with at least N followers, and so forth? Not sure.
17 Mar 11, 18:17:11
The problem with such rules is that they will eventually be gamed. See Stack Overflow's current situation, it'll become a circlejerk and/or spamfest.
17 Mar 11, 18:18:18
@Miguel: yes it's a problem indeed, I happen to be the guy running the largest digg-alike site in Italy and I know a lot how bad this problem is...
17 Mar 11, 18:19:44
If your goal is to educate these programmers instead of scaring them away from open source you have to encourage serious reviews under which people put their real names.
Anonymous voting is simply not the right tool for the job.
Anonymous voting is simply not the right tool for the job.
17 Mar 11, 19:26:04
Completely agree with honest guy. Two stars in performance would tell you nothing about where your code is lacking anymore than "it's too slow!" would.
Of course people can always open an issue or fork the project to help out.
Of course people can always open an issue or fork the project to help out.
17 Mar 11, 20:00:38
If you want this functionality, you should build it on top of the github API and solve all the hard problems yourself.
Rankings and Ratings do not belong on github.com
Rankings and Ratings do not belong on github.com
18 Mar 11, 03:32:46
Why not add a 140 character "request" text field into the code quality rating that other users can "rate" based on how much value they think it adds. Users that originate a high number of high value fix requests over various projects can have their votes weighted higher.
This would incentivize:
People to give higher quality requests in order to level up.
People to be the first to request in order to be originators.
high value account holders to minimize spam in order to maintain their reputations.
It would dis-incentivize
Low value account holders, the most likely spammers, to post spammy ratings, as they won't be weighted much.
Can you game it? Sure. A high value player can build up several accounts and then sell them. But that can be fixed by tying the ratings to your profile's code quality score as well. So to become a high value player would take more of an investment and increase the cost of gaming the system. Thoughts?
This would incentivize:
People to give higher quality requests in order to level up.
People to be the first to request in order to be originators.
high value account holders to minimize spam in order to maintain their reputations.
It would dis-incentivize
Low value account holders, the most likely spammers, to post spammy ratings, as they won't be weighted much.
Can you game it? Sure. A high value player can build up several accounts and then sell them. But that can be fixed by tying the ratings to your profile's code quality score as well. So to become a high value player would take more of an investment and increase the cost of gaming the system. Thoughts?
18 Mar 11, 11:48:18
@Marak, Great point. Github has excellent APIs but even more so, it's really just a URL. Sounds like a good project for those interested in to take on as a side project.
20 Mar 11, 20:36:14
In addition to the proposed rating system, It would be nice that each tagged commits could be updated with user's ratings. By default it could take the last tagged commits user's ratings. So ratings could evolve only on tagged releases.
I would be also nice github as a "similar projects" feature so you could compare portential projects you would like to use using those ratings.
I would be also nice github as a "similar projects" feature so you could compare portential projects you would like to use using those ratings.
23 Mar 11, 12:07:12
Hey antirez this sound interesting, would be a nice feature in the startup I'm planning to launch in the second half of 2011.
http://www.codesake.com
Can be a good deal, since I'm planning to connect user's github repo to provide code review facilities, to provide also some rating and commenting (in a social network style).
It can be also interesting to add a "Translate comments to Issues" feature to pull all the social interaction made on codesake.com in action to github repo... (of course only the report owner will see this button).
What do you think about it? Does it make sense?
Paolo
http://www.codesake.com
Can be a good deal, since I'm planning to connect user's github repo to provide code review facilities, to provide also some rating and commenting (in a social network style).
It can be also interesting to add a "Translate comments to Issues" feature to pull all the social interaction made on codesake.com in action to github repo... (of course only the report owner will see this button).
What do you think about it? Does it make sense?
Paolo