Beware of hardware benchmarks
Apr 18, 2013 3:33:25 GMT
Post by CharlieChomper on Apr 18, 2013 3:33:25 GMT
Periodically, I see this get brought up in the community where someone will post a link to recommendations for certain hardware (in particular, video cards).
The issue of power supplies/other components aside (I've been slowly working on writing up something related to that for a while to post here to share with you), one thing that I never see get brought up--regardless of what site/publication gets mentioned--is in the most common benchmark that also gets cited for results. This is an extremely important point to mention for the reason I'm about to get into.
Up until several years ago, the company that released this benchmarking program used to have full support of all the hardware companies out there and it had been unbiased for a period of time--this is also amongst the reasons (along with a lack of similar programs and a sad lack of independent, qualified hardware testers) as to why this program is so widely cited even now. However, at some point over the past several years, this changed. Specifically, what happened--and why many in the know now condemn this program and any citation of it and know to ignore it--was that either the company began asking more money from vendors to provide more favorable results in the benchmark results against their competition or that the vendors, themselves, began offering up more money to make their results appear better (AMD, in particular, is amongst the known companies who had been guilty of this both in terms of their processor/CPU results along with their video card results (both when it was still badged as ATI and now just as AMD), but tried to present it as supposedly "supporting the project"--however, I should mention they're hardly the only company who was found guilty). It remains unclear as to which one of the two approached the other, but regardless, it's been no secret that money has been changing hands between the vendors and the company behind this program to "boost" the results. In turn, Intel and nVidia amongst numerous other companies have publicly condemned this company/program and this practice and have severed all ties to them, wanting nothing to do with them--at one point, there had been some discussion of a possible lawsuit or call to investigate the matter legally, but on the latter front never ever was done about it and on the former, there was never any public mention of a lawsuit. However, I do know that both nVidia and Intel along with a host of other companies have kept their word as far as washing their hands of this company and this product--and that the suspect or biased benchmark results still coming into the picture.
The company, for its part, has denied this publicly despite the evidence and instead has claimed that supposedly that AMD (who is the best known/biggest name linked to this scandal) had just been "more cooperative" in providing them with "better data" that supposedly led to the skewed performance results whereas nVidia and Intel (amongst the others in the camp that wants no part of them, but again, those two companies being the largest players/biggest names involved) had been completely uncooperative/"difficult" and had not provided enough data to properly measure their devices against the competition which is what led to the (suspect) results (although, the company does not view their results as suspect so much as the rest of the industry who is aware of what has been going on sees it this way).
Yet, sadly, this is not the first instance of money changing hands in exchange for "better" results--just a different approach to it. In years past (although, this practice no longer takes place nearly on the level/scale it once had, at one point it was very widespread and affected a number of magazines and websites and other media that many consumers, in particular, relied upon when making purchasing decisions), there had been another practice involved of where some hardware reviewers were known to take "kickbacks" or "financial incentives" from certain hardware vendors to give their products better reviews--if they made the competition look even worse, so much the better at times. This, in turn, led to two reactions from just within the tech/techie communities with one angry/frustrated enough over the situation to where some set up independently-run review sites (the better ones run by actual hardware testers and engineering backgrounds) where they not only refused to take any money, but sometimes went out of their way and had to purchase the products they were reviewing themselves. The other reaction was not so much a "blind acceptance" of this (as overwhelmingly in the community the "buying of 'good' reviews" practice was reviled--and much of the industry knew this was going on) so much as some pointing out the lack of education that the public generally has when it comes to such matters and in how many turn to the same mass market review sites and magazines (with some being suspected of or known to have been involved in this practice) and rely so heavily upon their evaluations and reviews of products they often don't question them (funnily enough, there was a similar finding a number of years ago about computer games of where the magazines used to have so much sway and power over purchasing decisions made by consumers when it came to reviews with many gamers relying upon those reviews and taking it almost at its word, that it could make/break a game easily--however, with the decline of the publishing industry as a whole, it's also led to a breaking down of this situation as well). Regardless, in the hardware industry, it's hard to say with any certainty at one point this practice fell off out of favor (although, it hasn't gone away completely, it's nowhere near as common to encounter this as it had been), but as things evolved and some of the indie sites became more popular amongst system builders and hardcore gamers (who also tend to spend more on hardware than most consumers), many of those who engaged in the practice tried to distance themselves more from it publicly and sometimes went so far as to fire reviewers who were caught engaging in it in an effort not to appear as biased and gain more credibility. Ironically and sadly, however, there are now very few independently-run sites actually left and one former site has now actually ironically gone on to be suspected of engaging in the very practice its founder (who has long since sold and left the site) condemned (which along with its sometimes questionable methods and some questions raised about the qualifications of those know working on it and their knowledge/or lack thereof at times, has led to the site appearing to be almost on "life support" these days and with many in the community having long since gone on to avoid it--I personally found the situation actually sad to see happen as when it used to be run by the founder, it had been my main "go to" site for hardware back in the day. However, that hasn't been the case in years now sadly).
The issue of power supplies/other components aside (I've been slowly working on writing up something related to that for a while to post here to share with you), one thing that I never see get brought up--regardless of what site/publication gets mentioned--is in the most common benchmark that also gets cited for results. This is an extremely important point to mention for the reason I'm about to get into.
Up until several years ago, the company that released this benchmarking program used to have full support of all the hardware companies out there and it had been unbiased for a period of time--this is also amongst the reasons (along with a lack of similar programs and a sad lack of independent, qualified hardware testers) as to why this program is so widely cited even now. However, at some point over the past several years, this changed. Specifically, what happened--and why many in the know now condemn this program and any citation of it and know to ignore it--was that either the company began asking more money from vendors to provide more favorable results in the benchmark results against their competition or that the vendors, themselves, began offering up more money to make their results appear better (AMD, in particular, is amongst the known companies who had been guilty of this both in terms of their processor/CPU results along with their video card results (both when it was still badged as ATI and now just as AMD), but tried to present it as supposedly "supporting the project"--however, I should mention they're hardly the only company who was found guilty). It remains unclear as to which one of the two approached the other, but regardless, it's been no secret that money has been changing hands between the vendors and the company behind this program to "boost" the results. In turn, Intel and nVidia amongst numerous other companies have publicly condemned this company/program and this practice and have severed all ties to them, wanting nothing to do with them--at one point, there had been some discussion of a possible lawsuit or call to investigate the matter legally, but on the latter front never ever was done about it and on the former, there was never any public mention of a lawsuit. However, I do know that both nVidia and Intel along with a host of other companies have kept their word as far as washing their hands of this company and this product--and that the suspect or biased benchmark results still coming into the picture.
The company, for its part, has denied this publicly despite the evidence and instead has claimed that supposedly that AMD (who is the best known/biggest name linked to this scandal) had just been "more cooperative" in providing them with "better data" that supposedly led to the skewed performance results whereas nVidia and Intel (amongst the others in the camp that wants no part of them, but again, those two companies being the largest players/biggest names involved) had been completely uncooperative/"difficult" and had not provided enough data to properly measure their devices against the competition which is what led to the (suspect) results (although, the company does not view their results as suspect so much as the rest of the industry who is aware of what has been going on sees it this way).
Yet, sadly, this is not the first instance of money changing hands in exchange for "better" results--just a different approach to it. In years past (although, this practice no longer takes place nearly on the level/scale it once had, at one point it was very widespread and affected a number of magazines and websites and other media that many consumers, in particular, relied upon when making purchasing decisions), there had been another practice involved of where some hardware reviewers were known to take "kickbacks" or "financial incentives" from certain hardware vendors to give their products better reviews--if they made the competition look even worse, so much the better at times. This, in turn, led to two reactions from just within the tech/techie communities with one angry/frustrated enough over the situation to where some set up independently-run review sites (the better ones run by actual hardware testers and engineering backgrounds) where they not only refused to take any money, but sometimes went out of their way and had to purchase the products they were reviewing themselves. The other reaction was not so much a "blind acceptance" of this (as overwhelmingly in the community the "buying of 'good' reviews" practice was reviled--and much of the industry knew this was going on) so much as some pointing out the lack of education that the public generally has when it comes to such matters and in how many turn to the same mass market review sites and magazines (with some being suspected of or known to have been involved in this practice) and rely so heavily upon their evaluations and reviews of products they often don't question them (funnily enough, there was a similar finding a number of years ago about computer games of where the magazines used to have so much sway and power over purchasing decisions made by consumers when it came to reviews with many gamers relying upon those reviews and taking it almost at its word, that it could make/break a game easily--however, with the decline of the publishing industry as a whole, it's also led to a breaking down of this situation as well). Regardless, in the hardware industry, it's hard to say with any certainty at one point this practice fell off out of favor (although, it hasn't gone away completely, it's nowhere near as common to encounter this as it had been), but as things evolved and some of the indie sites became more popular amongst system builders and hardcore gamers (who also tend to spend more on hardware than most consumers), many of those who engaged in the practice tried to distance themselves more from it publicly and sometimes went so far as to fire reviewers who were caught engaging in it in an effort not to appear as biased and gain more credibility. Ironically and sadly, however, there are now very few independently-run sites actually left and one former site has now actually ironically gone on to be suspected of engaging in the very practice its founder (who has long since sold and left the site) condemned (which along with its sometimes questionable methods and some questions raised about the qualifications of those know working on it and their knowledge/or lack thereof at times, has led to the site appearing to be almost on "life support" these days and with many in the community having long since gone on to avoid it--I personally found the situation actually sad to see happen as when it used to be run by the founder, it had been my main "go to" site for hardware back in the day. However, that hasn't been the case in years now sadly).
