Domas Mituzas about the MemSQL vs MySQL benchmark:
Though I usually understand that those claims don’t make any sense, I was wondering what did they do wrong. Apparently they got MySQL with default settings running and MemSQL with default settings running, then compared the two. They say it is a good benchmark, as it compares what users get just by installing standard packages.
That is already cheating, because systems are forced to work in completely different profiles.
The first paragraph of the post summarizes very well the general feeling about benchmarks:
I don’t like stupid benchmarks, as they waste my time.
I think that most of the generic benchmarks are stupid, even if some generic numbers are considered interesting by software engineers. Benchmarks designed around specific scenarios of applications will most of the time give more realistic results. But even those are difficult to design and account for all the configuration options, scaling, or changes of the use cases.