CAD Olympics
Roopinder Tara has mentioned an interest in an event he calls CAD Olympics before, but I guess I never took him seriously. His latest mention came in a CAD Insider blog post entitled CAD Market — Life’s Not Fair. The post was about how CAD users select software to purchase. Ideally, Roopinder says, a product is picked on its merits. [pullquote]In a perfect world, a product is picked on its merits.
– Roopinder Tara[/pullquote] But in reality, people tend to just pick the leader or the perceived leader.
What would CAD Olympics look like? Who would run the software? Who would determine the events? Who would score the results? Is there any way at all to make an event like CAD Olympics “fair”? This is something I could really get behind, but it could also devolve into a marketing circus. I’m not sure about Roopinder’s goals for this, but I would hope that the goal would be to give people making CAD purchases a clearer idea of the real and comparative capabilities of individual software packages, without having to rely on the vendors themselves, whatever scattered semi-objective info exists on the web, or the user’s own evaluation, which is probably not very thorough.
When I worked for resellers, we would sometimes be called in to do head-to-head benchmarks against AutoCAD, Pro/E or Unigraphics. We always killed AutoCAD, but Pro/E or Unigraphics tended to depend on if there were any surfaces involved. The Pro/E sales guys tended to lose competitions based on personality alone – they would only compete if they could set the rules. Yeah, they were that condescending and arrogant. UG had good stuff, but back then the interface would get in the way. Anyway, I guess cheating was expected, and the customer would typically watch (and even count) keystrokes.
We had to create parts, put them together into assemblies, make drawings, make changes, save files, and print, and do it all live with a stopwatch. It wasn’t so bad. We always won these competitions with SolidWorks.
And then there are the infamous shootouts against Inventor at some user group, or the debacle with Ray Kurland. Somehow the Inventor guy was always an ace and the SolidWorks guy was always … well, not. I’ve used Inventor enough to know that if you skim the surface, it’s about the same as SolidWorks, but if you go deeper, Inventor cannot keep up.
How to make a CAD Olympics fair? There has to be an operator. You want a knowledgeable operator who will behave. Someone not involved in sales, but probably from the vendor.
There must be a set of events. Who chooses these? It would be possible to load the events so that they favored either direct edit or history-based modeling. It would be possible to severely punish non-surfacing modelers. It might be slanted toward software that can do multiple size variations within a single file. Part of the competition would be determining what you expect baseline CAD functionality to be. You can’t have vendors setting up the events. Journalists? They get paid by advertisers. Analysts? all the CAD companies are their clients. Is there any such thing as CAD agnostic? Even users are biased. Is it possible to make the results quantitative so there is no interpretation required to find a winner?
Maybe we set up an anonymous set of judges. Would that work? Wouldn’t you want to know who the judges were?
Maybe a vote? Do you have any faith in design by committee?
So, in the end, I believe that Roopinder really is right. Life is unfair. Or at least the CAD Olympics would not be able to be proved fair to everybody. The best we could do would be to state the conditions, name all those involved and then whoever looks at the results would have to form their own opinions based on the information given. I don’t think it’s really possible to name a single winner, anyway. There is no one CAD product that is good at everything.
Anyway, I just wanted to maybe help raise awareness of the possibility and maybe get people thinking about it.
Roopinder, where do we start? Anyone else have any ideas?
I agree with you, it is never fair. I have been working with Autodesk Inventor for 5 years, I am the design department, I do models, assemblies, changes, drawings and it is never fast enough for my boss, that says that it is too slow… I don´t know what are the average time to model, assemble, make drawings of a drawing die, for example, and then get the changes from my cnc department. My boss was impressed with Solidedge, because he saw that only with handling the grips, you can change the 3d model, but I worked with it, but, for me the software in very unpractical, when you use it with assemblies….
Why not something like the CSWP exam, but for everyone? It’s quite possible another CAD system could actually complete the CSWP as it stands (or at least portions of it), and it is timed, and even with prior knowledge and prep isn’t super easy to beat.
It’s also timed, and allows for multiple users of each software to attempt the task, allowing for “heats” or “trials” to select the best users.
Matt, at the risk of over doing my participation here I would say it *is* worth trying to bring back integrity to CAD evaluations and reporting. I remember, a few years ago now, maybe 2001-2005ish, there was a guy Joe Greco – unfortunately died young – who was someone attempting to do decent comparisons and tabulate features. I remember reading his articles in my own quest. Even at that time though the CAD journalism waters were fairly mirky and he was a rarity. I think actually they instituted an award after him in recognition of his efforts? Really though to do the same or better today I think you would want to self publish via the internet to be truly independant and charge for it to support your work and costs. If its well done people won’t mind paying a bit to access genuinely useful info.
Perhaps you could have a companion forum/blog where users and prospective users go to exchange experiences and opinions. There was a time when usenet was the place to go exactly because users related to users without being suffocated by vendor treacle and it was a defended space. Although that hangout deteriorated perhaps it could come back in another form.
I think someone with your experience and character would be a good candidate to do these in depth comparisons. Go for it. It gives you some diversification from just being involved with SW books and as I said elsewhere you are probably underachieving ATM.
Rick, put the pilot and a hostess in the toilet and close the door?..perhaps put a TSA goon in there too for added realism..dunno..
On the second issue I think you can use decals like that these days in the model view.
I think that we need more design challenges. Invite participants to use other CAM software. Encourage “me too” responses that might copy another entrant and model it differently.
I am facing a couple of design challenges:
I have a mannequin that needs clothing. I do not want the distraction of a naked pilot in my new design airplane. I want to wrap a paint scheme on the fuselage and wings. Split surfaces work ok but I need fine control of both top and side views. Coincident points seem to produce some gaps.
Sorry I posted by accident before I finished – fat fingers on an Android 😉
I think if you tried to host a CAD report the corporations would be all over you trying to discredit or excuse the results. They may sue you even. Lets face it when you get big you get corrupt or compromised.
You have legal teams creating mischief and your whole fascade is controlled and managed. Nothing is real.
The product is always wow! when its obviously meh… Its always a good upgrade even if its a fail…or its not as bad as you might think especially as you are aware how your hotel and travel to the launch was paid for.
I am sorry to say when you are thinking big serving the market truth and facts is the first casualty and there are usually any number of media whores and toading employees or dependants willing to/go along with the. deceit.
For instance the new SW feel good engineer/CEO comes out in a bit of corporate theatre and makes placating noises about evolution not revolution and yet we know that nothing has changed behind the scenes. They are still working on the revolutionary cloud as before. We were assured SW is going to be actively developed but as anyone with half a brain can tell looking at 2012 its not. And what is the point of him doing the redundant SW exam other than to sucker some SW users a bit longer by appearing to be committed to the existing.
Its all BS. People are immersed in BS. They are paid to BS. They get ahead by BS. They are enmeshed in BS.
The CAD industry in a ponzi scheme.
Well there is nothing written that isn’t to the vendors liking Matt. This is the sick self interest of large corporations.
Journalists are bought parrots. Bloggers are obligated or mindless lackies. Even wikipedia articles are massaged. We get carefully cultivated user meetings with SW personalities and SWW events that have some resemblance to cults. You know I have often spoken up about this pseudo social engineering and usually I get some moron coming back and labelling me negative or something.
Say something truthful somewhere and you get attacked by pets, VARs and lurking corporate minders. Possibly as a last resort you might even get your site hacked.
In the same way that vendors try to screw cad olympic events they would probably try to l a consumer report
You know Matt I almost entirely agree. Part of the reason many CAD industry publications closed was because articles on CAD software failed, time and again, to do anything different other than to review with rose coloured glasses and one in particular case issued 5 stars ratings to software which should have been lampooned. Their re/views became irrelevant.
That said you (thru’ your blog etc) have already come very close to providing usefull comparisons – how valuable was “puffy box” and some of your other challenges? I, personally, think they were or great value – and I don’t use Solidworks.
What you have already done may be an indication of one direction to consider/follow.
“I want to know what are the relative strengths or weaknesses of the SolidWorks Sketcher compared to the Solid Edge Sketcher”, ok! [Somebody] Roopinda, you or another simply needs to choose a task or area (Sketchers) with a particular intent, objectives and outcome and let the hounds loose; the results and the debate about the results/solutions will provide the comparison and – maybe – much more besides.
@R.Paul Waddington
Aw, geez, I feel like I’ve asked this question badly. I think this kind of comparative information is important, and the more objective it is, the more important it becomes. I know that we all have different tastes, and different needs. I don’t care about that stuff, that’s for you to deal with. I want to answer objective questions, to the extent possible, about comparing specific functions. This is not necessarily just a “check list” sort of comparison.
Maybe “olympics” is the wrong metaphor. I would prefer an event where the comparison would be between softwares, not between demo skills. I know, that’s virtually impossible to separate.
Do you all recognize the problem? Right now, there is no publicly available even remotely objective comparison between popular CAD products. Or if there is, I just don’t know about it. CAD journalists either don’t have the expertise to do a real comparison or the unwritten buddy arrangements they have with all the vendors prevents them from doing anything that would even remotely look like a definite opinion. That’ll offend a half dozen people or so. CAD journalists don’t write this stuff. Consultants may produce private comparisons for organizations, and I guess you can pay for honesty, or you can pay for a particular point of view. I’d like to see something that is free for anyone to access.
I want to know what are the relative strengths or weaknesses of the SolidWorks Sketcher compared to the Solid Edge Sketcher. I don’t want a slick demo, or a nail-biter competition. I want a point-by-point comparison. I’m looking for the type of data engineers would find useful, not reality television, or marketing fodder. I think people are smart enough to be able to interpret the results in a way relevant to them. I recognize there are a million ways for it to go wrong, but that doesn’t mean it can’t be done.
There is no Consumer Reports for CAD software. Maybe that’s the model we should be looking at rather than “olympic competition”.
CAD shoot-outs/an Olympics (would/could) achieve what? Questions that can only be answered measured against an outcome. For CAD it means, choosing a task(s) with a defined outcome first then, define the rules of the contest to ensure those participating are aiming for the same goal.
Problem; each user of CAD has different goals and reasons to apply and use CAD. Whilst Bolt’s incredible ability is important to those who compete against him, it is of no relevance to me. A CAD competition is more likely to be successful if it is for the competitors; their achievements could then be translated by the spectators comparing the tasks achieved and the methods used to their own. Shoot-outs (I have participated in) have attempted this and have had a measure of success when the audiences’ expectations matched the chosen tasks.
Matt, I smiled when I saw your comment, “we always killed AutoCAD”. One of my sweetest sale wins was Mech’ Desktop vs Solidworks. An existing AutoCAD site of some note gained a new engineering head who wanted to turf AutoCAD for Solidworks, when his crew were looking to move to Mech’ Desktop. He presented me with a list of twenty (20) points – “things MDT could not do” – provided by Solidworks. He insisted my demo MUST address each of those points else he was not even interested in listening to me. I did: I not only answered, and showed MDT could do each of the tasks set, it then gave me the opportunity to set a similar goal for Solidworks which they could not match: they lost.
There were several lessons to be learned from that stoush. The most important was that the Solidworks sales guys targeted their competitions product and its presenter. They completely forgot to address what their product needed to do for the customer. In setting the goals as they did, along with the complicity of the engineer they did, in fact, create a biased competition in which had any CAD product done what I did with MDT could have beaten them out of the sale. Once Solidwork’s goals had been met then the things MDT could do that Solidworks could not became more obvious and of more value to the customer. The two faces of the competitive coin.
Could a competition for driving nails with a claw hammer be used by a watching prospective buyer to choose a new hammer?
Neil mate you clearly have too much time on your hands 🙂
How about a CNC toaster – http://www.coolest-gadgets.com/20070502/custom-toast-printing/
This would also test the CAM capabilities of the software 🙂
LOL. Here you go Matt. This is a messy napkin type doodle for a ‘wireless laser toaster’. If you are desperate for original ideas you could use this for your new ‘reality CAD’ series of books.
You can burn artwork and messages on the toast – even different ones on each side at once and it will also burn toast for 2 people with different types of bread simultaneously and let you take yours if it is finished!
Will hold and reheat the finished toast inside if no one is there to take it and many other useful features…
Just the thing to make breakfast fun again 😉
hmmm.. you could have a random ‘thought for today’ or even 1+1=? for kids…or download a SWv6 tweet from the cloud – “having toast now, black..”…the possibilities!
[img]http://www.dezignstuff.com/blog/wp-content/uploads/2011/09/toaster002.jpg[/img]
Interesting topic and idea. I guess I am in the camp of not being sure what it would prove other than an indication as to what you might want to look into if you were shopping. When I was looking for a new CAD program I was not interested so much in the absolute outer limits of the capabilities of the program as much as I was interested in what it would do with 90% plus of my regular parts where the vasy majority of my time is spent. Software has grown to be so multifunctional, with a lot of it never used by the typical user I suspect, that one has to look at what they need VS bells and whistles. So would this Olympics have to be broken down into two categories perhaps with one for day to day mechanical parts where most of us live and then the super cool surfacing stuff where not that many go? They are in many ways two different worlds after all.
It may be impossible to declare an overall winner in the CAD Olympics. There is no gold medal for track and field, for example. But nations can tally their medal totals. Similarly, CAD Olympics should have specialized events: sheet metal, surfaces, etc.
Neil makes a very good suggestion there! Taking it one step further, this is a very valid way to compare different applications. But it needs to be in detail and ideally would involve actually designing as opposed to recreating. This would also work as a video series as well showing dead ends, and options. Mark B. of SolidWorks’s always does a bit of this in his surfacing demos and they are all the better because of it.
The average user and buyer is not interested in how fast an expert can recreate a model, but rather how they approach designing using the system, and what strategies they attempt for different types of design.
I think if this were tried in all the main CAD systems I think the results would show that there is not actually that much of a difference in packages that have a the Part/Assembly/Drawing approach. Comparing them to the Parts and Assemblies/Drawing systems would show (in my experience of using both) that upfront design is simpler and faster in these (latter) ones as there is less to get in the way. Dare I say, it would be less of direct model vs history model then.
As an educational series – design and model typical consumer products – perhaps something relatively static like an electronic product with internal PCBs etc and a curvy plastic case, then maybe something with a mechanism (‘cos designing linkages in any CAD system is still a bit of a black art), then maybe just something with a very complex but subtle shape (but not a car please!). I’d buy it.
So Matt why don’t you test them out and publish the results yourself? I mean sell your independant findings.
Slightly OT but related I have been thinking your best abilities are a bit wasted on updating SW Bibles to include the latest mouse gesture enhancements. I mean it is clear in your writing you are a very knowledgeable and capable chap – more so than your editor or scope of your current books would allow – and I think you could unleash yourself on more demanding pursuits. There are many folks who do the CAD ABC stuff but almost no one will ever say anything about how good or otherwise something is or talk about how it performs in a whole project. I did think of suggesting that you pick on something like a toaster and follow it through completely – sheetmetal, plastic, electronics, drawings, analysis etc etc. discussing strategies and practical issues. I mean the whole book focuses on actually doing something with real world relevance with the software and even using other software where needed. I guess you could start out with a SW version and adapt the book to other programs ie make a series.
When it comes to people evaluating CAD for their purposes there is really only a diet of feature lists on glossy paper and toy examples out there. Give us some grit. 😉
How about ..er..’Matt’s 3d Garage – Real design projects in SW’. ..dunno just an idea..
Matt,
The only thing missing in some of the excellent design challenge topics was posts that used other CAD software. I hope the users who have switched will come visit and participate.
@Kevin Quigley
Really, I see this as just an additional data point for someone who is looking to get the best software for a particular task, or generally anyone looking for additional CAD capabilities. It’s simply not practical for everyone to take training on SolidWorks, Solid Edge, Inventor, Pro/E, Unigraphics, Alibre, Rhino, SolidThinking, CoCreate, Spaceclaim, VX, Top Solid, and whatever else is available, nor does it make sense. It would be nice if someone had done the research for comparative strengths, and was able to state the results in a way that still allows potential buyers to make up their own minds.
@Rick McWilliams
Yeah, Rick, but even you are interested in the capabilities of other software packages. It would be nice to know that one software is best for press break sheet metal, another is good for formed or drawn sheet metal, another is best for plastics design, another is best for mechanism design. I’d be interested to know all that.
I would agree with your comments Kevin.
However many years ago the mag’ MultiCAD ran “competitions” at our engineering trade shows. Their success was questionable, but what was more interesting was the reaction from some vendors. At one event a vendor withdrew when they found out a “less superior” product was also participating: they did not want to be seen, on the podium and compared with, what they saw, as an inferior product. I, personally, thought their arrogance and inconsideration to the organizer – a CAD publisher – was appallingly unprofessional and told them so.
Out of that I suggested, to the publisher, an alternative event where a particular industry group chose a design task. Companies involved in that industry, and training institution in the same industries, would enter “teams/individuals” from their design offices/classes into the competition.
A competition contested not by vendors but by people from a particular industry.
Part of my reasoning/conditions was the teams chose their tools for the published tasks. Vendors had no control of the event, NONE.
Imagine a competition that threatened to showcase a teams design ability, using CAD software, but one which could mean a particular piece of vendors software would not feature in the event or the subsequent publicity. Possible exclusion would dampen most vendors’ arrogance.
No, it was never done – MultiCAD vanished as have many other CAD publications.
I do not see CAD as a sport. I am much more interested in nice examples of good ways to model shapes. The bottle design topic, and propeller design topics were terrific. These let modelers show off good design and great CAD techniques. It was entertaining and informative.
Matt in all honesty I think this would be a waste of time except from an ego point of view by those competing. The only way to properly benchmark software is for the end user to use it to model the parts they need to, and to create drawings etc.
Back in 1997 I attended a SolidWorks training course alongside a guy who ran a business manufacturing disability products. What her was doing was trying several different products by attending training courses, then using the software for a time and so making an informed decision. In those days this was a common sales tactic by SolidWorks – pay for a training course, use the software for a month, then decide. It was a great tactic as it worked. It weeded out the time wasters (‘cos if you were not prepared to invest in training you were unlikely to spend £5k on a seat). I did this as well and I was sold. After 30 days of constant use (after a 3 day training course) we bought it. The guy I met on the course bought 5 seats.
For me, only the end user can decide what they need. A head to head modelling 100m event is great for driving blog traffic and PR but nothing more IMHO.
No person wins the Olympics. People only win individual events. I assume the same would be true for a CAD Olympics. Best For sheet metal, surfacing, etc. Maybe.