I had no interest in the original EVE Project Discovery, but I thought maybe I'd give EVE Online: Project Discovery Exoplanets a shot. I mean, surely the activity couldn't be as awkward as the name. Right?
What I found was a metaphor for everything that's wrong with CCP's management of EVE Online at this point.
The new-user experience is pretty horrific. PDE rolled out unbelievably buggy, and the bugs haven't stopped. There's a tutorial, which I diligently worked through, but which failed to give some pretty important hints. Is there a way to zoom in on the data? There is: please show it to me. Is there a way to adjust the "detrending" filter? There is a little bar. Point it out to me.
Compounding the new-user experience is the fairly horrific user interface. Interaction with the tool is a random combination of clicks, click-drags, mouse scroll-wheels and widget manipulation. If you can't figure out how to do something, don't bother to look for the non-existent tooltips. What is "detrending", anyway? Oh, a high-pass filter.
Clicking "Next" at the end of the tutorial took me...back to the last screen of the tutorial. Finally figured out how to get past that one. The intermediate home screen between trials has mysterious objects at the bottom, including a spaceship. No idea. Want to look at the tutorial again once you're past it? There's probably a way, but I don't know what it is.
The tools provided to me are, by my in-real-life computer scientist standards, utterly inadequate and lame. CCP claims it would be "too hard" to implement any kind of DFT, in spite of the fact that they seemed to do a high-pass filter just fine. Honestly, an FFT running client-side in Python would be fast enough to provide a lot of help.
Why not provide a bandpass filter instead of just a high-pass one, to get rid of all that horrible high-frequency noise everywhere? From the comments I've seen, I'm not alone in thinking that the whole exercise would be orders of magnitude easier if you gave me a sensible analysis toolbox. Squinting at the terrible tiny graphs is just silly.
Anyway, having gotten all but the last example in the tutorial I figured I was fine. Let's try Level 1 training. Accuracy not so good, but I got through it OK. On to Level 2. Oops, haven't so far gotten a single example "right" in 10 or 15 tries. In most of these, I have a hard time believing the correct answer even after it is shown to me. There's no way I'm getting out of Level 2, let alone to Level 5.
I think PDE is bad science and bad use of our time. Please just give me API access to the few kilobytes of data that appear to be in each sample and let me build tools to answer via the API. I'm pretty sure that in a few hours I'd be doing better than a human could.
Hire a real UX team and let them do their job. This is classic EVE — the interface was obviously designed by some combination of astronomers and software devs, and it really shows.
Test things properly before release. The obvious bugs in the training data that folks have been posting are easily detectable with simple tools. Build them.
Get somebody who understands modeling and mechanics in a domain to build the algorithms for that domain. In this case, that would be someone with DSP and probably machine learning experience who's had some experience with this kind of dataset. Picking weak periodic signals out of noise has not been a big deal for 30+ years. If it's too hard for machines in this case, at least let the machines get the user part-way there.
Do all these lessons apply to EVE Online as a whole? Certainly. After a few hours of PDE, I was done. If CCP had been trying to sell PDE, I would have been yet another of their quit-during-the-free-trial customers.
Stupidly steep learning curve? Check. Insufficient help getting started? Check. Awful UX? Check. Must be playing EVE Online.