ultra_prescriptivist t1_jd09n75 wrote on March 20, 2023 at 10:07 PM

Appreciate the effort that went into this - I do like to see it when people go the trouble of actually testing these things out rather than just blindly (!) following the internet consensus.

That said, there a few possible issues with this test.

Firstly, what exactly were you trying to establish - A) that Hi-Res tracks sound better than CD quality tracks, B) that exclusive mode sounds better than the system mixer, or C) that Qobuz sounds better than Deezer?

There are a lot of potentially confounding variables here, so establishing what your specific goal is is essential in order to control for those variables.

For example:

Did you ensure that both services were using the same master recording for each track on the playlist? If so, how?
Shouldn't the DAC have been set to 44.1 KHz
instead of 48KHz in system settings, seeing as that is the native sample rate used by the music playing through Deezer?
Why didn't you use an electronic level meter to precisely match the playback volume levels of each app? Even a very slight disparity here could have affected your decisions.
Doing only 1 trial per track doesn't sufficiently rule out the possibility that you were simply guessing. When doing blind ABX tests, for example, it is customary to do at least 5 for each A/B comparison because you already have a baseline 50% chance of getting the answer right if you only do 1. So your result of getting 10/14 correct isn't really much better than the 7/14 you would have gotten simply by flipping a coin.

So, while I most certainly applaud you for setting all this up, I think narrowing your focus to achieve more reliable results would have been a good idea.

mmry404 OP t1_jd0ciie wrote on March 20, 2023 at 10:28 PM

Thank you for your thoughtful response! I will try to address your valid points:

There is really a two-fold problem here, being Deezer not HiRes and Deezer not bit perfect. But HiRes only makes sense with bit perfect playback, so I decided that these problems kinda entail one another. Primarily I was just choosing between two services while my free month on Qobuz is still going. That's concerning A and B, and C just sounds too vague to me without any technical explanation as to why would that be the case
On the same master recording I commented under another comment (shortly, I just used modern recordings that probably don't have many masters)
I haven't set the DAC to 44.1 because I forgot (hehe) and also I mostly use this dongle with my android phone with no way to fiddle with its sample rate (that I know of) and it should also be 48kHz by default. I used a laptop instead because Qobuz on the laptop has exclusive bit perfect mode, while on my phone it doesn't, as I haven't bought the USB Audio Player
I should have definitely used the meter to get the volumes right, I will next time
That's a cool idea on trying the same track multiple times, thanks! Yeah, I did have my statistics courses back in the university and I do remember some things (vaguely), and yeah, the goal was to get as far away from 7/14 as possible. Also, the results do sort of align and make sense in hindsight... While the result is not overwhelming, still it's there and at least I didn't skew to Deezer. Further investigations will be carried out, since it was much fun:)

Thank you very much for your insightful comments again!

ultra_prescriptivist t1_jd0i27x wrote on March 20, 2023 at 11:07 PM

>- There is really a two-fold problem here, being Deezer not HiRes and Deezer not bit perfect. But HiRes only makes sense with bit perfect playback, so I decided that these problems kinda entail one another.

True, but they still are two distinct factors. It's always best to isolate them and tackle them individually in my experience.

>- On the same master recording I commented under another comment (shortly, I just used modern recordings that probably don't have many masters)

I dunno, I see several tracks on there that have had multiple remasters and reissues over the years, and even 2020s music can occasionally be issued as different masters to different streaming services. You can't just rely on faith that they are probably the same.

Again, it's all about narrowing down to what you are trying to establish. Using different streaming apps when you haven't ruled out that the tracks you are listening to are not the same just introduces too many unknown factors, IMO.

For instance, if your goal is to establish whether or not it's better to use exclusive mode rather than the Windows mixer, it would be better to 1) use the same music app for both so you are not introducing additional, unforseen points of failure by using two, and 2) use local files instead of streaming services.

Perhaps you could use an app like Foobar2000 that allows you to switch between WASAPI exclusive mode and Windows Directsound. By using the same app and same track, that would eliminate a couple of your unknowns - the potentially different masters being used on streaming services and the volume disparity issue.

I'd definitely also shorten the list of tracks and do multiple trials (at least 5) per track.

Either way, let us know if you do any more follow-up tests!

mmry404 OP t1_jd0k9t1 wrote on March 20, 2023 at 11:22 PM

Thank you very much! I generally agree with all your points. That experiment design choice was mainly dictated by my streaming strategy (continue to use deezer or switch to qobuz), also I wanted to give qobuz the chance to sound the best it could. First try is kinda janky, as always, isn't it? I didn't think I would be able to tell them apart and got excited, maybe a little too soon.

Also, looks like I shouldn't bother my girlfriend too much with these tests and maybe wait till I have a friend over or something, lol

ultra_prescriptivist t1_jd0mkq1 wrote on March 20, 2023 at 11:39 PM

Ah, well if comparing streaming services is the goal then I certainly have done my fair share of that, as my comment and post history will attest 😄

I did a similar thing, although for me it was testing Spotify versus other streaming services.

As an aside, my testing on Spotify/Deezer showed that they generally use the exact same masters, for whatever reason.

Anyway, I used Audacity and WASAPI loopback recording to take samples of the different services and analyze the results though a combination of A-B comparisons using Audacity and blind ABX testing using Foobar.

These methods may prove useful to you if you want to continue your testing without bothering your girlfriend, lol.

mmry404 OP t1_jd0nwtq wrote on March 20, 2023 at 11:48 PM

Great! Thank you very much! I wil definitely check those out! With these gizmos, I can satisfy my OCD urges alone and any time :)

mmry404 OP t1_jd0l9po wrote on March 20, 2023 at 11:29 PM

Could you explain what do you mean by multiple trials? Like, the same track is played to me 5 times and each time I don't know where it comes from and I have to guess? That seems like a strange approach, since I can't go back to the previous playback state and compare, and I kinda just roll with whatever I'm hearing at the moment

Or, do I just ABABAB the same track, but each time I make a decision the person in charge flips a coin that decides which source is A and which is B and I start again (repeat 5 times)?

ultra_prescriptivist t1_jd0nqwb wrote on March 20, 2023 at 11:47 PM

>Or, do I just ABABAB the same track, but each time I make a decision the person in charge flips a coin that decides which source is A and which is B and I start again (repeat 5 times)?

Pretty much this. The idea is that you do the same comparison (e.g. the Depeche Mode track) multiple times and each time choose which one you think is which, with it being randomized after each decision. The more trials you do, the more you reduce out the possibility that you just happened to pick the right one by chance.