Everybody knows A/B test, as it became essential tool for exploring new user preferences or patterns and also a way to systemically innovate through chain of (managed) experiments. But A/A test, seriously? Yes, this is not typo (after all on QWERTY keyboard A and B are quite far-away from each other 😉 ) and there are good reasons for this test to exist (and being used). But let’s get to it step-by-step …
Ups, it happened again …
Imagine you had a glitch in system that led to sudden (maybe even undesired) A/B test of the customer experience. Some people received the service as expected, while others have been cut off from this element of experience. You do a post-mortem and you see that this positively influenced their shopping behavior. However, as this was a complicated glitch, you can’t easily replicate the test to find out if actually switching off the experience element completely would be a better long-term strategy. I mean, it was still valid A/B test, but as the assignment to groups was not controlled you are not sure if the up-tick in B-group shopping behavior can be attributed to the change in experience or just skewed (not random) assignment of user sample by glitch itself. How would you tell?
Wow, really?
Your existing A/B testing platform had fallen behind the curve and thus you decided to shop for alternative solution. After implementing the new tool, all of the sudden some of your experiments start to show significantly larger gaps between test and control group. Some of the gains are almost too good to be true. But you already cancelled your previous software subscription, so you cannot replicate the same experiment on previous platform any more. How would you find out if all of the sudden your campaigns started to work better or just new tool is “wired differently” to run the tests?
There is a way
As you can read above, both of the depicted scenarios are stemming from real life of online marketers. I bet you might have experienced some variation of those firsthand. But luckily, for both there is solution, you don’t have to throw away the data and start all over the again. And, yes, the solution is indeed our mysterious A/A test. So how does it really work?
A and A (again) , seriously?
The original idea of A/B test is quite simple: If mutually comparable groups of users are subjected to different treatment (and this treatment is the only substantial difference they have), if any of the groups behaves significantly differently as a result of the test, then there is high probability (yes, don’t forget it is still probabilistic finding) that these change in experience and change in behavior are linked to each other. The major vulnerability of this experiment, unfortunately, is the assumption that user samples A and B are really comparable. So what happens if we already have result of A/B testing but we have no proof/info how correctly the selection into groups happened?
Well, luckily this logic of the experimenting works also the other way around. If we have two groups undergoing identical treatment and resulting in also same behavior, we have high probability (yes, probabilistic here again), that the groups have had similar user distribution as well.
Therefore, if we have two groups and want to find out if they are some-what similar in user mix, we can subject them to same treatment and watch if they receive significantly similar result on high confidence level. And that exactly is the essence of our A/A test. Here the A+A signifies not that same group was used twice, but that different groups (in original experience A- and B-group) are subjected to same treatment in second experiment. This way we can try to “learn something about similarity’ of the groups ex post, already after completed A/B test.
Using A/A test is thus easy way to double check (or compensate for lack of) initial user assignment. Please note that while A/B test uses sample similarity to point out behavior difference, it’s A/A cousin uses (absence of) behavior difference to point out samples similarity. That also means that A/A does not:
- say anything (additional) about the strength of behavioral difference from original A/B test, nor it serves as any prove of it. (if the difference in final behavior between A/B groups was statistically insignificant, it remains so even after successful A/A test).
- prove general similarity of original A and B groups in general. It signals just similarity for behavior(s) relevant for original A/B experiment.
- generate any new insights about the users (which is often contested by opponents of A/A as waste of testing capacities).
The main value added of the A/A testing is that it is possible to run (almost) no matter what the original experiment was and is easy to set-up. After all, you just need to wire the groups to same branch of the process. Therefore, A/A test is a quick remedy to “unusual” set-ups/hick-ups in proper A/B testing.
Its simplicity, of course, comes with some controversy. Some practitioners argue against A/A tests as not being the most robust way to prove A/B groups similarity (heavy multinomial distribution comparisons are), being not fully possible to rerun (e.g., if original B-group condition altered long-term perception of service) as well as being opportunity (or real) costs of not running other experiments instead.
I am far from promoting A/A tests as silver bullet, evade it if any of the (above mentioned) counter-arguments are very true for your own situation. However, the mere A/A tests existence and its proper set-up should be part of your toolbox; May the situation turn it to be the cheapest (and quickest) way to heal the experiment (improper) set-up. Especially so, if you are to assess the results of A/B tests conducted by somebody else before.
Späť na domovskú stránku
If I was asked to describe CBH without using words audio and social media, I would probably wrap my explanation into Podcasts. That is because, from all the thigs of the World, audio podcasts are (both in form and benefit for end-user) probably as close to CBH as you can get. One might think of CBH being actually “just extension of podcasts” (which I would deem mistake). So what exactly is CBH likely to summon on its nearest podcast sibling?
If you look for exemplary victim of the CBH rising, then Meetup platform is the one. The simple reason for this bold statement is that CBH is something like superset of Meetup functions (whose mission is mainly to bring to attention of sizable (and likeminded) crowd to event happening). And offering this as-a-service to B2B customers mainly. Well, CBH can do all that Meetup was doing and it can actually host the very event itself. And its all for free (at least for now). Meetup’s outlook got already doomy with pandemic cutting the option of meetups take physically place. In respond to this, Meetup took the only logic step to motion towards virtual events. And .. BANG! That was the trap drop of CBH. I would be shocked if Meetup ever recovers from this.
When assessing the face-off with giants of social media, it is a bit of the moving target shot. Facebook has proven already early Feb 2021 that it does not intend to sit and wait for CBH to eat into its kingdom (and profits attached to it). So commenting on clash of these two forces is bit more like chess match commentary. However, Zuckerberg indicated that CBH is deemed to be serious foe and will face all the competitive treatment from Facebook. From my point of view, CBH can only claim the “bored so scrolling through FB feed” part of Facebook screen-time. Networking, direct messaging, sharing video and pictures, that all is missing in CBH. On the other hand, until CBH gets Android resolved, Facebook can effectively counter-strike with own audio-chat platform carving enough breathing room for itself in audio social media space. 
Assessment of traditional radios’ hopes is looks a bit more difficult on first sight. But if you strip out layers of historical sentiments, picture resembles the CBH vs. Meetup stand-off in sense of CBH can offer everything that FM radio has to offer. Not surprisingly, there are already 24/7 airing radio formats on CBH running. Therefore, the fight with CBH boils down to 2 factors: A] the coverage of Internet signal vs Radio signal ; and B] IT literacy of the radio audience. Thus, yes, in some geo areas Radio will maintain monopoly and also true that senior listeners are less likely to appear on CBH platform. But you can smell this holds true only to certain time. Because on both dimensions the development is running against the traditional radio odds. 
Conference (tourism) is yet another industry with nothing-to-envy fate. Completely trashed by 2020 pandemic, on its needs and now facing serious threat or replacement by ap like CBH. Without too much further due, let me comment that traditional (and virtual) conferencing is facing “Embrace Change or Bleed Out” dilemma. For those seeing opportunity in CBH (after all you spare all the rooms rent and equipment rental and people even eat they own food along), they might revive on/with CBH. But for majority of the others, this is probably pretty bleak game-over. 
The most complex (and thus most difficult to evaluate) is the relationship of CBH to video streaming services. Upon shallow consideration, CBH should not dare to think of hitting Youtube. After all, Youtube videos have all the audio content + video (and its monetization) on top of it. Also many YouTubers have vast influencer power which they don’t easily let go. I have to credit 
LinkedIn as predominantly the professional social network stands a bit aside from the main battlefield of CBH entry. Also the use-cases and value added of networks like LinkedIN (or German speaking Xing) has little overlap with CBH features. Therefore, we can assume CBH will have no means to challenge LinkedIn’s positioning of “place to look for talents” (or jobs which is just other side of the same coin). Though to be fair, recruiters and headhunters are pretty populated among early adopters of CBH. However, whatever (speaker) reputation you manage to build on CBH you might need to cement this into resume readable/usable in recruitment process. So in countries where CBH userbase will surpass LinkedIn account penetration, CBH rise might even (slightly) facilitate the growth of LinkedIN. As a side note, the HR services industry’s interest into CBH might be well fueled by chance to both scout (or showcase) the potential candidate for hiring organization. So be careful with answering the CBH questions, as some of them might come from you potential boss. 
Besides President Trump’s effort to weir it, TikTok was (deemed to be) the next big thing on social media scene. With its unique short term video content appeal, it was gaining traction with mainly younger cohorts of the internet population. So how does CBH rise impact this? Well, video still trumps audio, but we debated that this might be myopic way looking at this already in YouTube discussion above. Also TikTok is more one-way communication prone and hence you need to ask: Would you , as teenager, try to win heart of your dream girl with trying to hook her on funny videos or rather chose option to talk her into dating in private CBH chamber? Attention of once-to-be-adults is flickering. And CBH (also officially forbidden for under-aged, or exactly because of it) is fresh breath in direction from TikTok. 
This comparison pair might surprise you, because CBH actually does not claim any foothold in telecommunication arena. Well, but take 2 steps back and think about it again. Isn’t the CBH room just a group conf-call? Don’t get me wrong, I am not trying to suggest that CBH might be threat to Telco’s, quite the contrary, as Telecoms are on pathway to walk away from voice calls anyway. So they don’t mind any other platform morphing yet another (minor) portion of voice traffic into internet packets. What is also important to note that given the choice between streaming video content and audio content, the latter is certainly less bandwith drag. Ultimately massive adoption of CBH would help to take some download stream drain and lead to customers more happy with capacity at hand. CBH thus might indirectly help to make the cell network more stable and satisfying.
Working for TeamViewer, remote work and teleconferencing software provider, I can tell you that sizable part of the video calls either abstain from or would not mind move from video call to audio-only meeting. After all, this was how the things had been done before the internet boom. In COVID times the video extra is often warmly welcomed extra, but when CBH-like service is still acceptable fall-back option. Therefore, CBH will probably not endanger (or slow down) the growth of Zoom, 
