How we found the best streaming services: Behind the curtain with Allconnect’s metrics

Joe Supan
JS
Joe Supan

Feb 13, 2020 — 5 min read

Live TV streaming metrics

On-demand streaming metrics

How we scored the best live TV streaming services

Channel lineup

For live TV streaming services, we focused heavily on the quality of each service’s channel lineups. While each service boasts about the raw number of channels you’ll get, not all channels are created equal, and we didn’t want to give the same weight to Cheddar News as we did to CNN.

Enter TiVo’s Video Trends Report. The report comes out four times a year, and (among other things) asks 3,330 respondents, “Which channels would you be interested in including in your TV package?”

Image credit: TiVo

All in all, 112 channels are included in the list. We then ordered these from most desired to least, and assigned values to each channel on the list so that the most popular channel (ABC) was “worth” three times as much as the least popular one (YTV). Once each channel had a score, we simply checked which channels were included in each streaming package and added up the scores. We then standardized them to a 10-point scale for our “channel lineup” scores. 

When services had more than one plan available, we averaged the scores for each of their plans. Here’s how these channel lineup scores compare for each live TV streaming service:

*As of 01/17/20

Value

For our value scores, we simply divided the raw channel score for each service by its price. This gave us a good idea of how much you’d be paying for each channel. We also standardized these scores on a 10-point scale. Like we did with the channel lineup number, if a service had more than one plan, we averaged the scores for each plan. Here’s how each service scored for value:

*As of 01/17/20

There was one small exception here. Because Hulu + Live TV includes Hulu’s popular on-demand streaming service, we subtracted the price of a subscription ($5.99/mo.) from its base price ($54.99/mo.). That raised its value score from 5.6 to 6.0. 

Extra perks

With this score, we tried to quantify the quality of supplemental perks that each service offers. We focused on hours of cloud DVR storage included in each service’s base plan, how long recordings store for and the number of simultaneous streams allowed. We also considered any sign-up bonuses here, like Sling’s offer of a free Amazon Fire TV Stick when you prepay for two months of service. 

With unlimited cloud DVR storage that stores for nine months and three simultaneous streams, YouTube TV received our highest score in this section with 9.5. 

User experience

This is the only section that was truly objective. We tested each service out for about five hours each on four devices: a Roku Streaming Stick, Apple TV, Amazon Fire TV Stick and MacBook Pro. We looked for things like an easy-to-navigate interface, helpful recommendations and buffer-free streams. 

Overall score

To get our overall score, we weighted each section according to their importance. Our channel lineup and value scores each accounted for 35%, while extra perks and user experience were assigned 15% each. 

How up-to-date are Allconnect’s live TV streaming service metrics?

We do an official audit of our live TV streaming metrics on a monthly basis, but if any major changes occur outside of that schedule, we update them immediately. For instance, AT&T TV NOW increased its DVR storage from 50 hours to 500 hours in January 2020. That instantly raised its “extra perks” scores from 5.3 to 8.7 and its overall score from 5.3 to 6.3.

How we scored the best on-demand streaming services

Movie and TV scores

To evaluate each streaming service’s movie collection, we factored in both quality and quantity. For the sheer raw numbers of movies available, we utilized data from JustWatch.com, which tracks the titles that are on every streaming service. 

When it came to analyzing the quality of each service’s library, we used IMDb’s lists of the top 250 TV shows and movies, which combine user ratings and popularity to determine their rankings.

Like our live TV streaming scores, we awarded more points to the titles at the top of the list than the ones at the bottom. In the top 250 movies, for example, The Shawshank Redemption (number one on IMDb’s list) was worth about three times more than Guardians of the Galaxy (number 250). The same process was repeated for TV shows. 

Finally, we added up the scores for each service and standardized them on a 10-point scale. Here’s how the major streaming services stacked up for movies and TV shows:

*As of 01/17/20

Netflix came in first with 33 movies and 69 TV shows on the list, while Apple TV+ came in last place with zero on movies and TV shows in IMDb’s top 250. 

Original content

To evaluate each service’s original content, we focused on three metrics: critical reception, audience ratings and awards recognition. For each streaming service, we averaged user ratings from IMDb and Rotten Tomatoes for every original show and movie, provided each title had at least 100 ratings. 

To gauge critical reception, we used Rotten Tomatoes’ Tomatometer® score. The metric scores TV seasons and movies once at least five professional critics have weighed in. If an original title had enough critic reviews for a score, we included it.

Finally, we counted the number of Emmys and Academy Awards that each service has garnered since 2013, the first year a streaming service was recognized (Netflix was nominated for four Emmys for House of Cards that year and once for Arrested Development). These three scores were then averaged for an overall original content rating.

User experience

Like our user experience rating for live TV streaming services, we spent about five hours with each service (and often much more) to determine how user-friendly it is. We alternated between a Roku Streaming Stick, Apple TV, Amazon Fire TV Stick and MacBook Pro to grade how the services fared across devices. We looked for things like easy search and navigation, helpful recommendations and uninterrupted streams. 

Overall score

Unlike our overall score for live TV streaming services, we weighed each category equally for on-demand streaming services. 

How up-to-date are Allconnect’s on-demand streaming service scores?

Our TV and movie scores are updated every three months, while our original content ratings are typically updated every six months. That said, we also revise the original content scores when Emmy and Academy Award nominations are announced, and again after the awards ceremonies take place.

Our streaming reviews

To see our TV streaming metrics in action, check out all of our reviews below: 

For more of the latest streaming news, keep an eye on our Resource Center and follow our experts on Facebook and Twitter.

What to read next Read more

Latest