Controlled Experiments: Crash Course Statistics #9

Controlled Experiments: Crash Course Statistics #9



hi I'm Adriene Hill and welcome back to crash course statistics famous tech guy and likely future space dweller Elon Musk once told interviewers that there's a high probability that we're all living in a simulation now that might sound outlandish and it's an interesting statement about probability but today we're going to focus on the simulation part Elon Musk as well as some philosophers futurists and technologists argue that it's entirely possible probable in fact that in a few thousand years we will be able to create simulations like World of Warcraft or minecraft that are so real they'll be indistinguishable from real life living in a world like that there's no way for you to tell if you're living in the true world or a simulation or maybe a simulation inside a simulation if we were living in Musk's vision of a future world any time we wanted to test something like whether that bus would have splashed us if we didn't stop to answer a text or whether the new drug we invented can cure lung cancer all we would need to do is run two identical simulations in one simulation we'd give people with lung cancer the new drug and watch to see what happens taking notes on things like lung capacity and remission rates well that's happening we do exactly the same thing in another simulation except we'd give patients a placebo drug something that looks and feels like a real drug but isn't and we've record our results once both simulations are done we'd be able to look at our data and have a good idea of whether it works now we can't yet run a parallel universe just to satisfy our curiosity about whether the amount of time we spend sleeping next to our cellphone increases our chance of brain cancer but we are doing the next best thing researchers have started using simulations to study cancer treatments to predict the impacts of climate change researchers are using vr to simulate disaster situations and to help train people how to respond in earthquakes or floods thanks thought-bubble still we need other ways to answer the burning questions that keep us up at night and until scientists can pull through with the necessary technology we're limited to two main methods of collecting data to answer our questions one way to get around our inability to create and destroy universes at will is by using experiments experiments try to mimic parallel universes by taking the one universe we do have and splitting it randomly into groups imagine High School where test scores are low and we want to see whether buying cappuccino machines for students families will improve test scores we don't have two parallel versions of the high school to test our idea but we can randomly split it into two groups one groups families will get the free cappuccino machine and the others won't and then we let life go on just like in the simulations and we record the test scores of both groups because we randomly assigned each persons group every single family had an equal chance of being in the cappuccino or in the no cappuccino groups randomness allows us to claim that before the cappuccino makers were given out there were no systematic differences between the groups usually researchers will use a random number generator to assign subjects into random groups random assignment reduces the chance that the bias of the people running the experiment will affect the group's for example it prevents them from doing things like putting subjects who they think will respond to caffeine in the cappuccino group and those who don't in the no cappuccino group it also makes it impossible for people to choose their own groups that way the coffee lovers don't all sign up for the free cappuccino machine while the tea enthusiasts don't these two problems are called allocation bias and selection bias respectively because of randomness it's unlikely though not impossible that all the wealthy people ended up in one group and then not so wealthy and the other or the vegans in one group and the omnivores any other randomness is usually our best method for ensuring that our groups are similar before we give them any treatment but the slight possibility that our groups might be really uneven is why it's important to replicate experiments in some situations researchers can also force the number of wealthy people or the number of vegans to be the same in each group this is something called randomized block design but randomness usually does a pretty good job in our example a treatment was either getting a cappuccino machine or not getting a cappuccino machine in general treatments are conditions that we want to test like new medicines or educational interventions like reading extra books to your kids treatments can also have levels for example when we look at whether exercise is related to weight loss we might have three groups one group does no exercise one group does five hours of exercise a week and the third group does 10 hours of exercise a week these levels can help us see whether there's a linear relationship between our treatment and our outcome or whether five hours of exercise is just as good as 10 one of the treatments in an experiment is usually nothing these treatments are called controls and they play a huge role in experiments it's our way of pretending that we have a universe where there's no treatment let's say you got a little too eager to get your freshly baked chocolate chip cookies out of the oven and you didn't put the mitt on right and you burned your finger you put some Neosporin on it and after a few days your finger heals you're so happy that you practically turn into neosporin next spokesperson but it's possible that the burn would have healed just fine on its own a control treatment aka no treatment would let us compare what happens between two similar burns when treated with an antibiotic cream and one left to heal by itself these types of control groups are great at helping us divide up the changes we observe into changes due to treatments and changes that are just due to time or circumstance if your finger heals faster with ointment then without it we can more confidently claim that there's a relationship between neosporin use and burn healing than if we didn't have a control sometimes we want to control for more than just time and circumstance with our control treatments with medical trials you as the person taking the medicine would want new medicines to work better than nothing but you also want to make sure that it's not just the act of doing something that's making you feel better it's been shown that just the act of taking a fake medication or having a fake surgery seriously can make people feel better these are called placebo effects placebos allow us to control these effects by pretending to treat everyone subjects and medical studies are often given sugar pills or saline drips to make it seem like both groups are being treated non medical studies also use placebos studies that claim first-person shooter video games like Call of Duty improve cognition should probably have a control group that plays a non first-person shooter like Mario Kart then the researchers can ensure that it's not just the act of participating in research or learning a new video game console that's causing the observed changes essentially good placebos and controls should look and feel as close as possible to the actual treatment so that the only difference is whatever we're actually interested in like first-person shooter video games sometimes there are undetected factors causing changes in an experiment we just don't know about well chosen placebos and controls allow us to better account for them sometimes it's just not possible to shield subjects from knowing the difference between conditions take diets for when people agree to be put into a clinical study which compares a group of subjects who do not change their diet to a group that goes low-carb it's hard to make both groups think they're doing the same treatment people can see what they're eating when subjects don't know which treatment they're receiving usually because they're getting a good placebo they're considered blind to the treatment an experiment where subjects don't know what treatment they're getting but the researchers do is called a single blind study which leads to better experiments because all groups experience the act of being treated but even in a single blind study it's possible that researchers might be biased when observing the subjects the researchers still know which treatment the subject is getting a researcher who spent years creating funding and planning a study probably thinks their treatment works I mean why would you dedicate your life to developing a drug to cure cancer if you don't think it works the problem is that belief in the value of the treatment can cause researchers to subconsciously project those beliefs onto their subjects in the 1990s a group of researchers looked at almost two dozen studies about whether sugar causes hyperactivity in children and they concluded that in general it doesn't parents who thought their children were given sugar reported them as being hyperactive but it turned out they were probably just being kids when parents are blinded to whether their children receive sugar both groups of parents reported roughly equal levels of hyperactivity even when we think our subconscious biases don't get in the way they still sometimes do we try to solve this problem by having double-blind studies in which both the subjects and the researchers have no idea which treatment the subject is getting just like with blinding patients sometimes it's impossible for the researcher to not be able to tell the difference but double-blind studies are the gold standard in many fields whenever they're physically and financially doable and while we still can't bend space and time to make parallel universes there are a few other ways to pretend we do have them one is matched pairs experiments just like the name implies these experiments use pairs of subjects that are very similar and give one of them treatment a and the other treatment be identical twins for example are about as close as we can get to a parallel universe since twins are genetically similar and often grow up in the same situation were able to get our treatment groups to be almost exactly the same or at least way closer than random assignment alone the more similar the groups are before treatment the better researchers can spot treatment effects we can also have a matched pairs experiment with non twins in this case each pair is matched on one or more features that the researchers decide are important like age race gender or weight then these pairs are treated like twins one is assigned treatment a and the other treatment B for simplicity here we're assuming there are two groups there could be more but the best subject researchers can pair you with is yourself many experiments will give the same subject multiple treatments one at a time to see how they react to each this ensures that each treatment group is the same since it's all the same people this type of matched pairs design is often called a repeated measures design and while it comes with its own set of limitations it can be better than regular random assignment and sometimes the stars align and we get to see the results of the interesting experiments that we don't have the power to implement ourselves recently the city of Philadelphia passed a sugar tax that would charge companies an extra one and a half cents per ounce of sugary beverage sold researchers can't just assign cities to have extra taxes when cities do vote these things into law some very interesting things can happen and they did in the Philadelphia International Airport because the airport straddles the city's border some of those otherwise identical stores had to comply with the law while others didn't through this data researchers were able to find that contrary to previous assurances chain stores in tax terminals raised their prices about point eight three cents per ounce more than their non tax terminal counterparts which means the tax did what it was supposed to do increase the cost of sugary drinks and encourage people to buy less we all consume products that are the result of experimentation from Amazon's web design to the prescription drugs we take which is why it's useful to know why researchers feel like they can make the decisions and claims that they do knowing the theory behind experimentation also allows us to be more informed consumers so the next time you find yourself staring down the homeopathic medicine section wondering if that effervescent tab developed by in instructor of students will really stop your cold ask how they tested it ask who tested it dftba cue you know I'm not sure why this isn't taking off I mean where's my t-shirt anybody crash course statistics is filmed in the chad and stacey emigholz studio in indianapolis indiana and it's made with the help of all these nice people our animation team is thought cafe if you'd like to keep crash course free for everyone forever you can support the series at patreon a crowdfunding platform that allows you to support the content you love thank you to all our patrons for your continued support crash course is a production of complexly if you like content designed to get you thinking check out some of our other channels at complexly calm thanks for watching