Blog—Max Turgeon

Hidden powerful features of the tidyverse

Thu, 21 Sep 2023 00:00:00 +0000

Lately, I have had to write R code that would run from the command line (through Rscript) and whose expected behaviour was to summarise datasets that have standardized column names, but where the existing columns may differ from project to project. In developing this code, I ran into a couple of (new to me) features of the tidyverse, and specifically the packages tidyselect and dplyr. These are: tidyselect::any_of: variable selection by character vector that ignores missing variable.

PCA and MDS

Tue, 21 Sep 2021 00:00:00 +0000

Principal Component Analysis (PCA) and classical Multidimensional Scaling (MDS) are two closely related dimension reduction methods. In fact, under certain conditions, there are exactly the same. Their main difference is the type of input they take: PCA: A dataset where each row is an observation, and each column is a feature. MDS: A distance matrix. Let’s look at an example. We will apply PCA to the USArrests data that is included with base R.

Multivariate t distribution

Thu, 06 Feb 2020 00:00:00 +0000

I am currently teaching a graduate course in Multivariate Analysis (the course website can be found here). A few weeks ago, I introduced the family of elliptical distributions. In this blog post, I want to discuss the multivariate t distribution, how to generate samples, and highlight the issue of uncorrelatedness vs independence.

Elliptical distributions

If we generate samples from a multivariate normal, we can easily see that the contour lines are ellipses:

set.seed(7200)
library(mvtnorm)
n <- 10000
p <- 2
Sigma <- matrix(c(1, 0.5, 0.5, 1), ncol = p)
Y <- data.frame(rmvnorm(n, sigma = Sigma))
# Plot the data
library(ggplot2)
ggplot(Y, aes(X1, X2)) +
geom_point(alpha = 0.2) +
geom_density_2d() +
theme_minimal()

Elliptical distributions are a generalization of the multivariate normal distribution that retain this property that lines of constant density are ellipses.

Oscar 2019 predictions

Sat, 23 Feb 2019 00:00:00 +0000

Again this year, I will be using a prediction model to try and predict the winners in the top four categories: Best Picure, Best Director, Best Actor, and Best Actress. As in previous years, I still provide predictions for the other categories, but they were derived on a more ad hoc basis.

According to most pundits, the Best Picture race is wide open. But as you’ll see below, my prediction is less equivocal. To see this, it’s informative to look at how the winning probabilities have changed over time as more information was coming in:

As we can see, my model does not weight all guild awards the same way: a win at the Producers' Guild Awards (PGA) is worth a lot less than a win at the Directors' Guild Awards (DGA) or at the Bafta. Still, given the preferential ballot system used by the academy, my model is probably underestimating the chances of a movie with broad appeal like Green Book.

My predictions are below, in bold. After the Academy Awards tomorrow, I will update this post and point out the winners–I will indicate them in italics.

Update (2019/02/25): I got 16 out of 24 right. This year really convinced me that my model for Best Picture is missing something about the broad appeal movies like Spotlight, The Shape of Water, and Green Book, and so it may be time to update it… Stay tuned!

What I (currently) do at the Saskatchewan Health Authority

Fri, 28 Sep 2018 00:00:00 +0000

During the summer of 2016, my wife and I moved to Saskatoon so that she could pursue her medical residency training at the University of Saskatchewan. At the point, I was only wrapping up the third year of my PhD, and so I wasn’t necessary looking for a job. But one thing led to another, and I ended up applying for a new biostatistician position with the Saskatoon Health Region (SHR).

The job description talked about being a connection point between the health care system and the Saskatchewan Centre for Patient-Oriented Research (SCPOR), by providing both study design and methodological support for researchers. I thought I would be a good fit: McGill’s focus on data analysis and report writing provided me the tools to do the job; meanwhile I could learn more about the health care system itself and use my academic background to provide effective consultation to researchers.

Predictive model for the Oscars

Sun, 25 Feb 2018 00:00:00 +0000

A few years ago, as part of the graduate course Data Analysis and Report Writing in the Department of Epidemiology, Biostatistics and Occupational Health at McGill University, we explored the topic of predictive modeling using a dataset containing movies, directors and actors who were nominated for an Academy Award. The goal was to select some variables and build a predictive model for the winner in four categories: Best Picture, Best Director, Best Actor, and Best Actress. As a movie fan, this was the dream assignment: I could combine my love of movies with my love of statistics! And it payed off: I was the only one in my class to correctly predict all four winners.

Oscar 2018 predictions

Sat, 24 Feb 2018 00:00:00 +0000

For the past few years, I have tried to predict the winners in all categories at the Academy Awards. Again, I will be using statistics and data analysis to inform my decision in some categories: Best Picure, Best Director, Best Actor, Best Actress, Best Supporting Actor, and Best Supporting Actress.

As for the last three years, I stick to what the model tells me for my prediction in these categories. However, I’m skeptical about the predictions I have for best picture: several pundits see The Shape of Water as a front runner, but my model only gives it a 16% chance of winning. Due to rule changes that now require a preferential ballot for Best Picture, the winner has been difficult to predict in recent years. Since The Shape of Water possibly has a broader appeal than Lady Bird and Three Billboards Outside Ebbing, Missouri, it may prevail in the end. But I still believe Three Billboards Outside Ebbing, Missouri is the actual front-runner; but I do think my model is under-estimating The Shape of Water’s chances and over-estimating Lady Bird’s.

In the next few days, I will write another post in which I’ll describe how my model works. I’ll take the opportunity to try and explain why my model is so bearish for The Shape of Water.

My predictions are below, in bold. After the Academy Awards next weekend, I will update this post and point out the winners–I will indicate them in italics.

First steps with Leaflet

Wed, 05 Jul 2017 00:00:00 +0000

I should probably be working on my thesis, but instead I started reading through the introduction to the R package leaflet. And the following made me feel excited: Code for America has GeoJSON data on their Github page for several cities in the world. In particular, they have both Saskatoon and Montreal! I used the leaflet package to draw the boundaries of each neighbourhoods. You can see the results here. (I wasn’t able to host the maps directly on this webpage.

Installing multiple R versions

Sat, 01 Apr 2017 00:00:00 +0000

Sahir Bhatnagar and I are currently wrapping up the first version of our package casebase. In short, it’s an R package for survival analysis, where we use case-base sampling to fit smooth-in-time hazards. (I could write a post on this package, but there’s no need: check out the website and the four vignettes.) As part of our workflow, we perform continuous integration using Travis CI, and we test our package against both the current and development versions of R. Recently, some tests began to fail against the development version, and so I had to install R-devel on my local machine in order to debug our code. This blog post is a summary of how I did it.

To be fair, this is already documented online, and I made use of these resources; see the official R installation docs and this RStudio support post. I’m writing yet another post simply as a reference for myself and my colleagues. But I also ran into a compilation error that I wanted to document here. That error was “caused” by following closely the (amazing) book R packages by Hadley Wickham. Stick around to learn what the problem was!

Oscar 2017 predictions

Sun, 12 Feb 2017 00:00:00 +0000

For the past few years, I have tried to predict the winners in all categories at the Academy Awards. And for the past two years, I’ve also used statistics and data analysis to inform my decision in four categories: Best Picure, Best Director, Best Actor, and Best Actress.

As for the last two years, I stick to what the model tells me for my prediction in these categories. However, I’m skeptical about the predictions I have for the acting categories. First, for best actor, Denzel Washington and Casey Affleck are the only two front-runners–there is no way Ryan Gosling will win this award. Second, for best actress, although I would argue that Emma Stone is now the favourite, Isabelle Huppert definitely has a chance too. Therefore, I expect to miss one of these categories.

Finally, this year is definitely La La Land’s show: 14 nominations, tied with Titanic and All About Eve for the most ever. I don’t think it will set a new record (which is 11 wins), but I expect them to win around 10 awards. I don’t expect them to run the tables in the sound/music categories–which would be a first for a musical. I think the only one of these four awards they could miss is Best Sound Editing, which I predict will probably go to Arrival.

My predictions are below, in bold. After the Academy Awards, I will update this post and point out the winners–I will indicate them in italics.

Update (2017/02/27): Wow! What a finale! As for my predictions, I did better than last year: 16/24.

US Presidential Inaugural Addresses

Fri, 20 Jan 2017 00:00:00 +0000

Earlier this week, on January 20th 2017, Donald J. Trump was inaugurated as the 45th president of the USA. He also gave what seemed like a very short inaugural address, and so I was curious to see how short it really was compared to previous addresses. It was also an opportunity to have a quick look at other properties of his speech.

The Instability of Forward and Backward Selection

Sun, 29 May 2016 00:00:00 +0000

Classical statistics often assumes that the analyst knows which variables are important and which variables are not. Of course, this is a strong assumption, and therefore many variable selection procedures have been developed to address this problem. In this blog post, I want to focus on two subset selection methods, and I want to address their instability. In other words, I want to discuss how small changes in the data can lead to completely different solutions.

Removing all R CMD check warnings

Fri, 08 Apr 2016 00:00:00 +0000

Making R packages is an important aspect of the statistician’s work. Or at least it should be: it is quite annoying when a new method appears in the literature but no implementation is readily available.

A favourite mantra of mine when making R packages is the following: an R package is more than the sum of its functions. A functioning R package needs to be able to interact properly with the R environment (through the NAMESPACE); a good R package also needs great documentation; a great R package will also include a vignette to guide new users and explain how all the functions interact with one another.

The main reference for how to make R packages is Writing R extensions. Everything you need to know is there, if you know what you are looking for. Another, very useful reference is Hadley Wickam’s book on R packages. This book explains the different components of an R package, and it also serves as an introduction to his devtools package.

In what follows, I don’t want to go over how to make an R package; the above references do a better job than I could hope to do. Rather, I want to share my experience about some of the most annoying part of making an R package: passing the R CMD check. Removing the errors is the most important part, and what kind of errors you get really depends on the package (the log file is typically quite useful in figuring out what triggered the errors). On the other hand, you also want to minimize the number of warnings and notes, and most warnings you probably want to remove altogether.

By how much will Clinton win?

Tue, 08 Mar 2016 00:00:00 +0000

American politics is great for statistics: there are huge amounts of polls being conducted every week, some positions are up for re-election every other year, and there is really only two parties. Moreover, the complicated nature of the whole election process, which for example involves the electoral college for the presidential election, makes it more interesting than most democracies around the world. It’s for all these reasons that an incredible website like FiveThirtyEight is possible.

Oscar 2016 predictions

Sun, 07 Feb 2016 00:00:00 +0000

For the past three years, I have tried to predict the winners in all categories at the Academy Awards. But last year, I was able to combine my passion for both movies and statistics: as part of my Data Analysis course at McGill University, we had to come up with a prediction model for four categories: Best Picture, Best Director, Best Actor, and Best Actress. And my model performed quite well: it was the only one to predict correctly the four winners.

This year, I decided to repeat the experience again, especially since the Best Picture category is more competitive this year than last year. I have shared my predictions below, for all categories; however, I have used a statistical model only for the four categories mentionned above. All other categories are based on my own judgement (and readings I have done). My predictions are in bold font.

After the Academy Awards, I will update this post and point out the winners (I will indicate them in italics). I may also write a post on my prediction model.

Update (2016/02/28): Well, I didn’t do as well as I would have liked: 14/24.

Makefile and Beamer presentations

Mon, 07 Dec 2015 00:00:00 +0000

I have been wondering about Makefiles for some time now, and recently I finally got around learning about them so that I could use make to regenerate all the different versions of a manuscript I’m working on. And I thought I would take the opportunity to explain how they can be useful for Beamer presentations.

Test case: Optimising PCEV

Fri, 11 Sep 2015 00:00:00 +0000

I will give an example of code optimisation in R, using Noam Ross’s proftable function and Luke Tierney’s proftools package, which I discuss in my tutorial on optimisation. The code we will optimise comes from the main function of our PCEV package. A few months ago, while testing the method using simulations, I had to speed up my code because it was way to slow, and the result of this optimisation is given below.

For background, recall that PCEV is a dimension-reduction technique, akin to PCA, but where the components are obtained by maximising the proportion of variance explained by a set of covariates. For more information, see this blog post.

Tutorial: Optimising R code

Thu, 10 Sep 2015 00:00:00 +0000

The R language is very good for statistical computations, due to its strong functional capabilities, its open source philosophy, and its extended package ecosystem. However, it can also be quite slow, because of some design choices (e.g. lazy evaluation and extreme dynamic typing).

This tutorial is mainly based on Hadley Wickam’s book Advanced R.

Before optimising…

First of all, before we start optimising our R code, we need to ask ourselves a few questions:

Is my code doing what I want it to do?
Do I really need to make my code faster?
Is considerable speed up even possible?

Using MathJax

Fri, 14 Aug 2015 00:00:00 +0000

I had some trouble rendering correctly the mathematical equations in my previous post: at first I could only see the untransformed markup, then the text simply disappeared, without being transformed, and finally the equations appeared, but coloured orange. As you can see, everything now looks fine, but to fix this I ended up learning a bit more about how markdown, HTML, CSS and javacript all work together to create the website you are currently visiting. The purpose of this post is to share some of what I learned, so that future visitors can be spared some of the pain that accompanied the learning.

Principal components of explained variance

Thu, 06 Aug 2015 00:00:00 +0000

I have been spending most of my time on a very interesting technique that has, unfortunately, received little attention (I’ll come back later about some possible reasons). The purpose of this post is to introduce this method and give motivation for its use.

Hello World!

Sun, 12 Jul 2015 00:00:00 +0000

I am finally launching my personal website! It will mainly be used to share my projects and my thoughts about statistics. I will also share the slides from my talks and the occasional tutorial, whenever I feel I’ve learned some interesting stuff in my work (be it statistical or related to programming). In the meantime, I encourage you to take a look at my Github page, where you can see some of the projects I’ve contributed to.