Revisting ROC curves for biased datasets

In a recent discussion with a lab mate, I raised question as to whether or not ROC curves are sensitive to imbalances in the number of positive and negative samples.

This question comes in the the context of protein science where it is common for a scientist to collect a small data set of observed functional proteins, and then later develop a model from this data set to predict other functional sequences. This objective boils down to a classification task where the training data set is a small sample of true functional proteins without true negatives. The naive assumption is to assume that all other proteins are non-functional, however this hypothesized negative space is often times magnitudes larger than the number of true functional proteins.

Typically ROC (Receiving Operator Curves) are used to evaluate classifier performance. To generate a ROC curve, we start off with a test set of positive and negative samples. Each test sample is evaluated with the model, which maps it a numerical value. These values are sorted by the predicted value, and then varying thresholds are evaluated to generate the ROC curve. At each threshold, a False Positive Rate and True Positive Rate is calculated and plotted. The area under this curve is known as the AUC, and it provides a simplified number that describes the model performance.

True Positive Rate = TP / (TP + FN) = Percent positive correctly classified as positive
False Positive Rate = FP / (FP + TN) = Percent of negative samples incorrectly classified as positive

(A model with a threshold capable of separating the two populations has 100% TPR and 0% FPR. )

In the extremely imbalanced case as described above, we have many more hypothesized negatives than positive samples. The question I would like to address is whether or not varying the number of negatives change the AUC?  - Does it matter if we have 1000 negatives or 10,000 negatives

In short, it depends on how you vary the negative population.

First, it should be noted that FPR and TPR are independent of each other - The TPR only depends on how the positives were classified and FPR only depends on how the negatives were classified. Given that they are independent, I will only discuss what happens when we vary the population of negatives.

AUC is insensitive to the number of negatives if the FPR for given thresholds remain unchanged for the small and large populations. If the small and large negative population comes from the same distribution, the FPR should not change and thus the AUC should not change.

However, if the small and large populations vary the FPR, then AUC will change. If you add a large number negatives which are easy to classify as negative, then your AUC will be inflated. This large number of easy to classify negatives increases the TN, thus pushing FPR closer to zero.

It makes more sense once we see an example.
In the test case, we observe 1000 positives. This is unchanged. We want to see whether or not it makes a difference if we see 1000 negatives or 11000 negatives. In the notebook, we see see that when we have 10,000 more negatives in the same population, AUC is insensitive. However when we add 10,000 negatives which are very easy for the model to classify, then the model AUC is inflated.

http://nbviewer.jupyter.org/gist/xuevin/edc323adfe9de2ed4c2d34d39f906c7b


By applying AUC analysis to imbalanced data is that we can inflate the AUC score by adding easily classified negative sequences. If we vary the the number of negatives to disrupt the FPR, then AUC scores can be inflated. That is, if we add many easy negative samples, the FPR is pushed closer to zero, and AUC is closer to 1.

Coming full circle back to the protein space, it is clear that the selection of the negative samples can influence the reported AUC. If millions of "easy" negatives are added in the evaluation of the test set, then we should be wary of the reported AUC.

Focus on Data Science

This blog will reflect a few of the topics that I am interested in that I would like to learn.  The best way of learning something is through teaching it so I am excited to begin this journey.

Find and copy

A quick way to find and list all the presentations and copy them to another directory.

find /home/user/ -type f -name "*.pptx" -exec cp -s --parents \{\} /home/user2/Presentations/ ';'

Splitting PDB by delimiter

csplit  --prefix=chain --suffix="%1d".pdb 3FDL_Repair.pdb "/^TER/+1" "{*}"

Renaming Files

#!/bin/bash
for file in foo[0-9]*.png; do
  # strip the prefix ("foo") off the file name
  postfile=${file#foo}
  # strip the postfix (".png") off the file name
  number=${postfile%.png}
  # subtract 1 from the resulting number
  i=$((number-1))
  # copy to a new name in a new folder
  cp ${file} ../newframes/$(printf foo%08d.png $i)
done

http://stackoverflow.com/questions/55754/how-to-zero-pad-numbers-in-file-names-wit-a-bash-script

Sum up columns

awk '{print $1}' * | paste -sd+ | bc

String manipulation with bash

String manipulation with bash

for each in *.txt; do echo ${each/.txt/.pdb};done

${string:position}

paste -d '\n' file1 file2




Replace the 4th character with another letter.

echo "abcdefghijklmnop" | sed 's/./X/4'

Using Rpy2 with Anaconda and Ipython notebook

Typically in ipython notebook, the only command required to print a plot inline is the following:

=====================
%load_ext rmagic
=====================

=====================
%%R 
x<-c(1,2,3)
y<-c(2,3,4)
plot(x,y)

=====================

While working on AWS, my plots would not show up inline. I found out that this is because rpy2 only works for python 2.7.5. 

The easy fix is 

conda install python=2.7.5=2

Parameters for Negative Binomial

The negative binomial is a useful distribution for modeling the probability that a successful event occurs after r failures in q trials.

To put this into context:

Let's say that Joe is selling candy bars in his local neighborhood. He must sell 5 candy bars to meet his quota. The chance that a household will buy his candy is 0.4. What is the probability that he sells his 5th candy bar to the 10th household.

The parameters extracted from this problem are as follows:

\[r=5=\textrm{number of failures }
\\k=5=\textrm{number of successes (aka size)}
\\p=0.4=\textrm{probability of success}
\\k+r=10=q=\textrm{total number of trials}\]


The PDF for the negative binomial follows as
\[{{r+k-1} \choose k-1} (p)^k (1-p)^r\]

The answer that follows is 0.100329, which means that Joe has about a 10% chance of selling his 5th candy bar at the 10th house. Or rephrased, it also means that Joe has 10% chance of failing 5 times and succeeding 5 times at the 10th house. 

In R, you can calculate the CDF of the first 10 failures with the R code below.

Note that by the 10th house, we will have 5 failures and 5 successes.

pnbinom(1:10,5,0.4)

 [1] 0.0409600 0.0962560 0.1736704 0.2665677 0.3668967 0.4672258 0.5618218

 [8] 0.6469582 0.7207430 0.7827223

and the PDF

dnbinom(1:10,5,0.4)

 [1] 0.03072000 0.05529600 0.07741440 0.09289728 0.10032906 0.10032906

 [7] 0.09459597 0.08513638 0.07378486 0.06197928

Parameters for R

dnbinom(q, size, prob)
q - the number of failures
size - the number of successes
prob - the probability of success

Another quick example:

Joe decides it's time for a new job. Instead of selling candy bars he now sells notebooks. If he sells three notebooks, then he will meet his quota. However, the probability that he will sell a notebook is now 0.09. What is the probability that he sells his third notebook at the 10th house.

dnbinom(7,3,0.09)
0.013561876

Some more useful commands

takes the name of the file.
basename
or
cut -f1 -d '.' file.pdb

print the nth column

awk '{print $0}'

remove all dashes 

sed "s/-//g"

remove new lines

awk '{printf "%s", $0}' 

skips first line

sed 1d <File>

Uses the second file as a search.

cat temp.txt | grep -f temp2.txt 

Moves all items in subdirectories to another folder.

find -mindepth 2 -type f -print -exec mv {} ./newDir/ \;

Print everything except the first column

awk '{first = $1; $1 = ""; print $0, first; }' 

Prints out one line after the given search item

grep -A1 key $file

Remove all lines which don't begin with Y....[WC]

sed '/^Y.....[WC]/!d' AnnotatedOrfs.tab > CodingOrfs.tab

Delete All Parentheses

tr -d "\"" 

Take a look only at characters 17-20

cut -c 17-20 

If the 17th position has a B, then delete the line

cat file.txt | sed -r '/^(.{16})B(.*)$/d'

If the 17th position has an A replace it with a space

cat file.txt | sed -r 's/^(.{16})A(.*)$/\1 \2/'

Moving Directories which have a file of interest.

This command looks for files that have a ".ape" extension. It then takes the parent directory and moves it (the directory) to another location.

find . -name "*.ape" -type f -exec dirname {} \; | uniq | while read each; do mv "$each" dir2/; done;

How to mark all of your mail as read.

In Gmail, Create a filter like this

Matches: to:(*)
Do this: Skip Inbox, Mark as read

A one liner to rename files sequentially

This one line script will rename image files into nice sequential files such as

IMG_000.JPG
IMG_001.JPG
etc.

EII=0; for each in `ls -rt`; do mv $each IMG.`printf "%03d" $EII`.JPG; EII=`expr $EII + 1`; done;

Screencasting in Linux

I discovered a very useful command to record a desktop window using FFMPEG.
Credits to egrounds.org

Using the video captured from FFMPEG, and audio captured from an external Zoom H1 digital recorder,
I can combine these two recordings into a decent screencast.



#!/bin/sh

INFO=$(xwininfo -frame)
WIN_GEO=$(echo $INFO | grep -oEe 'geometry [0-9]+x[0-9]+' | grep -oEe '[0-9]+x[0-9]+')
WIN_XY=$(echo $INFO | grep -oEe 'Corners:\s+\+[0-9]+\+[0-9]+' | grep -oEe '[0-9]+\+[0-9]+' | sed -e 's/+/,/' )

echo $WIN_GEO
echo $WIN_XY

ffmpeg -f x11grab -r 15 -s $WIN_GEO -i :0.0+$WIN_XY -vcodec mpeg4 -sameq -y /tmp/$1.avi

Remove Spaces In Files

This command is very useful.

Credits to : http://www.swflug.org/index.php?option=com_content&task=view&id=71&Itemid=47

for file in *; do mv "$file" `echo $file | sed -e 's/  */_/g' -e 's/_-_/-/g'`; done 

How to record Games in Linux with RecordMyDesktop and PulseAudio

This tutorial is for those of you who have been struggling to record games on Linux. On the surface, this task may sound trivial, but getting started will reveal that it isn't so easy. The current online instructions that I have found are complicated and unnecessary. In my exploration, I spent about 3 hours playing around with JACK and other resources with no luck. JACK doesn't play very well with PulseAudio which comes on the more recent Ubuntu distributions. Working around it can be quite nightmarish so here I offer you an alternative.

TASK: Record Desktop Games With Game Audio.

INSTALLATION:

The applications you will need are gtk-recordmydesktop and PulseAudioDeviceChooser. In the terminal you can install with the following line:

sudo apt-get install gtk-recordmydesktop padevchooser

or with the more friendly Ubuntu Software Center.

INSTRUCTIONS:

Once these two applications are installed, open the PulseAudioVolumeControl

Applications > Sound&Video > PulseAudio Volume Control

In this window, go to the Input Devices tab and show All Input Devices

Scroll down to Monitor of Internal Audio Analog Stereo and click the checkbox. This will change your default input audio what you normally hear as output. I lowered the volume a little for quality, but you may have different preferences.

(When you are finished, restore the default by clicking on Internal Audio Analog Stereo)

Now you can open gtk-RecordMyDesktop

Applications > Sound&Video > Desktop Recorder

I think that using gtk-RecordMyDesktop is pretty straight forward. Clicking on Advanced I modified the Performance as shown. Nothing special on the Sound tab. 

All that's left is to click Save As  to name you file, Select your Window, and then hit record!

That's how you record audio and video together.

An idea for the future...

I want to work on developing a webapp music player that integrates MusicBrainz services. The webapp should play local files with MusicBrainz meta-data.

List files in FTP and Download/Unzip it

curl ftp://ftp.aoml.noaa.gov/phod/pub/IES_FTP/abaco/ > temp.txt


for each in `cat temp.txt | awk '{print $9}'`; do wget ftp://ftp.aoml.noaa.gov/phod/pub/IES_FTP/abaco/$each && unzip $each; done;

Exporting path...

I always seem to forget this line...

I currently have this in .bashrc, but it may not be the best place to put it.

export PATH=$PATH:/new/path/to/add:/other/new/path