Table Exploration

June 2018

Feel free to add your tips and tricks, anything useful/semi-useful will have a home here while being periodically updated and improved by users.

Table of Strong Collections

​​data_corr = data.corr()
​​# Set the threshold to select only highly correlated attributes
​​threshold = 0.5
​​# List of pairs along with correlation above threshold
​​corr_list = []
​​#Search for the highly correlated pairs
​​for i in range(0,size): #for 'size' features
​​    for j in range(i+1,size): #avoid repetition
​​        if (data_corr.iloc[i,j] >= threshold and data_corr.iloc[i,j] < 1) or (data_corr.iloc[i,j] < 0 and data_corr.iloc[i,j] <= -threshold):
​​            corr_list.append([data_corr.iloc[i,j],i,j]) #store correlation and columns index
​​#Sort to show higher ones first
​​s_corr_list = sorted(corr_list,key=lambda x: -abs(x[0]))
​​#Print correlations and column names
​​for v,i,j in s_corr_list:
​​    print ("%s and %s = %.2f" % (cols[i],cols[j],v))

Search for Highly Correlated Pairs

​​#Search for the highly correlated pairs
​​for i in range(0,size): #for 'size' features
​​    for j in range(i+1,size): #avoid repetition
​​        if (data_corr.iloc[i,j] >= threshold and data_corr.iloc[i,j] < 1) or (data_corr.iloc[i,j] < 0 and data_corr.iloc[i,j] <= -threshold):
​​            corr_list.append([data_corr.iloc[i,j],i,j]) #store correlation and columns index
​​
​​#Sort to show higher ones first
​​s_corr_list = sorted(corr_list,key=lambda x: -abs(x[0]))

Correlation with Target

​​# Correlation amongst top feautres and target
​​corr = df.corr()
​​abb = corr["target"].sort_values(ascending=False)[::5].index.values.tolist()
​​corr = corr[corr.index.isin(abb)]
​​corr = corr[abb]
​​corr.ix[:,0].abs().sort_values(ascending=False)
​​corr.shape

Classification Report