Model Interpretability - LIME

Neural network models are difficult to understand. When a network makes a prediction - why did it choose that specific prediction? Is the network biased in some way? Is it ignoring relevant information?

The ability to interpret the behaviour of models is important. Approaches like Random Forests or Decision Trees allow interpretation by looking at the features that they use for classification. A neural network takes inputs and combines them in complex ways. Looking directly at the structure of the network does not show why it is making a decision.

In a recent workshop the LIME tool (Local Interpretable Model-Agnostic Explanations) was demonstrated. This explores the way that a model classifies by changing the input and observing the changing output. The LIME tool can then present the impact of the different features on the prediction.

I’m interested in exploring how LIME works and what conclusions can be drawn from it. This post is an exploration of LIME.

The first thing to do is to get a dataset and a model and see if I can get LIME to make some predictions about it. LIME can work with:

• Text classifiers
• Models that take tabular data
• Image classifiers

Since the workshop covered tabular data already I’m inclined to use an image classifier. It’s quite easy to get a model that is pretrained for resnet.

Code

from fastai.vision.all import *

Code

model = models.resnet18(pretrained=True)
model.eval()

def open_image(path: Path, size: int = 224) -> Image.Image:
    image = Image.open(path)
    image = image.convert('RGB')
    image = image.resize((size, size))
    return image

def to_array(image: Image.Image) -> np.array:
    array = np.array(image, dtype="float")
    array = array / 255.0
    array = array - 0.5
    array = array * 2
    return array

def to_tensor(array: np.array) -> torch.Tensor:
    tensor = torch.Tensor(array)
    if len(array.shape) == 3:
        tensor = tensor[None, :]
    return tensor.permute(0, 3, 1, 2)

cat_image = open_image(Path("cat.jpg"))
cat_array = to_array(cat_image)
cat_image

Code

def predict(array: np.array) -> torch.Tensor:
    with torch.no_grad():
        tensor = to_tensor(array)
        return model(tensor).softmax(dim=-1).numpy()

np.flip(
    predict(cat_array)[0].argsort()
)[:5]

array([285, 284, 281, 478, 457])

I can decode this using the imagenet labels (see a json file here).

Code

labels = json.loads(Path("data/2021-03-07-model-interpretability/imagenet-simple-labels.json").read_text())

probabilities = predict(cat_array)[0]
for idx in np.flip(probabilities.argsort())[:5]:
    print(f"{labels[idx]: >15}: {probabilities[idx].item() * 100: >5.2f}%")

   Egyptian Mau: 73.68%
    Siamese cat:  6.75%
      tabby cat:  6.02%
         carton:  2.56%
        bow tie:  0.99%

An Egyptian Mau is a specific breed of cat, so I’ll take it. Lets try running LIME over this. I’m following the LIME basic image notebook.

Code

from lime import lime_image
from skimage.segmentation import mark_boundaries

explainer = lime_image.LimeImageExplainer()

Code

%%time

explanation = explainer.explain_instance(
    cat_array,
    predict,
    labels=labels,
    top_labels=5,
    hide_color=0,
    num_samples=1000
)

CPU times: user 1min 18s, sys: 3.17 s, total: 1min 21s
Wall time: 8.82 s

Code

masked_image, mask = explanation.get_image_and_mask(
    label=285, # Egyptian Mau
    positive_only=True,
    num_features=5,
    hide_rest=True
)
plt.imshow(mark_boundaries(masked_image / 2 + 0.5, mask)) ; None

Code

masked_image, mask = explanation.get_image_and_mask(
    label=457, # bow tie !
    positive_only=True,
    num_features=5,
    hide_rest=True
)
plt.imshow(mark_boundaries(masked_image / 2 + 0.5, mask)) ; None

We can also see what detracts from a specific label.

Code

masked_image, mask = explanation.get_image_and_mask(
    label=285,
    positive_only=False,
    num_features=5,
    hide_rest=False
)
plt.imshow(mark_boundaries(masked_image / 2 + 0.5, mask)) ; None

This doesn’t have anything that detracts from the prediction because it really just is a cat. Let’s find a more ambiguous image.

Code

cat_and_dog_image = open_image(Path("cat-and-dog.jpg"))
cat_and_dog_array = to_array(cat_and_dog_image)
cat_and_dog_image

Code

probabilities = predict(cat_and_dog_array)[0]
for idx in np.flip(probabilities.argsort())[:20]:
    print(f"{labels[idx]: >25}: {probabilities[idx].item() * 100: >5.2f}% ({idx})")

      German Shepherd Dog: 70.01% (235)
                 Malinois: 10.23% (225)
               Bloodhound:  2.36% (163)
 Chesapeake Bay Retriever:  1.97% (209)
               Leonberger:  1.63% (255)
                     lion:  1.55% (291)
                    dingo:  1.19% (273)
             Afghan Hound:  0.97% (160)
        Australian Kelpie:  0.81% (227)
               Great Dane:  0.61% (246)
                   cougar:  0.60% (286)
          Irish Wolfhound:  0.46% (170)
                tabby cat:  0.35% (281)
                tiger cat:  0.34% (282)
             Egyptian Mau:  0.33% (285)
         Golden Retriever:  0.28% (207)
       Norwegian Elkhound:  0.26% (174)
             Basset Hound:  0.26% (161)
                grey wolf:  0.24% (269)
         Alaskan Malamute:  0.21% (249)

Code

%%time

explanation = explainer.explain_instance(
    cat_and_dog_array,
    predict,
    labels=labels,
    top_labels=20,
    hide_color=0,
    num_samples=1000
)

CPU times: user 1min 17s, sys: 4.33 s, total: 1min 21s
Wall time: 8.65 s

Code

masked_image, mask = explanation.get_image_and_mask(
    label=235,
    positive_only=False,
    num_features=20,
    hide_rest=False
)
plt.imshow(mark_boundaries(masked_image / 2 + 0.5, mask)) ; None

It mainly seems pretty happy with the dog prediction. The muzzle of the cat is the detractor, and I guess that is the most distinctively non doggy.

---

Overall I think this evaluation went well. It was quite tricky to format the data in a way that LIME accepts. I think the problem there is the requirement that LIME understand the data enough to manipulate it effectively.