Assignment 2

[tutoringjedi]

What changes are you trying to make? (e.g. Adding or removing code, refactoring existing code, adding reports)

Completing Assignment 2: zero-shot image classification on Fashion-MNIST using a pre-trained CLIP model (openai/clip-vit-base-patch32). This includes:

• A baseline zero-shot classifier using bare class names as text prompts, with accuracy and a confusion matrix.
• An improved-prompt experiment using more descriptive, visually-detailed prompts to try to reduce confusion between visually similar classes (e.g. T-shirt/top vs. Shirt, Pullover vs. Coat).
• A UMAP visualization of CLIP's image embeddings across the test set, colored by true label.
• A mini-experiment implementing Top-K classification (checking whether the true class appears in the model's top-2/top-4 most-similar prompts, not just its top-1 guess) plus a short report on the results.

What did you learn from the changes you have made?

Zero-shot CLIP underperforms a CNN trained specifically on Fashion-MNIST, mainly due to the domain gap between low-resolution grayscale garment images and CLIP's natural-image pretraining. More descriptive prompts gave a modest improvement, especially for the upper-body garment classes that were most confused in the baseline. The UMAP plot showed those same classes overlapping heavily in CLIP's embedding space, which suggests the confusion is a genuine visual-representation issue rather than purely a prompt-wording problem. The top-K experiment showed that many of CLIP's "wrong" top-1 answers still had the correct class ranked close behind, meaning a lot of the error is near-misses between visually similar classes rather than the model being completely off-base.

Was there another approach you were thinking about making? If so, what approach(es) were you thinking of?

I considered fine-tuning a lightweight classifier on top of CLIP's frozen image embeddings (using a small number of labelled examples) as an alternative to pure prompt engineering, to see how much of the accuracy gap to a fully-trained CNN could be closed with minimal labelled data. I went with the prompt-engineering and top-K approaches instead since they were the ones outlined in the assignment, but this would be a natural follow-up.

Were there any challenges? If so, what issue(s) did you face? How did you overcome it?

Ran into an issue where clip_model.get_text_features() and get_image_features() returned a raw BaseModelOutputWithPooling object instead of a plain tensor in my installed transformers version, causing an AttributeError on .norm(). Resolved by calling the underlying text_model/vision_model submodules directly and manually applying the text_projection/visual_projection layers to their pooled outputs, replicating what the convenience methods are supposed to do internally.

How were these changes tested?

Ran the notebook top-to-bottom after each fix (restarting the kernel to avoid stale state) to confirm the baseline, improved-prompt, UMAP, and top-K sections all execute without errors and produce sensible output (accuracy scores, confusion matrices, and the UMAP scatter plot).

A reference to a related issue in your repository (if applicable)

N/A

Checklist

• I can confirm that my changes are working as intended

[github-actions]

Hello, thank you for your contribution. If you are a participant, please close this pull request and open it in your own forked repository instead of here. Please read the instructions on your onboarding Assignment Submission Guide more carefully. If you are not a participant, please give us up to 72 hours to review your PR. Alternatively, you can reach out to us directly to expedite the review process.