IEEE VIS 2024 Content: ImageSI: Semantic Interaction for Deep Learning Image Projections

ImageSI: Semantic Interaction for Deep Learning Image Projections



 Jiayue Lin - Vriginia Tech, Blacksburg, United States

Rebecca Faust - Tulane University, New Orleans, United States

 Chris North - Virginia Tech, Blacksburg, United States

 Screen-reader Accessible PDF

 Download Supplemental Material

 Room: Bayshore VI

2024-10-17T17:45:00ZGMT-0600Change your timezone on the schedule page
2024-10-17T17:45:00Z

Exemplar figure, described by caption below — An example using a collection of images of sharks and snakes. We want the dimension reduction (DR) to organize images based on the feature "open mouth" vs "closed mouth". (A) shows the initial projection, with added contours to highlight the locations of images with open mouths (yellow) and closed mouths (blue). The DR is not able to identify the open vs closed mouth feature. (B) illustrates the user’s interaction to convey this feature. (C) shows the DR after using ImageSI to update the embeddings. The DR now captures this feature much better than in it did with the original embeddings.

Fast forward

Keywords

Semantic Interaction, Dimension Reduction

Abstract

Semantic interaction (SI) in Dimension Reduction (DR) of images allows users to incorporate feedback through direct manipulation of the 2D positions of images. Through interaction, users specify a set of pairwise relationships that the DR should aim to capture. Existing methods for images incorporate feedback into the DR through feature weights on abstract embedding features. However, if the original embedding features do not suitably capture the users’ task then the DR cannot either. We propose ImageSI, an SI method for image DR that incorporates user feedback directly into the image model to update the underlying embeddings, rather than weighting them. In doing so, ImageSI ensures that the embeddings suitably capture the features necessary for the task so that the DR can subsequently organize images using those features. We present two variations of ImageSI using different loss functions - ImageSI_MDS−1 , which prioritizes the explicit pairwise relationships from the interaction and ImageSI_Triplet, which prioritizes clustering, using the interaction to define groups of images. Finally, we present a usage scenario and a simulation-based evaluation to demonstrate the utility of ImageSI and compare it to current methods.