The major computer software company Adobe is seeking a patent for an AI image scanner that will enable “diversity auditing.”
The proposed system would automatically audit images’ diversity and issue a scoring. The patent defined “diversity auditing” as a type of data auditing that assesses whether a data set contains certain levels of diversity in race, gender, and age using physical traits referred to as “sensitive attributes.”
“By using machine learning to compute a diversity score based on a distribution of a sensitive attribute (such as race, age, or gender) in a set of images, where the distribution is based on the automatic classification of faces in the images based on the sensitive attribute, at least one embodiment of the present disclosure can automatically audit a set of images for a predetermined level of diversity of the sensitive attribute, thereby allowing a user to avoid manually identifying and tagging information related to the sensitive attribute in each image in the image set, and without requiring manual curation of a control set and labeling the images in the image set for calculating a diversity metric.”
Adobe filed the patent request, “System and Methods for Diversity Auditing” (Patent No. 20230267764), last February, with the application publication issued last week. (Search for the patent here).
Leading up to the patent filing, the inventors published a research paper stating their desire to manipulate online media in pursuit of a “utopia” of representative diversity.
The inventors — Mehrab Tanjim, Ritwik Sinha, Moumita Sinha, David Thomas Arbour, and Sridhar Mahadevan — published the paper, “Generating and Controlling Diversity in Image Search,” declaring that “generations of systemic biases” resulted in racial and gender demographic patterns existing in certain professions. The researchers also stated that stock image and image search engines reflect that bias, and that AI-generated images would be necessary because algorithm tweaking wouldn’t go far enough to correct the issue. (Archive here).
“The pursuit of a utopian world demands providing content users with an opportunity to present any profession with diverse racial and gender characteristics,” stated the report. “To remedy these problems, we propose a new task of high-fidelity image generation by controlling multiple attributes from imbalanced datasets.”
The paper is provided by the Computer Vision Foundation, platformed by Microsoft Azure and sponsored by Amazon, Facebook, and Google. Arbour was formerly a research scientist for Facebook’s Core Data Science group.
CLICK HERE TO GET THE DAILY WIRE APP
In an apparent extension of this research interest, the Adobe patent proposes an AI system that can not only audit the diversity of images displayed on a website or database, but tweak the content given to a user searching for certain images on a database with a “representative set” of images in the return results. The latter will occur if the AI system determines that the diversity score for the image set is too low. At that point, the system may inform the user of the diversity score of the images as well.
“[T]he system augments the set of images to increase diversity in the set of images,” states the patent. “For example, the machine learning apparatus may compute a diversity score […] and determine that the set of images is below a predetermined threshold of diversity in the number of different types of the sensitive attribute that are depicted by the set of images.”
The patent noted that the system may retrieve data to determine appropriate diversity thresholds from census data or websites.
The AI system would consist of a machine learning model that may include one or more artificial neural networks (ANNs), a type of technology inspired by the structure and function of human brain neurons. The machine learning model would consist of an image collection component, which leads to a face detection network, which leads to an image classification network, which leads to a distribution component, which leads to a scoring component, and results in a generator network.
The image collection component identifies and collects both specific images and websites, as well as those identified and collected to achieve a higher diversity scoring. The face detection network includes a convolutional neural network (CNN), a deep learning algorithm that takes an input image and assigns meaning to learn over time. The image classification network generates an image feature vector — a translation characterizing and numerically quantifying images — with special attention to sensitive attributes that lend to a diversity score.
The distribution component generates a distribution of images’ sensitive attributes based on their classification, which then directly lends to the diversity score computation function of the scoring component. Then, the generator network may generate additional images based on the diversity score. Some of these images may be AI-generated creations “that look authentic to a human observer” using a generative adversarial network (GAN).
The Daily Upside, which first reported on the patent application, suggested that the system could also be used to conduct diversity audits of a company’s employees based on images.
Adobe last reported ownership of over 200 million photos, 115 million vectors and illustrations, 26 million videos, and 73,000 music tracks. Their Creative Cloud reached nearly 30 million subscribers last year, per their 2022 fiscal report.