Seeing artworks requires the perception of objects and compositional or formal characteristics of an image. The scene is thus viewed in its entirety or decomposed in segments. Eyes are the main instrument of art historians, who study and analyze art works in a comparative manner. In this way, smaller image collections can be evaluated with regards to similarities and dissimilarities. However, for larger image collections – as many digitization projects have produced over the last years – this method is insufficient. Since access to digital images is easy, fast and independent of place, art has become inclusive, but also confronted its user with a great amount of images, which eventually led to the question: What do I actually search and how do I find it? To answer this question, the disciplines of computer science and art history were joined.
Within the Computer Vision Group of the Heidelberg Collaboratory for Image Processing, an interdisciplinary project was established, which tackled the task of automatic object recognition and analysis of large art historical image corpora. Computer-based methods are successfully applied onto collections and assist art historians by evaluating large image-sets. Thus, relations between artists and workshops become visible. Computer scientists and art historians work side by side and allow a constant and direct exchange of ideas from which both sides benefit substantially. The project is concerned with questions of image understanding, aspects of machine learning and automatic perception. So far, image-immanent relations were detected and reconstructed based on object recognition, whereby visual similarities were measured. A web-based prototype, which was developed by the group, allows the automatic study and evaluation of large-scale image collections by searching for repetitive objects. Eventually, the focus shifted from object recognition to the analysis of entire images. Algorithms were used to understand scenes, compositions, genres and style of an art work.
Previous works included throughout evaluations on various datasets: The Sachenspiegel, a medieval law book which is attributed to Eike of Repgow, was analyzed and results were validated by an art historian. The law book exists in four different versions, which were named after their current location (Heidelberg, Dresden, Wolfenbüttel, Oldenburg). The search for repetitive gestures or individual objects revealed similarities and deviations between the four manuscripts. Additional art historical data was analyzed, including copies of Nazarene art by J.A. Ramboux (1790-1866), early modern architecture and treatises or depictions of Christ’s crucifixion. When applied to different sets, the algorithm proved its effectiveness, even when confronted with different search inquiries or research questions.
Future works will also focus on animated images, poses and relations between objects and individual parts (as can be found in sculptures). Tests, which were carried out on a Olympic-sports dataset, will serve as a basis for evaluations. Convolutional Neural Networks were used to automatically – and without supervision – detect visual similarities, whereby homogeneous groups were formed.
Video examples
Previous Work
Objectrecognition and Automatic Comparision of the Sachsenspiegel Codices:
https://hci.iwr.uni-heidelberg.de/compvis/research/mwk/
Evaluation of 3620 crucifixion images from the Prometheus Image Archive. Results can be viewed under the following address:
https://hci.iwr.uni-heidelberg.de/prototype_image_search_crucifixion
Computer-assisted detection and analysis of medieval legal gestures:
https://hci.iwr.uni-heidelberg.de/compvis/research/gestures/
Reconstructing the drawing process of reproductions from medieval images:
https://hci.iwr.uni-heidelberg.de/compvis/research/manesse/
Object recognition in in datasets of pre-modern images:
https://hci.iwr.uni-heidelberg.de/compvis/research/se/
Artificial and artistic seeing, computer vision and art history in collaboration (in German):
https://hci.iwr.uni-heidelberg.de/compvis/research/win/
Art historical analysis of architecture via computer vision:
https://hci.iwr.uni-heidelberg.de/compvis/research/composito/
Retrieving artworks from installation photographs:
https://hci.iwr.uni-heidelberg.de/installation-photographs/
For further information please contact:
Dr. Sabine Lang
Tel: 06221/5414811