Rensselaer Polytechnic Institute computer scientist Chuck Stewart will join a newly created institute aimed at using images of living organisms to understand biological processes like species evolution, genetic inheritance patterns, and even behavior. Stewart, an expert in the ecological applications of artificial intelligence and computer vision, will join the new Imageomics Institute, which is supported by a $15 million grant from the National Science Foundation (NSF).

“One of the grand challenges in research is discovering the rules of life,” said Stewart, a professor and head of the department of computer science at Rensselaer. “The goal of imageomics is to use AI and machine learning to help infer phylogenetic traits from images to understand everything from the structure of species down to individual characteristics and behaviors. At its core, we will use images to make biological predictions which can be verified by biologists or ecologists.”

Stewart is an expert in the ecological applications of computer vision and a co-investigator in the Wildbook project with Wild Me, which builds open source software and artificial intelligence for the conservation research community. The project draws heavily on a computer program he and his students created, known as the Image-Based Ecological Information System, which can identify individual animals in ordinary photographs and match them against images of known animals stored in a database.

“Professor Stewart’s extensive experience in extracting detailed, individualized information from wildlife imagery across multiple species makes him an essential member of the new Imageomics Institute,” said Curt Breneman, dean of the School of Science at Rensselaer. “His innovative computational solutions represent core enabling technology underpinning the new field of imageomics.”

The Imageomics Institute will be led by The Ohio State University and is an inaugural institute for data-intensive discovery in science and engineering created by the NSF as part of its Harnessing the Data Revolution initiative. As such, the Imageomics Institute will be part of forming a national collaborative research network dedicated to computation-enabled discovery.

The institute will create a new field of study in which scientists use images of living organisms as the basis for understanding biological processes of life on Earth. This new approach, called imageomics, will utilize machine learning methodologies to extract from images biological traits such as the behavior or physical appearance of an individual, or even the distinguishing skeletal structure of a species. Imageomics will leverage and extend the paradigm of knowledge-guided machine learning by structuring the algorithmic underpinnings around biological knowledge and continually guiding and refining them using new biological knowledge as it is generated.

Similar to genomics before it, which applied computation to the study of the human genome, imageomics will leverage computer science to help scientists extract meaning from an otherwise unwieldy amount of natural image data. Images to be studied include digital collections from museums, labs, and other institutions, as well as photos taken by scientists in the field, camera traps, drones, and even members of the public who have uploaded their images to platforms such as eBird, iNaturalist, and Wildbook.

The ability to use image data for scientific discovery will get scientists closer to answering compelling questions such as: Are a baby zebra’s stripes similar to its mother’s and what might the genetic mechanism be for inheriting the pattern? How do the skulls of bats species vary with environmental conditions and what evolutionary adaptation drives that change? What subtle morphological features distinguish closely related species of fish from each other and why?

“So many pictures of organisms exist with scientific value that we’ve never been unable to analyze at the right scale,” said Tanya Berger-Wolf, faculty director of the Translational Data Analytics Institute and principal investigator for the Imageomics Institute. “This will change how we are able to see and understand the natural world. Computers help humans ‘see’ images differently and expose what we may otherwise miss.”

In addition to Berger-Wolf and Stewart, the project team includes biologist and computer scientist co-investigators from Tulane University, Virginia Tech, and Duke University; senior personnel from Ohio State, Virginia Tech, and six additional institutions; and collaborators from more than 30 universities and organizations around the world.