Crossmodal Correspondences is a research project in Art, Science and Technology co-directed by Sebastián Tedesco and Bruno Mesz that focuses on the study of crossmodal correspondences between the senses of smell, taste, hearing, vision and touch in a primarily musical context.

Umwelt

Directors: Nelo Akamatsu, Bruno Mesz (UNTREF) y Sebastián Tedesco  (UNTREF)

Scents: Maria Zegna

Collaborators: Martin Ezequiel Reyes / Jessica Loto

Umwelt emerged from a collaborative process between the japanese artist Nelo Akamatsu and the argentines Bruno Mesz and Sebastián Tedesco, as a result of a common interest in the signals and sensoriums of organisms very different from humans. Akamatsu recently worked with the light produced by the enzymes of fireflies that lived 100 million years ago and Mesz and Tedesco with the biochemical quorum sensing of bacteria involved in fermentation. This led the team to base the present setup on a classical biosemiotic concept proposed by Jakob von Uexküll, precisely that of Umwelt. The idea he expresses is that while diverse organisms – human and non-human – share the same physical universe, we do not inhabit the same world. Each life form (including humans) is seen by von Uexküll as immersed in its own bubble, through which it perceives the world, mediated by signs, in its own time and space; the coexistence of the various species is a counterpoint, a “music of life”.

This installation suggests a certain estrangement produced by this counterpoint of biological-emotional signals, which cross other sensory devices.

 

---

Crossmodal pairings

Directors: Bruno Mesz (UNTREF) and Sebastian Tedesco (UNTREF)

Collaborators: Juan Cruz Amusategui (UNTREF) and Diego Alberti (UNTREF)

The “Cross-modal Glass” is an interactive cup that has capacitive sensors and an accelerometer, which detect specific movements and actions such as the act of drinking, aerating the cup, tilting it, etc. The data collected by the sensors is sent wirelessly to a system that uses them to trigger sounds or activate various actuators.

The device was developed with the aim of being used in performances, works and scientific experiments that explore cross-modal associations (correspondences between the senses). Through these, sound, visual, tactile or olfactory stimuli can modify the sense of taste, as has been demonstrated in numerous publications of empirical psychology.

This project obtained the 1° mention in the Federal Contest of Ideas-Project for Technological Innovation and Economic Reactivation of the Agencia I+D+i, Ministerio de Ciencia y Tecnología (2022).

 

---

Emotional Plates 2

Directors: Bruno Mesz (UNTREF) and Sebastian Tedesco (UNTREF)

 

The basic purpose of the project is to elaborate a tableware design based on the results of a scientific experiment on associations between musical emotions and shapes and materials. Each of the dishes would correspond in its shape and materiality to a musical emotion, thinking of serving food on them accompanied by emotionally congruent music that would function as a “sound condiment”, in order to obtain a gastronomic experience where the visual and musical elements would be in harmony. 

A preliminary experiment was already available, but it was decided to develop a new and much more rigorous research that was the basis for the design of four emotional dishes. This online experiment, with 176 participants from 10 countries, can be summarized as follows. Participants were asked to associate visual shapes (by drawing them) and a material from a list of 20 materials with emotions from the GEMIAC (Geneva Music-Induced Affect Checklist) scale induced by a set of musical excerpts. The intention of the study was to assess: (1) cross-modal correspondences (between music and shapes/materials) arising from music-induced emotions; and (2) predictability of music-induced emotions from extramusical information related to shapes and materials; that is, we sought to see to what extent visual elements captured emotionality.

The results suggest that music-induced emotions and their underlying core affect (i.e., the valence and degree of emotion activity) can be accurately predicted by the visual shape characteristics and materials that emerge in cross-modal correspondences with these emotions.

From these results it was then possible to derive the materials and characteristics of the shapes (their symmetry, curvature and complexity) that participants preferentially associated with each emotion and based on this, choosing four contrasting emotions, a Melancholic plate, an Energetic plate, a Powerful plate and a Tense plate were designed and constructed. 

This project was funded through a 2021 Creation Grant from the Fondo Nacional de las Artes.

 

---

Contemplation Surveillance

Directors: Bruno Mesz (UNTREF) and Sebastian Tedesco (UNTREF) and María Zegna (UBA)

Facial expression analysis: Jean-Christophe Sakdavong (Universidad de Toulouse)

Research Students: Victoria Balay (UNTREF) and Tomás Levita (UNTREF)

Contemplation Surveillance is a quadruple network of olfactory contemplations. The women in the portraits of Lía Correa Morales are contemplated via Zoom by other women; they are “watched” by a facial emotion recognition algorithm and by the authors, who trans-sensorialize the emotions delivered by the algorithm in perfumes inspired by the aromatic atmospheres of the 1930s. 

 

Unlike the static gazes that emerge from the portraits, their dynamic contemplation, computationally analyzed, turns out to be almost like a Zen meditation, empty of emotions.

---

V.I.T.R.I.O.L

Directors: Bruno Mesz (UNTREF) and Sebastian Tedesco (UNTREF) and María Zegna (UBA)

Collaborators: Nicolas Platner, Christian Marcelo Petersen, Guillermo Faivovich, TecnoLab.

 

V.I.T.T.R.I.O.L. is a work in dialogue with the exhibition “VÍCTOR GRIPPO / Preexistencias'” held at MUNTREF Center for Contemporary Art.

The acronym that gives it its title responds to the Latin phrase “Visita Interiora Terrae Rectificando Invenies Occultum Lapidem” (Visit the interior of the earth and purifying you will find the hidden stone). The author of this phrase is said to be Basilius Valentinus, a 15th century alchemist, who included it in his work L’Azoth des Philosophes, but it is possible that it was not actually a single person, but a pseudonym used by several writers of the time, who wrote about alchemy and hermetic philosophy.

The work consists of a device for the distillation and diffusion of a sound-olfactory essence. The sound textures were generated from Erik Satie’s Gnossiennes and the perfume is a formulation called “Ignite your inner sun”, composed of solar essences according to the mysticism of Ancient Egyptian perfumery, which have properties to activate the solar energy itself, in tune with the alchemical teachings and those of the Taoist masters in the secret books of Ancient China.

---

A Synesthete’s Still Life

Directors: Bruno Mesz (UNTREF), Marcos Trevisan (AR), Mariano Sigman (AR), Felipe de Ávila Franco (BR) and Roberto Azaretto (AR).

Aromas:IFF (International Flavors & Fragrances) 

Acknowledgments: Javier de Verda (IFF), Diego Shalom. 

 

A flattened ontology of the sensorium underlies this installation: the traditional vertical hierarchy between “primary” and “secondary” senses is relativized and horizontal transmodal correspondences that translate taste into other sensory modalities are emphasized. Bitter, sweet, salty and sour incarnations of “Rhapsody in Blue” emerge among objects inspired by these basic flavors from which a New York olfactory landscape emerges.

---

Cheese and Music

Research team: Bruno Mesz  (UNTREF), Mari Sandell (University of Turku), Anu Hopia (University of Helsinki), Seppo Salminen (University of Turku), Sami Silen (University of Turku), Gabriel Vinderola (CONICET), María Andrea Farina (Universidad de La Plata), Gustavo Basso (Universidad de la Plata), Daniel Vilte (INTA). 

We study the influence of the frequency and the energy of  sound on the biomass on cheese bacteria during fermentation, and the effects on the sensory evaluation of cheeses exposed to music during their maturation process

---

Emanations of Discourse

Directors: Sebastian Tedesco (UNTREF) and Bruno Mesz (UNTREF)

Aromas: IFF (International Flavors & Fragrances) 

Research Students: Leonardo Potenza (UNTREF) and Camilo Alvarez (UNTREF)

The installation focuses on a cross-modal artistic intervention based on proclamations by two iconic activists: Martin Luther King and Greta Thunberg. The discourses create a universe of relationships based on a primary layer of senses (sight and sound). At the same time, they open up to other senses that are not so obvious, capable of completing the discursive universe or diverting their logic. It is precisely there that this project begins, where certain sensory modalities are veiled by the pre-eminence of the word.

---

Emotional Plates

Directors: Bruno Mesz (UNTREF) and Sebastian Tedesco (UNTREF) 

Music for the Energetic Plate: Santiago Santero (UNA) and  Bruno Lo Bianco 

Research Students: Leonardo Potenza (UNTREF) and Camilo Alvarez (UNTREF)

Installation that includes videos, sounds, images and scents, inspired by the conceptual design of tableware that explores the use of digital devices at the table, conceived from the results of a cognitive pre-test on cross-modal correspondences generated by musical emotions, and aimed at creating multisensory gastronomic atmospheres.

---

Evasion Plan

Directors: Bruno Mesz (UNTREF) and Sebastian Tedesco (UNTREF) 

Research Students: Leonardo Potenza (UNTREF) and Camilo Alvarez UNTREF)

 

The work explores possible relations of conceptual and olfactory compatibility between the parts of a written universe. It hypothesizes that it is possible to extract, from a text and a series of selections and editing operations analogous to chemical distillation and re-synthesis processes, a condensed essence of the sublimated text in aromatic essence. 

The project works from the text of Adolfo Bioy Casares’ novel “Evasion Plan”, where synesthesia is one of the central themes. The piece consists of a large green glass jar from whose interior the perfume of the novel emerges. 

The liquid inside the bottle contains an aroma and sound release device that reproduces aromatic references dispersed within the text that converge in a single perfume due to its spatial proximity; from there a weak voice arises (it is necessary to physically approach to perceive it) that tells fragments of the book.

---

Souveillance de la Performance

 “Sousveillance of performance”, the affective processes of musicians recording themselves during performance, expressed both in their faces and in their playing, are computationally analyzed and collapsed into one synaesthetic perfume of affective involvement. We highlight, in equal terms, two sources of emotional communication, the face of the musician and the musical sounds, to examine their parallelisms, confluences and divergences, to reflect on how they interact to convey meaning, and to extract an olfactory portrait of the performance that could translate, across the senses and in a single instant, the synthesis of the emotions deployed in time.

In this piece we emphasize that the musical experience is always mediated not only by sounds but by other situational elements that contribute to the creation of an atmosphere of performance. We adhere to the conception of the philosopher Gernot Böhme, according to whom atmospheres have generators in several different senses (visual, auditory, tactile…), but they constitute in themselves the primary objects of perception, instead of the unimodal sensory impressions which he considers secondary. In our installation, the perfumes are intended to synthesize the emotional expression of the atmosphere as an exteriorized feeling, an irradiation or ek-stasis in Böhme’s terms.

---

Olfactorized Hearing 1

Directors: Bruno Mesz (UNTREF) and Sebastian Tedesco (UNTREF) 

Aromas: IFF (International Flavors & Fragrances) 

Research student: Facundo Suasnábar (UNTREF)

 

Four insulation domes release scents and sounds associated with four emotional states: Stimulant, Irritant, Depressive and Narcotic. 

---

Other Senses of Smell 

Directors: Bruno Mesz (UNTREF) and Sebastian Tedesco (UNTREF) 

Aromas: IFF (International Flavors & Fragrances) 

Video installation based on research on transmodal correspondences between aromas and music. The possible existence of an emotional connection between aromas and music, as well as a spontaneous link between colors and shapes and smells inspire the videos. Four aromas with different emotional effects were chosen (mandarin, stimulating; violets, melancholic; extreme amber, irritating: and sandalwood and vetiver, heavy and narcotic), which we tried to portray from the visual and auditory aspects.

---

Perfume of Yuyo and Alfalfa

Directors: Bruno Mesz (UNTREF) and Sebastian Tedesco (UNTREF) 

Piano performance: Bruno Mesz (UNTREF)

Collaborators: María Zegna (UBA) and Carlos Sztaynberg (UNTREF)

Research Students: Leonardo Potenza (UNTREF) and Camilo Alvarez (UNTREF)

Past research Students: Nicolas Gorla (UNTREF)

 

The work consists of a performance where a “smell organ” is used to execute a series of tangos that have been smellified in order to explore the cross-modal effects of aroma on the perception of music. 

The “smell organ” is a cross-modal system that associates notes of a piano with a series of spatialized aroma release devices. The selection of the aromas that smellify each tango arose from an investigation of the aromatic references in the universe of tango from its lyrics

---

Smell it, Smell it. What do you see?

Directors:Sebastian Tedesco (UNTREF), Maria Zegna (UBA) and Bruno Mesz (UNTREF)

This project is a multisensory installation that makes use of “Huelo”, a device developed to combine an aroma with the tasting of a liquid in the context of experiments on the sensory phenomenon of cross-modality. In this installation, the “Huelo” is used to create two sensory experiences through pairings between flavor, aroma, sound and vision, the latter being stimulated through the presence of an object of use, which evokes or reinforces the sense of certain associations.

---

Trans-Emotional Scriabin

Directors: Bruno Mesz (UNTREF) and Sebastian Tedesco (UNTREF)

Collaborators: María Zegna (UBA) and Carlos Sztaynberg (UNTREF)

Research Students: Leonardo Potenza (UNTREF) and Camilo Alvarez (UNTREF)

Past research Students: Nicolas Gorla (UNTREF)

 

“Trans-emotional Scriabin” consists of the presentation of an aroma piano that responds to the synesthetic associations note-color of the Russian musician Alexander Scriabin, producing combinations of perfumes and colors.

 The main objective of this project is the empirical study of cross-modal correspondences between the senses of smell, taste, sight, touch and hearing in a musical context. In the field of applications, work is done the development of technologies for the creation of multisensory atmospheres for performances, increased interactions with food and perfumes, the promotion of healthy eating behaviors, new musical interfaces; Sensory representations of data with chemical senses (smell and taste).

---

T2M

 

Director: Bruno Mesz (UNTREF)

Collaborator: Lucas Samaruga (UNQ)

T2M is a sound installation for interactive performances. It takes words from tweets produced by the audience or Twitter trending topics and composes music using Freesound® sound-files (labeled with those words) as raw material. 

T2M explores the intra-sonic and extra-sonic semantics of sounds that are given by the tags of audio files in the Freesound database, with the purpose of generating soundscapes from textual input.

The motivation for this installation was research on music- taste crossmodal correspondences which studied musical representations of the extra-musical semantic domain of taste words.

---

Undocumented Emotions of Touch and Scent

 

Directors: Bruno Mesz (UNTREF) and Sebastian Tedesco (UNTREF) 

Research Students: Leonardo Potenza (UNTREF) and Camilo Alvarez (UNTREF)

Past research Students: Nicolas Gorla (UNTREF)

 

The installation consists of the recreation of a laboratory environment in which the public can experience a series of abstract vibe-touch-aromatic compositions sitting on a chaise-longue equipped with vibrotactile devices that, through the skin, transmit tactile sensations to the body accompanied by aromas that are facilitated by an operator.

 

After testing the compositions, the viewer becomes participant of a  scientific experiment on correspondences between touch and aroma, for which she is exposed to a series of pairs of touch-aroma correspondences of which (s)he has to evaluate in the form of a series of combination characteristics.

---

Undocumented Emotions: Aestheticized Capítalism – Dance for the Sea

 

Directors: Bruno Mesz (UNTREF), Sebastian Tedesco (UNTREF) and María Zegna (UBA)

Aromas: IFF (International Flavors & Fragrances) 

Research Students: Leonardo Potenza (UNTREF) – Camilo Alvarez (UNTREF)

Past research student: Nicolás Gorla (UNTREF)

 

In Aestheticized Capitalism, the public can choose between two vibro-tactile-aromatic abstract compositions and experience the selected composition by leaning their backs on transducers that produce tactile sensations, accompanied by an aroma. Scents are interpretations and / or olfactory translations of the concepts of “money” and “pollution”.

 

Dance for the Sea: In “Dance for the Sea” I annihilate myself and get entangled with the sea.

(Maria Zegna)

---

VR, Emotion, Music

Directors: Bruno Mesz  (UNTREF) and Jean-Christophe Sakdavong (Université de Toulouse II)

 

This project explores the effect of different VR contexts on the perception of musical emotion.

 

Emotional dinner

Bruno Mesz (AR/UNTREF), María ZEGNA (AR/UBA)  and Janice Wang (CA/GB, Oxford University)

Exhibition design: Sebastian Tedesco (AR / UNTREF)

Aromas: IFF (International Flavors & Fragrances) 

Past research student: Facundo Suasnabar (AR / UNTREF)

 

This performance consisted on four multisensory scenes: Narcotic (Faure’s music with a heavy “arroz con leche” and smell of roses), Irritant (spicy bread served on the spines of a cactus, with the performance of “Child of Tree” by Cage), Depressive (blueberries put in cold water, dance, music by Radu Malfatti), and Estimulant (champagne and music by Xenakis).

 

Emotional Plates at Distat Terra

December 1st, 2022 – December 18th, 2022

The work of Bruno Mesz and Sebastian Tedesco participated in the Distat Terra Festival, held in Choele Choel, Río Negro, Argentina.

---

A Synesthete’s Still Life at Museum of the City of New York

From September 2022 to September 2023

 

A Synesthete’s Still Life is part of the exhibition “Food in New York: Bigger Than the Plate” at the Museum of the City of New York.

---

Sour Tango by Bruno Mesz at Manchester Science Festival and Experimenta, The Science Center

2022

The piece Sour Tango was part of two science exhibitions during 2022: the Manchester Science Festival at the Science and Industry Museum and A Matter of taste at Experimenta, The Science Center in Cologne, Germany.

---

Contemplation Surveillance at the Yrurtia Museum

From October 2021 to December 2021

The work Contemplation Surveillance by Bruno Mesz, Sebastián Tedesco and María Zegna participated in the exhibition The divine anomaly within the framework of BIENALSUR 2021, curated by Anthoniy Val and Marina Aguerre.

---

Talk The taste of Music and how virtual reality can change our perceptions.

September 23rd, 2020

Bruno Mesz and Jean-Christophe Sakdavong gave the talk The taste of music and how virtual reality can change our perceptions, organized by the Ministerio de Ciencia, Tecnología e Innovación, the French Embassy in Argentina and Centro Cultural de la Ciencia (C3).

---

Emotional Plates selected for the Premio Fundación Andreani 2020/2021

September 2020

The work Emotional Plates was selected to be part of the exhibition of the seventh edition of the Premio Fundación Andreani.

---

Emanations of Discourse in the exhibition Resistance and Essence

The work Emanations of Discourse by Bruno Mesz and Sebastian Tedesco was exhibited in the exhibition “Resistance and essence” at the LOOP Festival in Barcelona, curated by Gabriela Berti.

---

Official Statement of Benevolence for the organization of Brain & Wine

June 2019

The National Senate declared an official declaration of approval for the realization of the Brain & Wine event, the neuroscience and wine meeting that took place in March 2018 in Barcelona and was organized by Bruno Mesz, Sebastián Tedesco, Universidad de Tres de Febrero, the University of Oxford, INECO Foundation, ElBulliFoundation and the CenSeS Institute of London University.

 

---

Perfume of Yuyo and Alfalfa: Tango Aromas

May 2019

Bruno Mesz and Sebastián Tedesco achieved the experimental sound-olfactory performance Perfume of Yuyo and Alfalfa: Tango Aromas at the Museo Casa Carlos Gardel. It was performed in collaboration with Camilo Álvarez, Leonardo Potenza, María Zegna and Carlos Sztaynberg.

---

Science and Wine World Congress 2019

May 2019

Bruno Mesz participated in the First World Congress of Science and Wine 2019 that took place in Porto (Portugal). There he gave the talk Analyzing the impact of music on the perception of red wine through the temporal domain of sensations (done with Qian Janice Wang, Pablo Riera, Marcos Trevisan, Mariano Sigman, Apratim Guha and Charles Spence) and the poster Transmodal Cup (done together with Juan Cruz Amusategui and Sebastián Tedesco). Both works are part of Bruno’s research projects within the framework of MUNTREF Center of Art and Science.

Science & Wine 2019 addressed different topics related to wine technology, chemistry and biochemistry, health and environmentally sustainable development of this industry.

---

Sebastián Tedesco and Bruno Mesz at the exhibition Among Senses at BIENALSUR 2019

May 2019

Sebastián Tedesco and Bruno Mesz presented the work they carry out in the research projects they direct at MUNTREF Center of Art and Science at the exhibition Among Senses, held at the Museum of the Universidad Nacional de Tucumán. They also participated in the talk Transmodality: Science and poetics of the senses.

---

Presentation of Crossmodal Pairings Performance

April 23rd, 2019

A presentation of the Crossmodal Pairings Performance using the Crossmodal Glass was held at Vico Wine Bar. This project is carried out in the context of the research project Crossmodal pairings, directed by Bruno Mesz and Sebastián Tedesco with the collaboration of Diego Alberti and Juan Cruz Amusategui at MUNTREF Center of Art and Science.

---

Trans-Emotional Scriabin at the New Gardens Conference at the Servente

October 2018

The multisensorial installation Trans-emocional Scriabin was exhibited at the Conservatorio Provincial de Música Gilardo Gilardi in the city of La Plata.

---

Installation at the exhibition The Senses: Design Beyond Vision

From April to October 2018

Bruno Mesz, Marcos Trevisán and Mariano Sigman presented the installation The Taste of Music at the Cooper Hewitt, Smithsonian Design Museum, New York, USA. In it, four buttons reproduce short musical compositions inspired by the sense of taste. Several musicians were asked to create improvisations in response to basic taste words. The “sour” compositions tend to have short, sharp, dissonant notes. “Sour” compositions tended to be slower, lower, dissonant, and continuous (legato). The study found similar correlations for “sweet” (long, consonant, soft) and “salty” (staccato, with short, discontinuous notes).

---

Talk at Cooper Hewitt, Smithsonian Design Museum – New York

April 2018

As part of the exhibition The Senses: Design Beyond Vision, Bruno Mesz, with the collaboration of Pablo Di Liscia (UNQ) and Damián Anache (UNTREF), gave a talk on the influence of music on the taste of chocolate. Compartés® Chocolates were used.

---

Brain & Wine

March 2018

Brain & Wine was the first international forum of enological neuroscience. It took place in the city of Barcelona and it brought together the leading experts and researchers of the subject. It was organized by Universidad Nacional de Tres de Febrero (UNTREF), ElBulliFoundation, University of Oxford, INECO Foundation and Centre for the Study of the Senses (CENSES)

In addition to the scientific presentations, the UNTREF team, together with the University of Oxford, conducted a series of experiments with the public on the perception of the taste of wine in accordance to a combination with auditory stimuli. 

Several artistic works were also presented in relation to the undertaken investigations:  Smell it, Smell it. What do you see?, T2M (Text to Music) and Crossmodal Pairings.

---

Transmodal Translations with Alexánder Apóstol in MALBA

From December 2017 to February 2018

Bruno Mesz and Sebastian Tedesco worked together with Alexánder Apóstol on the site specific project “Salida de los obreros del museo. Taller y República a partir de Tucumán arde” for the Malba Museum. They also proposed a journey through the mathematical “B side” of the exhibition to, through a conference and a trans-sensorial performance, restore the concepts of the trans-modal translations present in their works.

The activity included a brief guided tour through the exhibition, that took into consideration the mathematical mechanisms -the Graph theory and the information theory- which were used to make a translation and interpretation of the text of Tucumán Arde, and afterwards a talk on the works and investigations developed at MUNTREF Center of Art and Science.

Furthermore, in the context of the activity they presented to the public the work “T2M”, made together with Lucas Samaruga, which explores correspondences and trans-modal translations between aroma, text and sound.