Students with disabilities, particularly those with autism, experience unequal outcomes in STEM education and employment. Despite frequently reported strengths and interests in STEM disciplines, individuals with autism often do not receive the support—those related to communication, transitions, and flexibility—needed to succeed. In working with teachers, parents, and employers, this project will create pathways to position students with autism spectrum disorder for success in school and also the larger community.

This NSF-funded project commences with three workshops in Fall 2021, during which a pilot group of supported teaching fellows will explore the potential of custom-designed, wearable musical instruments to embed and teach computational thinking concepts to neurodiverse middle school students. An initial set of instruments, described below, have been designed by co-principal investigator Prof. Seth D. Thorn of the School of Arts, Media & Engineering at Arizona State University. During these three workshops, teaching fellows will explore these instruments while co-designing curriculum, exploring ensemble activities, and proposing new designs and instrument modifications and elaborations. We propose the following themes for these initial workshops, but will adapt these to the recommendations of our teaching fellows:

Workshop 1: Playful exploration of wearable instruments.

Workshop 2: Curriculum development.

Workshop 3: Social affordances and ensemble activities.

How does it work?

Like an acoustic rainstick, this computational version contains a finite number of "grains" that trickle from one end to the other when the rainstick is tilted. The more vertical the tilt, the faster the grains fall.

In the simplest version, the grains fall at a linear rate. The student records a short piece of audio that is arbitrarily sliced into a number of individual grains. The student can dynamically change this number into fewer or more grains, so that the grains become longer or shorter in duration, respectively. As the rainstick is tilted, these grains will play back in ascending or descending order, according to the state of the rainstick.

Variations can be introduced. For instance, the playback order of the grains can be randomized using a scramble function, or sorted back into ascending/descending order. The timing can be altered so that the grains play back asynchronously rather than synchronously. A set of timing delays can be added to better simulate the physics of an acoustic rainstick, so that the grains encounter multiple "twigs" as they trickle down, multiplying the sound generation. A probability parameter can also be added determining the likelihood of a grain encountering a twig.

Playful improvisation is enhanced by having students record their own sounds "into" the rainstick. They can use speech, singing, musical sounds, and foley effects using everyday objects.

How can it be playfully used in group ensemble activity?

Rainsticks can be divided between two students: one pours the contents of their rainstick into someone else's rainstick. The other then pours it back. (This is also easy to implement telematically). This dyadic instrument could be extended to a group involving multiple individuals (3+), with a finite set of "grains" being poured around, with unexpected sonic transformations occurring after each exchange, or other creative modulations.

Alternatively, statistical operations such as minimum, maximum, and mean of the angle of multiple sensors can be selected for group control of the "same" rainstick.

How does it work?

This instrument is not so much a shaker as a clacker: jolting it causes a single sound event to occur. Stronger jolts increase the magnitude of this sound. Events can be composed of clicks, clacks, sonic "grains" of a recording, other percussion instruments, and/or melodic notes. These can be played independently or combined to create a richly layered instrument.

The sounds produced by this instrument can be composed very precisely using programmable probability tables that favor certain sound events over others. It is even possible to "load" the instrument with a certain "quantity" of each sound; whenever an event is triggered, it is removed from the pool and no longer available, and eventually the shaker becomes empty.

How can it be playfully used in group ensemble activity?

Event-based instruments like the shaker are easy to scale up to group instruments (any number of sensors can be coupled to a "single" shaker).

How does it work?

An exponential increase in frequency is needed to hear a linear increase in pitch. This presents an opportunity to explore different curves (exponential functions) while listening to a ramp between two pitches, which can be continuous or step-wise (like a musical scale). The formal term for the continuous slide is the Italian word "glissandi." Students listen to a few examples, then begin producing their own.

Raising or lowering the arm determines the slope of the glissandi. Touching the screen while rotating the sensor "winds up" the catapult, which determines its duration when the screen is released. The instrument is polyphonic, so that multiple rising and falling glissandi with different durations and curves can overlap in time.

At particular pitch ranges and curves, glissandi begin to sound like familiar sci-fi effects ("laser blasts"). Moreover, thick polyphony generates interesting and unique perceptual results. Students will have fun activating this chaos.

How can it be playfully used in group ensemble activity?

Similar to the shaker, the catapult is an event-based instrument, meaning that a musical event is triggered, then follows its own course according to the parameters (in this case, the amount of winding and the slope). The instrument could be set up so that angles of multiple sensors are averaged to determine the slope, winding is additive and done collectively, and triggering or "release" of the catapult is determined by a logical operation:

• AND: everyone must release the catapult for it to trigger

• OR: only one person must release for it to trigger 

How does it work?

Students are shown a simple melody player that generates a chromatic scale when the hand is raised and lowered. Raising the hand climbs up the scale; lowering it climbs down. Students then hear a different version that plays the familiar pentatonic scale, ubiquitous in folk and pop music. This new version transforms the basic chromatic scale (0, 1, 2, 3, 4) by using a lookup table that takes those numbers as input and outputs a new set of numbers (0, 3, 5, 7, 10).

This instrument can be enriched in many ways that makes it very engaging and fun to play. For instance, accompanying harmonies can be used the shift the scales into a new key by adding a fixed offset (e.g., 0, 3, 5, 7, 10 --> 1, 4, 6, 8, 11). Bass tones and duplicate melodies can be used as well, or drum beats added after a series of notes are played to enrich the instrument. Students can also record their voice and "melodize" it using a familiar autotune effect that is widely used in contemporary pop music, increasing affordances for playful improvisation and experimentation. A student can create a rich musical ensemble instrument that they activate by spinning around, raising the hand, or twisting the arm, generating harmonies, melodies, and drumbeats all at once.

How can it be playfully used in group ensemble activity?

The rich ensemble instrument can be divided among several students, so that one plays the drums, another the melody, and another changes the harmony. Overall tempo or other transformations can be an effect of averaging the movements of the group as a whole.

How does it work?

A pedometer detects footsteps by isolating spikes in acceleration signals from noise and other variations. Students will use an interface that offers simple filtering and threshold operations to carefully "tune" a sensor worn on the foot to detect footsteps. Students can then use this signal to create different sounds as they walk around the room. Pre-recorded samples of walking on gravel, snowpack, or in puddles of water can be used, or footsteps can trigger individual musical tones or play bits of looping musical melodies. Students also have the option to record their own sounds and parse them into smaller "grains" (as with the rainstick) that can be triggered by footsteps. Parameters can be manipulated to modify these recordings.

How can it be playfully used in group ensemble activity?

Students can walk around together to playfully explore the different sounds their footsteps are making. It is also possible to have everyone's shoes be related to the same sound source. A measure of relational activity, such as the elapsed time between the most recent footstep and the last one, can set the musical pitch of an instrument, which individual footsteps trigger. This relational feature enlivens the instrument whether it is used in a group or by an individual. Musical shoes should also work well telematically - both sonically and conceptually - by imparting the sense that everyone is walking in the same space.