Examples (1 through 8):

 

Example 1

File: control-t.avi

Frame: No gesture synchronized, with table

 

Example 2

File: f37-g-t

Frame: gesture synched with frame 37, with gesture, with table

 

Example 3

File: f39-g-t

Frame: gesture synched with frame 39, with gesture, with table

 

Example 4

File: control-nt

Frame: No gesture synchronized, no table

 

Example 5

File: f29-g-t

Frame: gesture synched with frame 29, with gesture, with table

 

Example 6

File: f37-g-nt

Frame: gesture synched with frame 37, with gesture, no table

 

Example 7

File: f43-g-t

Frame: gesture synched with frame 43, with gesture, with table

 

Example 8

File: f33-g-nt

Frame: gesture synched with frame 33, with gesture, no table

 

Note. This is only a few (8) of the examples used in the experiment. The full study used many more examples with the hand gesture synchronized at every frame along the sentence.

Results

 

1. Speech signal and corresponding frame positions for the synchronisation of the middle of the gesture (5 frames long):

Click here

 

2. Results for categorization: Click here

Note how “there” is perceived or categorized (50% or more of time) even when the gesture is synchronized on frame 38 or 39 – “there” is perceived even before the speech audio for “there” has begun:

Gestures influence what a listener perceives or hears as emphasised!

 

3. Results for clarity ratings. Click here

Note how “book” is perceived or categorized most clearly when the gesture is synchronized with the centre of the speech and that the affordance of a table influences clarity of perception (i.e., does it make sense?).

Gestures influence how clearly a listener perceives the word categorised - and affordances or context changes the meaning of the perceived speech!

 

 

For full explanation, download the full publication.

 

www.metaffordance.com