Open Standards for Sign Languages

Digital Media, Open Source, standards, video accessibility, women in IT, , , , , ,

Looking at accessibility for video includes sign language. It is a most fascinating area to get into and an area that still leaves a lot to formalise and standardise. A lot has happened in recent years and a lot still needs to be done.

Sign languages are different languages to spoken languages: they emerged in parallel to spoken languages in communities whose boundaries may not overlap with the boundaries of spoken languages. However, most developed means to translate spoken language artifacts (i.e. letters) into sign language artifacts (i.e. signs). So, a typical signer will speak/write at least 3-4 “languages”: the spoken language of their hearing peers, lip reading of that spoken language, letter signs of the spoken language, and finally the native sign language of the community they live in.

Encoding sign language in the computer is a real challenge. Firstly, there is the problem of enumerating all available languages. Then there is the challenge to find an alphabet to represent all “characters” that can be used in sign across many (preferably all) sign languages. Then there is the need to encode these characters in a way that computers can deal with. And finally, there is the need to find a screen representation of the characters. In this blog post, I want to describe the status for all of these.

Currently, sign language can only be represented as a video track by recording sign speakers. Once a sign character list together with an encoding and representation means for them and a specification of the different sign languages is available, it is possible to encode sign sentences in computer-readable form. Further, programs can be written that can present sign sentences on screen, that translate between different sign languages, and between sign and spoken languages. Also, avatars can be programmed that actually present animated sign sentences.

Imagine a computer that instead of presenting letters in your spoken language uses sign language characters and has keys with signs on them instead of letters. To a sign speaker this would be a lot more natural, since for most sign is their mother tongue.

Listing all existing sign languages
It was a challenge to create codes for all existing spoken languages – the current list of language codes has only been finalised in 1998.

Until the 1980s, scientists assumed that it is impossible to develop as rich a language with signs as with writing and speaking. Thus, the native languages of deaf people were often regarded as inferior to spoken languages. In many countries it was even prohibited to teach the language in schools for the deaf and instead they were taught to speak an oral language and read lips. In France this prohibition was only lifted in 1991! Only in about 1985 was it proven that sign languages are indeed as rich as spoken languages and deserve the right to be called a “language” and be treated as a fully capable means of communication.

So, there hasn’t actually been much time to map out a list of all sign languages. The best list I was able to find is in Wikipedia. It lists 28 N/S American, 38 European, 34 Asia-Pacific-AU/NZ, 30 African, and 13 Middle Eastern sign languages – in summary 143 sign languages. This list contains 177 sign languages.

Interestingly, there is also a new International Sign Language in development called Gestuno which is in use in international events (Olympics, conferences etc.) but has only a limited vocabulary.

In 1999 the Irish National Body, Deaf Action Committee for SignWriting, proposed the addition of sign language codes to ISO-639-2. Instead, a single code entered the list: sgn for sign language. In 2001, this led to the development of IETF language extension codes in RFC 3066 for 22 sign languages. In September 2006, this standard was replaced by RFC 4646, which defines 135 subtags for sign languages, including one for the International Sign Language and a generic “sgn” one.

While not complete, the current IANA subtag language registry now regards sign languages as valid derivatives of a country’s languages and therefore handles them identically to spoken languages. It’s also extensible such that any sign language not yet registered can still be specified.

Characters for sign languages
The written word is very powerful for preserving and sharing information. For a very long time there has been no written representation of sign languages. This is not surprising considering that there are still indigenous spoken languages that have no written representation. Also, the written representation of the spoken language around the community of a sign language would have served the sign community sufficiently for most purposes – except for the accurate capture of their thoughts and sign communications. It would always be a foreign language.

To move sign languages into the 20th century, the invention of characters for signs was necessary.

It is relatively easy to map the alphabets of spoken languages to signs (e.g. American (ASL) manual alphabet, British, Australian and NZ (AUSLAN) manual alphabet, or German manual finger alphabet, also see fingerspelling). Interesting the AUSLAN manual alphabet is a two-handed one while the ASL one is single-handed.

Fonts are available for these alphabets, too, e.g. British Sign Font, American Sign Font, French Sign Font and more.

The real challenge lies in capturing the proper signs deaf people use to communicate amongst themselves.

This is rather challenging, since sign languages uses the hands, head and body, with constantly changing movements and orientations for communication. Thus, while spoken language only has one dimension (sound) over time, sign languages have “three dimensions” and capturing this in characters is difficult. Many sign languages to this date don’t have a widely used written form, e.g. AUSLAN. Mostly in use nowadays are sequences of photos or videos – which of course cannot be computer processed easily.

Two main writing systems have been developed: the phonemic Stokoe notation and the iconic SignWriting.

Stokoe notation was created by William Stokoe for ASL in 1960, with Latin letters and numbers used for the shapes they have in fingerspelling, and iconic glyphs to transcribe the position, movement, and orientation of the hands. Adaptations were made to other sign languages to include further phonemes not found in ASL. Stokoe notation is written left-to-right on a page and can be typed with the proper font installed. It has a Unicode/ASCII mapping, but does not easily apply to other sign languages than ASL since it does not capture all possible signs. It has no representation for facial and body expressions and is therefore a relatively poor representation for sign.

SignWriting was created by Valerie Sutton in 1974, a dancer who had two years earlier developed DanceWriting and later developed MimeWriting, SportsWriting, and ScienceWriting. SignWriting is a writing system which uses visual symbols to represent the handshapes, movements, and facial expressions of sign languages. It is a generic sign alphabet with a list of symbols that can be used to write any sign language in the world.

SignWriting can be easily learnt by signers and is more popular now than Stokoe. Signers compose the symbols together in a spatial way to represent their signs. They then write the composed symbols from top to bottom on a page, similar to other iconic character sets. SignWriting currently supports 73 different sign languages, whose dictionaries and encyclopedias are captured in SignPuddle. This will eventually allow the creation of complete corpora for all sign languages.

Unicode encoding of SignWriting and visual representation
Because of its unique challenges of having to cover the spatial combination of symbols as a new symbol rather than just the sequential combination of symbols, it took a while to get a Unicode representation of SignWriting.

About a year ago, on 19th September 2008, Valerie Sutton released the International SignWriting Alphabet (ISWA 2008).

A binary representation of SignWriting is defined in ISWA 2008. It is based on a representing 639 base symbols and their potential 6 fill and 16 rotation variants in 61,343 code points, that completely cover the subset of 35023 valid symbol codes. The spatial aspect of SignWriting are encoded in a 2-dimensional coordinate system. The dimensions go from -1919 through 1919 to place the top left corner of the symbol.

SignWriting base symbols are encoded in plane 4 of Unicode, which provides 65,536 code points, easily covering the defined 61,343 Binary SignWriting code points. Further special control and number characters are used to encode the spatial layout.

Visual Representation of SignWriting
Valerie Sutton created over 35k individual PNG images for ISWA 2008, which have been reformatted for standard color & reduced file size, and renamed to the character code. They are a font used to represent the signs. The images can be accessed on Valerie’s server.

Closing
After learning all this today, I have to say that Valerie Sutton has just turned into a new idol of mine. The achievements with SignWriting and the possibilities it will enable are massive.

Now I just have to figure out what to do when we hit on a sign language track that has been encoded in SignWriting and it represents captions. Maybe it is possible to display sign as overlay but on the left side of the video. This would be similar to some other languages that go from top to bottom rather than left to right.

25 thoughts on “Open Standards for Sign Languages”

  1. This is why there are no sign language Wikipedias yet – no written form, let alone a widely-used written form. We would LOVE to have sign language Wikipedias … this work is very promising!

  4. @Joe – I tend to compare SignWriting to other language representations that consist of iconic “characters” – things like Chinese and Japanese. Even where a full word or concept is represented in a scribbling, when it is encoded, it is still called a “character”. I am fully aware that the transliteration is always poor – but so is any written representation of a language.

    I also believe that it is advantageous to have a written representation that is not video – it is important that computers understand what a person is expressing and is able to process that, e.g. to translate into other languages or even index it for search.

    So, while I do understand the desire to not lose any expressivity of a sign expression by recording it with video, I also believe it is of utmost importance to have a computer-representable representation of that. And from what I have read, SignWriting isn’t doing too bad a job there.

    I am pretty sure from what I read that a sign speaker would prefer having their computer represent the user interface with SignWriting rather than with e.g. English, since it is much closer to their mother tongue than to the language “forced” upon them by the speaking community around them. However, I’d be keen to hear the opinion from actual HOH people.

  7. @Ben I actually have a bit of a background in automated audio-visual content analysis and indeed found your link highly interesting. It is very useful for automated translations between sign and English.

    However, I think it is impossible to decode a full sign sentence through the words that have an English representation. I think it is equally if not somewhat more important to create corpora of the “words” / “expressions” that sign languages are made up of. It would be great if the research you pointed to could help there by providing sign boundaries, so catalogs can be built of the used expressions. I’ve seen such corpora starting to build!

  8. an example of where this is useful: recording testimony. It’s not the same thing at all to record the translated written version of a Deaf person’s testimony in a court of law.

    To have a written version of signs means it’s less abstracted.. I’d want to know exactly what sign the accused used, not what word the translator picked.

    but i have no idea how one would encode all the spacial grammar – and all the meaning added by facial expression in New Zealand Sign Language — and gosh there must be yet more intricacies in other signing languages.

  9. Thank you Silvia for providing a great account of the story so far regarding the coding and representation of sign languages.

    The need to represent Sign Language information.
    Your post makes it very apparent that there is a need to represent sign languages with symbols, because it is a more efficient encoding of the language (information theory) than video is, for instance. Even if we create a video codec dedicated to encoding the psycho-visually and psychophysically important information of sign languages in moving pictures, we will not be able to provide, in a computationally efficient way, such high symbolic functions as search (for a specific movement or gesture in the sign language content) or even editing functions such as copy-paste-this-piece-of-sign-language on this representation of the language.

    Writing Sign Language VS Representing Sign Language
    The confusion, as Joe pointed out accurately, is that of text literate people who needs sign language to be written down to become usable for their analytic tools (gloss, tag etc) or for the common needs in society (legal transcripts, contracts must all be in writing). Sign Language users are very functional without writing. The fact that we need to provide a symbolic representation for sign languages is not saying that we are advocating a writing system.
    The ambiguity in the name “SignWriting” often has the adverse effect of leading sign language representatives to ignore and stay away from the notation system because it seems like advocating a writing policy.
    The advantage of SignWriting is that it is very intuitive : once a few rules have been learned, the signs can be understood and short handed very easily. Sutton Sign Writing is very good for taking notes : I used it when I was learning SASL at Wits. My notes would be in SASL thanks to the notation, and not in English like my classmates, which means when I went back home I could review my notes in SASL, not the English words or sentences. However there are limitations in terms of the expression of movement and ambiguity in 3D positioning that is more than acceptable in the same way that writing English never managed to reproduce all that spoken English could communicate (stress, tone of voice, speed etc ).

    There are other notation systems for Sign Languages that are also very good depending on the function they are to be used for.
    HamNoSys for example is a lot more difficult to use for taking notes. In a sense, it’s a notation that is more phonological in nature and is better suited for transcribing sign language articulations accurately for animation purposes for example. In the same domain, SLIPA is a very interesting attempts at using International Phonetic Alphabet characters that can all be represented using Unicode. The merit of the SLIPA project is that it provides some rational on how the unicode characters were chosen in relation the particular phonology of Sign Languages. I am also aware of other notations that have been recently created (French and Czech).

    What is required.
    As it is, most of this notation systems use their own symbols (SSW, HamNoSys) which are available as (very incomplete) fonts for visual representation based on usual characters. Additionally, these notation systems also can be represented using XML (SWML and SignML) which make them computationally viable for linguistic, animation and other kinds of applications. But it still does not solve the problem that there is no symbolic representation for sign languages on information systems, and this needs to be addressed using Unicode.

    I do think that both a conceptual notation system such as Sutton Sign Writting AND a phonological system such as HamNoSys are both needed in the Unicode range, in the same way that Japanese as two writing systems that coexist in their texts, one is more conceptual and the other more phonological.

    Finally, to go back to Writing VS Video, editing sign languages doesn’t necessarily need to involve video OR writing for presentation, it can be done using the hybrid of animation with signing avatars. What keeps signing avatars from really making an impact is that creating 3D animated content is a very difficult, non intuitive and a time consuming task. Learning to type HamNoSys to generate animations is also not particularly easy for someone that wants to send a message in sign language (eSign project). The solution I’m investigating is to provide an hybrid environment that would use the notation to drive the animation and the animation modeling to infer parametrized notations so has to guide the capture process of a particular sign language. Existing approaches such as Vcom3D software use two separate environments : one for capturing new signs and another to edit sign language using the prerecorded signs. This approach also makes it fast to edit sign language but again, it doesn’t use a common base to encode the language.

    The base line is that these notations need to be used more often to capture sign language content. The representation of these notations do not only result in their graphical notation forms but can be used as annotation tracks in videos to encode the specific gestural information of sign languages, with avatars to generate animations and to have a specific mean on information system in general to encode sign language contents.

  12. @Guillaume Thanks so much for all the additional detailed information! I think what you are working on is extraordinary and will certainly contribute to making sign language more integrated with information systems. I agree that this is the biggest challenge right now.

    @Daniel Thanks for the links to SiGML. That research also falls into the space of animated sign language, which I think Guillaume is working in, too. Great stuff!

    @Roberto’s Flash+SMIL example is a nice way to present video with captions and sign language video. With the new HTML5 video element, we want to enable these kinds of displays without need to install additional browser plugins!

  13. @Roberto on Flash+SMIL : thank you so much, I have been looking for this exact technical implementation for a long time! I just couldn’t implement the Flash, it’s great !

  15. hello, i went through your site, you have portrayed your blog in a really new and dynamic style. you have adopted the slow and steady method this is what have attracted me towards your site. you have done a great job.

  16. Native speakers of sign language provably have no interest in writing down their utterances. You have no business telling them otherwise.

    Your computer-scientist obsession with boiling everything down to characters (I note you misrepresent even Chinese in this regard) will serve you ill.

  18. @Joe Never before in my life have I received this much gratitude for the work I am doing (and note: SignWriting is not my work – I am just reporting on it). The gratitude of those who care is enough proof for me to continue my work. Those who don’t want to make use of new technology don’t have to. There is no obligation.

  19. @Joe Thank you for supporting the point I was making in my earlier post.
    It is important to remember that sign language notations such as Stokoe have been instrumental at proving to the world that sign languages had to be recognized and given the status of natural languages. This goes to show that sign language notations are essential in the study of sign languages – although not necessarily their practice.
    For information systems (computers, cell phones) to provide proper sign language support, notations are needed. They might not be shown to the user but they would be used in the background to provide functionalities such as searching on gestures and indexing. The Sign Writing Sign Puddle dictionary search feature using part-of-signs provides such a proof of concept.

  20. Your conclusions are quite correct. It would be good if all written literature easily could be transferred to any Sign Language by using a computer. For those who cannot understand the written language and are part of the target group it would be a very large step towards available information and literature.

    However, there are a few things that make this solution hard to carry out. Firstly, Sign Language which is a

    1. @Patric Thank you soo much for this clarification and for providing this many detais. I have indeed probably under-estimated the complexities of trying to transcribe Sign Languages – in particular with regards to spatial locations. Have you been able to look at Sign Writing? And whether it is useful to do such transcriptions in a mostly lossless manner? As I stated in the blog post, I was just trying to learn more about sign languages when I stumbled upon Sign Writing and found it so compelling that I had to blog about it to try and find out more.

  21. Thank you Patric for your very clear illustration of the problem. This is the kind of explanation that has been given many times in various written articles over the years, but your comment scores high for brevity and conciseness ! This blog post by Silvia and its contributed comments should now have been picked-up by search engines, and I am sure this will benefit many readers in the future (even though less productive remarks sometimes slip-in and create noise…..just like mine ? :o) ).
    Regards, Daniel

  22. I want to confirm some aspects of writing: mainly that the written form of a language cannot convey all aspects of a language. Indeed, simply comparing the lyrics of a song to the song itself, it becomes very evident that the lyrics cannot convey the song. Writing a language – any language – requires a different way of communicating. A narrator of a play, for instance, could say “The elephant looked at the squirrel,” and the actor playing the elephant could be looking at a squirrel in a tree but to write it with the same meaning, it would need to be written, “The elephant looked at the squirrel in the tree.” Written forms of sign languages can similarly be more defined.. Signwriting can be used for more than just a transliteration of a signed language, it can be a creative medium for sign language authors.

  23. Writing sign language is no longer a problem. The elephant/squirrel examples above, from a linguistic standpoint merely say that the words have inflections. Most languages do–it’s not a barrier. Using a script like Sutton’s, we can write sign with no loss of information. This has been settled.

    Comparing such a transcription to the written form of a spoken language is largely irrelevant. For one thing writing sign is far less abstract. With speech you have to change the phonological sounds to visual units, which is arbitrary and complex. With a visual language the phonological units are already visual. Sign language written with SignWriting is processed in the brain the same way as actual conversation.

    There is no need to change the phonological units (signs), just transfer them to a surface, and omit extraneous details. Microsoft Paint will let you do ordinary handwritten ASL, although it’ll be sloppy and ugly.

    SignWriting can also be animated, which is even more like actual sign language. It’s debatable whether that can be called writing. It’s more like an avatar–a better one in my opinion, since it shows nothing except meaningful morphemes.

    The only real problems that remain to be solved concern doing all this with computers, and it’s basically the same problem presented by tone languages. Chinese has five vowels, each with one of four tones, and possibly a diacritic mark. So programmers have to choose: 11 symbols in three sets, or 40 symbols in one set. We’ve chosen separate symbols sets for the tones, which works but it’s a nuisance.

    ASL just presents the same choice way magnified: lots and lots of symbols, in a half dozen sets. Using one set of symbols and lining them up in a row is not an option since you would need a nearly infinite number of symbols. ASCII is out, and I’m not geek enough to offer intelligent alternatives, but I have faith you guys will, or are.

  24. @Skwiver It’s great to hear this confirmed. I was indeed under the impression that sign writing is basically just painting what is being signed.

    The introduction of character encodings for sign writing is an interesting challenge, though, and the Unicode encoding is rather challenging. I hope browsers and other applications will eventually support it and also find a good font to represent it. It would be rather interesting to see a browser with sign writing!

Comments are closed.