CompSci Colloquium: Tom Yeh on “Build a Better Book”

CompSci Colloquium: Tom Yeh on “Build a Better Book”

December 14, 2019 0 By Kailee Schamberger


Join me in welcoming our
own Professor Tom Yeh, [audience applauds]
for our weekly [applause drowns out voice] series. Tom did his undergraduate work
at Simon Fraser University before heading off to
MIT, where he got a PhD in 2000? Nine. Nine [laughs]. [audience laughs]
This century. This century, yes. And from MIT, he proceeded to do a post-op from the University of Maryland, before joining us in 2012 as an assistant professor. Tom’s done research in a wide
number of different areas. Since he got here he has
best-paper awards at CHI, Wits, and at other conferences,
and he’s also the recipient of the faculty affair student. Some office.
[audience laughs] Faculty of the Year
Award here at CU Boulder. I think he’s the only
member of our faculty who’s gotten that award. So today he’s gonna tell us
how to Build a Better Book. So welcome, Tom. [audience applauds] First I would like to lead
you through a brain exercise, and then, especially the
imaginative part of your brain. I want you to imagine, and you can close your
eyes, but it’s not required. I would you to imagine that
you are standing in a field, of Colorado, it’s hot, sunny, very dry. You can feel the heat, and then you see there’s a
tree, just right down there, there’s a tree. Now imagine how the tree looks like. And the tree gives you
some hope of cooling down. So you walk toward the tree, and you get into the shade. Oh, it’s really cool,
that feels much better. And you reach your hand to touch the tree, and feel the trunk. And it feels so good, you open your arms and give the tree a big hug. And you just had an experience of a tree, and that experience is the exercise of your imagination, that’s actually locked in your head. And in order to share the
experience with others, so somebody else can know,
so understand, experience it, and you might draw a picture to convey what’s in your head. To convey so somebody
else can actually see it. What can be better? Could be, you could also build a book. You can build a book
using various material, using pieces of paper, pieces of fabric, pieces of plastic, pieces of wood. And to illustrate the scene. And I would argue that it is
actually a better picture, because it is more accessible. Accessible to children who are
blind and visually impaired. And for children who are blind, they experience a tree in a
very different way than you do. They probably wouldn’t
be able to see the tree far down in the distance, and probably they have
some little bit hard time actually walking toward it, and being able to actually get to the tree, and the experience could
be more like being held in their hand, walking toward the tree. But the part about being
able to enjoy the shade, being able to touch it, being able to give the tree a big hug, it’s all accessible experience,
but they are lost in the very common, traditional
visual representation, what we know as pictures. So there’s a Chinese proverb, that I’d like to share with you, [speaks in Chinese] one generation plants the trees, [speaks in Chinese] in whose shade another generation rests. When we plant a tree,
when we write a book, when we design a product,
when we draw a picture, when we develop a system, then the action it should have some sort of impact on the younger generation, children. And it could be my own children, could be other people’s children. And sometimes the impact might
not be immediately obvious, it might take a while to materialize, such as a shade, a tree might take a year,
years, even 20 years, to develop the shade
for the next generation to rest. And I’m really curious about how people build better things
for the next generation, in other words, children. The previous generation builds something for the next generation, in a way that to make sure
those are the better things, so those impacts actually
will be positive, rather than negative. For instance, when you’re
planning how people plant a tree, how people build better
products, safer products, how people design better books, and a more accessible book, and build a better system and so on. And so I take this as a research that requires more effort,
especially for children with visual impairment. There just isn’t a lot
of research about this, particularly there’s a lot more research about just children in general. And for children with visual impairment, a lot of common sense, or intuition, or the things that we make
assumptions we made about based on our experience
with our own children, no longer held, might not be that valid. For instance, how does
a child draw a tree? How would you draw a tree? You probably would draw a tree like this. But how would a blind child draw a tree? Some blind children would draw a tree as a circle. And now you think about,
that has to do with their personal experiences, how they got to experience
a tree for the first time. Think about for a moment, why a circle could be a preferred
representation of a tree, especially when its being
drawn by a blind child. If you followed me in that brain exercise, I hope you probably might have an insight about the open arm, the
hugging experience of a tree. And so this is the research that I’ve dedicated myself and my lab, over the previous six, seven years. It’s a research project
called Build a Better Book, that’s what I wanna share with you today. And it started during my first year at CU, it started with two of
my first PhD students, Abigale and Jeeeun Kim,
Abigale Stangl and Jeeeun Kim. I’m very happy for them,
they both graduated. They signed up with the right project that would live for this long, and they managed to get
dissertations, two dissertations done out of this project. And Abby’s now at UC Austin’s post-op, and June’s at Texas A and M
as an assistant professor, both chose to go to Texas, and over time we develop partnership with Anchor Center for Blind Children, Colorado Center for the Blind, and the National Federation for the Blind. And in cooperation with Science Discovery, School of Education, SCU. Also with Mike, as a
correspondences for this work. Given it’s a research project, I would like to start
with describing the three key ingredients to get
the project started. The research question, number one, and the theory that the
research is grounded on, and also the method that we took. And in terms of the research question, what we’d like to study and find out is what’s going on in that yellow box leading to people, to build
better X for children, in this case X is children. In other words, what are the factors that might influence people, that they will build better product, better books, better
system, in this case, books. And then start to fill in
the spots, so what’s in them? Then, based on the literature, and there’s people
studying similar problems, different context, there
might be some theory about it. So, the theory that really
drove our work is this, so building a better product actually is a high level cognitive process, and it required high level thinking, with rationality and critical thinking, and Perkins and his colleague, in 1993 actually proposed a theory about, they’re trying to understand, what does it take for people to think at a high level, and to do certain things. And then they concluded that dispositions are kind of important, and they are three main dispositions, capacity, motivation, and empathy, that has connected to, and
that defined and influenced how people will actually
carry out high function, high cognitive process, and in our case we’re actually interested in building a better book. And in terms of research, we’re actually seeing whether we can test whether this theory actually works for our problem, in our context. And the method, not really be a surprise to you, is the method I grounded in HCI research. And so of course, so what is HCI research, and to me, and also pay tribute to my mentor, Ben Schneiderman, when I was in Maryland, and Krzysztof Gajos, I really see it as a three-step process. We start with understanding
how people interact with computers. And then create, now with that knowledge, we create better ways,
take better technologies, to facilitate into action. And lastly, we go about to educate people, research about those better ways. And it’s an eternity process, you go back to step one, step two,
step three, and so on. Three-step process, is to under stand people better, create better technologies, and educate people better. Step one, understand people better. Going back to the theory, initially we focused on capacity. Of the three dispositions,
if you remember, if you don’t that’s fine, we started with capacity, and that’s what we were mostly focused on during the early stage of this project. Capacity, that I can build better books. How come you’re not building better stuff? How come your stuff is
actually harming children? Because I can not. So I can, I can increase the capacity. And so, Richard Ladner is
a inspiration for me that in, it says in 2005, and he and his colleagues
showed that technology could help increase the capacity. They are working on technology for building better textbook, allow the textbook to have charts, like bar graph, they have pie chart,
they are not accessible to children and the students
with visual impairment, and so you see very simple
computer vision techniques, that are able to track all these lines, and then a map to a 2D tactile diagram that we are able to print using a this pretty cool machine
called an embosser, it can just print it. And so they showed that
was a way for technology to improve capacity, and then I actually spent a lot of time studying computer vision
as a PhD student at MIT, I thought, oh maybe that’s something where I should start at. But again, step one is
about understanding people, it’s not about just go
ahead and build technology. So we have formulated this
research question at the time, wondering about capacity, that who are the people who
are actually building all those accessible books and pictures
using different materials, about why aren’t they at capacity? What can they do? What can they not do? And also technologies,so, and Richard Ladner’s group was there studying computer vision, and I also was studying computer vision, and also at the time it was
this 3D printing technology that might be a better way of printing out tangible artifacts, so that’s the technology we
formed the question about, what are parent’s and teacher’s interests in new technologies as a
means of creating tactile learning material? And I have a couple publications, at CHI and IDC, Interaction Designed for
Children Conference at SIGCHI. And so, then our approach is
that maybe we should start it, making the example, and then we went back to the lab, and we exercised our best educated guess, and to illustrate a scene
from a children’s’ book. And to choose that children’s book, I looked at my shelf and I found I have just about
like five copies of this book. [audience laughs]
“Goodnight Moon” when people gave it to
me when we had Luke, and so then that’s when we started, so if you are familiar with the story, there’s a big green room. There are lots of objects. There’s trees, no tree, there’s a cat, there’s a bed, there’s a
cow flying over the moon. There’s a really rich
scene, yet inaccessible if you just look at a picture like this. And in this case, 3D modeling process
allowed us to pop those flat, inaccessible features up, and they got intimate and touchable, and that’s very rich, a lot
of stuff in the picture, and I found we felt quite good about it, and so we shared it with them, we took a trip to Anchor
Center for Blind Children, over time we visited like 15, that year we visited, during our first year of
study we visited 15 times. We also received inquiries
from parents and teachers from other parts of the world, and then we got into
the business of Amazon, with delivering lots of
samples over the world with tells us do they understand? Do you like it? Do you think there’s something
that can be improved, and do you want, for like the material, any material you’d like to see? And so on, so forth. And then we took data. So we got a lot of data
from that observation notes from the 15 site visits, when kids are playing with our book feedback from 28 parents and
teachers around the world, and we learned that our
picture is interesting, but didn’t quite work. Number one, it’s way too complex. That we felt pretty good
about having a lot of things on the same picture, but
it turns out it’s not. And so that’s an example of findings, about responsibility, justification doing these sort of
things, and the parents and teachers excited because
they really were feeling it’s their responsibility to help a child develop their curiosity about their environment and to hone, establish, and
redirect their sense of touch. Its justification for why does tactile learning material matter? Why can you not just
listen to audio books, how about those right? Why can’t we just listen to audio books? So this provides us justification, hey we actually do care about touch. And touch is important. And about capacity, and parents and teachers
have commented on, they find it really easy, they’ve been doing some
tactile picture book creations by hand. They found that, while
a simple vocabulary book is pretty easy, because it’s just considered
a bunch of objects, they could probably source
them from a local dollar store, buy a cow and cat, and put in the back, but nearly impossible to
create a tactile fiction story, with plot, with characters,
and with storyline, and about time. They are just, often the
school setting is too hectic, and just don’t have enough time. So this is an example
of some early findings, it also gives insight
about number one, capacity, number two, that technology’s promises to improve the capacity, so I feel that we’re ready
to move on to step two, create better technologies. Going with our theory, so
the the theory is that, perhaps with better technology, that it will feel like
from I cannot, to I can that it could be improvement in capacity. And so then we’ll look into
some work by Mitchel Resnick, that he’s an inventor of, anybody knows? Scratch? And then so that it was in
his opinion that a tool, there’s three desirable
properties of enabling technology. From you cannot to you can. Low floors, high ceilings, and wide walls. A low floor, a technology
to lower the floor, so the area of interest is easy to learn, but it was through example, it’s always easy to learn through example, you also want high ceilings. You can build from through example to something really complex, so that people would
like to invest their time to actually learn it. And also wide walls which, in addition to just serving one purpose, for instance, in our case, we care about building better books. Maybe they want to build
better something else, too, with the same tools that
will autotomize their investing and learning across market polls at locations’ domains. So low floors, high
ceilings, and wide walls. And so we started with our
low floors, automation, and my thesis is about
visual automation for search, and computer vision for
automation and search. And so automation, well I can do that, I have capacity to apply some computer vision techniques, and say maybe we can
take a textbook picture, from a biology textbook, such as a cell. There an input picture, and then with visual traction, we got, oh let’s try this,
polygon, some polygon set, and then it was this elevation map. Step two, we generate a
bunch of OpenSCAD code, a program that is for constructive solid, and that they will turn
into a 3D printed object, so we use a computer program
to write a computer program in this format. And so that will generate
a set of polygons, and a shape, for instance a polygon, which consists of a
point, bunch of points, and then we do linear extrude,
polygon is a 2D thing, kind of like this, a polygon its true is running through a space,
and it gives this a volume. So there’s a fundamental
concept of 3D printing and 3D modeling. And so in step three,
was that representation that we’re able to send in the 3D printer, and we got our very first prototype of our automated generated
tactile pictures, based on the input picture. So there was an old exploration
with Amy Hurst, at UMBC, that got a pretty good
taste of this potential of 3D printing, that we
decided we wanted to do more. And in terms of better representation, and I would like to say oh,
well you have got low floor, completely automated, but full automation means that there’s no room for anyone to inject ingenuity,
you cannot change stuff, difficult to change things. For instance, revisiting
the OpenSCAD code, so this is a scene from a children’s book called “Harold and the Purple Crayon.” Harold standing in front a tree, and behind the tree, there’s a monster. And what’s on the tree? On the tree, there are apples on the tree. And what time of day it is? In the evening, because
there’s a moon in the sky. But now my verbal description is reflected in the code, and the code you look at,
the code is linear extrude. Module two. Module two is a bunch of polygons that describe the shape of a tree, and so on, so it’s very
difficult to make some changes, for instance, some of
the parents were saying, well there’s a bit too many apples, can I reduce the number of apples? Where can we go in the code to
change the number of apples? The apple is a bit too thin, we want to increase the thickness so children can just touch. Where in the code can we
go in there and change it? So even though with automated process, could allow us to write
this code very quickly, but there’s a bit too
low a representation, perhaps there’s something
that’s more intermediate, to make a high level a little bit. So our approach is to
look into web programing for inspiration. Web programming, there’s a
very popular markup language, that are really good
for describing content. And when 3D modeling just came about, most publications about functional part, car components, for jet engines, and so there’s a very functional property before our purpose, accessible books. Content, is more important than function. For content, we thought perhaps we could design a language that will
look very similar to HTML, CSS, and JavaScript,
like writing web content, and we were able to
describe the same scene that I described to you using
a markup language like this. If you read the code, tree,
as a column of apple and trunk of the tree, an apple could
generate a sheet of apple. She might ask you, I don’t like 10 apples, I like to only five apples. Do you know where you should
go and change the code? Then you will be able
to go to repeat, and 10, and to just change the number. And so a machine can generate this code, and so is humans, so a human
can also write this code, or go modify this code. And so, in 2016, I gave a talk, when this language was
just in very early stage, and then I attempted to give
a live demo of the prototype and with spectacular failure. [audience laughs]
And so like, learned, and we spent many
years to improve this system, so I want to make a second attempt, [audience laughs] to show a live demo of our system. So we start with a very simple online game of how to craft a name tag, so it’s actually on our website, and so it goes through a process of adding complexities to get a name tag. A name tag is kind of an intro to the, so this is a name tag. And I would like to
change my name to Tom Yeh, so I can go and change it right there. And since we are really going
by making things better, better in terms of accessibility, so you now design a language with elevated accessible features, to the first-class language feature. For instance, to change this to braille, so braille text can become accessible, what does it take, using our tool, to change it to braille? So do we have to model different texts, actually lay them out? We don’t have to. So all I have to do is change
the division text to braille, and if I spelled it correctly, and then we’ll get a braille name tag, and then, so you can
still go read this code understanding why this has a structure that looks like that. Because it has a cube, and also have a cylinder, half cylinder, it got cropped like this, using a tag, using a transformation operator, so it was actually a cube and cylinder with some transformation, if I were gonna transfer it back, crop half in each direction. So I wouldn’t pretend
that it is a language you could just teach anyone
who walked into this room, well I could teach it to anyone
who walked into this room, [audience laughs]
but, then… So I’m pretty surprised that the demo actually did not crash.
[audience laughs] And so it gives me some
confidence to look at another, more complex example of this picture that we made. So this is a picture based on the book, “Penguin and his Big Adventure,” “Penguin and his Big Adventure.” and so this is, like,
does it look like this? Kind of matches this. So you look at the code, this is where the penguin, and the map I’ve laid
out makes it shorter. And so if I ask a number,
Gene, can you give me a number to define, to maybe I want to
rotate the map a little bit to a deeper rotation. Now it’s rotated by 90, or 70 degrees, so it will give the number. Can you give me a number
between zero to 360? GENE: 180 180. And so we’ll change to 180, and so we can immediately see that reflect to get updated, and you can also. What if I just add, I would
like to have two maps, so the penguin was really, really lost, he need two map,
[audience laughs] in order for the story to work, and then we’ll just add it
to where it’s supposed to be. So. And I’ve succeeded in
demonstrating two example live, and I encourage you to go to the website to play with it if you want to. And so, live demo one, live demo two. And so that was pretty easy, I hope that you appreciated how
easy it is to read the code, yet being able to modify it as well, compared to the baseline system, OpenSCAD, as it’s too low level. And can we build something
more, like, complex? And it turns out you can build
computational structures, like fractal, as a
computer science professor I’m quite proud that we
can build fractal structure using this simple HTML-like language. And for wide walls, what can we do other
than laying out a book, accessible book? We can also model buildings,
various buildings, they’re all 3D printable
and very colorful. CSS turns out to be really
powerful for adding color to it. And then the question is that, you might know 3D printing
only takes one to two colors, what is the meaning for this color? So we have some idea
about we can eventually make color to different
pictures, automatically. I do have another paper
that I could talk about in the future about how we automatically generate pictures, then add the color. And I’m also quite glad to see examples of helping children with other impairment, motor impairment, for instance. Children with, maybe they have, for instance, difficult to hold a cup, we can model a cup like this, a cup holder, and there’s parametric, so we can change the sizes very easily to fit these people’s, a
child’s hand, and so on. So this is a example that
we show to demonstrate that this tool, you can lower a floor to achieve wide walls and high ceiling. And step three, with this tool, how about we go and educate people better, tell people, educate people, teach people the process of how to design, and make, and build better books. We got our work hours,
and we would like to run this workshop, who would like to come, we will teach you, and
pick a children’s book, and then we will try to model the scenes, and using 3D printers, we will teach you through the involving
software, they will have various software you can choose from, TinkerCAD, SketchUp, also our tool, and so I get really good responses. And what surprised us is that, so as we started this work, we had a lot of conversations with TVI, Teachers for Visual
Impairment, also parents. And then we did not expect
that by doing this workshop, there’s a diversity of people, the diversity represented
by the people who attended the workshop, this sort of surprised us, we see librarians, we saw
the makers, we saw engineers, we saw just volunteers,
and designers, and so on. We thought, well we do have a set of tool we can choose from, we did manage to apply capacity, a tool-centric
capacity approach. Give your tool low floors,
high ceilings, and wide walls. But then it doesn’t
quite explain why people who have nothing to do with blind children might want to come to do it. And so, why do they come? What motivated them? And we realized we had to do a study, actually research,
understand their motivation. And so that was the second disposition, around the time 2014, ’15, that’s when we started
to pay more attention. And that was coincided with, I had the fortune to collaborate with Jenny Preece and Mary Lou Maher and actually found a project on science and crowd-sourced design, and there we really
studied about motivation. There, I got some
education about motivation and trying to bring my own work and to Build a Better Book. And so, I realized we had
to go back to step one, understand people, and we had to form a new
research question about, number one, stakeholders. What are different, distinct
stakeholders’ groups, are interested in this. And why motivation for the capacity? And so we started with, we had the opportunity to
study six distinct groups, six different workshops. Question is, what is their motivation? Data found workshop, and 67 participants, who have field notes, and design products, and we did interviews, and this is the motivation that we found. The TVIs, not too surprisingly, T stands for teacher,
V-I, visually impaired, teachers care about education. Reason why they would like to come, because they want to learn how
to educate their kids better. And access librarians, they
are talking to a library, they send out audiobooks. Their motivation is, hey maybe I… They relate to the transcription process of turning something visual to a different modality, they’ve been working on audio modality, so they can see a similar process might be important for
transcribing to a tactile modality. And children’s libraries care about, as we illustrate a story, do we stay true to author’s intent? Just all of those things. And engineering just like technology, oh it’s a new way to apply
3D printers, it’s cool. And designers like to play
with the 3D modeling software. And it’s kind of a different motivation. And then we start going back to step two, create technology, can
we react to those finding in our technology creation? And so we decided to put together an online collaboration platform, and so this is what that, we have a website that
feature all our models, so this is the model, it’s another example I choose to show this example, another book, “Noah’s Ark,” that did not have any copyright
issue when we actually did this book, whereas we had some other
volume was locked earlier, but in this case of “Noah’s Ark” and it was different
pages, and we sent out quite a few copies to different people, and so, with this collaborative platform, designers, who could come in to design, actually designed this,
the first pass to design, illustrate a scene with a tangible output, and
a TVI came in to help us, commenting on the educational value. They were the one that
were able to tell us, hey you need to introduce a caption. So, this is, oh it’s not a caption, margin, so this is like a key, where there’s all of the
different recurring component, and also help us to
simplify some of the text. And the librarian, children’s librarian, had came in to kind of give a feedback on the story, was it
representative of the story, in some cases, how could we
remove religious references, could be used in the
mainstream classroom and so on. There’s some issues we never
imagined we would encounter. And so then we have engineer, that came in to download this model, help us 3D print them, they
just like to print stuff, and it’s not trivial to 3D print them, they like to solve the typical problem. And volunteer helped us distribute them, and to test them, and so on. So, there was an example
of our online platform, that catered, as a
reaction to the diversity, a diverse range of motivation. So, again, educate people better. Now we have this platform which is a tool. And so we have two
people who have received previous education in my lab working on this project, and it’s Jeeeun and Abby, and Jeeeun represented a big,
more like engineering side, she’s engineering by heart. And Abby’s the designer. So when you think about that, when did they actually become designer or become engineer? Did they, somewhere in
their life trajectory, something might have happened
that led them to pick a particular orientation. And so in 2014, I received
email from Stacey, who was there, but just left
because of something else. So she emailed me, hey I
read about your project in the news, so you are
interesting on the workshop, to share about your work
with a bunch of teenagers, and teach them about your process, and I thought, and my students thought, oh it would be pretty cool, maybe we could do that. And so, and we’ll have the workshop in one of the libraries here, and teens, a lot of teenager students came, we teach them how to design, for accessibility,
model based on the book, and then how to name them,
you can use clay and Play-Doh to model before you jump on
the 3D modeling software, to model them, and so on. And so we found that there are two things, capacity and motivation, if you remember up to this point, and we found this learning
experience has provided motivation for teens to be super engaged, they spent tons of time, to provide multiple crucial entry points, for instance, they could come in because they like the story part,
they like to write stories. And some students they
really like the art and craft component of this. Some students came in like,
oh I really like to learn 3D printing, it’s pretty cool, and some students just
liked to help people with visual impairment, I thought it was really meaningful things. So multiple entry point maybe helped to cater to their motivation. And the capacity we saw that
they managed to develop, 3D printing, 3D modeling,
technical skill as well. And that we also saw something
that couldn’t quite capture that capacity and motivation. They said something like, oh
we never thought about seeing, and experiencing from
a different perspective using modality. And why we do not see a
lot, you say, we don’t see kids saying something, oh I
feel sorry for blind children. Oh I feel so bad for them, we don’t really hear
that, we hear more about it opened up, they got
opened up thinking about there’s a different way
of seeing the world. A different way of experiencing the world. And for instance, if our
goal is to get people to draw trees, as many trees as possible, capacity, with capacity, with better tool, we can make people say,
so I can draw the tree. And also with the right motivation, right who, I want to draw the tree, but, there’ll be a issue that
this is the only kind of tree everybody draw, if this
is the only kind of tree they’ve drawn. So we realized that we
need these sort of trees get drawn as well. Our kids need to learn to
see from their perspective, it’s a different perspective,
as well as experiences. And then we realized that
is what empathy means. So empathy became the last
missing piece of seeing their different work. Even though at the beginning I showed you there were three different dispositions, to be honest with you,
I started with a very system and tool oriented mindset, where I cared about capacity,
building better tools. Only after, and over time,
learning from my mistake, and then, that we’ll
feel really good about them having them having
identified empathy, and in a lot of conversation,
we got a lot of help from Justin Hess, and he is
one of the leading scholar in the role of empathy
in engineering education. He’s the one who showed me empathy is not limited to feeling, it’s more than, not just about the affective component, about oh, you step on something sharp, and you’re painful, I
feel bad about about it. It’s more about, it’s also
a cognitive aspect of it, being able to take perspective, for instance, there is a
different way of experiencing the tree, taking others’ perspective. So, going back, I thought maybe
we should really understand that phenomenon, how
empathy can be developed as one of the factors, that influencing building
a better product. And then so we went from a
couple workshop, locally, and we expanded our cooperation, we sent Discovery to include Bridget, that’s in the audience, from
our School of Education, and she has expertise
in children literacy, and really gives us a
lot of literacy chop, and with Ann Cunningham, who
has been a tactile artist, teaching blind people how to make and appreciate art for many years. And we received additional
funding from the ITEST Program, in partnership with various libraries, and it is the Build a Better Book Project, and we’re trying to design a team-based experiences, based on our understanding of the three dispositions,
empathy, motivation, and capacity. And so activities that are trying to connect to one of
the three dispositions. And then so that will give
us the opportunity to study how the three things interplay
in the way that influences future engineers, future
scientists, when they become actual scientists and
engineers down the road, maybe they will remember
to carry that empathy, motivation, and capacity with them. And since I gave a talk in
this arena three years ago, so this Build a Better
Book has grown nationwide, to 12 states, more than 12
states, and has 2,000 youths, middle school and high
school student has taken, participated in our
workshop, could go into the process of building a better book, learning about a design
aspect, developing the way to see from different perspective, and then I also have the technical side of 3D printing and learning. And so books also are the
opportunity to exhibit, for instance, “Harold
and the Purple Crayon” book that I showed you
earlier, has been shown in our Gemmill Library right here, our “Noah’s Ark” book has
been different places, for instance in Peanas Con Piezas, Italy, there is our “Naoh’s
Ark” book right there, and we have the “Penguin’s
Big Adventure” book that I showed you earlier,
has been displayed at the Smithsonian
Design Museum last year. And we just have a lot
of data from 12 states, 2,000 youths, whose interviews, surveys, and design products, we’re
in the middle of like, processing those data, trying
to understand what’s going on. And we like to see signs
that capacity, and empathy, and motivation. So there’s an early example that we found from students’ quotes. For instance, in this
student, Atzin Palacios Luna, [speaks quickly] for their
popular library, she said, “I never even thought about this before, “but now I’m thinking about
all the ways that we could,” or all the ways we couldn’t before, “improve things for people
who have different abilities.” So that seems to suggest that
there’s some gain in capacity, and she believes she can actually do more, whereas she couldn’t do before. And, “Something that’s accessible “isn’t just for blind people, “but for everyone, race, gender, “everyone has to be able to enjoy,” there’s a clear evidence of empathy. And, “We feel like we
are part of the project “that’s helping our community.” The helping community is
clear that for everything, that is the motivation,
and she’s, people have a volunteer mindset, in this case. And some other quotes I’ve
heard about some creativities, also shows some really signs of motivation based on her interest in
art and craft and so on. So, going forward, we’ll
continue this three-step HCI designed three-step process of understanding people better, creating better technologies, and educating people better. And so just to give you a
taste of where we are gonna go, so bringing back to the
theories that would, when we care about how
people build better books, and we believe, or we found, as some of the evidence suggests, that the theory of disposition really work in this context as well, that empathy, and
capacity, and motivation. And I have argued that, in
terms of responsibility, we would like to help kids. But so far we haven’t
really had a chance to really measure the final outcome. Do children really get
better off as a result? Where there’s question marks. So we gather indirect evidence from parents and teachers,
say oh, our kids read more. And see we never had a chance, nor did we feel comfortable
of knocking on people’s door, can we study your kids right away. So we’ve been kind of,
wait and wait and wait, and building this trust and so on. Over the many years we feel like our project has reached maturity, that it’s earned enough
trust that maybe we could do something about children’s outcome. And one other outcome that,
based on the literature, that’s about brain development, exercise, the brain exercise you had right in the beginning of my talk, that has stimulated a
certain part of your brain. You don’t even know, but
even the this imagination that I could actually trigger
that, more can be done, more is possible if that is actually a tangible experience of touch. And so we thought maybe what
we would like to make sure would be really study the differences that are in the brain development. And so going back to step one, understand people better, we’re designing, and getting all the approval for, studying a tactile picture’s effect using functional
near-infrared spectroscopy, it’s a way they use LED light
to make your blood flow, so we like to collect evidence that some of the brain area
actually get activated when kids get to touch tangible material early in their life. And statistics show that
maturity of blind kids who are born with congenital blindness, that didn’t get to touch
their first graphic until they are in the middle school, or high school, they just, like, listen to it, or audio. So, we can see evidence that they could actually support the
needs of early exposure of tactile stimuli, we hope, so that could be beneficial
for the community. And then so that is also
a reaction to, kind of, feedback from our community, really want it to have it
be more scientific evidence to support, so they could
make a better claim to, one, to get the government
to support more a position of tactile literature in the early child development stage. And I have a new student, Nicole Johnson, joining all that, who’s also 18, and she worked at the office creating tens of thousands tactile
graphics all throughout her undergrad as a kind of part-time job. But then she’s a full-time student, and also working as a part-time tactile, so with her experiences,
and we finally found a NSF fellowship, and in collaboration
of University of Denver Family and Child Neuroscience Lab, we hope we could be up to
actually get the evidence once and for all. And so I intend to create
better technologies, and so we’re going back to
the collaborative platform, this is actually anchored
on this really large online repository of 3D models,
people can come in and make changes, and collaboratively work on the 3D models. So, I will have an opportunity to join a team led by Benil Starly, and he is a professor of
mechanical engineering, so there are actually a
lot of similar efforts happening other spaces, for instance, mechanical engineering,
they are trying to, oh maybe we should put
together a 3D repository model. So it might be possible with 3D models. And then we have medicine,
or we have education, all trying to do similar things. Perhaps we should come together and talk and to join our efforts. So we were selected as part of
NSF Convergence Accelerator, trying to put together
a phase two proposal to really bring this into reality. And, in terms of better, so
I have Build a Better Book, is focusing on better
as in more accessible, so outside of this Build a Better project, all that also been looking
at better as in more ethical. And so my student Layne Hubbard, who is there, and Xu Han,
who is away at a workshop, they have been working
on a loner dimension of how to build better
robots, X equals robot, who interact with children. And Layne’s example, and she’s
interested in social robot, it talks to the kids, and Xu’s example is a
special voice assistant, they talk to the kids. And it’s partnership was
National Head Start Association, and funded by Piton
Foundation, NIH, and NSF. And so, up to this point, I
will share with you how my lab takes the HCI research
method of understanding people better, creating
better technologies, and educating people better, and repeat. It’s an eternity process. And so I’ll share with
you a personal journey. As we started with a
very system-tool oriented researcher, that cared about
building tool, high ceiling, wide walls, and lower floors, and he eventually, over time, discovered the importance and roles
of motivation and empathy, and then we’re trying to
imbed them into my framework that support my lab’s research
and continuing to do so. For Build a Better
product, software, system, or you name it, and for children. So, going back to the Chinese proverb about planting trees, when is the best time to plant trees? Is it during the
summertime, or wintertime, or during the spring, or during the fall? 20 years ago, yes. Best time is 20 years ago. What is the second best time? The second best time is actually now. So, the tree that we planted,
the software that we designed, the system we developed,
the book we’ve written, the picture we’ve drawn, these will be impact on our children, and some impact might take
20 years to materialize. And in terms of public
outreach, if you have a chance to talk about your work,
that might influence children also in another way, in that 20 years from
now, perhaps that kid that you talked to in a classroom about technology, not just about capacity, but also about motivation, empathy. That kid, 20 years from now,
might be in the position to build better things for the world. And so, with that I would
like to take questions about the Build a Better Book Project, and thank you for listening. [audience applauds] Yes, Young. YOUNG: Yes, all right, so thanks for this talk,
it was really great, seven years is a lifetime in computing, and so I was wondering
if you could talk about, sort of this data, you know
you’re looking at these questions around technology,
and around design tools, and I wonder if you could
tell us about, like, how things are now in 2019, versus earlier in the project. Is this getting easier, or is
this getting more difficult? So, seven years ago there
was a lot of promises, when you heard about 3D
printers, and then that there’s a promising vision about
how a 3D printer will be in everybody’s living room, and we know this has not happened yet, just like how self-driving
car has not really happened this year, and so that has become, that led to adjustment of our work, to look for something
a bit more permanent, something that’s not
depending on a technology of the trained, and something a bit more intrinsic and fundamental,
that’s why we’re starting to incorporate, perhaps we can
work on the motivation level, working at the empathy level,
that would be the properties that could have more that indirect impact of what we wish to achieve, for instance, we would like
to plant a lot of trees. Suppose… My goal is that we hope
there will be a lot of better products built, similar
to a lot of better trees. So I could go ahead planting
a lot of trees myself, get my student to do it,
but more respectively, is that how can I cultivate
the next generation of UB engineers, or UB scientists, so they will become tree
planters one day themselves. Basically increase the
population of people who actually care, so I
think I kind of answered your question directly,
so I think maybe chasing the technology gap, might not be as productive as trying to keep up with technology might not be as productive, so, for instance, try to,
think learning right now, it’s being covered in books, [speaks quickly] and so we’d
like to maybe have, of course, it’s still important, so we
still care about capacity, but we really need to
balance it with empathy and motivation as well. Thank you for the question. Hi Tammy. TAMMY: So, Tom, what have, I have one early on [speaks quietly]. I’m hearing a lot about [speaks quietly], so one of the issues is
that it’s not just the book. It’s the interactive reading
skills between parents and the child, and that’s
actually crucial for vocabulary. Have you thought at all
about [speaks quietly]? So the question is that, interactive experiences,
do we design books so that books could become
like iPads, like become like reading nannies? TAMMY: No, no, no. Psychology studies.
Psychology studies? TAMMY: Needs the interactions between parent and child.
So you mentioned at the end we will be working together, we have a paper on– MAN IN AUDIENCE: Can
you repeat the question? The question is that have
we looked at the effect of co-reading, how the books
actually support parents and child reading together? And so– TAMMY: How does this process change? How the process changes facilities. Like change.
So let me mention, Eliana Colunga is a faculty in the Institute of Cognitive Science also of psychology, also computer science, and we actually worked
together, and in one study that she used one of our books
in her study that delivered material to children, zero
through three-years-old, and then also another
condition that delivered just regular books with a lot of pictures, and with words, and so one condition is tangible pictures with words. And we studied in the
vocabulary acquisition and the way that the
things are being said, we found in one condition that kids make, the number of words being
spoken is about the same, but there’s the essence, the kind of word and also the pattern of
word that’s being said is slightly different. I wish I am not misquoting the data, but one of the things I remember is that kids tend to repeating more about their referencing the object in the tangible condition, whereas the kids tend to
talk about something else other than the object in the
traditional medium condition. So that’s one of the things
we’d like to study more, and hopefully next time
I share some results, in fact Eliana has formally
joined our Build a Book team, we have a pending proposal that hopefully will get funded and we can study for next five years. Aaron? AARON: I’m interested
in the [speaks quietly] for converting the buildings. Seems like, there’s a lot of
stuff that’s being hidden, in terms of laying out where things go and in order to provide
us someone in your place, you have to hide the tips. So I’m wondering, how robust is that system? How easy is it for someone
who wants to design a scene to sort of run into the walls or run into the rails of this thing, and start to kind of
struggle with the fact that the algorithms underneath, or algorithms cannot be [speaks quietly]. Replacing things and [speaks quietly]. Right, so, Aaron’s question
is that how is the process like for somebody to
design a scene from scratch using our tool. So our finding is still
a programming tool, so people with basic programming skills, programming skills are
still a pre-requisite of being able to use the tool to design something from scratch, it’s similar to HTML,
and the web programmer. There are two different kinds,
there are web programmer who develop program from scratch, and there is a web designer
who can take a web program and make a few changes to modify. So we found a significant
phenomenon in our work as well, with two kind of authors, one is that you design
a scene from scratch, two is they come in and hey,
I want to move the penguin to the left a little bit
and make the penguin larger. So our code program
interface is simple enough to support a second group of users who might not have ability to
build something from scratch by writing code, but they
could understand the code, read the code, and bring
their design mindset into it, and be able to
go to the right place to make changes. Does that answer your question? But so, it’s definitely not like a drag and drop interface, you can just go in and
just make things happen. But it does provide, like, a medial ground of customization, in
terms of customization, our program language
strikes the right balance of being able to provide parameters, so you can turn everything to parameters, turn the number of apples,
the thickness of the apple, the size of the apple,
the shape of the apple, all become parametrize-able, so if you’re interested I can
show you a live demo offline about how it can be done. Time for one more. [Man Two In Audience] So I
was really intrigued by this notion of like, making things better, but since I’m not that much in the area, what I’m more uncertain
of is how you actually quantify these things are better, so maybe with CraftML you
might quantify it one way, when you’re building these books for kids you might quantify it in another and it seems like there’s
probably like, best practices like interviews, and case
studies and whatever else, but with kids it seems
a little challenging because they are probably unreliable or hard to get information out of, you might be exposing
them to new technologies that no one else has ever,
kind of, measured before. So are there some kind of
challenges in quantifying this, or lessons learned from quantifying, or new ways that you kind of– Right so, the question is
how can we quantify better, more accessible? And so one thing we
would now have to clarify we would have two kind
of interconnected goals, the primary goal is how
can we enable people to be able to build things better? So from the baselines
that don’t quite care, they don’t think about disability
when they design products, when they write their web pages too, when they actually do care about it, nowhere to add old text and so on. So that is our primary goal. Secondary goal is to be
the architect themselves. So I think your question is more about the architect-self, right? [Man Two In Audience]
So you could even ask the first part, how do
you know they care more now that they’ve done it, but I’m sure even just the second part, how would you– So we have codings we use
to look at the product that are actually produced
by the participant in our workshop, and then so,
the pre-test and post-test, that will look at their
product, whether they’re really adding signal or sign that showed design, something that we take
into consideration others’ perspective, it’s not just
designing for themself. And then in the post we
look for those signs, for instance, are they
thinking about adding braille, are they thinking about adding colors? And because colors are
important for the parents who might not have visual impairment, ’cause they still want to
be able to read the books. And so kind of, in
response to your question, about to also learn about the importance of the picture has to be
pretty as well, visually, in addition to tangible properties. HOST: Thanks, Tom. Thank you!
[audience applauds] HOST: There are more cookies left.