EFTA02702508

"in the control room of the banquet" Richard P. Gabriel IBM Research deep in the dark— awake In the dark— Abstract the power of snow the edge of the water can walking in the deepness spread in your presence The Turing Test, AI, programming, creativity, and mystery. Categories and Subject Descriptors D.2.10 [Software Engi- scrupulous in the twilight— time of life issue: neering, Designl; D.3.0 [Programming Languages, General]; the price of gold chases a bird of prey pulls up the way of the world in power out of the way into the palm 1.2.7 (Artificial Intelligence, Natural Language Processing General Terms Artificial Intelligence, Natural Language Processing those four haiku are good—not just human-like, but good Keywords Science; programming; natural language generation poetry with two of them close to being exceptional. I worked on the system more over the next six months, broadening and expanding the template language to give more control I am writing this essay because I am puzzled. In July 2015 to InkWell, deepening its understanding of language and I took eighteen haiku-like poems to a writers' conference the music of language, and adding more observations Ink- and presented them as my own work. In reality, a program Well could make of its drafts and along with them more I created called "Inkwell" wrote them, and I intended to revisions. Over those months InkWell produced a lot more execute a variant of the Thring Test. The results were better haiku, and I selected fourteen of them to add to the above than I had hoped for in verifyingInkWell as a good poet, but four for a Turing Test. I was left with disquiet about what the experience meant for understanding the Turing Test, programming, the artificial intelligence research program, and what consciousness is. In October 1950, the British journal Mind published an essay by Alan M. Turing titled, "Computing Machinery and Intelligence," in which Turing proposed an operational In the Winter of2014 I programmed my English language definition for "intelligence" [2]. This definition would come revision system [1] to write haiku—just to see whether it to be called "the Turing Test." Turing himself called it "the could do so plausibly. I let the system run overnight generat- imitation game," in which a questioner separated from two ing about 2000 haiku. Among them were the four at the top contestants would submit questions to those contestants. of the next column. They stopped me in my tracks because read their replies, and ultimately choose one as human and the quick program I wrote was not of the monkeys typing the other as machine. at keyboards variety—instead I programmed the system to Interestingly, Thring introduced this game with a similar determine its own topic and then write coherently about it but different one in which the interrogator was to attempt to using a few dozen haiku templates as starting points. And determine the gender of two contestants, one male the other female. Interesting because Turing was homosexual and Permisslon to make digital or hard copies of all or part of this work (or petsonal or perhaps accustomed to such an imitation game. But in the classroom use is granted without fee provided that copies art not made or distrib• matter of intelligence, such an operational definition made toed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others some sense. Thring wrote the following: than the author must be honored. Abstracting with credit is permitted. To con otherwise. or republish. to post on servers or to redistribute to lists, requires prior May not machines carry out something which ought specific permission andfor a fee. Request permissions from permissionseacm.org, to be described as thinking but which is very different ChswardS14. October 20-2a.201OrPorthodrOvegotRUSA. from what a man does? This objection is a very strong Consightithekibyths es.h.,.. ..sou.., lh...,.dtu ACM. A04970-1-1503-25tertrlittInAl0. one, but at least we can say that if nevertheless, a ma- hurifdx.dotorgil0.1145r266ttifir2661I55- EFTA_R1_02074939 EFTA02702508 chine can be constructed to play the imitation game rhythmic: "compact fluid energy: 'wry and elliptical: and satisfactorily, we need not be troubled by this objection. "whimsical elegance." -Turing, Computing Machinery andIntelligence, 1950 I didn't consider this as evidence that InkWell passed the Turing Test. it was the first day of the conference and people The Turing Test is at (or near) the heart of the research were jet-lagged and not entirely prepared for the rigors of program called artificial intelligence. In my youth I described the conference; and my reading took about four minutes of artificial intelligence research as an exercise in trying to write an allocated ten. Most writers stretched their reading time at programs one doesn't know how to write—at least for engi- least a little, thus my short reading of short pieces stood out neering-type Al research. Some of us generalized this to the as energetic and sudden. I was still uncertain whether the idea of "exploratory programming: in which one had a gen- hint was noticed—the hint contained in the title and abstract eral sense of what the program should do, and only a partially for my lecture. formulated idea of how to achieve it. In recent years the idea of what programming is has drifted away from including this view toward specifiable. routine. infrastructurish programs Here is what a haiku is: and systems. i've heard this referred to as the static-verse. in a famous debate / discussion, Michael Polanyi and Alan Turing A haiku in English is a very short poem in the Eng- talked about whether the mind / the brain was unspecifiable lish language,following to a greater or lesser extent the or merely not-yet specified [3]. And what would an incorrect form and style of the Japanese haiku. A typical haiku but Thring-Test-passing system be? is a three-line, quirky observation about a fleeting mo- In his discussion of how the imitation game might go in ment involving nature. the computer version, Thring wrote this as the first example -Wikipedia 141 of a question in the game: For many, the quintessential haiku poet is Basho in the 17" Q: Please write inc a sonnet on the subject ofthe Forth century; an exemplar ofhis haiku is the following 151: Bridge. A: Count me out on this one. I never could writepoetry. On a withered branch A crow has alighted: -Turing, Computing Machinery andIntelligence, 1950 Nightfall in autumn. The nature of haiku is complex and has changed over the In the Summer of20151attended the Warren Wilson Alum- centuries—time and place are still essential: counting on is ni Writing Conference, which is held annually for graduates not (some mistakenly conflate on and syllables). of the Warren Wilson MFA program. i am such a graduate. in poetry. The conference was held at Lewis & Clark College in Portland, Oregon. The week was unusually hot and humid InkWell is a small program (about 45,000 lines ofCommon for Portland, and this physical difficulty was reflected in an Lisp code), but it has a lot of data (about 15gb when all the edginess to the conference. My plan was threefold: read the dictionaries, databases, and tables are loaded). Turing wrote eighteen haiku aloud on the first night of the conference to all "I should be surprised if more than 10' [binary digits] was re- attendees: participate a few days later in a writers' workshop quired for satisfactory playing of the imitation game." InkWell as the writer of those eighteen haiku; and on the final day of has more than 10". InkWell "knows" a lot about words, per- the conference, give a lecture entitled "is My Program a Better sonality, sentiment, word noise, rhythm, connotations, and Writer than You?" The abstract for that lecture was as follows: writing.Its vocabulary is probably more than five times larger than yours, gentle reader. The core engine works by taking a I've been working on a program that thinks like a template in a domain-specific writing language along with a poet andproduces nice stuff. I'llshow you how it works set of about fifty writing-related parameters and constraints, and why it's not like the kinds ofprogmms that do your a description ofa writer to imitate, and other hints, and com- banking or predict the weather. But everythingI'll talk piles all that into an optimization problem which the writing about is really about writing. engine works to find a good way to express what the template and constraints specify. Although some parts ofInkWell were i read on Sunday night. After the reading a few of the writers created through machine learning, the overall approach is came up to me and commented on my reading. My reading optimization, not machine-learned transformations. was short because the poems were short—and the attendees The primary research question is to try to isolate and codify knew i didn't normally write haiku. Their comments includ- what separates information transfer from beautiful writing. ed these: "terse condensations: "evocative: "took the top of Here is an example of information transfer: my head or "funny and profound: "natural, personal, and 2 EFTA_R1_02074940 EFTA02702509 The summer homes on LongIsland were closed. Tonight To the objection that the nervous system is continuous and I watched aferry begin its crossing to Connecticut. The digital computers discrete. Turing remarks that an interro- moon was rising, andas it rose I thought about how the gator couldn't take advantage of this because the right sort houses are not part of the natural world and what the of answer could be made anyhow by the remote machine. island looked like to early Dutch sailors coming upon To the objection that humans have informal behavior, 'lur- it—like something new. ing remarks that a machine can easily have laws of behavior, which is really what people have. To the objection of ESP (!!!), Turing admits fear but concludes that a telepathy-proofroom And here is how F. Scott Fitzgerald wrote "the same thing" in will solve that problem. `The Great Gatsby" (6J: This leave the objections of consciousness and originality. These are subtle aspects of thinking, and though Turning ad- Most ofthe bigshoreplaces were closednow and there dressed them, I will address them in the context ofInkWell, were hardly anylights except the shadowy, movingglow which answers the objections well, and in unexpected ways. ofa ferryboat across the Sound. And as the moon rose higher the inessential houses began to melt away until gradually I became aware of the old island here that The Turing Test is about an interrogator and two subjects: flowered once for Dutch sailors' eyes—a fresh, green a person and a machine. The test is described as ifit happens breast of the new world. once, and all the people—and the machine—are ordinary. It -Fitzgerald, The Great Gatsby doesn't look at extraordinary talents, special skills, and ex- pertise; and the test is presented so that clever avoidance of There is more going on in this version. But what is it? Images, the question is within the rules. mood, a "vivid and continuous dream" as John Gardner would Can the interrogator tell the machine and person apart? put it (7j. The first gives us the bits; this one gives us the story. Here is your chance to be an interrogator. At the end of this The haiku writer is a driver program that produces the essay in the Appendix is a page called "Thirty-two Haiku." It template and constraints the core engine works from. contains the eighteen haiku I took to the writers' conference. plus fourteen more. Four of the ones InkWell wrote were re- vealed on the first page of this essay, so of the twenty-eight I propose to consider the question, "Can machines others on that page of thirty-two, half were written by Ink- think?' Well and the other half by Ban'ya Natsuishi, Annie Bachini, -Turing, Computing Machinery andIntelligence, 1950 and John Ashbery. Have fun deciding which. But the task I just set demonstrates an important problem Turing begins his essay thus. A large question. And largely with the original 'flaring Test viewed sixty-five years after its his essay aims to explore it. The word "think" is the disturb- conception: being unable to distinguish a computer from a ing part for many—at least when Turing wrote this essay. person once is not always enough. No one in their right mind Thinking seems like something only humans can do. But and being honest could argue that it's clear which fourteen even sixty-five years later the full meaning of the word can are which—all the poems seem like they were written by a confuse us. Is thinking puzzle solving, creativity, empathy. person or by people. The question is whether there is a dis- wonder, faith, curiosity, ideas, reasoning, reflection, recol- tinction to be reasonably observed between those written by lection, intention, attention, care, imagining, consciousness, a poet and those not. A single non-expert interrogator could language, metaphor, judgment—all of these? Some of these? easily mistake InklA'ell for a person. Multiple sessions and Turing addresses objections to the idea that machines could multiple interrogators are needed. think, and offers some suggestions on how to approach achiev- The issue of the proper interrogator has been addressed in ing mechanical thought. the past by pitting an expert human against a computer. If To the objection that only souls can think, Turing asks the computer can "defeat" that expert, it has some human- whether God lacks the power the grant souls to machines. like chops. One of the first examples was the checkers play- To the objection "I hope not," Baring turns away—though ing program written by Arthur Samuel in the late 1950s today thinkers like Hawking, Musk, and Gates embrace the i8j. It was also one of the first programs to improve itself fear. To the objection that Godel took care of that for us. TUr- through machine learning—although a very simple type. ing points out that G0del took care of only a specific form The program was able to play advanced amateurs quite well. ofincompleteness, which might not be relevant, and besides. In 1995, a checkers playing program called "Chinook" won a why are humans immune from it? To the objection that ma- special checkers championship called "Man-Machine World chines have certain disabilities ("can't feel, can't fall in love. Championship." (Chinook won the year before, but its creator, can't make mistakes..."). 'flaring generally derides the idea Marion Tinsley. withdrew from the competition because of as not entirely relevant or as not something to be proud of. pancreatic cancer.) Chinook had no machine learned aspects. 3 EFTA_R1_02074941 EFTA02702510 In 1997, Garry Kasparov lost to Deep Blue at chess—Kasp- standards. Moreover, each of these tests was subject to the arov was the reigning world chess champion. In one pivotal Moravec paradox, which states that high-level performance game Kasparov remarked on the "superior intelligence" of the on "intelligent" problems—playing chess or other games, machine during the first game (won by Kasparov) by avoiding simulating abstract thought, theorem proving, and skill in a dangerous position that had short-term advantages; some arenas requiring expertise—is relatively easy to accomplish have reported that this realization shook Kasparov, who lost with not much computational strain, while perception, mo- the second game. And according to other reports, this unusual bility, and other low-level cognitive tasks are comparatively move turned out to be due to a bug in the software. 'Airing difficult 1111. Moravec and others speculate that long evolu- himself created a paper-machine-based chess-playing pro- tionary work developed the latter, while higher cognition gram, which Kasparov described as a "competent" player (9j. appeared recently in animals, and it likely represents a thin Beyond chess and checkers are backgammon and other veneer on deep sub- and unconscious foundations. games, which machines are good at. The game Go has re- The Turing Test is squarely on one side of this paradox. cently started to succumb to machine play. Away from games, machines have challenged some language-oriented human performances. One of the early examples was PARRY, writ- In a writers workshop, a group of writers comment in a ten by Kenneth Colby and his students at Stanford Univer- loosely structured way on the work brought to the work- sity (101. PARRY simulated what was then called a paranoid shop. The work is distributed well before the workshop so schizophrenic, using a simple model of the condition and a participants can prepare. Each workshop session looks only fairly sophisticated English parser. PARRY is considered to at the work of one writer. In general, the writer whose work be the first program to pass the Turing Test, or a version of is being discussed remains silent. The comments begin with it. A group of psychiatrists analyzed a combination of real an overview, then what's good about the work, then how to patients and computers running PARRY through teleprint- improve it, and finally the writer can ask questions about the ers. Another group of thirty-three psychiatrists were shown comments. Sometimes there is a teacher or workshop leader; transcripts of the conversations. The two groups were then because the Warren Wilson graduates are well-practiced, the asked to identify which of the "patients" were human and workshops at this conference operate without such. which were computer programs. The psychiatrists were able My workshop group consisted of four people: CG (woman), to make the correct identification only 48% of the time—the MN (woman). DC (man), and me. CG, MN, and DC had five same as random guessing. published books of poetry between them, and many magazine More recent was the IBM Jeopardy!-playing program called publications. I was the last writer workshopped, and my slot "Watson." In early 2011, Watson beat the two of the most suc- was on Wednesday. the day before my lecture. cessful contestants on the show, Ken Jennings and Brad Rut- I recorded the workshop. and I will present paraphrases of ter. Watson was a stand-alone system with about 3000 cores, some of the comments. When I do,I'll use a sans serif font so 16TB ofRAM, and a pretty large store of encyclopedias, dic- it's clear it's not a direct quote. In most cases the paraphrases tionaries, thesauri, newswire articles, databases, taxonomies, are close to being quotes. and ontologies—some of which InkWell also uses. Watson DC was the most published of the workshop participants. was not connected to the Internet. He started with a "flyover," which is a kind of overview of There were many reasons Watson was able to win—some the work. having to do with the Turing Test aspect of the problem, but many having to do with hardware and algorithms. For ex- These are extraordinary and extraordinarily small, large ample. Watson was routinely able to exploit the difference poems. The writer of these—this guy, Richard, or who- between humans and the machinery in response speed when ever—he is not a random person, he's not a random guy. the signal was given that "buzzing in" was permitted. Watson I think he understands randomness, so it's all the more was able to use the many previous Jeopardy! games it was scary. He doesn't do things—as a rule—by accident. He tested with to be able to better predict where Daily Doubles makes choices. The variety is amazing on every level: were, and it was able to do better betting based on game number of syllables, subject matter, syntax, whether theory. The software used an ensemble approach that com- they start out specific and go to the general, or start bined about a hundred different ways to (statistically) solve out general and go to the specific. Some of them are the answer, and Watson would buzz in only when there was simple, some of them are complex, some of them are enough confidence in the early results of this analysis—and funny, some of them are dead serious, some are kind-of it then used the time Alex Trebek used to recognize Watson in the natural world (but mostly not); there are different to continue the analysis. persons in them; "by myself" is repeated; music seems In these experiments, it wasn't an "ordinary" interrogator, important. Some are observations, some are moments, an "ordinary" person, and a machine, but expert-level com- some are philosophical and very large (and not just the petitors, and performance was judged according to difficult words, but the ideas). "Murder" is already a big word; 4 EFTA_R1_02074942 EFTA02702511 "murderous" is a bigger word; "murderousness" is about from 1.0 to 3.0, biased toward 1.0. Combining the same sense not this fatalist murderousness. as big as you can get. Lots of "ness" with the template senses ensures some degree of coherence deathwatch, words. "Depth" is more boring than throughout the haiku. but your dead subroutine "deepness: The next step is to assign random weights to the 32 con- -oc straints InkWell uses for haikus. This includes the language model for InkWell to imitate—in this case it's a collection of The language of this comment is typical ofhow working po- my daily poems from 2011 and 2012. InkWell constructs a ets talk to each other. misfit (unction for these constraints where the function re- Notice he said, "The writer of these—this guy, Richard, or turns 0.0 when all the constraints are satisfied. Inkwell se- whoever...." I asked him about this later and he said that he lects words and phrases to try (28.785 underivative narrative entertained the idea that my program wrote the haiku, but in this case), and then optimizes the lighting after considering that for a while, he rejected it as not likely— misfit function over these choices. A on the half-randomized number however, he kept a small hedge. table with all the chosen constraint weights is in Appendex Table 1. In the last step, InkWell reviews the top several haiku for Here's how InkWell produces haiku. The topic for the hai- sense (using ngrams) and uses the most best. ku comes from two sources: input from a person and input The final haiku seen just above is not great, but it's an hon- from one ofInkWell's 110 source texts. Input from a person est look at the sorts of haiku InkWell routinely produces. is used if the interactive haiku maker is used; otherwise the One of the remarkable things about this haiku is that Ink- topic input comes solely from InkWell's database of texts. I'll Well selected the word "underivative" for the specified word describe the two-source process using an example. "first." This is a choice not many writers or poets would dis- First a person inputs some words. These words represent cover. And for technical people the idea ofChalf-randomized a topic suggestion. Suppose the words input are as follows: number" is interesting. If one were to consider this a poem number, random, player, narrative. InkWell next randomly written by a person, one could analyze it as commenting on determines a number of words to select from its textual da- how an artificial writer based on random processes could pro- tabase to add to the input words. In this case it decides to duce a story unlike any seen before. Could a half-randomized choose five words taken from a randomly selected passage number be one produced by an algorithm—a pseudo-random from Steinbeck's "The Grapes of Wrath" I12): fire, shifting, number? I find the more I look at this haiku—which I selected rusty, stow, lids. Because they come from a small region in because the parameters it chose illustrated InkWell's writing the text, they are not random words—they are related. For process even though I didn't like the final haiku—the more each set of words, InkWell constructs a sense, which is a word- meaning and tangents it has. Very human in a spooky way. vector-like structure from the supplied words and close-by synonyms directed by a complex spreading algorithm which also assigns weights or relevance coefficients to the entries. The other two poets made flyover comments; MN remarked: Then the two senses are combined as follows: cS, S,, where c is a linear factor, S, is the person's input sense, and S,is the I think he is writing these as a release after a day's work, sense Inkwell chose from Steinbeck. In this case, c=2.04. The and they were written over a period of time (not as a resulting sense (S) can be visualized as a word cloud with the group). I see two sorts of language—poetic, concrete sizes of the words proportional to their associated weights. language and things in the world, as well as technical Appendix Figure I shows the resulting word cloud. The linear or corporate language. It's as if there is a war going on factor is always at least 1.0, which has the effect of favoring between the two sides of his brain. But the same brain. the person's input. -MN Next.Inkwell chooses a haiku template (the one at the bot- tom of the page, in this case). The template is in a domain- Here MN reveals she is specifically reading these poems as specific language for haiku. This template specifies four senses mine, because she has been in writers' workshops with me indicating a season, transformation, completion, and a jour- before and knows my (real) poetic work as well as my scien- ney. The sense words are as follows: snow, snowfall, water, ice; tific work. CG was a little more terse: fall; complete,finish; and span, bridge. Each of these senses is linearly combined with the sense S above to create the senses The language is condensed but plain. that will be used for the haiku. The linear factor for S ranges -CG ((first adj no-auto-cap) ((local-sense snow) (noun-phenomenon noun-substance) ((snow ice] noun)) (return) ((local-sense falling) verb-weather ((fall] verb) gerund) (return) on the half (word-hyphen) ((local-sense finished) verb-change ((finish complete] verb) past) ((local-sense bridge) noun-artifact ((bridge] noun))) 5 EFTA_R1_02074943 EFTA02702512 The writers then went on to talk about some of the poems. pleasure for me. The talon isn't mentioned, but you can see it; (DC agrees); it isn't mentioned, but you can see it. Images do a lot of work, especially in haiku, and I like -CG to see movement in the haiku, so this one ("this gravel is my favorite—the one I felt so much movement from. MN said, "This is a great one? This one taught me something, and it changed Next, DC brought up "the maiden condominium" as an this grave— no one sees it something. The speaker is in the image even example of the variety of the poems. mortality, mortality though there is no "I." I even felt the image move. What I learned is that mortality is not It is different from the others. I really like the sound just when the body goes, but when the person is no in this one. I don't get the full sense. This doesn't turn longer remembered. That's just so beautiful. me off from being intrigued and trying to understand -CG it. These are big words that have never been put in the same line the maiden condominium opens its award-winning gametocyte I see it differently. I like all these readings, and I'm a together before in the history of in the control room of the banquet fan of this one too, even if we all read it a little differ- the English language. (Then he ently. One way is that people don't see the end coming, reads the whole poem while CG iaughs.rgametocyte" because they are living their lives and here "mortality" and "banquet" don't rhyme but they go together. "Ga- is perking up and saying "don't forget about me"; or metocyte" is a sign of life. (CG and MNrepeat "maiden also that the writer's current life is like a grave—the daily condominium" and "control room of the banquet" and routine, the getting and spending, and our day-to-day wonder what they could be. They are having fun and life is a kind of mortality. But this is because of the other laughing.) poems pointing this way. There is some super power going on in this one. And -DC big words. There is wonderful humor in these. Not standup comic DC brought up "time oflife issue" as one ofhis favorites. It humor, thank God. Not one liners. There is comedy in was from of the original 2000 poems written in 2014. these. Whimsy. Along with lots of seriousness too. A great combination. Definitely one of my favorites. There is no "I" in it, except -DC there is are "eyes"—someone is observing it, thinking it, and feeling it, and commenting about it. It's Then DC quickly mentions "day after day" as time of life Issue: powerful, and it's large and small at the same a bird of prey pulls up another example of good humor. time; or general and specific at the same time. out of the way into the palm "The maiden condominium" is a good example "time of life issue" could be abstract, but "a bird of something InkWell does well that poets have of prey pulls up" (CG says "wow" in the background) trouble with. InkWell is relentless in trying to find uncommon is very vivid and specific, and "out of the way into the things to say and ways ofsaying things. It's not a coincidence palm" is both. It has a sort-of opening up. One of the that "gametocyte" and "banquet" almost thyme—Ink- day after day ways good haiku and short poems work best is they look Well uses a concept called echoes to populate poems in the man's can and feel somewhat tight, concentrated, and highlighted with sonic echoes, a sort of musicality. a man can and momentary but there is a kind of opening up—and not just a fly-away, not an escape, necessarily, but an opening up. I feel this; this is a fantastic one. Poetry seems to be one of the tasks Turing and others con- -DC sider central to the idea of the Turing Test. Recall the first example exchange in a fictional exercise of the test: I want to sing the praises of this one too. I want the pleasure of saying how much I like it. Because it took Q: Please write me a sonnet on the subject ofthe Forth me two or three readings before I got it, before I had Bridge. an image, and then it was transformer time. You know, A: Count me out on this one. I never could writepoetry. everything just transformed. This one shows the power of the form because everything is working together, and The Forth Bridge is iconic and considered a symbol of Scot- I just got a strong image. And it changed, too—it wasn't land.In his critique of Thring's idea, Geoffrey Jefferson wrote just given to me. I had to work; there was space in the the following in the British Medical Journal [13]: poem for me. I got the connection and that was the Not until a machine can write a sonnet or compose a concerto because ofthoughts and emotionsfelt, and 6 EFTA_R1_02074944 EFTA02702513 tuned adrenalin not by the chancefall ofsymbols, could we agree that my musk, machine equals brain—that is, not only write it but a beat•boogled heedful know that it had written it. -Jefferson, The Mindof Mechanical Man, 1949 I believe this is...well, you decide. What does InkWell tell us about this? InkWell selects topics to write about, and then chooses a set of personality traits to CG pointed out one that seemed funny—"the powerful display, a set of controlling mood words to use to steer what head." DC commented on it as follows: it says about the topic, and overarching subsenses to direct its inner gaze. Indeed InkWell uses randomness as part of Those words are all deadly—potentially deadly. Unpo- its composition strategy, but as DC pointed out, "<InkWell> etic, right? They're abstract. Who ever has used "cogni- is not a random person, <InkWell>'s not a random guy." But tion" in a poem? There are some world records being does InkWell feel these thoughts and emotions? That's basi- set here. After three lines you realize the the powerful head cally what the ThringTest is trying to define. Recent work on poem has turned itself upside down—this designates its powerful head consciousness (e.g. "The Ego Tunnel" by Thomas Metzinger poem undercuts itself. Maybe because to support cognition [141) has something to say about that, but perhaps the best "powerful head" is already the brain or thought is that in writing this, Jefferson mistakes or misun- mind, and it's passing the buck to either itself or some derstands the poetic / creative process. sub-brain or sub-mind, but to support cognition, which Writing a poem is not fundamentally an emotional, expres- means it's thinking about passing the buck on thinking. sive explosion—it's a deliberate task using practiced skills. It's I didn't want to go there. I'm feeling sorry for whoever not Walt Whitman's "1sound my barbaric yawp over the roofs is caught up in this (meaning the speaker), because it's of the world" [15]. The poem "Howl" by Allen Ginsberg [16] just the opposite of what it just said. It's "support cogni- (see Appendix) was mythologized as being a performance tion," but.. thank goodness I didn't quite go there, even piece that was recorded and published (this was part of the though it wants me to all the time. testimony at the obscenity trial surrounding the poem), but -oc it was written over a period of nearly two years with critical evaluation by friends brought to bear and specific writing "Deep in the dark" is the first poem to catch my attention techniques explored and exploited. Ginsberg himself com- from the original 2000 InkWell wrote. mented on the intellectually directed choices and investiga- tions he made while creating the poem. The great thing about it ("deep in the dark") I like is InkWell can be thought of as operating deliberately too. that the word "dark" of the first line contrasts with the Like Ginsberg, InkWell can decide to experiment with long unexpressed "white" of the snow in the second line. The lines; InkWell can decide the degree and nature of deep in the dark— last line puts them together. musicality using rhythms and sounds; InklArell can the power of snow -MN decide to make sense or be crazy; and many other walking in the deepness things like this, but all are deliberate artistic choices. I see an echo of "stopping by woods." This is a Like real poets, InkWell uses skills to create art. Poets who good echo to have. I really do like "the deepness." It res- use feelings alone are the best targets for the criticism of cues it. I really can't say why but I know. I tried changing "chance fall of symbols." it to "depth." But it's a musical thing or an aural thing. After Inkwell writes a poem. does it know that it had writ- Or "depth" is too familiar and conventional. Each line ten it? In a literal sense it does—it records each poem in a log, has a "the" and one could play around with removing sometimes (depending on parameters I set) also noting the them. But removing any of them removes also the par- artistic choices it made. But in the sense Jefferson meant, no. ticularness of the image. "The" slackens the lines—makes There is no phenomenal self model in play. That is. Inkwell them looser—but it also makes them more immediate, doesn't maintain an internal representation of what it is do- familiar, and more specific ing aside from representing its artistic choices. -oc What about the question 'Baring imagines: "Please write me a sonnet on the subject of...." Recall that Inkwell can be directed to look at a topic based on a set of words suggested According to the most extremeform of this view the to it. In Spring 20151was demoing InkWell to a former long- only way by which one could be sure that machine time colleague; he asked me "can you ask it to write a haiku thinks is to be the machine and tofeel oneself thinking. about this: blues guitar and loud music." I asked InkWell to -Turing, computing Machinery andIntelligence. 1950 write five poems, and this was one of them: 7 EFTA_R1_02074945 EFTA02702514 This is the consciousness argument. In its extreme form Turing wrote: the only unequivocal way to look at consciousness is solip- sism—it's just me, babe. But 'Raring rejects that and works We also wish to allow thepossibility that an engineer toward Jefferson's objection about writing a sonnet by consid- or team of engineers may construct a machine which ering whether a viva voce would satisfy him—an oral exam works, but whose manner of operation cannot be sat- in which the interrogator asks detailed questions about the isfactorily described by its constructors because they sonnet.' This leads to this interesting question: to what degree have applied a method which is largely experimental. does InkWell know about the poems it writes? -Turing, Computing Machinery and Intelligence. 1950 Inkwell certainly is not programmed to respond to ques- tions such as "why did you use these particular words right Today we read this as referring to machine learning. Some here," but it has an accessible representation of the reasons parts ofInkWell are machine learned—judgments about per- for all its choices. InkWell decides which artistic choices to sonality and emotions in a text, for example. In some cases make,either through whimsy or by reading a text, how much these judgments are baked into dictionaries and databases. to weigh them against each other, and which moods or out- The central part ofInkWell is a meta-heuristic optimization side influences to consider. These choices are enshrined in a process, the basis of whose operation can be explained, but misfit function InkWell constructs—InkWell composes the whose detailed operation in any particular instance is a bit source code for this function and then compiles it—and all mysterious. The construction of the misfit function is sym- the choices sit in data structures. You might comment, "Ga- bolic, and unpacking how that function directed the result briel, you're exaggerating all this," but these explicit traces of the optimization is explainable. are my answer to the difficult question, "how do you debug InkWell?" I need to see how and why all the decisions were made, because the only significant bugs arise from domain- That bird of prey poem: I felt a lot of doublenesses, related mistakes, which manifest as surprising utterances and I love doublenesses. I wouldn't describe it as really and never exceptions or type errors. And to figure them out, dark, even though there is darkness in it. I find it also I need to examine InkWell's state of mind, as it were. And comical—not really funny. There's whimsy to it, a whimsy were I so inclined, I could program InkWell to access more tone to it, both. This is a form of doubleness—dark and gently this self model when quizzed—more gently than by comical / whimsical—and I don't know how you do d— using data structure inspectors and debuggers. how you, Richard, do it. This is a very large, small poem. "I chose this pair of words because they sparked off each It sounds quiet to me. The last line is not threatening, other well with syllable sounds without being blatant rhymes: but the poem starts out threatening. Not to the exclu- because I wanted to come offas extraverted while channeling sion of others, but this one is really terrific. remorse: because I was trying to include a subtext of explo- ration and discovery. They were also very Hemingwayesque. And the best other choices were these, and they just didn't measure up." Inkwell can't say that, but looking at its param- Raring remarks that Lady Lovelace wrote in her memoir the eters, it's sense descriptions, its halos, its musicality settings. following (17]: its target personality, the writer's ngrams it's trying to mimic, the recorded results of the component factors measured in The Analytical Engine has nopretensions to originate InkWell's misfit function, etc. for a particular poem. I can anything. It can do whatever we know how to order trivially report it. it to perform. One way to look at it is that InkWell has an effective, op- -Countess of Lovelace, Translator's Notes.... 1843 erational self model, but Inkwell itself is not yet in that self model, and thus Inkwell is only partway toward being con- This leads to Thring's "surprise" concern: scious. Inkwell modifies its own self model to change how it makes art. When we "talk" to InkWell about these inner A better variant of the objection says that a machine changes and factors, we do so in a nonhuman language. and can never 'take us by surprise.' InkWell responds in the same language. -Turing, Computing Machinery and Intelligence. 1950 Is this ok? Is this enough? Everyone—including 'TVring—who has ever programmed has been taken by surprise by their code. And not because of bugs and errors—and not because ofrandomness particularly. For Inkwell it's because ofits relentlessness trying to find in- . This is calkd the-Pickwick' lest, becauseTuring's essay describes a series teresting things, such as the phrase in the King James Bible ofqueslions about Charks Dickcns's "The Pickwick Papers." Sec Appendix. 8 EFTA_R1_02074946 EFTA02702515 with the best set of sonic similarities to the noun "computer purpose-built software robots like those that play checkers, programmer": "provoked me to anger." chess, backgammon, and Go. These are intended to show that I can ask InkWell specifically to surprise me because it has human-level expertise in narrow domains can be exhibited or ngrams for millions of texts written since 1890 118) including at least simulated. The Jeopardy!-playing Watson is close to frequencies ofappearance, and I can ask for rare or never-seen these narrow robots because the domain is trivia—the sort of combinations. I can ask InkWell to search for unusual syn- stuff that Google is good at finding. Watson moves closer to onyms; I can ask it to write unlike particular writers. When the universal notion of thinking that Turing was approaching. there are dozens of constraint types with both positive and negative weights, there are few limits to surprise. Turing ended his essay with an appeal to a learning ap- proach to get machines close to human abilities. As noted The poets were surprised, too. CG: "'sampling in chocolate'" learning is generally taken as machine learning these days. is surprising language"; "'guitar-shaped coloring' is surpris- As I write this essay, AlphaGo just marched to victory against ing. It evokes brown / beige because guitars are made of wood, a very strong human Go player (Lee Se-dol, a 9-dan profes- a reasonable assumption— and it's interesting / surprising that a shape sional Go player). As David Silver et al wrote [201: by myself, could evoke a color." DC: "I don't know ex- sampling in chocolate actly the sense of this, but i like the surprise. We have developed, for thefirst time, effective move the sound, the sonic surprise of'scrupulous selection and position evaluation functions for Go, in the twilight'"; "i recognize all ofit as poetry because of the based on deep neural networks that are trained by a surprisingness of the language." novel combination of supervised and reinforcement I asked about the use oflanguage in the poems. CG: "Awe- learning. We have introduced a new search algorithm some." DC: "Good noise. Surprising in a good way. a few days— that successfully combines neural network intriguing. Lots ofvariety. Big, small, long short, y "Ws° • evaluations with Monte-Carlo rollouts. Our browsing guitar-shaped coloring loud, soft, complex, plain. Not a single track." program AlphaGo integrates these components together, at scale, in a high-performance tree search engine. The Turing Test is ofcourse bogus. At least in the form Tur- -Silver et al, Mastering the Game of Go with Deep Neural ing envisioned. A common strategy for passing is to dodge Networks and Tree Search, 2016 questions. typically using humor and distractions. Turing's scrupulous in the twilight— own example shows a dodge as an accept- Here the issue of viva vice comes up—how would AlphaGo the price of gold chases able response; "Count me out on this one. explain why it made a particular move? Answers of the form the way of the world in power I never could write poetry." "7 is better than 6" won't work well, but perhaps the people Beginning in 2008 a series ofpractical who developed AlphaGo can intuit such answers. AlphaGo Turing tests have been conducted under academic scrutiny, lost game four, and here is what was reported in the press 1211: run using the best interpretation of Turing's specifications.In June 2014 an extensive set of interrogations were conducted According to tweets from DeepMind founder Denies at the Royal Society [19J. This produced ISO parallel tran- Hassabis, however, this time AlphaGo really did make scripts, each of which contains a single interrogator posing mistakes. The Al "thought it was doing well, but got questions for five minutes to a human and a chatterbot, with confused on move 8Z" Hassabis said, later clarifying the responses being returned side-by-side at the same time on that it made a mistake on move 79 but only realized the same screen. In the Appendix you can see a sample par- its error by 87. allel transcript. In this sample the LHS (left-hand side of the -httpwwww.theverge.com/2016/3/13/11184328/alphago- screen) was a female adult human, and the RHS was Eugene, deepinind-go-rnatch-4-result a chatterbot. The judge misjudged the LHS to definitely be a machine and the RHS to be a non-native English speaking And Ttiring would seem to respond when he wrote: human. The judge got it backward. The human on the LHS had weak responses while the machine on the RHS tried to May not machines carry out something which ought dominate the conversation and was definitely more lively than to be described as thinking but which is very different the LHS. The chatterbot pretending to be Eugene Goostman, front what a man does? This objection is a very strong a 13-year-old Ukrainian boy, was declared to have passed the one, but at least we can say that if nevertheless, a ma- Turing Test for having fooled more than 30% of the judges. chine can be constructed to play the imitation game Part ofThring's idea was that the unexpected scope ofques- satisfactorily, we need not be troubled by this objection. tions would be the key to deciding whether the computer was -Turing, Computing Machinery andIntelligence, 1950 thinking sufficiently like a human. This is in contrast with the 9 EFTA_R1_02074947 EFTA02702516 An early goal of artificial intelligence was to understand authorship are the haiku themselves—forming a feedforward how people think and act. Most Al research from the 1%0s, loop in the best case. After some evidence is found that the 1970s, and 1980s was directed toward symbolic Al, which was writer is human, the poem is examined more thoroughly, writing programs whose inner workings—being directed to- finding even more evidence of this. Recall what CG said: "... ward emulating thought—could be understood and explained. it wasn't just given to me. I had to work; there was space in In the 1980s it became apparent that programs that could do the poem for me. I got the connection and that was the plea- the mental work humans do could form the basis for an in- sure for me. The talon isn't mentioned, but you can see it...." dustry. Around the same time, progress in machine learning •-••••-• was accelerating alongside advances in computer power, and the idea that it was important to understand how Al programs Consider surprise and the Lovelace objection—programs did what they do was swept aside. One could always fall back do only what we tell them to do and therefore cannot be on the pop idea that human intellectual performance had considered "Turing human." An extreme form ofsurprise is an intuitive, only faintly understandable side, along with a for the program to do something far removed from its basic deliberate, conscious side—that is, a machine learning side programming. InkWell has never created a recipe nor has and a symbolic side. This is the heart of the Moravec paradox. it proved a difficult theorem. How could it and why would Perhaps this pop idea has some merit. it? You'd need a program designed to survive and thrive in a dynamic environment to discover and achieve novel capa- bilities. To quote the fictional character from "The Martian" After I revealed to the poets in my writers workshop that I221, "you solve one problem, and you solve the next one, and the poems I presented were actually produced by a program, then the next. And if you solve enough problems, you get to two of them were good with it though they expressed surprise. come home." This is the universal version of the evolution- The third, though. was quite upset. CG said that it was un- ary and learning objective function—you reward behaviors fair for me to keep that information secret—because in such and ideas that enable you to live. The environment provides workshops it's assumed implicitly that the work is produced opportunities for learning, and simple, built-in mechanisms by the writers sitting right there, and all the comments are ratchet that into new capabilities, habits, and proclivities. made with that in mind, including that it is proper to be gentle This is reminiscent of unsupervised and semi-supervised with those comments. This because all the participants are learning. This is finding patterns in data never seen before; graduates of the same writing program, and hence are linked there is no reliable way to do this, but when the learner frosted winter, by a special. "caring" bond. CG said that perhaps critiques stumbles across some sort of reinforcement, this can bridge black. could have been more blunt had the rouse not been in place. turn into a weak form of supervised learning. And in ice white I countered by saying that a poem boils down to the words the real world, refinement is always possible. on the page, and everything else is contextual interpretation. All of this leads to Searle's Chinese Room argument 1231 We can assume that the person believed to have written the and the role ofconsciousness in artificial intelligence. Here is poem is important to its effect on a reader, but it all begins a greatly simplified and distilled rephrasing ofSearle's issue: with the words on the page—even if "by the chance fall of there must be something inside ofor causally created by the symbols' [13j. brain that is operating on meaning and intentions, not sym- The upshot of CG's comment is that if it's known that a bols and syntax; this thing is consciousness. He argues this poem was written by a machine. then the poem's inhuman- by putting a human (himself, actually) in place of, essentially, ity could be explored and perhaps highlighted. CG said that the CPU in an Al program that seems to pass a Turing-like knowing the haiku were written by a computer would open Test in Chinese. He remarks that mechanically processing the door for blunt comments. However, the other side of this the rules ofChinese question answering does not constitute coin is more interesting: if it's known that a poem was writ- "understanding" Chinese because he (Searle) playing the part ten by a human, then the poem's humanity can be explored of the CPU wouldn't: and highlighted. This is the side of the coin CG said was the default for the workshop. ...it seems to me quite obvious...that I do not under- But how does one know that a poem was written by a hu- stand a word of the Chinese stories. I have inputs and man? By seeing evidence. The entity claiming authorship looks outputs that are indistinguishablefrom those ofthe na- human, acts human. The Turing Test? Could we say that the tive Chinese speaker,...but I still understand nothing. Turing Test is what makes us human—at least in the eyes of For the same reasons, Schank's computer understands of other people (or other Thring Test passers)? In this case, nothing ofany stories, whether in Chinese, English, or aside from my claim of authorship,' the evidence of human whatever, since in the Chinese case the computer is me, and in cases where the computer is not me, the corn- 2. Actually, my claim was literally truthful: "Ere been working on short poems recently." 10 EFTA_R1_02074948 EFTA02702517 puter has nothing more than I have in the case where feeling that the poet is a real person, and that increases the 1 understand nothing. likelihood of finding more and deeper human elements in -Searle. Minds, Brains, and Programs, 1980 the poem—or at least increases the incentive to look for hints of humanity and the energy to look with. Yet somewhere inside or nearby a person there is a thing or In writing InkWell I was trying to explore how poets and a clutch of things such that if you replace it or those with "real" writers write—I was trying to capture in a program Searle. Searle would understand what the person does solely what I had learned while studying how to write poetry. I in- by doing what it or they do. I believe Searle is talking about tentionally started at the word-choice end of things where consciousness. many of the effects writers use are hidden in sound (noise). But Turing argued that relentlessly seeking out conscious- connotation, mood, author personality, and influence. By ness in a program is asking more of the program than we pursuing this I intentionally made the internals of Inkwell normally ask of other people. As CG might say, we afford as expressive as was reasonable so I could see the effects of people the courtesy of believing they are people. changes and additions, as well as study how the different in- Yet the Turing Test seems to be about findinghints someone tended effects interacted to produce different texts. Neverthe- is at home in(side) the test subject. The Pickwick test, which less, InkWell has many learned aspects—machine learned at Turing made up, is about exploring inner thought processes. my hands, machine learned by others, and learned through Jefferson is after this too. curated and automatically produced dictionaries and data- bases. The interplay between the symbolic and learned aspects of InkWell can be observed decently well, and perhaps the But what of programs? InkWell is the program whose hu- nature of this interplay could provide useful research results manity we are exploring—does it exhibit traits generally asso- into the possible nature of the mind. ciated with people, especially creative people who are typically I have written a poem every day since March 18, 2000. thought ofas feeling beings? And if that question even makes That's a lot of poems. Some nights when I sit down to write sense, is this realm of questions part of a reasonable battery my daily poem, I "don't have it," as they say. My talent has ofrequirements and specifications? We talk about the "ilities" taken the night off, nothing happened during the day to but are these external and to some extent internal traits part serve as a trigger, or I'm simply a little too fatigued to crush of an "ility" we can discuss? The traits describe being able to it. For the past few years when this happens, I have turned to act like a human mind, and to be able to answer questions InkWell to help me. I tell it some odd topics to consider and about how and why the program performed certain human ask it to write a few dozen poems. And from those I'll revise mindful behaviors—the Pickwick test. to a good poem or will use the sequence as a starting place. To be able to explain a program's mental behavior probably InkWell is a good helper. requires some sort ofunderstandable reification ofits innards. And this also implies that the innards of a program are im- portant to its humanity or at least to our understanding of The Thring Test provides an interesting lens or instrument its humanity. This means that this kind of programming is useful for exploring what we can make of the semi-living na- not like typical software engineering, where getting the job ture of programs that are designed for a little bit more than done with good performance, correctness, and maintain- their useful effects. We could go crazy exploring all the ins ability is paramount. and outs of philosophy of mind, strong and weak AI, con- k.a..• •• • sciousness, machine learning versus symbolic deliberation, and intuition versus reasoning. but all I'm wondering about In the end a group of expert writers and poets believed is the lesson to learn from a group of hardcore poets taking that eighteen haiku written by Inkwell were worthy of be- Inkwell as a colleague. ing considered real and sometimes real good poetry. These writers and poets believed I wrote these haiku, and the ques- tion is how much that mattered. One of three poets in my writers' workshop believed this made a difference, and the other two did not. There also seemed to be opportunities for Turing ratchet- ing: finding some human elements in a poem increases the 11 EFTA_R1_02074949 EFTA02702518 References IIJ Richard P. Gabriel, Jilin Chen, Jeffrey Nichols, InkWell: 1131 Geoffrey Jefferson, The Mind ofMechanicalMan. British A Creative Writer's Creative Assistant, Creativity & Cog- Medical Journal, 1949. nition, 2015. 1141 Thomas Metzinger, "The Ego Tunnel: The Science of the Pi Alan M. Turing, ComputingMachinery and Intelligence, Mind and the Myth of the Self," Basic Books, 2010. Mind, October 1950. DM Walt Whitman, Song of Myself. "Leaves of Grass," self- (3] Alan M. Thring et al, The Mind and the Computing Ma- published, 1855. chine, The Rutherford Journal. 1947. (161 Allen Ginsberg, "Howl and Other Poems," City Lights. httpt://en.wikipedia.org/wiki/Haiku 1956. (5] Robert Han, editor, "The Essential Haiku: Versions of (171 Luigi Federico Menabrea, Ada Lovelace, Sketch of the Bubo, Buson, & Issa," The Ecco Press, 1995. Analytical Engine invented by Charles Babbage...with (61 F. Scott Fitzgerald, "The Great Canby," Charles Scrib- notes by the translator, translated by Ada Lovelace, in ner's Sons, 1925. Richard Taylor, "Scientific Memoirs 3," Richard and (71 John Gardner, "The Art of Fiction: Notes on Craft for John E. Taylor. pp. 666-731, 1843. Young Writers," Vintage. 1991. (181 http://storage.googleapis.comThooksMgrams/books/datasetsv2. (81 Arthur Samuel. Some studies in machine learning using html the game of checkers, IBM Journal of Research and De- 1191 Kevin Warwick & Huma Shah. Can machines think? A velopment Volume 44, Issue: 1.2. 2000. report on Turing test experiments at the Royal Society, (9] Garry Kasparov, The Chess Master and the Computer, Journal of Experimental & Theoretical Artificial Intel- review of Chess Metaphors: Artificial intelligence and ligence. 2015. the Human Mind by Diego Rasskin-Gutman, in New (20] David Silver et al. Mastering the Game of Go with Deep York Review of Books, Volume 57, Number 2, Febru- Neural Networks and Tree Search, Nature 529, pp.484- ary 11, 2010. 489, 28 January 2016. (101 Kenneth Colby, Turing-like Indistinguishability Testsfor (211 http://www.theverge.com/2016/3/13/11184328/alphago-deep- the Validation ofa Computer Simulation ofParanoidPro- mind-go-match-4-result cesses, Artif. Intel]. 3(1-3) 1972. pp. 199-221. (221The Martian (2015 movie). http://www.imdb.comititlei HI] Hans Moravec,Mind Children, Harvard University Press, U36593881 1988. (231 John Searle. Minds, Brains, and Programs, Behavioral (12] John Steinbeck, "The Grapes of Wrath," The Viking Press, and Brain Sciences. 1980. 1939. EFTA_R1_02074950 EFTA02702519 Appendix Thirty-two Haiku deep in the dark— an on-the-far-side summer night— the power of snow whipping up high tea. walking in the deepness [1276911 we stripped pickles 14242311 I'm not the same aboard a boat on an island a round table dancing— with destructive rhythm (467771) an old song [2665271 behind a rock shoved off the stairs— on the green slope falling I become dead soldiers' spirits [4833111 a rainbow [359659) the powerful head the maiden condominium designates its powerful head opens its award-winning gametocyte to support cognition [3660191 in the control room of the banquet 1238801) this grave— a reasonable assumption— no one sees it by myself, mortality, mortality [4833371 sampling in chocolate 13635891 shopping parade— from the boulder people step over smiling up at heaven the broken cassette [3571911 the continent begins 1159947) a bitch. old lift: this deep in trick through the grille a fortiori not a man [263573) three women in pastel t-shirts [1733171 not this fatalist murderousness, rural signal, deathwatch. cannot understand Oregon but your dead subroutine [357781) —agricultural 13844731 time of life issue: parted in the middle— a bird of prey pulls up the authority of the air conditioner out of the way Into the palm (1087911 perfection In the brightness 11356971 awake in the dark— too late: the edge of the water can the last express passes through spread in your presence [306473) the dust of gardens 14907571 day after day a blue anchor— in the man's can grains of grit a man can [471853) in a tall sky sewing I361597) scrupulous in the twilight— pirates Imitate the price of gold chases the ways of ordinary people the way of the world in power [3030191 myself for instance [102941] tuned adrenalin dental hospital— my music. dead files line a beat-boogied headful [275309) the light casings 12592831 under the sea the hostile defense a fish becomes human leads its problematic rear, In an air pocket (2656511 the rear of frustration 13482091 a crooked rag day— a few days— by myself by myself, dunking distracted sardines [494497) browsing guitar-shaped coloring 1160967) bare branches. in rags and crystals. tonight again stars, stars sometimes with a shred of sense arc misprints (269239) an odd dignity 13498231 Haiku labeled with primes were written by Inkwell 13 EFTA_R1_02074951 EFTA02702520 .. ... 7 mrc ... ° atomic-number" scorer i „ d folktale . r 01 cornetist m fibonacci-number . i,_ 2_.. I temente numerousness Tri — ”., complex-quanrinc1/10iinist preyalenceadsup 4 b_ or-miss u a ,_unselected ognksman w 4- clarinetist shooterlutanist t at, n:_ C.0 .... ,,, §:violoncellist> .- ›..'6..L. trumpeter. prinitcluantity communicatory t !LI 0, 1 ..o.c 13).—a >r e alcard—playerstringer = sorft-"" e rhyg w palpebraumplano-player participanO ( .- e ac o =G o) rix O - hockey-player floating-point-numberree. 2 u C fiddler pan om SOD-SION - 47, c-6 isoccer- playerwolistom ic-t ' ° V,hornist oxidation-numbet 0, .w iso _ billiard-player v i vjaavolleyball-playerlutist fl Clarinettist gam ist to E. recorder-player ,..2 c— c i_.0r° nurn rharpist luirawl football-player. shifting tr-.it keyboardist 2most-valuable-player2 E 3-5, jazzim cea-nhol, e 2 accompanist tlergodic B y- rilayerMUSiCiall baseball-player c worii-our rhythm-and-blues-musicianoymerositym 2 .- 1- -player payer constant ta: 2 g-xnarrat 'vets. bowl-- 41 4- d lacrosse-player rock-'n'-roll-musician 71:11 t. E ci ":3- °>-...- ru — .t C O. 0 ° ig a er bell-ringerws ire chess-player o. u ballplayerin koto- player E ibistguitar -player .E bassooniis$ it anClt iti cSieaend edie?'ict esri Ortist I count1,1 grandmaster- sitar-player porous-wove 0, — . 0̀w::: -_ cro 6:. harmoniser= IV binist 'wore osa „,. . 1,_ / ooMal er serverwtco u vxopjfonist r aim d ionist._= ,y,6 :electric-2Ow to .0 w a a , IfittJadrate e r i 0ecimal m golfer i=2 1tnu r °pianist er an sman 0, or „s o i. cirri t muitiplier-facror prim mu =risKer w v, . = re piper soloist fourth-Power.° al mvp oboist — — ° -0 majority — Appendix Figure 1 Agreeableness 10.0 Echo Bonus 37.29 Conscientiousness -10 Synonym Search Depth 2 Extraversion 0.0 <choose> Search Depth 1 Neuroticism 10.0 Synonym Decay Rate 0.57 Openness 10.0 Maximum Synonym Sense,. 2 Openness-To-Change 0.0 2-gram-bonus 75 Big5 Strength 3.19 3-gram-bonus 85 Common Word Bonus 2.90 4-gram-bonus 85 Global Halo Frost.halo 5-gram-bonus 75 Global Halo Bonus 434 Writer-2-gram-bonus 27.698803 Local Halo Bonus 5.76 Writer-3-gram-bonus 45.65159 Local Sense Bonus 40.00 Writer-4-gram-bonus 71.91725 Local Predicates Bonus 938 Writer-5-gram-bonus 44.505795 Synonym Proximity Bonus 4.97 Writer File Poemsrpg.txt Constraint Bonus 3.41 Long Words True Rhyme Bonus 47.21 Appendix Table I Howl, Allen Ginsberg, first three lines I saw the best minds ofmy generation destroyed by madness, starving hysterical naked. dragging themselves through the negro streets at dawn lookingfor an angryfix. angelheaded hipsters burningfor the ancient heavenly connection to the starry dynamo in the machinery ofnight.... 14 EFTA_R1_02074952 EFTA02702521 Royal Society Turing Test, June 2014 Transcript I -LHS Transcript I—RHS (16:20:48) Judge what isyour worst memory in your life so far? (16:21:00) Judge: what did you have for breakfast today (16:21:06) Entity: I don't understand. Explain. (16:21:10) Entity: Is it 'the third degree'? I'm not going to (16:21:35) Judge: memory of something that happened to you answer such a provocative question! Well, let's go on though. that was bad or painful (16:21:56( fudge: do go on...your call? (16:22:581 Entity: Hmm that a tricky question. My life is won- (16:22:06) Entity: It is highly possible. By the way, what's derful. I overcome many bad things that have happened to me your occupation? I mean—could you tell me about your work? in the past. However, the most painful memory I have had is (16:22:30( Judge: so you are asking me now eh? I am an performing very poorly in my GCSE's when I was younger. actress and writer (16:23:561 Judge: did you fail? (16:22:381 Entity: Well, I'll read something of your books. (16:24:12) Entity: Partially. if they ain't too tedious (which is highly possible:-) (16:24:35) Judge: explain? (16:23:151 ►udge: grammer...aren't too tedious! A bit rude (16:25:04) Entity: Some grades turned out to be good. Some of you... turned out to be quite bad. (16:23:17) Entity: Calm down, please. (16:25:37) Judge: and that is the worst thing that has happened (16:23:49) Judge: i asure you i am very calm ... are you? to you so far? Mmm how old are you? (16:24:01( Entity: Don't even bother me with the fact that you are very calm ... are mine. And I forgot to ask you where you are from ... (16:25:01) Judge: I am from Dorset...where are you from? (16:25:08) Entity: I am from Ukraine, from the city called Odessa. You might have heard about it. The Pickwick Test Interrogator. In the first line of your sonnet which reads "Shall I compare thee to a summer's day," would not "a spring day" do as well or better? Witness: It wouldn't scan. Interrogator: How about "a winter's day." That would scan all right. Witness: Yes, but nobody wants to be compared to a winter's day. Interrogator: Would you say Mr. Pickwick reminded you of Christmas? Witness: In a way. Interrogator. Yet Christmas is a winter's day. and I do not think Mr. Pickwick would mind the comparison. Witness: I don't think you're serious. By a winter's day one means a typical winter's day, rather than a special one like Christmas. t5 EFTA_R1_02074953 EFTA02702522