From loebner@ACM.ORG Fri May 5 16:06:54 EDT 1995 Article: 29535 of comp.ai Path: cantaloupe.srv.cs.cmu.edu!rochester!udel!gatech!swrinde!hopper.acm.org!ACM.ORG!LOEBNER From: loebner@ACM.ORG (Hugh Loebner) Newsgroups: comp.ai,comp.ai.games,comp.ai.nat-lang Subject: 1995 Loebner Prize - Call for Entries & Rules Date: 4 May 1995 16:39:16 GMT Organization: ACM Network Services Lines: 352 Message-ID: <3oavvk$8e5@hopper.acm.org> Reply-To: loebner@ACM.ORG NNTP-Posting-Host: acm.org Xref: glinda.oz.cs.cmu.edu comp.ai:29535 comp.ai.games:1222 comp.ai.nat-lang:3310 LOEBNER PRIZE COMPETITION IN ARTIFICIAL INTELLIGENCE ANNOUNCEMENT OF, AND APPLICATION INFORMATION FOR, THE 1995 PRIZE COMPETITION (Rev 4/1995) The Loebner Prize Medal and a cash award is awarded annually to the designer of the computer system that best succeeds in passing a variant of the Turing Test. In 1995, $2,000 and a bronze medal will be awarded to the designer of the Most Human Computer if the median rank of that computer is below the overall median rank of all of the hidden human confederates in the contest. The Loebner Prize Competition in Artificial Intelligence was established in 1990 by Dr. Hugh Loebner and the Cambridge (Massachusetts) Center for Behavioral Studies. In accordance with the requirements of Dr. Loebner as published in the June 1994 Communications of the ACM, the winner of the Grand Prize must be prepared to deal with audio visual input (pattern recognition). However, to celebrate the first unrestricted Turing Test, this rule will be waived for the 1995 contest only. Should the median rank of a computer equal or exceed the overall median rank of the confederates -- in other words, should that computer prove to be indistinguishable from the typical human in the event -- the Grand Prize of $100,000 and a gold medal will be awarded, and the contest will be discontinued. The highest-ranking human confederate will receive an award for being the Most Human Human. Applications for the 1995 competition must be postmarked by November 1, 1995. Submissions will be evaluated and no more than eight finalists selected by November 21, 1995. The 1995 finalists will compete in a simultaneous and real-time competition on Friday, December 15, 1995, in New York NY (date and location subject to change). There is no entry fee. Applications must be accompanied by printed or e-mail protocols recording interactions between the computer system to be entered and one or more human subjects. Protocols must not exceed ten typewritten double-spaced pages or 2,500 words. Applications may be submitted by e-mail. The Cambridge Center or the Prize Committee may request an opportunity to interact with candidate programs. Transcripts and scores from the 1991, 1992, 1993, and 1994 competitions may be obtained by contacting the Cambridge Center for Behavioral Studies at 617-491-9020 (e-mail 76557.1175@compuserve.com). Please review the accompanying rules carefully and provide the following information: Name of Entrant: Professional Title of Entrant (if applicable): Affiliation (if applicable): Contact Person (if applicable): U S Postal Address: Telephone Number(s): Fax Number(s): E-Mail Address(es): Computer hardware to be employed: Computer software to be employed (optional): Enclose a protocol recording interactions between the computer system to be entered and one or more human subjects (maximum 10 typewritten double-spaced pages or 2,500 words). Mail or e-mail completed applications to: Hugh Loebner, Project Director Loebner Prize Competition c/o Crown Industries, Inc. 155 North Park St. East Orange, NJ 07017 e-mail address: loebner@acm.org ============================================ OFFICIAL RULES 1995 LOEBNER PRIZE COMPETITION IN ARTIFICIAL INTELLIGENCE ============================================ 1. The objective of the 1995 Loebner Prize Competition in Artificial Intelligence is to identify the computer system that can best succeed in passing a modern variant of the Turing Test. Judges will attempt to distinguish computer systems (referred to henceforward as "contestants") from human beings (referred to henceforward as "confederates" or "human confederates") based on interactions with these computer entries. 2. Applicants may be individuals, organizations, businesses, schools, corporations, institutions, or other entities. Individuals need not have institutional affiliations. Applicants may be of any nationality or age. Applicants may submit more than one entry during the same year, but a separate application must be completed for each entry. 3. Employees, affiliates, and agents of the Cambridge Center for Behavioral Studies, Crown Industries, Inc., or of other official contest sponsors, as well as members of the families of such individuals, are not eligible to apply. 4. Applications for the 1995 Prize competition may be made on an official application form or facsimile or submitted by electronic mail. Applications must be postmarked by November 1, 1995. Applications should be submitted to: Dr. Hugh Loebner at the above address 5. If there are two or more entries there will be a Loebner Prize contest. The medal and cash award will go to the designer of the computer system with the highest median score, based on the ratings of the contest judges. Mean scores will be used only to break a tie. The Grand Prize of $100,000 and a gold medal will be awarded to the designer of the highest-ranking computer if the median rank of that computer equals or exceeds the median rank of the hidden human confederates. If there is only one entry, Loebner Prize Medal and the $2000.00 cash award will be awarded to that entry. If there are no entries in the 1995 Loebner Prize Competition, the $2000.00 cash prize will be added to the cash award for 1996, making the award for 1996 $4000.00. 6. It is the task of the computer entries to respond to the communications of the judges in such a manner as to imitate the responses of a human. No constraints will be placed on the judges' communications. Entries must be prepared to communicate for an indefinite period of time. 7. Computer entries may contain standard or customized software and hardware. The hardware may be of any type as long as it is inorganic and as long as its replies are in no manner controlled by human beings or other organic systems in real time. 8. Entries will be required to run on hardware located at the competition site. NO TELECOMMUNICATIONS WILL BE PERMITTED. Finalists chosen to participate may submit their entries as programs recorded on standard machine readable media (magnetic or optical storage) together with operating documentation. Macintosh and IBM compatible computers with standard mass storage peripherals will be available on-site for the contest. Efforts will be made to secure the use of other standard computers such as workstations, but this can not be guaranteed. Entrants requiring non-standard hardware will be required to provide that hardware for operation at the contest site. Personnel will be prepared to operate the computer entries according to documentation providedby the contestant. 9. Appropriate steps will be taken to prevent the unauthorized duplication or publication of contestants' entries, however neither the Cambridge Center, the Prize Committee, nor any of their agents can guarantee absolute security. Contestants, by entering this contest, imply their understanding of this and agree to hold the Cambridge Center, the Prize Committee and their agents harmless should there be any unauthorized duplication or publication of programs. Entrants who wish absolute security will be allowed to to operate their entries and/or provide their own hardware on site. 10. Computer entries will be required to record the conversations as text files on magnetic media. Information regarding the file format will be available after April 1, 1995 from the Contest Director. The recordings will remain the property of the Cambridge Center, which will also retain the copyright on transcripts or other representations, magnetic or otherwise, of the recordings. 11. Applications must be accompanied by printed protocols or e-mail transcripts recording actual interactions between the system to be entered and one or more human beings. The protocols may not exceed ten double-spaced typewritten pages or 2,500 words. 12. The selection process may entail interaction between selection personnel and computer entries. Applicants will be notified of the selection decision by November 21, 1995. No more than eight entries will be selected as finalists to compete in a real- time and simultaneous contest to be held in New York, NY, on Friday, December 15, 1995 (date and location subject to change). 13. Judges and human confederates for the 1995 competition will be selected who have no prior knowledge of any of the contestants or human confederates. 14. Judges will have one or more opportunities to interact with each of the computer terminals available concurrently during the contest. Judges will be allowed unrestricted communications. They will be informed that at least two of the terminals are controlled by human confederates and that at least two of the terminals are controlled by computers. 15. One or more referees may be present to limit communications of the confederates. 16. Judges will not be allowed to interact with each other and will be instructed to provide individual ratings of each of the appropriateness and responsiveness of the replies. 17. The names "Loebner Prize" and "Loebner Prize Competition" may be used by contestants in advertising only by advance written permission of the Cambridge Center, and their use may be subject to applicable licensing fees. Advertising is subject to approval by representatives of the Loebner Prize Competition. Improper or misleading advertising may result in revocation of the prize and/or other actions. 18. Computer programs must communicate in the English language. Data Entry, File Format, and Hardware Specifications 1. KEY ENTRY - TERMINAL DISPLAY: a. The key sequence @,@,nn,, (nn=00-99) when entered will indicate that a new judge, Judge nn, is now entering data. Each judge will thus identify himself/herself when moving to a new terminal. b. Judges' questions and comments can be multiline. Each question or comment will be entered one line at a time. Each line will be terminated by a carriage return .Judges will key in questions and comments in response to a ">" prompt from the program. c. The computer will prompt the judge with a ">" character and then echo on the screen, character by character, the judge's entry. d. Entry of two consectutive carriage returns will indicate that the judge's question or comment is complete and that the terminal (program or human) must respond. e. The terminal will display character data in a monotype font (all characters of equal width). EXAMPLE Assume program is responding to current interaction with Judge 3 [some program comment-completed] >@@04 ( New Judge, number 4 ) > ( two lines required ) Welcome judge 4 ( Comment by program (or human) ) > Do you think that the ( Multiline question from judge4) > Republicans can succeed ( question continued line 2 ) > in winning the White House? ( question ended (first ) ) > ( second ) Only if Newt succeeds in ( Answer, line 1 ) developing a more tolerant image. ( Answer, line 2 ) >_ ( ">" prompt - judge to respond ) ^ Cursor waits for input --------------------------- 2. DATA FILE FORMAT Intent: Each program entered in the Loebner Prize Contest will produce a text file transcript of the interactions with judges. The file should be readable by standard text reader programs. a. Each computer entry in the 1995 Loebner Prize Contest shall create a text file on 3.5" diskette containing the transcript of keyboard input and program output. b. The file will named LP95-nn.TXT eg LP95-01.TXT. The nn (00 -> 99) allows a program to record multiple executions on a single disk. That is, the first execution will write LP95-01.TXT to the diskette. The second execution will write LP95-02.TXT, etc. It is the responsibility of the program to determine the appropriate cycle (nn). c. The file will be in ASCII text format suitable for input into a standard word processing program. d. The first three lines are headers containing the following: (c)1995 Cambridge Center For Behavioral Studies all rights reserved Start at: e. Each succeeding line will EITHER mirror one line displayed on the screen, preceeded by the source "JUDGEnn" or "PROGRAM" and time in brackets OR indicate a change of judges:"*** JUDGEnn ***" eg: For the above interaction: *** JUDGE04 *** PROGRAM[14:12:25]Welcome judge 4 JUDGE04[14:12:32]Do you think that the JUDGE04[14:12:39]Republicans can succeed JUDGE04[14:12:55]In winning the White House? PROGRAM[14:13:15]Only if Newt succeeds in PROGRAM[14:13:17]developing a more tolerant image ------------------------------- 3. HARDWARE SPECIFICATIONS Intent: Computers under control of a program may access disc drives, CD Roms, or other peripherals in a distinctive manner, thereby providing visual and/or auditory clues In order to eliminate such clues, computer bodies and peripherals will be screened from judges. However, monitors and keyboards will be on judges' side of the sceens. Therefore, the following rule: a. Monitor and keyboard cables will be at least 1.5 Meters long. screening || || _______ ____ || [ ] [ c ]---||--[monitor] [ o ] || [_______] [ m ]---||-----/ \------[ keyboard ] [disk][ p ] ||____/___\___ _[____][____]___|| || || || || || || || || || || || |_||_| || ------------------------------------------------------------------------------- Hugh Loebner, Pres. Crown Industries, Inc. 155 N Park St, East Orange NJ 07017 Mfgs of Roll-Up Plastic Lighted Portable Disco Dance Floors and Other Essential Products for Home and Industry Crown Industries is *proud* to be a Corporate Sponsor of The Loebner Prize for Artificial Intelligence Tel:Business 1-800-GO-CROWN Personal Voice Mail 1-500-EGO-TRIP e-mail loebner@acm.org