I need assistance with this homework assignment, thank you!
Microsoft Word - ds2500_lab_nlp.docx DS 2501: Intermediate Programming with Data / Lab Practicum Prof. Rachlin, Northeastern University Hambot In natural language processing, n-grams refer to all of the n-length contiguous words occurring over the text. For example, if our input is: “She sells sea shells by the sea shore” the 2-grams would be (ignoring case): she sells, sells sea, sea shells, shells by, … etc. and our 3-grams would be: she sells sea, sells sea shells, sea shells by, shells by the, … etc. We’re going to try to use 2-grams and 3-grams to create a lyrics-generating bot for the musical Hamilton. Here’s the general approach: 1. Read the file containing Hamilton lyrics. Remove punctuation, headers that indicate the speaker such as [HAMILTON] or [HAMILTON & LAURENS] and convert all words to lowercase. 2. Catalog all the words that begin a sentence. 3. Catalog all the words that terminate a sentence 4. Catalog all the 2-grams as a dictionary where the key is the first word, and the value is a list of all the possible follow-on words. For example, in the tongue twister provided above we might have: start_words: [she] stop_words: [shore] 2-grams: {she: [sells], sells: [sea], sea:[shells, shore], … }. 5. Now generate random Hamilton lyrics by doing the following: a. Pick a random start word b. For each current word, pick a random next word from your 2-gram dictionary by looking up the possible follow-on words and choosing one at random. c. If the chosen word is a stop word, your sentence is done, otherwise go to step b. 6. Does this produce anything reasonable or complete gibberish? I suspect maybe the latter. Submit your code and your best lyrics. 7. To earn a 5 on the lab: Modify your code to work with 3-grams. Now the start and stop words become start / stop word pairs, and the 3 grams map pairs of words (tuples) to lists of possible 3rd words. This should produce sentences that are more coherent. Does it work? [HAMILTON] I am not throwing away my shot! I am not throwing away my shot! Hey yo, I'm just like my country I'm young, scrappy and hungry And I'm not throwing away my shot! I'ma get a scholarship to King's College I prob'ly shouldn't brag, but dag, I amaze and astonish The problem is I got a lot of brains but no polish I gotta holler just to be heard With every word, I drop knowledge! I'm a diamond in the rough, a shiny piece of coal Tryin' to reach my goal. My power of speech: unimpeachable Only nineteen but my mind is older These New York City streets get colder, I shoulder Ev'ry burden, ev'ry disadvantage I have learned to manage, I don't have a gun to brandish I walk these streets famished The plan is to fan this spark into a flame But damn, it's getting dark, so let me spell out the name I am the— [HAMILTON/LAFAYETTE/MULLIGAN/LAURENS] A-L-E-X-A-N-D E-R—we are—meant to be… [HAMILTON] A colony that runs independently Meanwhile, Britain keeps shittin' on us endlessly Essentially, they tax us relentlessly Then King George turns around, runs a spending spree He ain't ever gonna set his descendants free So there will be a revolution in this century Enter me! [LAFAYETTE/MULLIGAN/LAURENS] (He says in parentheses) [HAMILTON] Don't be shocked when your hist'ry book mentions me I will lay down my life if it sets us free Eventually, you'll see my ascendancy [HAMILTON & LAURENS] And I am not throwing away My shot (my shot) I am not throwing away My shot (my shot) Hey yo, I'm just like my country I'm young, scrappy and hungry And I'm not throwing away my shot And I'm not throwing away my shot [HAMILTON/MULLIGAN/LAURENS/LAFAYETTE] I am not throwing away my shot I am not throwing away my shot Hey yo, I'm just like my country I'm young, scrappy and hungry And I'm not throwing away my shot It's time to take a shot! [LAFAYETTE] I dream of life without a monarchy The unrest in France will lead to 'onarchy? 'Onarchy? How you say, how you say, 'anarchy?' When I fight, I make the other side panicky With my— [HAMILTON/LAURENS/LAFAYETTE/MULLIGAN] Shot! [MULLIGAN] Yo, I'm a tailor's apprentice And I got y'all knuckleheads in loco parentis I'm joining the rebellion 'cause I know it's my chance To socially advance, instead of sewin' some pants! I'm gonna take a— [HAMILTON/LAURENS/LAFAYETTE/MULLIGAN] Shot! [AARON BURR] How does a bastard, orphan, son of a whore and a Scotsman, dropped in the middle of a forgotten Spot in the Caribbean by providence, impoverished, in squalor Grow up to be a hero and a scholar? [JOHN LAURENS] The ten-dollar Founding Father without a father Got a lot farther by working a lot harder By being a lot smarter By being a self-starter By fourteen, they placed him in charge of a trading charter [THOMAS JEFFERSON] And every day while slaves were being slaughtered and carted Away across the waves, he struggled and kept his guard up Inside, he was longing for something to be a part of The brother was ready to beg, steal, borrow, or barter [JAMES MADISON] Then a hurricane came, and devastation reigned Our man saw his future drip, dripping down the drain Put a pencil to his temple, connected it to his brain And he wrote his first refrain, a testament to his pain [BURR] Well, the word got around, they said, “This kid is insane, man” Took up a collection just to send him to the mainland “Get your education, don’t forget from whence you came, and The world's gonna know your name. What’s your name, man?” [ALEXANDER HAMILTON] Alexander Hamilton My name is Alexander Hamilton And there's a million things I haven't done But just you wait, just you wait... [ELIZA HAMILTON] When he was ten his father split, full of it, debt-ridden Two years later, see Alex and his mother bed-ridden Half-dead sittin' in their own sick, the scent thick [FULL COMPANY EXCEPT HAMILTON (whispering)] And Alex got better but his mother went quick [GEORGE WASHINGTON & COMPANY, BOTH] Moved in with a cousin, the cousin committed suicide Left him with nothin' but ruined pride, something new inside A voice saying “Alex, you gotta fend for yourself.” He started retreatin' and readin' every treatise on the shelf [BURR & COMPANY, BOTH] There would have been nothin’ left to do For someone less astute He woulda been dead or destitute Without a cent of restitution Started workin', clerkin' for his late mother's landlord Tradin’ sugar cane and rum and all the things he can’t afford Scammin' for every book he can get his hands on Plannin' for the future see him now as he stands on (ooh) The bow of a ship headed for a new land In New York you can be a new man [COMPANY & HAMILTON, WOMEN, MEN] In New York you can be a new man (Just you wait) In New York you can be a new man (Just you wait) In New York you can be a new man In New York, New York Just you wait! [COMPANY] Alexander Hamilton (Alexander Hamilton) We are waiting in the wings for you (waiting in the wings for you) You could never back down You never learned to take your time! Oh, Alexander Hamilton (Alexander Hamilton) When America sings for you Will they know what you overcame? Will they know you rewrote the game? The world will never be the same, oh [BURR, MEN, & COMPANY] The ship is in the harbor now See if you can spot him Just you wait Another immigrant Comin’ up from the bottom Just you wait His enemies destroyed his rep America forgot him [MULLIGAN/MADISON & LAFAYETTE/JEFFERSON] We fought with him [LAURENS/PHILIP] Me? I died for him [WASHINGTON] Me? I trusted him [ELIZA & ANGELICA & PEGGY/MARIA] Me? I loved him [BURR] And me? I'm the damn fool that shot him [COMPANY] There's a million things I haven’t done But just you wait! [BURR] What's your name, man? [COMPANY] Alexander Hamilton! [COMPANY] 1776. New York City [HAMILTON] Pardon me. Are you Aaron Burr, sir? [BURR] That depends. Who’s asking? [HAMILTON] Oh, well, sure, sir I’m Alexander Hamilton, I’m at your service, sir I have been looking for you [BURR] I’m getting nervous [HAMILTON] Sir… I heard your name at Princeton. I was seeking an accelerated course of study when I got sort of out of sorts with a buddy of yours. I may have punched him. It’s a blur, sir. He handles the financials? [BURR] You punched the bursar [HAMILTON] Yes! I wanted to do what you did. Graduate in two, then join the revolution. He looked at me like I was stupid, I’m not stupid So how’d you do it? How’d you graduate so fast? [BURR] It was my parents’ dying wish before they passed [HAMILTON] You’re an orphan. Of course! I’m an orphan God, I wish there was a war! Then we could prove that we’re worth more Than anyone bargained for… [BURR] Can I buy you a drink? [HAMILTON] That would be nice [BURR] While we’re talking, let me offer you some free advice Talk less [HAMILTON] What? [BURR] Smile more [HAMILTON] Ha [BURR] Don’t let them know what you’re against or what you’re for [HAMILTON] You can’t be serious [BURR] You wanna get ahead? [HAMILTON] Yes [BURR] Fools who run their mouths off wind up dead [LAURENS] Yo yo yo yo yo! What time is it? [LAURENS/LAFAYETTE/MULLIGAN] Show time! [BURR] Like I said… [LAURENS] Show time! Show time! Yo! I’m John Laurens in the place to be! Two pints o’ Sam Adams, but I’m workin’ on three, uh! Those redcoats don’t want it with me! Cuz I will pop chick-a pop these cops till I’m free! [LAFAYETTE] Oui oui, mon ami, je m’appelle Lafayette! The Lancelot of the revolutionary set! I came from afar just to say “Bonsoir!” Tell the King “Casse toi!” Who’s the best? C’est moi! [MULLIGAN] Brrrah brraaah! I am Hercules Mulligan Up in it, lovin’ it, yes I heard ya mother said “Come again?” [LAFAYETTE & LAURENS] Ayyyyy [MULLIGAN] Lock up ya daughters and horses, of course It’s hard to have intercourse over four sets of corsets… [LAFAYETTE] Wow [LAURENS] No more sex, pour me another brew, son! Let’s raise a couple more… [LAURENS/LAFAYETTE/MULLIGAN] To the revolution! [LAURENS] Well, if it ain’t the prodigy of Princeton college! [MULLIGAN] Aaron Burr! [LAURENS] Give us a verse, drop some knowledge! [BURR] Good luck with that: you’re takin’ a stand You spit. I’m ‘a sit. We’ll see where we land [LAFAYETTE/MULLIGAN] Boooo! [LAURENS] Burr, the