뉴스 → 크롤링 → NER

Susan Li (Aug 17, 2018), "Named Entity Recognition with NLTK and SpaCy - NER is used in many fields in Natural Language Processing (NLP)"에서 구현된 웹사이트를 크롤링하여 텍스트를 가져오고 여기에서 개체명(Named Entity)를 추출하는 사례를 살펴보자.

뉴스기사 크롤링

requests를 이용하여 웹사이트를 가져오고 bs4 라이브러리를 활용하여 뉴스기사만 추출해낸다.

import requests
from bs4 import BeautifulSoup
import re

def url_to_string(url):
    res = requests.get(url)
    html = res.text
    soup = BeautifulSoup(html, 'html5lib')
    for script in soup(["script", "style", 'aside']):
        script.extract()
    return " ".join(re.split(r'[\n\t]+', soup.get_text()))

ny_bb = url_to_string('https://www.nytimes.com/2018/08/13/us/politics/peter-strzok-fired-fbi.html?hp&action=click&pgtype=Homepage&clickSource=story-heading&module=first-column-region&region=top-news&WT.nav=top-news')

print(ny_bb[:100])

     F.B.I. Agent Peter Strzok, Who Criticized Trump in Texts, Is Fired - The New York Times        

개체명 추출

spacy 라이브러리를 통해서 개체명을 추출해낸다.

import spacy
from spacy import displacy
import en_core_web_sm

nlp = en_core_web_sm.load()

article = nlp(ny_bb)

len(article.ents)

173

개체명 기초통계

추출된 개체명을 바탕으로 기초통계 작업을 수행해보자.

from collections import Counter

## labels
labels = [x.label_ for x in article.ents]
Counter(labels).most_common(5)

[('PERSON', 83), ('ORG', 39), ('DATE', 23), ('GPE', 16), ('CARDINAL', 5)]

## 토큰

items = [x.text for x in article.ents]
Counter(items).most_common(5)

[('Strzok', 28), ('F.B.I.', 13), ('Trump', 12), ('Russia', 7), ('Clinton', 5)]

기사에서 문장 무작위 추출

random 라이브러리에서 문장을 무작위로 추출한다.

import random

random.seed(77111)

sentences = [x for x in article.sents]
sample_sentence = sentences[random.randrange(0, len(sentences))]
print(sample_sentence)

” Mr. Strzok, who rose over 20 years at the F.B.I. to become one of its most experienced counterintelligence agents, was a key figure in the early months of the inquiry.

NER 시각화

displacy.render(nlp(str(sample_sentence)), jupyter=True, style='ent')

” Mr. Strzok PERSON , who rose over 20 years DATE at the F.B.I. ORG to become one of its most experienced counterintelligence agents, was a key figure in the early months DATE of the inquiry.

displacy.render(nlp(str(sample_sentence)), style='dep', jupyter = True, options = {'distance': 40})

전체 뉴스기사 시각화

displacy.render(nlp(str(sentences)), jupyter=True, style='ent')

[ F.B.I. Agent Peter Strzok PERSON , Who Criticized Trump PERSON in Texts GPE , Is Fired - The New York Times SectionsSEARCHSkip to contentSkip to site, indexPoliticsSubscribeLog InLog InToday’s PERSON PaperPolitics|F.B.I. Agent Peter Strzok PERSON , Who Criticized Trump PERSON in Texts GPE , Is FiredAdvertisementSupported byF.B.I., Agent Peter Strzok PERSON , Who Criticized Trump PERSON in Texts GPE , Is FiredImagePeter Strzok, a top F.B.I. counterintelligence agent who was taken off the special counsel investigation after his disparaging texts about President Trump PERSON were uncovered, was fired., CreditCreditT.J. Kirkpatrick PERSON for The New York TimesBy Adam Goldman PERSON and Michael S. SchmidtAug PERSON ., 13 CARDINAL , 2018WASHINGTON — Peter Strzok, the F.B.I. ORG senior counterintelligence agent who disparaged President Trump PERSON in inflammatory text messages and helped oversee the Hillary Clinton PERSON email and Russia GPE investigations, has been fired for violating bureau policies, Mr. Strzok PERSON ’s lawyer said Monday DATE ., Mr., Trump ORG and his allies seized on the texts, — exchanged during the 2016 DATE campaign with a former F.B.I. ORG lawyer, Lisa Page PERSON — in assailing the Russia GPE investigation as an illegitimate “witch hunt., ” Mr. Strzok PERSON , who rose over 20 years DATE at the F.B.I. ORG to become one of its most experienced counterintelligence agents, was a key figure in the early months DATE of the inquiry., Along with writing the texts, Mr. Strzok PERSON was accused of sending a highly sensitive search warrant to his personal email account., The F.B.I. ORG had been under immense political pressure by Mr. Trump PERSON to dismiss Mr. Strzok PERSON , who was removed last summer DATE from the staff of the special counsel, Robert S. Mueller III PERSON ., The president has repeatedly denounced Mr. Strzok PERSON in posts on Twitter GPE , and on Monday DATE expressed satisfaction that he had been sacked., Mr., Trump’s ORG victory traces back to June DATE , when Mr. Strzok PERSON ’s conduct was laid out in a wide-ranging inspector general’s report on how the F.B.I. ORG handled the investigation of Hillary Clinton’s PERSON emails in the run-up to the 2016 DATE election., The report was critical of Mr. Strzok PERSON ’s conduct in sending the texts, and the bureau ’s Office of Professional Responsibility ORG said that Mr. Strzok PERSON should be suspended for 60 days DATE and demoted., Mr. Strzok PERSON had testified before the House ORG in July DATE about how he had not allowed his political views to interfere with the investigations he was overseeing., But Mr. Strzok PERSON ’s lawyer said the deputy director of the F.B.I. ORG , David Bowdich PERSON , had overruled the Office of Professional Responsibility ORG and fired Mr. Strzok PERSON ., A spokeswoman for the F.B.I. ORG did not respond to a message seeking comment about why Mr. Strzok PERSON was dismissed rather than demoted., Firing Mr. Strzok PERSON , however, removes a favorite target of Mr. Trump PERSON from the ranks of the F.B.I. ORG and gives Mr. Bowdich PERSON and the F.B.I. ORG director, Christopher A. Wray PERSON , a chance to move beyond the president, ’s ire., Aitan Goelman PERSON , Mr. Strzok PERSON ’s lawyer, denounced his client’s dismissal., “, The decision to fire Special Agent Strzok is not only a departure from typical bureau practice, but also contradicts Director Wray’s PERSON testimony to Congress ORG and his assurances that the F.B.I. intended to follow its regular process in this and all personnel matters,” Mr. Goelman PERSON said.“This decision should be deeply troubling to all Americans NORP ,” Mr. Goelman PERSON added., “, A lengthy investigation and multiple rounds of congressional testimony failed to produce a shred of evidence that Special Agent Strzok’s personal views ever affected his work.”Mr., Strzok’s ORG text exchanges with Ms. Page PERSON demonstrated animosity toward Mr. Trump PERSON ., In one, Ms. Page PERSON asks: Trump ORG is “not ever going to become president, right?, Right?!, ”, Mr. Strzok PERSON responds: “, No., No he won’t., We’ll stop it., ”, The inspector general, who uncovered the messages, found no evidence that the pair imposed their political views on their investigative decisions but cited that exchange as “not only indicative of a biased state of mind but, even more seriously, implies a willingness to take official action to impact the presidential candidate’s electoral prospects.”The report by the inspector general, Michael E. Horowitz PERSON , that preceded Mr. Strzok PERSON ’s firing not only criticized his conduct in sending the texts but also his use of personal email accounts to handle sensitive information., In addition, the inspector general criticized Mr. Strzok PERSON ’s decision not to move swiftly to examine new emails related to the Clinton PERSON investigation just weeks DATE before the 2016 DATE election., Mr., Horowitz PERSON said in his report that he was “deeply troubled” by the text messages., Hundreds CARDINAL exchanged over months DATE were found in which the pair disparaged Mr. Trump PERSON and, to a lesser extent, Mrs. Clinton PERSON , exchanged work gossip and bantered., On Twitter GPE , Mr. Strzok PERSON said he was “deeply saddened by this decision,” adding, “, It has been an honor to serve my country and work with the fine men and women of the FBI.”Mr., Strzok became emblematic of Mr. Trump’s PERSON unfounded assertions that a so-called deep state of bureaucrats opposed to him was undermining his presidency., Mr. Trump PERSON contended that Mr. Strzok PERSON targeted the president and accused Mr. Strzok PERSON of being “treasonous” and a “disgrace., ” Mr. Strzok PERSON told lawmakers that he never leaked information about the Russia GPE inquiry, which could have upended the election and hurt Mr. Trump PERSON ’s chances of becoming president., After Mr. Horowitz PERSON uncovered the texts, Mr. Mueller PERSON , who had by then taken over the investigation, removed Mr. Strzok PERSON from his team., He was reassigned to the F.B.I.’s ORG human resources division., Ms. Page PERSON , who had left Mr. Mueller PERSON ’s team before the discovery of the text messages, quit the F.B.I. in May. DATE , The inspector general’s report also took issue with the reaction by Mr. Strzok PERSON and other F.B.I. ORG officials to the discovery of possible new evidence in the Clinton PERSON investigation, known internally as Midyear Exam PERSON , in late September 2016 DATE on a laptop that belonged to the disgraced politician Anthony D. Weiner PERSON , the husband of a top Clinton PERSON aide., At the time, Mr. Strzok PERSON was in the early stages of investigating whether any Trump ORG associates had conspired with Russia GPE ’s interference in the presidential election, and nearly a month DATE passed before agents and analysts began to act on the emails found on Mr. Weiner PERSON ’s laptop., Mr. Horowitz PERSON could not rule out that Mr. Strzok PERSON had slow-walked the examination of the new emails to help Mrs. Clinton PERSON , ’s presidential bid.“Under, these circumstances, we did not have confidence that Strzok’s ORG decision to prioritize the Russia GPE investigation over following up on the Midyear- DATE related investigative lead discovered on the Weiner PERSON laptop was free from bias,” he wrote., The delays were merely the “result of bureaucratic snafus,” Mr. Strzok PERSON ’s lawyer wrote last month DATE in USA GPE Today DATE ., But the justifications for the delay were “unpersuasive” and had “far-reaching consequences,” the inspector general said., James B. Comey PERSON , the former F.B.I. ORG director, has told investigators that if he had known about the emails earlier, it might have influenced his decision to alert Congress ORG to their existence days before the election., In addition, the inspector general said that Mr. Strzok PERSON had forwarded a proposed search warrant to his personal email account., The inspector general said the email, which included a draft of the search warrant affidavit, contained information that appeared to be under seal., In a heated congressional hearing last month DATE , Mr. Strzok PERSON expressed “significant regret” for the texts and rebutted the president ’s ORG attacks on the Russia GPE inquiry., “, This investigation is not politically motivated; it is not a witch hunt; it is not a hoax,” he said., Mr., Strzok’s ORG dismissal was not unexpected., He is the second ORDINAL senior F.B.I. ORG agent to be fired as a result of the inspector general’s investigation., In March DATE , Andrew G. McCabe PERSON , the former deputy director, was fired after the inspector general repeatedly faulted him for misleading investigators., The firing was politically motivated, Mr. McCabe PERSON has said, as an effort to discredit him as a witness in the special counsel investigation., Both men were fired before they were eligible for their pension and health benefits., Mr., Strzok, 48 DATE , a graduate of Georgetown University ORG , served as an officer in the Army ORG before he joined the F.B.I., He held several key positions in the F.B.I. ORG , eventually becoming a top deputy in the counterintelligence division., He handled many important espionage cases including one CARDINAL involving a former C.I.A. ORG officer suspected of working for China GPE and a group of Russian NORP spies who had been working undercover in the United States GPE ., Follow Adam Goldman PERSON and Michael S. Schmidt PERSON on Twitter: @adamgoldmanNYT and @nytmike PERSON ., A version of this article appears in print on , on Page A1 ORG of the New York GPE edition with the headline:, F.B.I. Banishes An Agent Who Reviled Trump., Order Reprints |, Today’s Paper | SubscribeRelated CoverageF.B.I. Agent Defends Actions in Russia GPE Inquiry in Contentious House FAC , TestimonyJuly ORG 12, 2018ImageReport Criticizes Comey ORG but Finds No Bias in F.B.I. Decision on ClintonJune ORG 14 PRODUCT , 2018ImageAndrew McCabe, a Target of Trump’s F.B.I. Scorn,, Is Fired Over Candor QuestionsMarch 16, 2018ImageAdvertisementSite IndexGo PRODUCT to, Home Page »newshome, pageworldU.S.politicsElection, 2020New CARDINAL , Yorkbusinesstechscienceclimatesportsobituariesthe upshottoday DATE 's papercorrectionsopiniontoday's opinionop-ed columnistseditorialsop-ed Contributorsletterssunday ORG reviewvideo: opinionartstoday's artsart & designbooksdancemoviesmusicPop Culturetelevisiontheaterwatchingvideo ORG :, artslivingautomobilesCookingcrosswordeducationfoodhealthjobsmagazineparentingreal estatestylet magazinetravellovelistings & moreReader CenterWirecutterLive EventsThe Learning Networktools & servicesN.Y.C., events guidemultimediaphotographyvideoNewslettersNYT, storetimes, journeysmanage my accountnewshome pageworldU.S.politicsElection, 2020New DATE , Yorkbusinesstechscienceclimatesportsobituariesthe upshottoday DATE 's papercorrectionsopiniontoday's opinionop-ed columnistseditorialsop-ed Contributorsletterssunday ORG reviewvideo: opinionartstoday's artsart & designbooksdancemoviesmusicPop Culturetelevisiontheaterwatchingvideo ORG :, artslivingautomobilesCookingcrosswordeducationfoodhealthjobsmagazineparentingreal estatestylet magazinetravellovemoreReader CenterWirecutterLive EventsThe Learning Networktools & servicesN.Y.C. events guidemultimediaphotographyvideoNewslettersNYT, storetimes journeysmanage my accountSubscribehome deliverydigital subscriptionsCrosswordCookingemail newsletterscorporate subscriptionseducation ratemobile applicationsreplica, editionSite Information Navigation ORG © 2019 , The New York Times ORG CompanyContact UsWork with usAdvertiseYour Ad ChoicesPrivacyTerms of ServiceTerms ORG of SaleSite MapHelpHelpSubscriptions , <iframe, src="https://www.googletagmanager.com/, ns.html?id= ORG , GTM-P528B3&gtm_auth= ORG , tfAzqo1rYDLgYhmTnSjPqw&gtm_preview= ORG , env-130&gtm_cookies_win=, x" height="0" width="0" style="display:, none;visibility:, hidden"></iframe> ]