您好, 欢迎来到 !    登录 | 注册 | | 设为首页 | 收藏本站

如何使用python从网站中提取带有匹配单词的html链接

如何使用python从网站中提取带有匹配单词的html链接

您需要india显示的文本中 搜索单词。为此,您需要一个自定义函数

from bs4 import BeautifulSoup
import requests

url = "http://www.bbc.com/news/world/asia/"
r = requests.get(url)
soup = BeautifulSoup(r.content)

india_links = lambda tag: (getattr(tag, 'name', None) == 'a' and
                           'href' in tag.attrs and
                           'india' in tag.get_text().lower())
results = soup.find_all(india_links)

india_links拉姆达发现都是标记<a>链接href属性,包含india(不区分大小写)中显示的文本的地方。

注意,我使用了requests响应对象 .content 属性。将解码留给BeautifulSoup!

演示:

>>> from bs4 import BeautifulSoup
>>> import requests
>>> url = "http://www.bbc.com/news/world/asia/"
>>> r = requests.get(url)
>>> soup = BeautifulSoup(r.content)
>>> india_links = lambda tag: getattr(tag, 'name', None) == 'a' and 'href' in tag.attrs and 'india' in tag.get_text().lower()
>>> results = soup.find_all(india_links)
>>> from pprint import pprint
>>> pprint(results)
[<a href="/news/world/asia/india/">India</a>,
 <a class="story" href="/news/world-asia-india-30647504" rel="published-1420102077277">India scheme to monitor toilet use </a>,
 <a class="story" href="/news/world-asia-india-30640444" rel="published-1420022868334">India to scrap tax breaks on cars</a>,
 <a class="story" href="/news/world-asia-india-30640436" rel="published-1420012598505">India shock over Dhoni retirement</a>,
 <a href="/news/world/asia/india/">India</a>,
 <a class="headline-anchor" href="/news/world-asia-india-30630274" rel="published-1419931669523"><img alt="A Delhi police officer with red flag walks amidst morning fog in Delhi, India, Monday, Dec 29, 2014. " src="http://news.bbcimg.co.uk/media/images/79979000/jpg/_79979280_79979240.jpg"/><span class="headline heading-13">India fog continues to cause chaos</span></a>,
 <a class="headline-anchor" href="/news/world-asia-india-30632852" rel="published-1419940599384"><span class="headline heading-13">Court boost to India BJP chief</span></a>,
 <a class="headline-anchor" href="/sport/0/cricket/30632182" rel="published-1419930930045"><span class="headline heading-13">India captain Dhoni quits Tests</span></a>,
 <a class="story" href="http://www.bbc.co.uk/news/world-radio-and-tv-15386555" rel="published-1392018507550"><img alt="A woman riding a scooter waits for a traffic signal along a street in Mumbai February 5, 2014." src="http://news.bbcimg.co.uk/media/images/72866000/jpg/_72866856_020889093.jpg"/>Special report: India Direct</a>,
 <a href="/2/hi/south_asia/country_profiles/1154019.stm">India</a>]

注意http://www.bbc.co.uk/news/world-radio-and- tv-15386555这里的链接;我们必须使用lambda搜索,因为带有text正则表达式的搜索不会找到该元素;包含的文本(特殊报告:India Direct )不是标签中的唯一元素,因此无法找到。

/news/world-asia-india-30632852链接也有类似问题。嵌套<span>元素使得 法院提升印度BJP首席 标题文本不是link标记的直接子元素。

您可以提取 链接

from urllib.parse import urljoin

result_links = [urljoin(url, tag['href']) for tag in results]

相对于原始URL解析所有相对URL的位置:

>>> from urllib.parse import urljoin
>>> result_links = [urljoin(url, tag['href']) for tag in results]
>>> pprint(result_links)
['http://www.bbc.com/news/world/asia/india/',
 'http://www.bbc.com/news/world-asia-india-30647504',
 'http://www.bbc.com/news/world-asia-india-30640444',
 'http://www.bbc.com/news/world-asia-india-30640436',
 'http://www.bbc.com/news/world/asia/india/',
 'http://www.bbc.com/news/world-asia-india-30630274',
 'http://www.bbc.com/news/world-asia-india-30632852',
 'http://www.bbc.com/sport/0/cricket/30632182',
 'http://www.bbc.co.uk/news/world-radio-and-tv-15386555',
 'http://www.bbc.com/2/hi/south_asia/country_profiles/1154019.stm']
python 2022/1/1 18:47:08 有328人围观

撰写回答


你尚未登录,登录后可以

和开发者交流问题的细节

关注并接收问题和回答的更新提醒

参与内容的编辑和改进,让解决方法与时俱进

请先登录

推荐问题


联系我
置顶