scrapy自定义pipeline类实现将采集数据保存到mongodb的方法_Python

scrapy自定义pipeline类实现将采集数据保存到mongodb的方法

2020-06-08 09:40pythoner Python

这篇文章主要介绍了scrapy自定义pipeline类实现将采集数据保存到mongodb的方法,涉及scrapy采集及操作mongodb数据库的技巧,具有一定参考借鉴价值,需要的朋友可以参考下

本文实例讲述了scrapy自定义pipeline类实现将采集数据保存到mongodb的方法。分享给大家供大家参考。具体如下：

				?

									# Standard Python library imports

									# 3rd party modules

									import pymongo

									from scrapy import log

									from scrapy.conf import settings

									from scrapy.exceptions import DropItem

									class MongoDBPipeline(object):

									  def __init__(self):

									    self.server = settings['MONGODB_SERVER']

									    self.port = settings['MONGODB_PORT']

									    self.db = settings['MONGODB_DB']

									    self.col = settings['MONGODB_COLLECTION']

									    connection = pymongo.Connection(self.server, self.port)

									    db = connection[self.db]

									    self.collection = db[self.col]

									  def process_item(self, item, spider):

									    err_msg = ''

									    for field, data in item.items():

									      if not data:

									        err_msg += 'Missing %s of poem from %s\n' % (field, item['url'])

									    if err_msg:

									      raise DropItem(err_msg)

									    self.collection.insert(dict(item))

									    log.msg('Item written to MongoDB database %s/%s' % (self.db, self.col),

									        level=log.DEBUG, spider=spider)

									    return item