scrapy自定义pipeline类实现将采集数据保存到mongodb的方法

本文实例讲述了scrapy自定义pipeline类实现将采集数据保存到mongodb的方法。分享给大家供大家参考。具体如下：

# Standard python/' target='_blank'>python library imports# 3rd party modulesimport pymongofrom scrapy import logfrom scrapy.conf import settingsfrom scrapy.exceptions import DropItemclass MongoDBPipeline(object):  def __init__(self):    self.server = settings['MONGODB_SERVER']    self.port = settings['MONGODB_PORT']    self.db = settings['MONGODB_DB']    self.col = settings['MONGODB_COLLECTION']    connection = pymongo.Connection(self.server, self.port)    db = connection[self.db]    self.collection = db[self.col]  def process_item(self, item, spider):    err_msg = ''    for field, data in item.items():      if not data:        err_msg += 'Missing %s of poem from %s\n' % (field, item['url'])    if err_msg:      raise DropItem(err_msg)    self.collection.insert(dict(item))    log.msg('Item written to MongoDB database %s/%s' % (self.db, self.col),        level=log.DEBUG, spider=spider)    return item

希望本文所述对大家的Python程序设计有所帮助。