基于python+scrapy+flask的二手房的采集和分析可视化系统-毕业设计源码+使用文档（高分优秀项目）

2020-04-23 10:16:30 [scrapy.utils.log] INFO: Scrapy 1.6.0 started (bot: LianjiaSpider) 2020-04-23 10:16:30 [scrapy.utils.log] INFO: Versions: lxml 4.3.4.0, libxml2 2.9.9, cssselect 1.1.0, parsel 1.5.2, w3lib 1.21.0, Twisted 18.9.0, Python 3.7.3 (default, Apr 24 2019, 15:29:51) [MSC v.1915 64 bit (AMD64)], pyOpenSSL 19.0.0 (OpenSSL 1.1.1e 17 Mar 2020), cryptography 2.7, Platform Windows-10-10.0.18362-SP0 2020-04-23 10:16:30 [scrapy.crawler] INFO: Overridden settings: {'BOT_NAME': 'LianjiaSpider', 'DOWNLOAD_DELAY': 0.25, 'LOG_FILE': 'Spiderlog20200423.txt', 'LOG_LEVEL': 'INFO', 'NEWSPIDER_MODULE': 'LianjiaSpider.spiders', 'SPIDER_MODULES': ['LianjiaSpider.spiders']} 2020-04-23 10:16:30 [scrapy.extensions.telnet] INFO: Telnet Password: 6b0540ee8c8de579 2020-04-23 10:16:30 [scrapy.middleware] INFO: Enabled extensions: ['scrapy.extensions.corestats.CoreStats', 'scrapy.extensions.telnet.TelnetConsole', 'scrapy.extensions.logstats.LogStats'] 2020-04-23 10:16:31 [scrapy.middleware] INFO: Enabled downloader middlewares: ['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware', 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware', 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware', 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware', 'LianjiaSpider.middlewares.ProxyMiddleware', 'LianjiaSpider.middlewares.UserAgentMiddeleware', 'scrapy.downloadermiddlewares.retry.RetryMiddleware', 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware', 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware', 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware', 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware', 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware', 'scrapy.downloadermiddlewares.stats.DownloaderStats'] 2020-04-23 10:16:31 [scrapy.middleware] INFO: Enabled spider middlewares: ['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware', 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware', 'scrapy.spidermiddlewares.referer.RefererMiddleware', 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware', 'scrapy.spidermiddlewares.depth.DepthMiddleware'] 2020-04-23 10:16:31 [scrapy.middleware] INFO: Enabled item pipelines: ['LianjiaSpider.pipelines.CsvPipeline', 'LianjiaSpider.pipelines.MongodbPipeline'] 2020-04-23 10:16:31 [scrapy.core.engine] INFO: Spider opened 2020-04-23 10:16:31 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2020-04-23 10:16:31 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023 2020-04-23 10:16:53 [root] ERROR: https://zz.lianjia.com/ershoufang/104102188631.html save CSV Field error 2020-04-23 10:17:04 [root] ERROR: https://zz.lianjia.com/ershoufang/104103786789.html save CSV Field error 2020-04-23 10:17:04 [root] ERROR: https://zz.lianjia.com/ershoufang/104103785799.html save CSV Field error 2020-04-23 10:17:05 [root] ERROR: https://zz.lianjia.com/ershoufang/104103785683.html save CSV Field error 2020-04-23 10:17:06 [scrapy.core.downloader.handlers.http11] WARNING: Got data loss in https://zz.lianjia.com/ershoufang/baisha/pg72/. If you want to process broken responses set the setting DOWNLOAD_FAIL_ON_DATALOSS = False -- This message won't be shown in further requests 2020-04-23 10:17:06 [root] ERROR: https://zz.lianjia.com/ershoufang/104103783057.html save CSV Field error 2020-04-23 10:17:06 [root] ERROR: https://zz.lianjia.com/ershoufang/104103783011.html save CSV Field error 2020-04-23 10:17:07 [root] ERROR: https://zz.lianjia.com/ershoufang/104103782964.html save CSV Field error 2020-04-23 10:17:12 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 https://zz.lianjia.com/ershoufang/104103670134.html>: HTTP status code is not handled or not allowed 2020-04-23 10:17:31 [scrapy.extensions.logstats] INFO: Crawled 186 pages (at 186 pages/min), scraped 136 items (at 136 items/min) 2020-04-23 10:17:34 [root] ERROR: https://zz.lianjia.com/ershoufang/104103909501.html save CSV Field error 2020-04-23 10:17:51 [scrapy.core.scraper] ERROR: Error processing {'city': '郑州', 'detail_url': 'https://zz.lianjia.com/ershoufang/104103307861.html', 'house_info_dict': {'上次交易': '暂无数据', '交易权属': '商品房', '产权所属': '暂无数据', '供暖方式': '集中供暖', '关注人数': '1', '单价': '8195元/平米', '套内面积': '暂无数据', '小区名称': '金程•名湖山庄', '建筑类型': '板楼', '建筑结构': '混合结构', '建筑面积': '144㎡', '总价': '118万', '户型结构': '平层', '房屋年限': '暂无数据', '房屋户型': '3室2厅1厨2卫', '房屋朝向': '南', '房屋用途': '普通住宅', '房本备件': '未上传房本照片', '所在区域': '中牟县', '所在楼层': '高楼层 (共5层)', '抵押信息': '无抵押', '挂牌时间': '2019-11-24', '梯户比例': '一梯两户', '装修情况': '其他', '配备电梯': '无'}, 'street': '中牟县', 'street_page_url': 'https://zz.lianjia.com/ershoufang/zhongmuxian1/pg99/'} Traceback (most recent call last): File "C:\Users\Shinelon\Anaconda3\lib\site-packages\twisted\internet\defer.py", line 654, in _runCallbacks current.result = callback(current.result, *args, **kw) File "C:\Users\Shinelon\Desktop\bishe_websheji\LianjiaSpider\LianjiaSpider\pipelines.py", line 105, in process_item self.writer.writerow(data) UnicodeEncodeError: 'gbk' codec can't encode character '\u2022' in position 2: illegal multibyte sequence 2020-04-23 10:17:51 [scrapy.core.scraper] ERROR: Error processing {'city': '郑州', 'detail_url': 'https://zz.lianjia.com/ershoufang/104103307403.html', 'house_info_dict': {'上次交易': '暂无数据', '交易权属': '商品房', '产权所属': '非共有', '供暖方式': '集中供暖', '关注人数': '18', '单价': '7155元/平米', '套内面积': '暂无数据', '小区名称': '金程•名湖山庄', '建筑类型': '板楼', '建筑结构': '混合结构', '建筑面积': '123㎡', '总价': '88万', '户型结构': '平层', '房屋年限': '暂无数据', '房屋户型': '3室1厅1厨1卫', '房屋朝向': '南 北', '房屋用途': '普通住宅', '房本备件': '未上传房本照片', '所在区域': '中牟县', '所在楼层': '高楼层 (共6层)', '抵押信息': '有抵押 25万元', '挂牌时间': '2019-11-24', '梯户比例': '一梯两户', '装修情况': '精装', '配备电梯': '无'}, 'street': '中牟县', 'street_page_url': 'https://zz.lianjia.com/ershoufang/zhongmuxian1/pg99/'} Traceback (most recent call last): File "C:\Users\Shinelon\Anaconda3\lib\site-packages\twisted\internet\defer.py", line 654, in _runCallbacks current.result = callback(current.result, *args, **kw) File "C:\Users\Shinelon\Desktop\bishe_websheji\LianjiaSpider\LianjiaSpider\pipelines.py", line 105, in process_item self.writer.writerow(data) UnicodeEncodeError: 'gbk' codec can't encode character