梅馨嫣栀 » Python实现简单网页爬虫及自动告警

文章目录

一、概述

Python是一个高效的脚本语言，用python撰写爬虫具有很大的优势，很多大牛已经分享了爬虫经验，各种花式爬，贴子、图片、视频（舅服爬了1.8PB视频的 beaston02），最近由于刚需驱动，也简单学习了下python爬虫，写了一个能够干活的小爬虫，在这里总结下，希望能给后来者提供一点帮助。

这里的爬虫的场景适合监控某个网站上公告信息，等感兴趣的信息发布出来后，爬虫能够自动监控到，然后可以发邮件或短信自动提醒，这个过程涉及系统登录，验证码识别，网页解析和定时任务。爬虫场景比较简单，这里重点介绍下整个过程所用到的工具，方便后来者可以根据自己需求进行扩展:

邮件提醒：利用smtplib进行邮件发送，利用MIME定义图片、HTML等不同类型的邮件内容；
短信提醒：利用Twilio的试用版本，进行实时短信通知；
模拟登录：利用Selenium+WebDriver进行系统登录，对于无图形化界面操作系统环境，利用PhantomJS进行登录；
网页解析：利用BeautifulSoup解析HTML，分析语法，找到感兴趣内容；
验证码识别：利用PIL图片库，进行图片裁剪，利用百度OCR开发API进行识别；
定时任务： BlockingScheduler自定义定时任务。

二、相关工具

2.1 Selenium2

Selenium 是用于测试 Web 应用程序用户界面 (UI) 的常用框架。 Selenium 能够在一个或多个浏览器中执行测试，WebDriver 由Google的Simon Stewart创建的一个类似的项目，Selenium 1 和 WebDriver 合并成一款性能更佳的产品 Selenium 2。Selenium 2 具有来自 WebDriver 的清晰面向对象 API，并能以最佳的方式与浏览器进行交互。Selenium 2 不使用 JavaScript 沙盒，它支持IE、Firefox、Chrome等多种浏览器和多语言绑定。

安装：

pip install selenium

1	pip install selenium

下载webdriver驱动：
Chrome: https://sites.google.com/a/chromium.org/chromedriver/downloads
Firefox: https://github.com/mozilla/geckodriver/releases
IE: https://npm.taobao.org/mirrors/selenium/3.4/

#!/usr/bin/env python简单示例：

# -*- coding: utf-8 -*-

import time
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
# 打开网页
driver = webdriver.Chrome('C:\Program Files (x86)\chromedriver_win32\chromedriver.exe')
driver.set_window_size(1024, 768)
driver.get('http://zjzone.cc')
# 通过Class名称页面元素，点击页面
elem_firt_post = driver.find_element_by_class_name('post-title')
elem_firt_post.click()
# 搜索框输入文字，回车进入
elem_search = driver.find_element_by_id('s')
elem_search.send_keys('openstack')
elem_search.send_keys(Keys.RETURN)

time.sleep(50)
driver.close()

# -*- coding: utf-8 -*-

import time

from selenium import webdriver

from selenium.webdriver.common.keys import Keys

# 打开网页

driver = webdriver.Chrome('C:\Program Files (x86)\chromedriver_win32\chromedriver.exe')

driver.set_window_size(1024, 768)

driver.get('http://zjzone.cc')

# 通过Class名称页面元素，点击页面

elem_firt_post = driver.find_element_by_class_name('post-title')

elem_firt_post.click()

# 搜索框输入文字，回车进入

elem_search = driver.find_element_by_id('s')

elem_search.send_keys('openstack')

elem_search.send_keys(Keys.RETURN)

time.sleep(50)

driver.close()

教程参考：http://www.seleniumhq.org/docs/03_webdriver.jsp#selenium-webdriver-api-commands-and-operations

2.2 PhantomJS

PhantomJS是一个基于WebKit的服务器端JavaScript API，无需浏览器的支持即可实现对Web的支持，且原生支持各种Web标准，如DOM 处理、JavaScript、CSS选择器、JSON、Canvas和可缩放矢量图形SVG。

PhantomJS是可编程的浏览器，可以执行自定义的js脚本，可以利用js脚本做一些网页测试、状态检测等。

下载安装：在http://phantomjs.org/download.html中下载，解压，将bin/phantomjs.exe放到环境变量中去，可以在CMD命令行中执行phantomjs命令即可。

教程参考：

http://phantomjs.org/quick-start.html

可以试试自带的好玩Demo：https://github.com/ariya/phantomjs/tree/master/examples

在这里，我们主要将它用在无图形化界面的执行脚本程序，比如某个节点最小化安装操作系统，没有安装Firefox或Chrome，我们想运行的selenium程序，或者我们想在后台运行脚本，不让浏览器弹来弹去。这时候可以用PhantomJS代替其他浏览器，比如上面的例子，我们用PhantomJS替换Chrome，一样可以运行，比如下面的例子，我们将Chrome替换为PhantomJS，并利用selenium给网页截图，并保存为文件：

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import time
from selenium import webdriver
from PIL import Image
from selenium.webdriver.common.keys import Keys

driver = webdriver.PhantomJS()
driver.set_window_size(1024, 768)
driver.get('http://zjzone.cc')

snap_image = 'D:\shotsnap.png'
driver.get_screenshot_as_file(snap_image)
im = Image.open(snap_image)
im.show()

time.sleep(50)
driver.close()

#!/usr/bin/env python

# -*- coding: utf-8 -*-

import time

from selenium import webdriver

from PIL import Image

from selenium.webdriver.common.keys import Keys

driver = webdriver.PhantomJS()

driver.set_window_size(1024, 768)

driver.get('http://zjzone.cc')

snap_image = 'D:\shotsnap.png'

driver.get_screenshot_as_file(snap_image)

im = Image.open(snap_image)

im.show()

time.sleep(50)

driver.close()

2.3 Baidu OCR API

百度OCR文字识别服务是百度AI开放平台上提供一项OCR服务，每天免费500次调用，这里主要用来进行验证码识别。

百度 OCR API文档：

http://ai.baidu.com/docs#/OCR-API/top

提供一个Python调用文字识别服务的Demo:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import urllib, urllib2, sys, base64
import ssl
import json

API_KEY = 'XXXXXXXXXXXXXXXXXXXXXX'
SECRET_KEY = 'XXXXXXXXXXXXXXXXXXXXXXXXXXXX'

def get_access_token():
    '''获取access_token'''
    host = 'https://aip.baidubce.com/oauth/2.0/token?grant_type=client_credentials&client_id=' \
       +API_KEY+ '&client_secret=' + SECRET_KEY
    request = urllib2.Request(host)
    request.add_header('Content-Type', 'application/json; charset=UTF-8')
    response = urllib2.urlopen(request)
    content = response.read()
    if (content):
        return json.loads(content)['access_token']
    else:
        return ''

def get_verification_code(access_token, image_path):
    '获取百度AI识别的验证码'
    url = 'https://aip.baidubce.com/rest/2.0/ocr/v1/general?access_token=' + access_token
    # 二进制方式打开图文件
    f = open(image_path,'rb')
    # 参数image：图像base64编码
    img = base64.b64encode(f.read())
    params = {"image": img}
    params = urllib.urlencode(params)
    request = urllib2.Request(url, params)
    request.add_header('Content-Type', 'application/x-www-form-urlencoded')
    response = urllib2.urlopen(request)
    content = response.read()
    if (content):
        return json.loads(content)['words_result'][0]['words']
    else:
        return ''

if __name__ == '__main__':
    code_image = 'D:\image_code.png'
    access_token = get_access_token()
    code = get_verification_code(access_token, code_image)
    print code

#!/usr/bin/env python

# -*- coding: utf-8 -*-

import urllib, urllib2, sys, base64

import ssl

import json

API_KEY = 'XXXXXXXXXXXXXXXXXXXXXX'

SECRET_KEY = 'XXXXXXXXXXXXXXXXXXXXXXXXXXXX'

def get_access_token():

'''获取access_token'''

host = 'https://aip.baidubce.com/oauth/2.0/token?grant_type=client_credentials&client_id=' \

+API_KEY+ '&client_secret=' + SECRET_KEY

request = urllib2.Request(host)

request.add_header('Content-Type', 'application/json; charset=UTF-8')

response = urllib2.urlopen(request)

content = response.read()

if (content):

return json.loads(content)['access_token']

else:

return ''

def get_verification_code(access_token, image_path):

'获取百度AI识别的验证码'

url = 'https://aip.baidubce.com/rest/2.0/ocr/v1/general?access_token=' + access_token

# 二进制方式打开图文件

f = open(image_path,'rb')

# 参数image：图像base64编码

img = base64.b64encode(f.read())

params = {"image": img}

params = urllib.urlencode(params)

request = urllib2.Request(url, params)

request.add_header('Content-Type', 'application/x-www-form-urlencoded')

response = urllib2.urlopen(request)

content = response.read()

if (content):

return json.loads(content)['words_result'][0]['words']

else:

return ''

if __name__ == '__main__':

code_image = 'D:\image_code.png'

access_token = get_access_token()

code = get_verification_code(access_token, code_image)

print code

注意：API_KEY 和SECRET_KEY 需换成个人的开发密钥。

2.4 PIL

PIL是Python图像处理的库，具备丰富的图像处理能力，功能主要包括图像储存、图像显示、格式转换以及基本的图像处理操作等。 Pillow 库则是 PIL 的一个分支，维护和开发活跃，Pillow 兼容 PIL 的绝大多数语法，推荐使用。

安装：

pip install Pillow

1	pip install Pillow

用户手册：

http://pillow.readthedocs.io/en/3.4.x/handbook/index.html

PIL图像处理简单示例如下，主要操作是图像模糊处理及旋转：

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import os,sys
from PIL import Image,ImageDraw,ImageFont,ImageFilter

origin_img = 'D:/tiger.bmp'
blur_img = 'D:/tmp.bmp'

# 图像打开
im = Image.open(origin_img)
# 图像滤波：模糊处理
im1 = im.filter(ImageFilter.BLUR)
im1.show()
# 图像保存
im1.save(blur_img)
# 图像旋转
im2 = im.transpose(Image.ROTATE_270)
im2.show()
im.close()

#!/usr/bin/env python

# -*- coding: utf-8 -*-

import os,sys

from PIL import Image,ImageDraw,ImageFont,ImageFilter

origin_img = 'D:/tiger.bmp'

blur_img = 'D:/tmp.bmp'

# 图像打开

im = Image.open(origin_img)

# 图像滤波：模糊处理

im1 = im.filter(ImageFilter.BLUR)

im1.show()

# 图像保存

im1.save(blur_img)

# 图像旋转

im2 = im.transpose(Image.ROTATE_270)

im2.show()

im.close()

2.5 BeautifulSoup4

Beautiful Soup 是一个可以从HTML或XML文件中提取数据的Python库.它能够通过转换器实现文档的导航，查找和修改。Beautiful Soup是一个强大的网页解析工具，支持正则表达式，能够大大提升文档解析的效率。

安装：

pip install beautifulsoup4

1	pip install beautifulsoup4

使用文档：

https://www.crummy.com/software/BeautifulSoup/bs4/doc/index.zh.html

BeautifulSoup4网页解析的一个简单示例：

#!/usr/bin/env python
# -*- coding: utf-8 -*-

from selenium import webdriver
from bs4 import BeautifulSoup

driver = webdriver.PhantomJS()
driver.set_window_size(1024, 768)
driver.get('http://zjzone.cc')
doc = driver.page_source

soup = BeautifulSoup(doc, 'lxml')
# 格式化缩进
print soup.prettify()
# 文档信息
print soup.title
print soup.title.string
print soup.p
print soup.a
print soup.find_all('a')
# 提取文档中所有超链接
for link in soup.find_all('a'):
    print link.get('href')

# 按CSS类名进行提取
for link in soup.find_all("a", class_="navbar-brand"):
    print link.get('href')

# 是否存在某个属性来提取
for link in soup.select('a[href]'):
    print link

# 按层级来进行提取
for link in soup.select("body a"):
    print link

#!/usr/bin/env python

# -*- coding: utf-8 -*-

from selenium import webdriver

from bs4 import BeautifulSoup

driver = webdriver.PhantomJS()

driver.set_window_size(1024, 768)

driver.get('http://zjzone.cc')

doc = driver.page_source

soup = BeautifulSoup(doc, 'lxml')

# 格式化缩进

print soup.prettify()

# 文档信息

print soup.title

print soup.title.string

print soup.p

print soup.a

print soup.find_all('a')

# 提取文档中所有超链接

for link in soup.find_all('a'):

print link.get('href')

# 按CSS类名进行提取

for link in soup.find_all("a", class_="navbar-brand"):

print link.get('href')

# 是否存在某个属性来提取

for link in soup.select('a[href]'):

print link

# 按层级来进行提取

for link in soup.select("body a"):

print link

2.6 STMP和MIME

smtplib模块无需过多介绍，python2自带的邮件客户端。

使用文档：https://docs.python.org/2/library/smtplib.html

一个简单的邮件发送客户端示例：

#!/usr/bin/env python
# -*- coding: utf-8 -*-

from smtplib import SMTP

SMTPSVR = 'smtp.163.com'
sender = 'HuaTengMa@163.com'
password = 'YourNerverGuess'
receiver = 'JackMa@163.com'

origHdrs = ['From:' + sender,
            'TO:'+ receiver,
            'Subject:Test msg from python client']
origBody = ['Test', 'Please do not replay', 'THK!']
origMsg = '\r\n\r\n'.join(['\r\n'.join(origHdrs),
                           '\r\n'.join(origBody)])

def send_mail():
    try:
        handle = SMTP(SMTPSVR)
        handle.login(sender, password)
        handle.sendmail(sender, receiver, origMsg)
        print "邮件发送成功！"
        handle.close()
    except:
        print '邮件发送失败！'

if __name__ == "__main__":
    send_mail()

#!/usr/bin/env python

# -*- coding: utf-8 -*-

from smtplib import SMTP

SMTPSVR = 'smtp.163.com'

sender = 'HuaTengMa@163.com'

password = 'YourNerverGuess'

receiver = 'JackMa@163.com'

origHdrs = ['From:' + sender,

'TO:'+ receiver,

'Subject:Test msg from python client']

origBody = ['Test', 'Please do not replay', 'THK!']

origMsg = '\r\n\r\n'.join(['\r\n'.join(origHdrs),

'\r\n'.join(origBody)])

def send_mail():

try:

handle = SMTP(SMTPSVR)

handle.login(sender, password)

handle.sendmail(sender, receiver, origMsg)

print "邮件发送成功！"

handle.close()

except:

print '邮件发送失败！'

if __name__ == "__main__":

send_mail()

MIME(Multipurpose Internet Mail Extensions) 多用途互联网邮件扩展是描述消息内容类型的因特网标准，其消息包含文本、图像、音频、视频以及其他应用程序专用的数据。它扩展了电子邮件标准，使其能够支持：

非ASCII字符文本；
非文本格式附件（二进制、声音、图像等）；
由多部分（multiple parts）组成的消息体；
包含非ASCII字符的头信息（Header information）。

借助MIME，我们可以发送一些负责的邮件，比如发送HTML格式的邮件，添加非文本格式的附件等等。

这里提供一个能够发送HTML格式的邮件示例，同时上传图片附件，并将附件添加到邮件正文，如下：

#!/usr/bin/python
# -*- coding: UTF-8 -*-

import smtplib,os
from email.header import Header
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText
from email.mime.application import MIMEApplication
from email.mime.image import MIMEImage
from email.utils import parseaddr, formataddr

sender = 'vps@163.com'
password = 'yournerverguess'
receivers = 'zj@163.com,vps@163.com'
attachments = 'D:\excel.xls,D:\logs.zip'
img1 = 'D:\home_count_1.png'
img2 = 'D:\home_count_2.png'

def format_addr(s):
    '''格式化From/To邮件地址,支持中文内容'''
    name, addr = parseaddr(s)
    return formataddr(( \
        Header(name, 'utf-8').encode(), \
        addr.encode('utf-8') if isinstance(addr, unicode) else addr))

    #构造附件
def uploadattachments():
    if attachments != -1 and attachments != '':
        for att in attachments.split(','):
            os.path.isfile(att)
            name = os.path.basename(att)
            att = MIMEText(open(att).read(), 'base64', 'utf-8')
            att["Content-Type"] = 'application/octet-stream'
            #将编码方式为utf-8的name，转码为unicode，然后再转成gbk(否则，附件带中文名的话会出现乱码)
            att["Content-Disposition"] = 'attachment; filename=%s' % name.decode('utf-8').encode('gbk')
            msg.attach(att)

def send_mail():
    try:
        s = smtplib.SMTP("smtp.163.com")  # 连接smtp邮件服务器,端口默认是25
        s.login(sender, password)  # 登陆服务器
        uploadattachments()
        for receiver in receivers.split(','):
            s.sendmail(sender, [receiver], msg.as_string())  # 发送邮件
        s.close()
        print '发送邮件成功！'
    except :
        print '发送邮件失败！'


msg = MIMEMultipart('alternative')
msg['Subject'] = u'资源可用提醒' # 利用Header设置utf-8中文编码
msg['From'] = format_addr(u'服务器自动通知 <%s>' % sender)
msg['To'] = receivers

mail_msg = """<html><body>
<p>邮件提醒：已经存在可申请资源,请尽快登录
</p><p><a href="http://www.bj.gov.cn">
http://www.bj.gov.cn</a></p>
<p>资源信息统计：</p>
<p><img src="cid:image1" height="300" width="300"></p>
<p><img src="cid:image2" height="100" width="300" ></p>
<p>本邮件来自服务器自动提醒服务，请勿回复</p>
</body></html>"""

# 如果收件人设备不支持查看HTML邮件，还可以查看纯文本内容
msg.attach(MIMEText(mail_msg, 'plain', 'utf-8'))
msg.attach(MIMEText(mail_msg, 'html', 'utf-8'))

# 指定图片为邮件正文图片元素
fp = open(img1, 'rb')
msgImage = MIMEImage(fp.read())
fp.close()
msgImage.add_header('Content-ID', '<image1>')
msg.attach(msgImage)
fp = open(img2, 'rb')
msgImage = MIMEImage(fp.read())
fp.close()
msgImage.add_header('Content-ID', '<image2>')
msg.attach(msgImage)

if __name__ == '__main__':
    send_mail()

#!/usr/bin/python

# -*- coding: UTF-8 -*-

import smtplib,os

from email.header import Header

from email.mime.multipart import MIMEMultipart

from email.mime.text import MIMEText

from email.mime.application import MIMEApplication

from email.mime.image import MIMEImage

from email.utils import parseaddr, formataddr

sender = 'vps@163.com'

password = 'yournerverguess'

receivers = 'zj@163.com,vps@163.com'

attachments = 'D:\excel.xls,D:\logs.zip'

img1 = 'D:\home_count_1.png'

img2 = 'D:\home_count_2.png'

def format_addr(s):

'''格式化From/To邮件地址,支持中文内容'''

name, addr = parseaddr(s)

return formataddr(( \

Header(name, 'utf-8').encode(), \

addr.encode('utf-8') if isinstance(addr, unicode) else addr))

#构造附件

def uploadattachments():

if attachments != -1 and attachments != '':

for att in attachments.split(','):

os.path.isfile(att)

name = os.path.basename(att)

att = MIMEText(open(att).read(), 'base64', 'utf-8')

att["Content-Type"] = 'application/octet-stream'

#将编码方式为utf-8的name，转码为unicode，然后再转成gbk(否则，附件带中文名的话会出现乱码)

att["Content-Disposition"] = 'attachment; filename=%s' % name.decode('utf-8').encode('gbk')

msg.attach(att)

def send_mail():

try:

s = smtplib.SMTP("smtp.163.com") # 连接smtp邮件服务器,端口默认是25

s.login(sender, password) # 登陆服务器

uploadattachments()

for receiver in receivers.split(','):

s.sendmail(sender, [receiver], msg.as_string()) # 发送邮件

s.close()

print '发送邮件成功！'

except :

print '发送邮件失败！'

msg = MIMEMultipart('alternative')

msg['Subject'] = u'资源可用提醒' # 利用Header设置utf-8中文编码

msg['From'] = format_addr(u'服务器自动通知 <%s>' % sender)

msg['To'] = receivers

mail_msg = """<html><body>

邮件提醒：已经存在可申请资源,请尽快登录

http://www.bj.gov.cn</a>

资源信息统计：

本邮件来自服务器自动提醒服务，请勿回复

</body></html>"""

# 如果收件人设备不支持查看HTML邮件，还可以查看纯文本内容

msg.attach(MIMEText(mail_msg, 'plain', 'utf-8'))

msg.attach(MIMEText(mail_msg, 'html', 'utf-8'))

# 指定图片为邮件正文图片元素

fp = open(img1, 'rb')

msgImage = MIMEImage(fp.read())

fp.close()

msgImage.add_header('Content-ID', '<image1>')

msg.attach(msgImage)

fp = open(img2, 'rb')

msgImage = MIMEImage(fp.read())

fp.close()

msgImage.add_header('Content-ID', '<image2>')

msg.attach(msgImage)

if __name__ == '__main__':

send_mail()

示例邮件发送的几个功能点：

邮件的正文既可以纯文本、又可以是HTML正文，MIMEMultipart(‘alternative’)和后面设定MIMEText(mail_msg, ‘plain’, ‘utf-8’）、MIMEText(mail_msg, ‘html’, ‘utf-8’)对应；
From和To中利用Header做了utf-8编码的设定，支持中文；
图片插入到Html邮件正文，注意用法；
支持多附件上传，遍历附件，一一上传；
支持多个收件者，遍历收件者，一一发送；

收到邮件的效果：

2.7 Twilio

“Twilio是一个专注通讯服务的开放PaaS平台、是一个提供技术能力的网站，也是美国较为知名的云计算通讯服务类的初创企业。Twilio通过将复杂的底层通信功能打包成 API 并对外开放，让 web、桌面及移动应用可以方便地嵌入短信、语音及 VoIP 功能，从而实现云通信的功能。”

作者：张十三来源：知乎链接：https://www.zhihu.com/question/19751074/answer/17669577
通知告警中，我们用到了Twilio的短信服务，这里简单介绍下使用：

安装：

pip install twilio

1	pip install twilio

然后在https://www.twilio.com注册，体验SMS服务即可，服务试用大概有500条短信，省着点用已经够了。

API调用文档：https://www.twilio.com/docs/api/messaging/send-messages

一个简单的Twilio发送短信的客户端示例：

#!/usr/bin/python
# -*- coding: utf-8 -*-

from twilio.rest import Client

def send_msg(msg_body):
    # Your Account SID from twilio.com/console
    account_sid = "XXXXXXXXXXXXXXXXXX"
    # Your Auth Token from twilio.com/console
    auth_token  = "XXXXXXXXXXXXXXXXXXXXXXX"
    client = Client(account_sid, auth_token)
    message = client.messages.create(
    to="+86155********",
    from_="+14433962255",
    body=" "+msg_body)
    if (message.sid):
        print 'Success'
    else:
        print 'Error'

if __name__ == '__main__':
    send_msg('Test')

#!/usr/bin/python

# -*- coding: utf-8 -*-

from twilio.rest import Client

def send_msg(msg_body):

# Your Account SID from twilio.com/console

account_sid = "XXXXXXXXXXXXXXXXXX"

# Your Auth Token from twilio.com/console

auth_token = "XXXXXXXXXXXXXXXXXXXXXXX"

client = Client(account_sid, auth_token)

message = client.messages.create(

to="+86155********",

from_="+14433962255",

body=" "+msg_body)

if (message.sid):

print 'Success'

else:

print 'Error'

if __name__ == '__main__':

send_msg('Test')

注意：account_sid和auth_token 需换成个人账户及授权码。

2.8 BlockingScheduler

定时任务的功能windows和linux操作系统都提供，Linux下配下crontab就行，window下设置下任务计划即可。这里介绍python下定时任务框架的BlockingScheduler，基于日期、固定时间间隔以及crontab类型的任务。

安装：

pip install apscheduler

1	pip install apscheduler

用户文档：

http://apscheduler.readthedocs.io/en/latest/userguide.html

一个简单的定时任务示例：

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import home_login
import logging
from apscheduler.schedulers.blocking import BlockingScheduler
from datetime import datetime

sched = BlockingScheduler()
@sched.scheduled_job('interval', minutes=8)
def timed_job():
    print(datetime.now().strftime("%Y-%m-%d %H:%M:%S"))
    home_login.do_work()
#@sched.scheduled_job('cron', day_of_week='mon-fri', hour='0-24', minute='30-59', second='*/3')
def scheduled_job():
    print('This job is run every weekday at 5pm.')

print('before the start funciton')
sched.start()
print("let us figure out the situation")

#!/usr/bin/env python

# -*- coding: utf-8 -*-

import home_login

import logging

from apscheduler.schedulers.blocking import BlockingScheduler

from datetime import datetime

sched = BlockingScheduler()

@sched.scheduled_job('interval', minutes=8)

def timed_job():

print(datetime.now().strftime("%Y-%m-%d %H:%M:%S"))

home_login.do_work()

#@sched.scheduled_job('cron', day_of_week='mon-fri', hour='0-24', minute='30-59', second='*/3')

def scheduled_job():

print('This job is run every weekday at 5pm.')

print('before the start funciton')

sched.start()

print("let us figure out the situation")

三、源码

经过上面工具的介绍，根据实际的场景构造自己的爬虫脚本就变的非常容易，下面简单介绍下之前实现的一个简单爬虫场景：需求是实时监控某个页面，当页面出现特定内容时，主动发送邮件和短信通知，这个场景在系统日常运维中十分常见，比如个人博客系统中有没有新的留言，集群管理中服务或节点有没有Down掉等等。

GitHub源码：https://github.com/zjmeixinyanzhi/Auto_Login-Alert

3.1 系统登录

系统登录之前的《两个简单的Python自动登录脚本》中已经详细介绍，

这里实现是将OCR文字识别将调用SDK换成调用百度API来实现：

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import urllib, urllib2, sys, base64, ssl, json, time

class BaiduAIOCR:

    def __init__(self, api_key, secret_key):
        self.api_key = api_key
        self.secret_key = secret_key

    def get_access_token(self):
        '''获取access_token'''
        host = 'https://aip.baidubce.com/oauth/2.0/token?grant_type=client_credentials&amp;client_id=' \
               + self.api_key + '&amp;client_secret=' + self.secret_key
        request = urllib2.Request(host)
        request.add_header('Content-Type', 'application/json; charset=UTF-8')
        response = urllib2.urlopen(request)
        content = response.read()
        if (content):
            return json.loads(content)['access_token']
        else:
            return ''

    def do_recognition(self, access_token, image):
        '获取百度AI识别的图片内容'
        url = 'https://aip.baidubce.com/rest/2.0/ocr/v1/general?access_token=' + access_token
        # 二进制方式打开图文件
        f = open(image, 'rb')
        # 参数image：图像base64编码
        img = base64.b64encode(f.read())
        params = {"image": img}
        params = urllib.urlencode(params)
        request = urllib2.Request(url, params)
        request.add_header('Content-Type', 'application/x-www-form-urlencoded')
        response = urllib2.urlopen(request)
        content = response.read()
        if (content):
            try:
                return json.loads(content)['words_result'][0]['words']
            except:
                print '识别文字错误！'
                return ''
        else:
            return ''


if __name__ == '__main__':
    API_KEY = 'XXXXXXXXXXXXXXX'
    SECRET_KEY = 'XXXXXXXXXXXXXXXXXXX'
    IMG = 'D:\home_count_1.png'
    ocr = BaiduAIOCR(API_KEY,SECRET_KEY)
    access_token = ocr.get_access_token()
    text = ocr.do_recognition(access_token,IMG)
    print text

#!/usr/bin/env python

# -*- coding: utf-8 -*-

import urllib, urllib2, sys, base64, ssl, json, time

class BaiduAIOCR:

def __init__(self, api_key, secret_key):

self.api_key = api_key

self.secret_key = secret_key

def get_access_token(self):

'''获取access_token'''

host = 'https://aip.baidubce.com/oauth/2.0/token?grant_type=client_credentials&client_id=' \

+ self.api_key + '&client_secret=' + self.secret_key

request = urllib2.Request(host)

request.add_header('Content-Type', 'application/json; charset=UTF-8')

response = urllib2.urlopen(request)

content = response.read()

if (content):

return json.loads(content)['access_token']

else:

return ''

def do_recognition(self, access_token, image):

'获取百度AI识别的图片内容'

url = 'https://aip.baidubce.com/rest/2.0/ocr/v1/general?access_token=' + access_token

# 二进制方式打开图文件

f = open(image, 'rb')

# 参数image：图像base64编码

img = base64.b64encode(f.read())

params = {"image": img}

params = urllib.urlencode(params)

request = urllib2.Request(url, params)

request.add_header('Content-Type', 'application/x-www-form-urlencoded')

response = urllib2.urlopen(request)

content = response.read()

if (content):

try:

return json.loads(content)['words_result'][0]['words']

except:

print '识别文字错误！'

return ''

else:

return ''

if __name__ == '__main__':

API_KEY = 'XXXXXXXXXXXXXXX'

SECRET_KEY = 'XXXXXXXXXXXXXXXXXXX'

IMG = 'D:\home_count_1.png'

ocr = BaiduAIOCR(API_KEY,SECRET_KEY)

access_token = ocr.get_access_token()

text = ocr.do_recognition(access_token,IMG)

print text

登录页面示例：

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import time, os
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from PIL import Image,ImageEnhance
from baidu_ai_ocr import BaiduAIOCR

def do_login():
    # 加载Chrome浏览器驱动，打开网页
    driver = webdriver.Chrome('C:\Program Files (x86)\chromedriver_win32\chromedriver.exe')
    driver.set_window_size(1302, 923)
    driver.get("XXXXXXXXXXXXXX")

    # 验证码图片截图，注意截图框的坐标
    snap_image = 'D:\shotsnap.png'
    code_image = 'D:\image_code.png'
    driver.get_screenshot_as_file(snap_image)
    im = Image.open(snap_image)
    box = (1265, 470, 1371, 505)
    region = im.crop(box)
    region.save(code_image)

    # 识别验证码
    API_KEY = 'XXXXXXXXXXXXXXXX'
    SECRET_KEY = 'XXXXXXXXXXXXXXX'
    ocr = BaiduAIOCR(API_KEY, SECRET_KEY)
    access_token = ocr.get_access_token()
    if access_token is '':
        exit()
    code = ocr.do_recognition(access_token, code_image)
    print code

    # 登录页面
    elem_user = driver.find_element_by_id("tbxUserName")
    elem_user.send_keys("XXXXXXXXXXX")
    elem_pwd = driver.find_element_by_id("tbxPassword")
    elem_pwd.send_keys("XXXXXXXXXXXXX")
    elem_verify_code = driver.find_element_by_id("tbxCaptcha")
    elem_verify_code.send_keys(code)
    elem_pwd.send_keys(Keys.RETURN)

    # 返回html
    html_doc = driver.page_source
    driver.quit()
    return html_doc

if __name__ == '__main__':
    doc = do_login()
    print doc

#!/usr/bin/env python

# -*- coding: utf-8 -*-

import time, os

from selenium import webdriver

from selenium.webdriver.common.keys import Keys

from PIL import Image,ImageEnhance

from baidu_ai_ocr import BaiduAIOCR

def do_login():

# 加载Chrome浏览器驱动，打开网页

driver = webdriver.Chrome('C:\Program Files (x86)\chromedriver_win32\chromedriver.exe')

driver.set_window_size(1302, 923)

driver.get("XXXXXXXXXXXXXX")

# 验证码图片截图，注意截图框的坐标

snap_image = 'D:\shotsnap.png'

code_image = 'D:\image_code.png'

driver.get_screenshot_as_file(snap_image)

im = Image.open(snap_image)

box = (1265, 470, 1371, 505)

region = im.crop(box)

region.save(code_image)

# 识别验证码

API_KEY = 'XXXXXXXXXXXXXXXX'

SECRET_KEY = 'XXXXXXXXXXXXXXX'

ocr = BaiduAIOCR(API_KEY, SECRET_KEY)

access_token = ocr.get_access_token()

if access_token is '':

exit()

code = ocr.do_recognition(access_token, code_image)

print code

# 登录页面

elem_user = driver.find_element_by_id("tbxUserName")

elem_user.send_keys("XXXXXXXXXXX")

elem_pwd = driver.find_element_by_id("tbxPassword")

elem_pwd.send_keys("XXXXXXXXXXXXX")

elem_verify_code = driver.find_element_by_id("tbxCaptcha")

elem_verify_code.send_keys(code)

elem_pwd.send_keys(Keys.RETURN)

# 返回html

html_doc = driver.page_source

driver.quit()

return html_doc

if __name__ == '__main__':

doc = do_login()

print doc

3.2 页面解析

TODO

3.3 邮件告警

TODO

3.4 短信告警

TODO

3.5 定时任务

TODO

3.6 运行示例

TODO

四、参考文档

https://www.ibm.com/developerworks/cn/web/wa-selenium2/index.html

http://www.infoq.com/cn/news/2015/01/phantomjs-webkit-javascript-api

https://liam0205.me/2015/04/22/pil-tutorial-basic-usage/

https://zh.wikipedia.org/wiki/%E5%A4%9A%E7%94%A8%E9%80%94%E4%BA%92%E8%81%AF%E7%B6%B2%E9%83%B5%E4%BB%B6%E6%93%B4%E5%B1%95

https://www.liaoxuefeng.com/wiki/001374738125095c955c1e6d8bb493182103fac9270762a000/001386832745198026a685614e7462fb57dbf733cc9f3ad000

http://blog.leanote.com/post/fengfengdiandia/%E4%BD%BF%E7%94%A8

赏

Markdown	Result
text	text
text	text
*text*	text
`code`	`code`
~~~ more code ~~~~	more code
[Link](http://www.example.com)	Link
* Listitem	Listitem
> Quote	Quote

Python实现简单网页爬虫及自动告警

一、概述

二、相关工具

2.1 Selenium2

2.2 PhantomJS

2.3 Baidu OCR API

2.4 PIL

2.5 BeautifulSoup4

2.6 STMP和MIME

2.7 Twilio

2.8 BlockingScheduler

三、源码

3.1 系统登录

3.2 页面解析

3.3 邮件告警

3.4 短信告警

3.5 定时任务

3.6 运行示例

四、参考文档

One comment

近期文章

浏览统计

分类目录

文章归档

近期评论

功能

Python实现简单网页爬虫及自动告警

一、概述

二、相关工具

2.1 Selenium2

2.2 PhantomJS

2.3 Baidu OCR API

2.4 PIL

2.5 BeautifulSoup4

2.6 STMP和MIME

2.7 Twilio

2.8 BlockingScheduler

三、源码

3.1 系统登录

3.2 页面解析

3.3 邮件告警

3.4 短信告警

3.5 定时任务

3.6 运行示例

四、参考文档

近期文章

标签云

浏览统计

分类目录

文章归档

近期评论

功能