Hacked By AnonymousFox
B
�A�[� � @ s� d dl mZmZmZ d dlmZ d dlmZ d dlm Z
m Z e
e_ e e_ dgZ
G dd� de�ZG dd� de�ZG d d
� d
e�Zd S ) � )�absolute_import�division�unicode_literals)�str)�urllib)�parse�request�RobotFileParserc @ sZ e Zd ZdZddd�Zdd� Zdd� Zd d
� Zd d � Zd
d� Z dd� Z
dd� Z dd� Z dS )r zs This class provides a set of methods to read, parse and answer
questions about a single robots.txt file.
� c C s, g | _ d | _d| _d| _| �|� d| _d S )NFr )�entries�
default_entry� disallow_all� allow_all�set_url� last_checked)�self�url� r �T/opt/alt/python37/lib/python3.7/site-packages/future/backports/urllib/robotparser.py�__init__ s
zRobotFileParser.__init__c C s | j S )z�Returns the time the robots.txt file was last fetched.
This is useful for long-running web spiders that need to
check for new robots.txt files periodically.
)r )r r r r �mtime& s zRobotFileParser.mtimec C s ddl }|� � | _dS )zYSets the time the robots.txt file was last fetched to the
current time.
r N)�timer )r r r r r �modified/ s zRobotFileParser.modifiedc C s& || _ tj�|�dd� \| _| _dS )z,Sets the URL referring to a robots.txt file.� � N)r r r �urlparse�host�path)r r r r r r 7 s zRobotFileParser.set_urlc
C s| yt j�| j�}W nH t jjk
rZ } z&|jdkr:d| _n|jdkrJd| _W dd}~X Y n X |� � }| �
|� d�� � � dS )z4Reads the robots.txt URL and feeds it to the parser.)i� i� Ti� Nzutf-8)
r r Zurlopenr �errorZ HTTPError�coder
r �readr �decode�
splitlines)r �f�err�rawr r r r < s
zRobotFileParser.readc C s, d|j kr | jd kr(|| _n | j�|� d S )N�*)�
useragentsr r �append)r �entryr r r �
_add_entryI s
zRobotFileParser._add_entryc C st d}t � }�xN|D �]D}|sL|dkr0t � }d}n |dkrL| �|� t � }d}|�d�}|dkrj|d|� }|�� }|sxq|�dd�}t|�dkr|d �� �� |d< tj� |d �� �|d< |d dkr�|dkr�| �|� t � }|j
� |d � d}q|d dk�r*|dk�rX|j � t
|d d �� d}q|d d
kr|dkr|j � t
|d d �� d}qW |dk�rp| �|� dS ) z�Parse the input lines from a robots.txt file.
We allow that a user-agent: line is not preceded by
one or more blank lines.
r r � �#N�:z
user-agentZdisallowFZallowT)�Entryr* �find�strip�split�len�lowerr r �unquoter' r( � rulelines�RuleLine)r �lines�stater) �line�ir r r r R sJ
zRobotFileParser.parsec C s� | j r
dS | jrdS tj�tj�|��}tj�dd|j|j|j |j
f�}tj� |�}|s\d}x"| j D ]}|�
|�rd|�|�S qdW | jr�| j�|�S dS )z=using the parsed robots.txt decide if useragent can fetch urlFTr
�/)r
r r r r r4 �
urlunparser �params�query�fragment�quoter �
applies_to� allowancer )r � useragentr �
parsed_urlr) r r r � can_fetch� s
zRobotFileParser.can_fetchc C s d� dd� | jD ��S )Nr
c S s g | ]}t |�d �qS )�
)r )�.0r) r r r �
<listcomp>� s z+RobotFileParser.__str__.<locals>.<listcomp>)�joinr )r r r r �__str__� s zRobotFileParser.__str__N)r
)
�__name__�
__module__� __qualname__�__doc__r r r r r r* r rE rJ r r r r r s
3c @ s( e Zd ZdZdd� Zdd� Zdd� ZdS ) r6 zoA rule line is a single "Allow:" (allowance==True) or "Disallow:"
(allowance==False) followed by a path.c C s( |dkr|sd}t j�|�| _|| _d S )Nr
T)r r r@ r rB )r r rB r r r r � s zRuleLine.__init__c C s | j dkp|�| j �S )Nr& )r �
startswith)r �filenamer r r rA � s zRuleLine.applies_toc C s | j r
dp dd | j S )NZAllowZDisallowz: )rB r )r r r r rJ � s zRuleLine.__str__N)rK rL rM rN r rA rJ r r r r r6 � s r6 c @ s0 e Zd ZdZdd� Zdd� Zdd� Zdd � Zd
S ) r. z?An entry has one or more user-agents and zero or more rulelinesc C s g | _ g | _d S )N)r' r5 )r r r r r � s zEntry.__init__c C sP g }x | j D ]}|�d|dg� q W x | jD ]}|�t|�dg� q,W d�|�S )Nz User-agent: rF r
)r' �extendr5 r rI )r �ret�agentr9 r r r rJ � s z
Entry.__str__c C sF |� d�d �� }x.| jD ]$}|dkr*dS |�� }||krdS qW dS )z2check if this entry applies to the specified agentr; r r& TF)r1 r3 r' )r rC rS r r r rA � s zEntry.applies_toc C s$ x | j D ]}|�|�r|jS qW dS )zZPreconditions:
- our agent applies to this entry
- filename is URL decodedT)r5 rA rB )r rP r9 r r r rB � s
zEntry.allowanceN)rK rL rM rN r rJ rA rB r r r r r. � s
r. N)�
__future__r r r Zfuture.builtinsr Zfuture.backportsr Zfuture.backports.urllibr �_parser Z_request�__all__�objectr r6 r. r r r r �<module> s
Hacked By AnonymousFox1.0, Coded By AnonymousFox