CURRENT PATH:
/
opt
/
gsutil
/
gslib
/
__pycache__
/
[ ⬅ KEMBALI ]
|
[ HOME ]
Upload File:
Upload Ke Sini
Dir Baru
File Baru
Editing:
boto_resumable_upload.cpython-39.pyc
a +(Wg�b � @ s d Z ddlmZ ddlmZ ddlmZ ddlmZ ddlZddlZddlZddl Z ddl Z ddl Z ddl m Z ddl mZ dd lmZ dd lmZ dd lmZ dd lmZ dd lmZ ddlmZ ddlmZ ddlmZ ddl m Z ddl m Z e j r�e Z!G dd� de"�Z#dS )ap Boto translation layer for resumable uploads. See https://cloud.google.com/storage/docs/resumable-uploads-xml for details. Resumable uploads will retry interrupted uploads, resuming at the byte count completed by the last upload attempt. If too many retries happen with no progress (per configurable num_retries param), the upload will be aborted in the current process. Unlike the boto implementation of resumable upload handler, this class does not directly interact with tracker files. Originally Google wrote and contributed this code to the boto project, then copied that code back into gsutil on the release of gsutil 4.0 which supports both boto and non-boto codepaths for resumable uploads. Any bug fixes made to this file should also be integrated to resumable_upload_handler.py in boto, where applicable. TODO: gsutil-beta: Add a similar comment to the boto code. � )�absolute_import)�print_function)�division)�unicode_literalsN)�urllib)� http_client)�config)� UserAgent)�AWSAuthConnection)� ResumableTransferDisposition)�ResumableUploadException)�InvalidUrlError)�GetMaxRetryDelay)� GetNumRetries)�XML_PROGRESS_CALLBACKS)�UTF8c @ s� e Zd ZdZdZejeej ej fZ dZ d dd�Z dd� Zd d d �Zd d � Zdd� Zd dd�Zdd� Zdd� Zdd� Zd dd�Zddefdd �ZdS )!�BotoResumableUploadz3Upload helper class for resumable uploads via boto.i )r ���Nc C s8 |r| � |� nd| _|| _d| _d| _|| _|| _dS )ae Constructor. Instantiate once for each uploaded file. Args: tracker_callback: Callback function that takes a string argument. Used by caller to track this upload across upload interruption. logger: logging.logger instance to use for debug messages. resume_url: If present, attempt to resume the upload at this URL. num_retries: Number of times to retry the upload making no progress. This count resets every time we make progress, so the upload can span many more than this number of retries. Nr )� _SetUploadUrl� upload_url� num_retries�service_has_bytes�upload_start_point�tracker_callback�logger)�selfr r � resume_urlr � r �*/opt/gsutil/gslib/boto_resumable_upload.py�__init__U s z BotoResumableUpload.__init__c C sd t j�|�}|j�� dvs |js,td| ��|| _t� ddd�pD|j| _ d|j |j f | _ d| _dS )z�Saves URL and resets upload state. Called when we start a new resumable upload or get a new tracker URL for the upload. Args: url: URL string for the upload. Raises InvalidUrlError if URL is syntactically invalid. )�http�httpszInvalid upload URL (%s)� Credentials�gs_hostNz%s?%sr )r �parse�urlparse�scheme�lower�netlocr r r �get�upload_url_host�path�query�upload_url_pathr )r �url� parse_resultr r r r r s � �z!BotoResumableUpload._SetUploadUrl�*c C s d||f S )Nz bytes %s/%sr )r � range_specZ length_specr r r �_BuildContentRangeHeader� s z,BotoResumableUpload._BuildContentRangeHeaderc C s. | � d|�dd�}tj|d| j| j|| jd�S )a� Queries service to find out state of given upload. Note that this method really just makes special case use of the fact that the upload service always returns the current start/end state whenever a PUT doesn't complete. Args: conn: HTTPConnection to use for the query. file_length: Total length of the file. Returns: HTTP response from sending request. Raises: ResumableUploadException if problem querying service. r0 �0)� Content-Range�Content-Length�PUT�r+ � auth_path�headers�host)r2 r � make_requestr- r* )r �conn� file_length� put_headersr r r �_QueryServiceState� s ��z&BotoResumableUpload._QueryServiceStatec C s� | � ||�}|jdkr"d|d fS |jdkr>td|j tj��d}|�d�}|r�t�d|�}|r�t|� d��}t|� d ��}d }n| j S |s�td t |� � � tj��|j dkr�| j� d ||� ||fS ) a� Queries service to find out what bytes it currently has. Args: conn: HTTPConnection to use for the query. file_length: Total length of the file. Returns: (service_start, service_end), where the values are inclusive. For example, (0, 2) would mean that the service has bytes 0, 1, *and* 2. Raises: ResumableUploadException if problem querying service. �� r � i4 z2Got non-308 response (%s) from service state queryF�rangezbytes=(\d+)-(\d+)� Tz7Couldn't parse upload service state query response (%s)z Service has: Range: %d - %d.)r? �statusr r � START_OVER� getheader�re�search�long�group�SERVICE_HAS_NOTHING�str� getheaders�debugr ) r r< r= �respZgot_valid_responser1 �m� service_start� service_endr r r �_QueryServicePos� s: � �� �z$BotoResumableUpload._QueryServicePosc C s� |j j}|jdkr | j�d� d| _i }|D ](}|�� dkrHtdtj��|| ||< q,d||j j < |� d|j j |j |�}|� � }|jdv r�td |j tj��n&|jd kr�|jd kr�td |j tj��|�d �}|s�td| tj��| �|� | �|� dS )z�Starts a new resumable upload. Args: key: Boto Key representing the object to upload. headers: Headers to use in the upload requests. Raises: ResumableUploadException if any errors occur. rA z Starting new resumable upload.r zcontent-lengthz5Attempt to specify Content-Length header (disallowed)�start�POST)� �� � zEGot status %d from attempt to start resumable upload. Will wait/retryr@ �� z>Got status %d from attempt to start resumable upload. Aborting�LocationzHNo resumable upload URL found in resumable initiation POST response (%s)N)�bucket� connectionrN r r r' r r �ABORT�provider�resumable_upload_headerr; �name�readrD �WAIT_BEFORE_RETRYrF r r ) r �keyr9 r< Z post_headers�krO �bodyr r r r �_StartNewResumableUpload� sN � ���� �� z,BotoResumableUpload._StartNewResumableUploadc C s@ |� | j�} |rL|dkr,|| j |d } n|dk r:d} nd} d} |||� |rX|�� ni } |r�||krv| �d|�} n| �d||d f |�} | | d< t|| �| d< tj|d | jd | | jd �}|� d |j � | D ]}|� || | � q�|� � |� d� | �r�tj�r |�| � |t| �7 }nDt| t��rD|�| � |t| �7 }n | �t�}|�|� |t|�7 }|�r�| d7 } | | k�s�| dk�r�|||� d} |� | j�} q�|� |j� |�r�|||� ||k�r�td ||f tj��|�� }|jd k�r |� d�|� d�|� d�fS |jdv �r tj }ntj}td|j|j f |��d S )ad Attempts to upload file bytes. Makes a single attempt using an existing resumable upload connection. Args: conn: HTTPConnection from the boto Key. http_conn: Separate HTTPConnection for the transfer. fp: File pointer containing bytes to upload. file_length: Total length of the file. total_bytes_uploaded: The total number of bytes uploaded. cb: Progress callback function that takes (progress, total_size). num_cb: Granularity of the callback (maximum number of times the callback will be called during the file transfer). If negative, perform callback with each buffer read. headers: Headers to be used in the upload requests. Returns: (etag, generation, metageneration) from service upon success. Raises: ResumableUploadException if any problems occur. rC r r r0 z%d-%drA r4 r5 r6 Nr7 z<File changed during upload: EOF at %d bytes of %d byte file.r@ �etagzx-goog-generationzx-goog-metageneration)i� rV rW rX z1Got response code %d while attempting upload (%s)) ra � BUFFER_SIZE�copyr2 rL r �build_base_http_requestr- r* � putrequestr+ � putheader� endheaders�set_debuglevel�six�PY2�send�len� isinstance�bytes�encoder rN r r r] � getresponserD rF rb �reason)r r< � http_conn�fpr= �total_bytes_uploaded�cb�num_cbr9 �buf�cb_count�ir> � range_header� http_requestrd Z buf_bytesrO � dispositionr r r �_UploadFileBytes s� �� �� � ��z$BotoResumableUpload._UploadFileBytesc C sx | j \}}|jj} | jr�z0| �| |�\}}|| _| jdkrF| j�d� W q� ty� } z2| jdkrr| j�d| j � | � ||� W Y d} ~ q�d} ~ 0 0 n | � ||� | j du r�|| _ |d } | |k r�|� | � |jj} | � | j| j| j�} | �| j� ztz$| �| | ||| |||�W W | �� S ttjf�y\ | �| |�} | jdk�rVtdtj��n� Y n0 W | �� n | �� 0 dS )a� Attempts a resumable upload. Args: key: Boto key representing object to upload. fp: File pointer containing upload bytes. file_length: Total length of the upload. headers: Headers to be used in upload requests. cb: Progress callback function that takes (progress, total_size). num_cb: Granularity of the callback (maximum number of times the callback will be called during the file transfer). If negative, perform callback with each buffer read. Returns: (etag, generation, metageneration) from service upon success. Raises: ResumableUploadException if any problems occur. rA zResuming transfer.z Unable to resume transfer (%s).Ni� z�Got 400 response from service state query after failed resumable upload attempt. This can happen for various reasons, including specifying an invalid request (e.g., an invalid canned ACL) or if the file size changed between upload attempts)rK r[ r\ r rS r rN r r �messagerf r �seek�new_http_connectionr* �port� is_securern r� �close�socket�errorr? rD r r] )r rc ry r= r9 r{ r| rQ rR r< �erz rx rO r r r �_AttemptResumableUpload� sP � $ � �� � z+BotoResumableUpload._AttemptResumableUploadc C s| |j tjkr(|dkr$| j�d|j� � nP|j tjkrP|dkrL| j�d|j� � n(|j tjkr`� n|dkrx| j�d|j� d S )NrA zWCaught non-retryable ResumableUploadException (%s); aborting but retaining tracker filezVCaught non-retryable ResumableUploadException (%s); aborting and removing tracker filez1Caught ResumableUploadException (%s) - will retry)r� r �ABORT_CUR_PROCESSr rN r� r] rE )r r� rN r r r � HandleResumableUploadException� s( � � �z2BotoResumableUpload.HandleResumableUploadExceptionr c C sz | j |krd| _n| jd7 _| j| jkr8tdtj��tt�� d| j t� �}|dkrl| j � d| j|� t � |� dS )a Tracks the number of iterations without progress. Performs randomized exponential backoff. Args: service_had_bytes_before_attempt: Number of bytes the service had prior to this upload attempt. debug: debug level 0..3 r rA zaToo many resumable upload attempts failed without progress. You might try this upload again laterrC zZGot retryable failure (%d progress-less in a row). Sleeping %3.1f seconds before re-tryingN) r �progress_less_iterationsr r r r� �min�randomr r rN �time�sleep)r � service_had_bytes_before_attemptrN �sleep_time_secsr r r �TrackProgressLessIterations� s" ���z/BotoResumableUpload.TrackProgressLessIterationsc C sZ |si }d}||v r&|| du r&||= |r6|||j j< t|d< |} |jjj} | jdu r^t� | _d| _| j } z@| � ||| |||�\} | _ | _ | j |_ | dkr�| j �d� W dS | j�y } zN| dkr�| j �d| �� � t| t��r | jtjk�r |jjj�� W Y d} ~ n:d} ~ 0 t�yD } z| �| | � W Y d} ~ n d} ~ 0 0 | j| | d� qddS ) a� Upload a file to a key into a bucket on GS, resumable upload protocol. Args: key: `boto.s3.key.Key` or subclass representing the upload destination. fp: File pointer to upload size: Size of the file to upload. headers: The headers to pass along with the PUT request canned_acl: Optional canned ACL to apply to object. cb: Callback function that will be called to report progress on the upload. The callback should accept two integer parameters, the first representing the number of bytes that have been successfully transmitted to GS, and the second representing the total number of bytes that need to be transmitted. num_cb: (optional) If a callback is specified with the cb parameter, this parameter determines the granularity of the callback by defining the maximum number of times the callback will be called during the file transfer. Providing a negative integer will cause your callback to be called with each buffer read. Raises: ResumableUploadException if a problem occurs during the transfer. z Content-TypeNz User-Agentr rA zResumable upload complete.zCaught exception (%s))rN )r^ � acl_headerr r[ r\ rN r r r� r r� � generation�metagenerationr �RETRYABLE_EXCEPTIONS�__repr__rs �IOError�errno�EPIPEr� r r� r� )r rc ry �sizer9 � canned_aclr{ r| � content_typer= rN r� �_r� r r r �SendFile s@ � ""�z BotoResumableUpload.SendFile)NN)r0 r0 )N)r )�__name__� __module__� __qualname__�__doc__rh r � HTTPExceptionr� r� r� �gaierrorr� rK r r r2 r? rS rf r� r� r� r� r r� r r r r r J s. � � : <wO � (�r )$r� � __future__r r r r r� r� rG r� r� ro � six.movesr r �botor r �boto.connectionr �boto.exceptionr r �gslib.exceptionr �gslib.utils.boto_utilr r �gslib.utils.constantsr r �PY3�intrI �objectr r r r r �<module> s2
SIMPAN
[ BATAL ]