CURRENT PATH:
/
opt
/
gsutil
/
gslib
/
__pycache__
/
[ ⬅ KEMBALI ]
|
[ HOME ]
Upload File:
Upload Ke Sini
Dir Baru
File Baru
Editing:
tracker_file.cpython-39.pyc
a +(Wg�` � @ s� d Z ddlmZ ddlmZ ddlmZ ddlmZ ddlZddlZddlZddl Z ddl Z ddl Z ddl Z ddl mZ ddlmZ dd lmZ dd lmZ dd lmZ dd lmZ dd lmZ dZdZdZ dZ G dd� de �Z dd� Z dd� Z!dd� Z"d8dd�Z#d d � Z$d9d d �Z%d d!� Z&d"d#� Z'd:d$d%�Z(d&d'� Z)d(d)� Z*d;d*d+�Z+d<d,d-�Z,d.d/� Z-d0d1� Z.d2d3� Z/d=d4d5�Z0d6d7� Z1dS )>z0Helper functions for tracker file functionality.� )�absolute_import)�print_function)�division)�unicode_literalsN)�config)�CommandException)�GetGsutilStateDir)�ResumableThreshold)�UTF8)�GetMd5)�CreateDirIfNeeded�d zCouldn't write tracker file (%s): %s. This can happen if gsutil is configured to save tracker files to an unwritable directory)�encryption_key_sha256�serialization_datac @ s$ e Zd ZdZdZdZdZdZdZdS )�TrackerFileType�upload�downloadZdownload_componentZparallel_uploadZsliced_download�rewriteN) �__name__� __module__� __qualname__�UPLOAD�DOWNLOAD�DOWNLOAD_COMPONENT�PARALLEL_UPLOAD�SLICED_DOWNLOAD�REWRITE� r r �!/opt/gsutil/gslib/tracker_file.pyr 4 s r c C sP t | tj�r | �t�}| }n| }| �t�}t�|�}d|�� d |dd� S )a" Apply a hash function (SHA1) to shorten the passed file name. The spec for the hashed file name is as follows: TRACKER_<hash>_<trailing> where hash is a SHA1 hash on the original file name and trailing is the last 16 chars from the original file name. Max file name lengths vary by operating system so the goal of this function is to ensure the hashed version takes fewer than 100 characters. Args: filename: file name to be hashed. May be unicode or bytes. Returns: shorter, hashed version of passed file name ZTRACKER_�.i�N) � isinstance�six� text_type�encoder �decode�hashlib�sha1� hexdigest)�filename�filename_bytesZ filename_str�mr r r � _HashFilename= s r+ c C s& t �ddtj�t� d��} t| � | S )a Looks up or creates the gsutil tracker file directory. This is the configured directory where gsutil keeps its resumable transfer tracker files. This function creates it if it doesn't already exist. Returns: The pathname to the tracker directory. �GSUtil�resumable_tracker_dirz tracker-files)r �get�os�path�joinr r )Z tracker_dirr r r �CreateTrackerDirIfNeededY s �r2 c C s( t �ddd| ||||f �}t|tj�S )aU Gets the tracker file name described by the arguments. Args: src_bucket_name: Source bucket (string). src_obj_name: Source object (string). dst_bucket_name: Destination bucket (string). dst_obj_name: Destination object (string) api_selector: API to use for this operation. Returns: File path to tracker file. �[/\\]�_z!rewrite__%s__%s__%s__%s__%s.token)�re�sub�_HashAndReturnPathr r )�src_bucket_nameZ src_obj_name�dst_bucket_name� dst_obj_name� api_selector�res_tracker_file_namer r r �GetRewriteTrackerFilePathh s ���r= c C s� |t jkr(t�ddd| j| j|f �}n�|t jkrTt�dddtj� | j�|f �}n�|t j kr�t�dddtj� | j�||f �}nf|t j kr�t�ddd| j| j||f �}n<|t j kr�t�dddtj� | j�|f �}n|t j kr�t� �t||�S )a� Gets the tracker file name described by the arguments. Args: dst_url: Destination URL for tracker file. tracker_file_type: TrackerFileType for this operation. api_selector: API to use for this operation. src_url: Source URL for the source file name for parallel uploads. component_num: Component number if this is a download component, else None. Returns: File path to tracker file. r3 r4 z resumable_upload__%s__%s__%s.urlz resumable_download__%s__%s.etagz#resumable_download__%s__%s__%d.etagz#parallel_upload__%s__%s__%s__%s.urlz sliced_download__%s__%s.etag)r r r5 r6 � bucket_name� object_namer r/ r0 �realpathr r r r �NotImplementedErrorr7 )�dst_url�tracker_file_typer; �src_url� component_numr< r r r �GetTrackerFilePath� sB �� �� �� �� �� rF c C s2 t t| tj|�� t| |�}|D ] }t |� q dS )z�Deletes all tracker files corresponding to an object download. Args: dst_url: StorageUrl describing the destination file. api_selector: The Cloud API implementation used. N)�DeleteTrackerFilerF r r �!GetSlicedDownloadTrackerFilePaths)rB r; Z tracker_files� tracker_filer r r �DeleteDownloadTrackerFiles� s � rJ c C s� t | tj|�}|g}|du r�d}zVz t|d�}t�|�d }W n( ttfyf | Y W |rd|�� S 0 W |r�|�� n|r�|�� 0 t |�D ] }|� t | tj ||d�� q�|S )a Gets a list of sliced download tracker file paths. The list consists of the parent tracker file path in index 0, and then any existing component tracker files in [1:]. Args: dst_url: Destination URL for tracker file. api_selector: API to use for this operation. num_components: The number of component tracker files, if already known. If not known, the number will be retrieved from the parent tracker file on disk. Returns: File path to tracker file. N�r�num_components�rE ) rF r r �open�json�load�IOError� ValueError�close�range�appendr )rB r; rL Zparallel_tracker_file_pathZtracker_file_pathsrI �ir r r rH � s6 � � � ��rH c C sF t � }t| �}dt|��� |f }d|tj|f }t|�tk sBJ �|S )z�Hashes and returns a tracker file path. Args: res_tracker_file_name: The tracker file name prior to it being hashed. tracker_file_type: The TrackerFileType of res_tracker_file_name. Returns: Final (hashed) tracker file path. z%s_%sz%s%s%s)r2 r+ �str�lowerr/ �sep�len� MAX_TRACKER_FILE_NAME_LENGTH)r< rC r- Zhashed_tracker_file_name�tracker_file_nameZtracker_file_pathr r r r7 � s ��r7 c C s | rt j�| �rt �| � d S )N)r/ r0 �exists�unlink)r\ r r r rG s rG c C sz | r*| j r*| jr*| jr*|r*|j r*|jr*|s.dS t� } | ||||||| ||| f D ]"} | durN| �t�| ��d�� qN| �� S )a` Creates an MD5 hex digest of the parameters for a rewrite call. Resuming rewrites requires that the input parameters are identical. Thus, the rewrite tracker file needs to represent the input parameters. For easy comparison, hash the input values. If a user does a performs a same-source/same-destination rewrite via a different command (for example, with a changed ACL), the hashes will not match and we will restart the rewrite from the beginning. Args: src_obj_metadata: apitools Object describing source object. Must include bucket, name, and etag. dst_obj_metadata: apitools Object describing destination object. Must include bucket and object name projection: Projection used for the API call. src_generation: Optional source generation. gen_match: Optional generation precondition. meta_gen_match: Optional metageneration precondition. canned_acl: Optional canned ACL string. max_bytes_per_call: Optional maximum bytes rewritten per call. src_dec_key_sha256: Optional SHA256 hash string of decryption key for source object. dst_enc_key_sha256: Optional SHA256 hash string of encryption key for destination object. fields: Optional fields to include in response to call. Returns: MD5 hex digest Hash of the input parameters, or None if required parameters are missing. Nr ) �bucket�name�etagr �updater! r" r# r' ) �src_obj_metadata�dst_obj_metadata� projection�src_generation� gen_match�meta_gen_match� canned_acl�max_bytes_per_call�src_dec_key_sha256�dst_enc_key_sha256�fields�md5_hashZ input_paramr r r �HashRewriteParameters s< ) �������ro c C s� d}|s dS z�zBt | d�}|�� �d�}||krN|�� �d�W W |rL|�� S W nF ty� } z.|jtjkr�tj� d| |j f � W Y d}~n d}~0 0 W |r�|�� n|r�|�� 0 dS )a~ Attempts to read a rewrite tracker file. Args: tracker_file_name: Tracker file path string. rewrite_params_hash: MD5 hex digest of rewrite call parameters constructed by HashRewriteParameters. Returns: String rewrite_token for resuming rewrite requests if a matching tracker file exists, None otherwise (which will result in starting a new rewrite). NrK � zGCouldn't read Copy tracker file (%s): %s. Restarting copy from scratch.) rN �readline�rstriprS rQ �errno�ENOENT�sys�stderr�write�strerror)r\ �rewrite_params_hashrI Z existing_hash�er r r �ReadRewriteTrackerFileK s, � �� �r{ c C s t | d||f � dS )a Writes a rewrite tracker file. Args: tracker_file_name: Tracker file path string. rewrite_params_hash: MD5 hex digest of rewrite call parameters constructed by HashRewriteParameters. rewrite_token: Rewrite token string returned by the service. z%s %s N)�_WriteTrackerFile)r\ ry Z rewrite_tokenr r r �WriteRewriteTrackerFilel s �r} c C s� | j s J �d}| jt� k r"||fS |j}|du r8tj} ntj} |d| 7 }t|| ||d�}d} �zz�t|d�} | tju r�| � � � d�} | | j kr�||fW W | r�| � � S nP| tju r�t � | �� �} | d | j kr�| d | jkr�|| d fW W | r�| � � S |�d | � W nZ ttf�yd } z<t| t��s:| jtjk�rP|�d |t| �f � W Y d} ~ n d} ~ 0 0 W | �r�| � � n| �r�| � � 0 | tju �r�t|d | j � n| tju �r�t|| |� ||fS ) a? Checks for a download tracker file and creates one if it does not exist. The methodology for determining the download start point differs between normal and sliced downloads. For normal downloads, the existing bytes in the file are presumed to be correct and have been previously downloaded from the server (if a tracker file exists). In this case, the existing file size is used to determine the download start point. For sliced downloads, the number of bytes previously retrieved from the server cannot be determined from the existing file size, and so the number of bytes known to have been previously downloaded is retrieved from the tracker file. Args: src_obj_metadata: Metadata for the source object. Must include etag and generation. dst_url: Destination URL for tracker file. logger: For outputting log messages. api_selector: API to use for this operation. start_byte: The start byte of the byte range for this download. existing_file_size: Size of existing file for this download on disk. component_num: The component number, if this is a component of a parallel download, else None. Returns: tracker_file_name: The name of the tracker file, if one was used. download_start_byte: The first byte that still needs to be downloaded. Nz component %drM rK rp ra � generation�download_start_bytezPTracker file doesn't match for download of %s. Restarting download from scratch.zOCouldn't read download tracker file (%s): %s. Restarting download from scratch.z%s )ra �sizer r? r r r rF rN rq rr rS rO �loads�readr~ �warnrQ rR r rs rt rW r| �!WriteDownloadComponentTrackerFile)rc rB �loggerr; � start_byte�existing_file_sizerE r\ Z download_namerC rI � etag_value�component_datarz r r r � ReadOrCreateDownloadTrackerFilez sh ! � � � �� � � �r� c C s | j s J �d}| jt� k r |S |du r.tj}ntj}t||||d�}d}z�z�t|d�}|tju r�|�� � d�} | | j kr�|W W |r�|� � S nL|tju r�t � |� � �} | d | j kr�| d | jkr�| d W W |r�|� � S W n ttfy� Y n0 W |�r|� � n|�r|� � 0 |S )af Returns the download starting point. The methodology of this function is the same as in ReadOrCreateDownloadTrackerFile, with the difference that we are not interested here in possibly creating a tracker file. In case there is no tracker file, this means the download starting point is start_byte. Args: src_obj_metadata: Metadata for the source object. Must include etag and generation. dst_url: Destination URL for tracker file. api_selector: API to use for this operation. start_byte: The start byte of the byte range for this download. existing_file_size: Size of existing file for this download on disk. component_num: The component number, if this is a component of a parallel download, else None. Returns: download_start_byte: The first byte that still needs to be downloaded. NrM rK rp ra r~ r )ra r� r r r r rF rN rq rr rS rO r� r� r~ rQ rR ) rc rB r; r� r� rE r\ rC rI r� r� r r r �GetDownloadStartByte� sL � � � � � r� c C s$ |j |j|d�}t| t�|�� dS )z�Updates or creates a download component tracker file on disk. Args: tracker_file_name: The name of the tracker file. src_obj_metadata: Metadata for the source object. Must include etag. current_file_pos: The current position in the file. )ra r~ r N)ra r~ r| rO �dumps)r\ rc Zcurrent_file_posr� r r r r� s �r� c C s� zXt �| t jt jB t jB d�}t �|d��}|�|� W d � n1 sJ0 Y W dS ttfy� } zt | |j ��W Y d}~n d}~0 0 dS )z/Creates a tracker file, storing the input data.i� �wNF) r/ rN �O_WRONLY�O_CREAT�O_TRUNC�fdopenrw rQ �OSError�#RaiseUnwritableTrackerFileExceptionrx )r\ �data�fdZtfrz r r r r| + s �(r| c C sN zt �|�}W n0 ty> } zt| |j��W Y d}~n d}~0 0 t| |� dS )zrCreate a tracker file and write json data to it. Raises: TypeError: If the data is not JSON serializable N)rO r� � TypeErrorr� rx r| )r\ r� �json_str�errr r r �WriteJsonDataToTrackerFile7 s "r� c C s� d}d}d}�zVzjt | d�}|�� }t�|�}|t |krDd}d}n4|t W W |rZ|�� |rj|�d| � |rvt| � S W n� t y� } z(|j t j kr�|�d| |j � W Y d}~n�d}~0 t tf�y2 } z\d}|dur�d}n>|W Y d}~W |r�|�� |�r|�d| � |�r t| � S W Y d}~n d}~0 0 W |�rD|�� |�rV|�d| � |�r�t| � n0|�rt|�� |�r�|�d| � |�r�t| � 0 dS )a� Reads tracker data from an upload tracker file if it exists. Deletes the tracker file if it uses an old format or the desired encryption key has changed. Args: tracker_file_name: Tracker file name for this upload. logger: logging.Logger for outputting log messages. encryption_key_sha256: Encryption key SHA256 for use in this upload, if any. Returns: Serialization data if the tracker file already exists (resume existing upload), None otherwise. NFrK Tz�Upload tracker file (%s) does not match current encryption key. Restarting upload from scratch with a new tracker file that uses the current encryption key.zKCouldn't read upload tracker file (%s): %s. Restarting upload from scratch.)rN r� rO r� � ENCRYPTION_UPLOAD_TRACKER_ENTRY�"SERIALIZATION_UPLOAD_TRACKER_ENTRYrS r� rG rQ rs rt rx �KeyErrorrR ) r\ r� r rI Zremove_tracker_fileZencryption_restartZ tracker_dataZ tracker_jsonrz r r r �GetUploadTrackerDataD sx �� ���� ��r� c C s t t| |f ��dS )z:Raises an exception when unable to write the tracker file.N)r �&TRACKER_FILE_UNWRITABLE_EXCEPTION_TEXT)r\ Z error_strr r r r� s �r� )NN)N)NNNNNNNN)N)N)N)2�__doc__� __future__r r r r rs r% rO r/ r5 ru r! �botor �gslib.exceptionr �gslib.utils.boto_utilr r �gslib.utils.constantsr �gslib.utils.hashing_helperr Zgslib.utils.system_utilr r[ r� r� r� �objectr r+ r2 r= rF rJ rH r7 rG ro r{ r} r� r� r� r| r� r� r� r r r r �<module> sn � � 3 � + � C! � ^ � F ;
SIMPAN
[ BATAL ]