CURRENT PATH:
/
opt
/
gsutil
/
gslib
/
__pycache__
/
[ ⬅ KEMBALI ]
|
[ HOME ]
Upload File:
Upload Ke Sini
Dir Baru
File Baru
Editing:
parallel_tracker_file.cpython-39.pyc
a +(Wg�0 � @ s� d Z ddlmZ ddlmZ ddlmZ ddlmZ ddlmZ ddlZddl Z ddl Z ddl Z ddl Z ddl mZ dd lmZmZ dd lmZ ed d �ZG d d� de�Zdd� Zdd� Zdd� Zddd�Zd dd�Zd dd�Z dS ) zAHelper functions for composite upload tracker file functionality.� )�absolute_import)�print_function)�division)�unicode_literals)� namedtupleN)�CommandException)�WriteJsonDataToTrackerFile�#RaiseUnwritableTrackerFileException)�UTF8�ObjectFromTrackerzobject_name generationc @ s$ e Zd ZdZdZdZdZdZdZdS )� _CompositeUploadTrackerEntryz7Enum class for composite upload tracker file JSON keys.� components�component_nameZcomponent_generation�encryption_key_sha256�prefixN) �__name__� __module__� __qualname__�__doc__�COMPONENTS_LIST�COMPONENT_NAME�COMPONENT_GENERATION� ENC_SHA256�PREFIX� r r �*/opt/gsutil/gslib/parallel_tracker_file.pyr & s r c C s d}d}g }d}z�z`t | d�}|�� }t�|�}|tj }|tj }|tj D ] }|�t |tj |tj �� qNW nv t y� } z(| j t jkr�|�d| | j� W Y d} ~ n@d} ~ 0 ttfy� } z d}t|�\}}W Y d} ~ n d} ~ 0 0 W |r�|�� n|�r|�� 0 |||fS )a� Read the tracker file from the last parallel composite upload attempt. If it exists, the tracker file is of the format described in WriteParallelUploadTrackerFile or a legacy format. If the file doesn't exist or is formatted incorrectly, then the upload will start from the beginning. This function is not thread-safe and must be protected by a lock if called within Command.Apply. Args: tracker_file_name: The name of the tracker file to read parse. logger: logging.Logger for outputting log messages. Returns: enc_key_sha256: Encryption key SHA256 used to encrypt the existing components, or None if an encryption key was not used. component_prefix: String prefix used in naming the existing components, or None if no prefix was found. existing_components: A list of ObjectFromTracker objects representing the set of files that have already been uploaded. N�rz^Couldn't read upload tracker file (%s): %s. Restarting parallel composite upload from scratch.)�open�read�json�loadsr r r r �appendr r r �IOError�errno�ENOENT�warn�strerror�KeyError� ValueError�_ParseLegacyTrackerData�close) �tracker_file_name�logger�enc_key_sha256r �existing_components� tracker_file� tracker_data� tracker_json� component�er r r � ReadParallelUploadTrackerFile/ s@ �� �$ � r4 c C sz | � d�dd� }d}g }|rr|d }d}|t|�d k rr|| ||d }}|sXd}|�t||�� |d7 }q*||fS )at Parses a legacy parallel composite upload tracker file. Args: tracker_data: Legacy tracker file contents. Returns: component_prefix: The prefix used in naming the existing components, or None if no prefix was found. existing_components: A list of ObjectFromTracker objects representing the set of files that have already been uploaded. � N���r � � )�split�lenr! r )r0 Zold_tracker_datar r. �i�name� generationr r r r) k s r) c C s� t jr.t|t�r|�t�}t|t�r.|�t�}|r�||kr�zR|�d| � g } |D ] } |�� } | j| _| j | _ qP|j || | t j j |jjd� W n, dd� |D �} |�dd�| �� Y n0 dg fS ||fS )a� Validates that tracker data matches the current encryption key. If the data does not match, makes a best-effort attempt to delete existing temporary component objects encrypted with the old key. Args: tracker_file_name: String file name of tracker file. existing_enc_sha256: Encryption key SHA256 used to encrypt the existing components, or None if an encryption key was not used. existing_prefix: String prefix used in naming the existing components, or None if no prefix was found. existing_components: A list of ObjectFromTracker objects representing the set of files that have already been uploaded. current_enc_key_sha256: Current Encryption key SHA256 that should be used to encrypt objects. bucket_url: Bucket URL in which the components exist. command_obj: Command class for calls to Apply. logger: logging.Logger for outputting log messages. delete_func: command.Apply-callable function for deleting objects. delete_exc_handler: Exception handler for delete_func. Returns: prefix: existing_prefix, or None if the encryption key did not match. existing_components: existing_components, or empty list if the encryption key did not match. z�Upload tracker file (%s) does not match current encryption key. Deleting old components and restarting upload from scratch with a new tracker file that uses the current encryption key.)� arg_checker� parallel_operations_overridec S s g | ] }|j �qS r )� object_name)�.0r2 r r r � <listcomp>� s z8ValidateParallelCompositeTrackerData.<locals>.<listcomp>zrFailed to delete some of the following temporary objects: %s (Continuing on to re-upload components from scratch.)r5 N)�six�PY3� isinstance�str�encoder r% �Cloner@ r= �Apply�gslib�command�DummyArgChecker�ParallelOverrideReason�SPEED�join)r+ Zexisting_enc_sha256�existing_prefixr. Zcurrent_enc_key_sha256� bucket_url� command_objr, Z delete_funcZdelete_exc_handlerZcomponents_to_deleter2 �urlZcomponent_namesr r r �$ValidateParallelCompositeTrackerData� sB � � �� rT c C s t t�dd�t| � d �S )aN Generates a random prefix for component objects. Args: encryption_key_sha256: Encryption key SHA256 that will be used to encrypt the components. This is hashed into the prefix to avoid collision during resumption with a different encryption key. Returns: String prefix for use in the composite upload. r7 l �c( l d( )rF �random�randint�hash�r r r r � GenerateComponentObjectPrefix� s �rY c C sr |�Z t | |�\}}}||kr2td|||jf ��|g}|| } t| || |d� W d � n1 sd0 Y dS )a� Rewrites an existing tracker file with info about the uploaded component. Follows the format described in _CreateParallelUploadTrackerFile. Args: tracker_file_name: Tracker file to append to. tracker_file_lock: Thread and process-safe Lock protecting the tracker file. component: ObjectFromTracker describing the object that was uploaded. logger: logging.Logger for outputting log messages. encryption_key_sha256: Encryption key SHA256 for use in this upload, if any. zygsutil client error: encryption key SHA256 (%s) in tracker file does not match encryption key SHA256 (%s) of component %srX N)r4 r r@ � WriteParallelUploadTrackerFile) r+ �tracker_file_lockr2 r, r �existing_enc_key_sha256r r. Znewly_completed_componentsZcompleted_componentsr r r �)WriteComponentToParallelUploadTrackerFile� s* ������r] c C sf t jrt|t�r|�d�}g }|D ] }|�tj|jtj |j i� q"tj |tj |tj |i}t| |� dS )a� Writes information about components that were successfully uploaded. The tracker file is serialized JSON of the form: { "encryption_key_sha256": sha256 hash of encryption key (or null), "prefix": Prefix used for the component objects, "components": [ { "component_name": Component object name, "component_generation": Component object generation (or null), }, ... ] } where N is the number of components that have been successfully uploaded. This function is not thread-safe and must be protected by a lock if called within Command.Apply. Args: tracker_file_name: The name of the parallel upload tracker file. prefix: The generated prefix that used for uploading any existing components. components: A list of ObjectFromTracker objects that were uploaded. encryption_key_sha256: Encryption key SHA256 for use in this upload, if any. �asciiN)rC rD rE �bytes�decoder! r r r@ r r= r r r r )r+ r r r Ztracker_componentsr2 �tracker_file_datar r r rZ s ��rZ )N)N)N) r � __future__r r r r � collectionsr r# r rU rC rJ �gslib.exceptionr �gslib.tracker_filer r �gslib.utils.constantsr r �objectr r4 r) rT rY r] rZ r r r r �<module> s. < I � % �
SIMPAN
[ BATAL ]