CURRENT PATH:
/
opt
/
gsutil
/
gslib
/
__pycache__
/
[ ⬅ KEMBALI ]
|
[ HOME ]
Upload File:
Upload Ke Sini
Dir Baru
File Baru
Editing:
boto_translation.cpython-39.pyc
a +(WgV � @ sV d Z ddlmZ ddlmZ ddlmZ ddlmZ ddlZddlZddlZddl Z ddl Z ddl Z ddl Z ddl Z ddlZddlZddlZddlZddlZddlZddlZddlmZ ddlmZ ddlZdd l m Z ddl Z dd l m Z dd l m Z dd l!m"Z" dd l#m$Z$ ddl%m&Z' ddl(m)Z) ddl*m+Z, ddl-m.Z. ddl/m0Z0 ddl1Z ddl2Z ddl3m4Z4 ddl5m6Z6 ddl5m7Z7 ddl5m8Z8 ddl5m9Z9 ddl5m:Z: ddl5m;Z; ddl5m<Z< ddl5m=Z= dd l5m>Z> dd l5m?Z? dd l5m@Z@ dd l5mAZA ddlBZCdd lDmEZE dd!lDmFZF dd"lGmHZH dd#lGmIZI dd$lJmKZK dd%lJmLZL dd&lMmNZO dd'lPmQZQ dd(lRmSZS dd)lRmTZT dd*lRmUZU dd+lVmWZW dd,lVmXZX dd-lYmZZZ dd.lYm[Z[ dd/lYm\Z\ dd0lYm]Z] dd1lYm^Z^ dd2lYm_Z_ dd3l`maZa dd4l`mbZb dd5lcmdZd dd6lemfZf dd7lgmhZh dd8limjZj dd9limkZk dd:limlZl dd;limmZm dd<limnZn dd=limoZo dd>limpZp dd?limqZq dd@limrZr ddAlimsZs ddBlimtZt ddClimuZu ddDlimvZv ddElwmxZx ejy�r�ezZ{e j|j}e j|j~e j|j=e j|j?e j|je j|j�fZ�e��� a�dFa�ej�dGej�dH�Z�e��dI�Z�dJdK� Z�dLdM� Z�G dNdO� dOe��Z�G dPdQ� dQe9�Z�dS )Rz?XML/boto gsutil Cloud API implementation for GCS and Amazon S3.� )�absolute_import)�print_function)�division)�unicode_literalsN)� parseString)� _exceptions)� http_client)�config)�handler)�Cors)�LifecycleConfig)�CORSConfiguration)� DeleteMarker)� Lifecycle)�Prefix)�Tags)�BotoResumableUpload)�AccessDeniedException)�ArgumentException)�BadRequestException)�CloudApi)�NotEmptyException)�NotFoundException)�PreconditionException)�ResumableDownloadException)� ResumableUploadAbortException)�ResumableUploadException)�!ResumableUploadStartOverException)�ServiceException)�CommandException)�InvalidUrlError)�GOOG_PROJ_ID_HDR)�PopulateProjectId)�GenerationFromUrlAndString)�StorageUrlFromString)�storage_v1_messages)�parallelism_framework_util)�ConfigureNoOpAuthIfNeeded)�GetMaxRetryDelay)� GetNumRetries)�ListToGetFields)�ValidateDstObjectMetadata)�DEFAULT_FILE_BUFFER_SIZE)�REQUEST_REASON_ENV_VAR)�REQUEST_REASON_HEADER_KEY)�S3_DELETE_MARKER_GUID)�UTF8)�XML_PROGRESS_CALLBACKS)�Base64EncodeHash)�Base64ToHexHash)� AddAcceptEncodingGzipIfNeeded)�multiprocessing_context)�EncodeStringAsLong)�AclTranslation)� AddS3MarkerAclToObjectMetadata)�CorsTranslation)� CreateBucketNotFoundException)�%CreateNotFoundExceptionForObjectWrite)� CreateObjectNotFoundException)�DEFAULT_CONTENT_TYPE)�HeadersFromObjectMetadata)�LabelTranslation)�LifecycleTranslation)�REMOVE_CORS_CONFIG)� S3MarkerAclFromObjectMetadata)�UnaryDictToXml)�TWO_MIBFz.*non-\s*existent\s*object)�flagsz^"*[a-fA-F0-9]{32}"*$c C sH dddd�}|� � D ].\}}t�d|d�}|durt| jj||� qdS )z:Update Boto Key object with user config's custom endpoint.�host�port� host_header)�gs_hostZgs_portZgs_host_header� CredentialsN)�itemsr �get�setattr�bucket� connection)�keyZ user_setting_to_key_attributeZ user_settingZ key_attributeZuser_setting_value� rQ �%/opt/gsutil/gslib/boto_translation.py�_AddCustomEndpointToKey� s �rS c C s t �� at�dd�adS )z�Perform necessary initialization for multiprocessing. See gslib.command.InitializeMultiprocessingVariables for an explanation of why this is necessary. �ir N)r& � CreateLock�boto_auth_initialized_lockr5 �Value�boto_auth_initializedrQ rQ rQ rR �"InitializeMultiprocessingVariables� s rY c @ s e Zd ZdZdd� Zdd� ZdS )� DownloadProxyCallbackHandlerzFIntermediary callback to keep track of the number of bytes downloaded.c C s || _ || _d S �N)� _start_byte� _callback)�self� start_byte�callbackrQ rQ rR �__init__� s z%DownloadProxyCallbackHandler.__init__c C s | j r| � | j| |� dS )z�Saves necessary data and then calls the given Cloud API callback. Args: bytes_downloaded: Number of bytes processed so far. total_size: Total size of the ongoing operation. N)r] r\ )r^ Zbytes_downloaded� total_sizerQ rQ rR �call� s z!DownloadProxyCallbackHandler.callN)�__name__� __module__� __qualname__�__doc__ra rc rQ rQ rQ rR rZ � s rZ c s" e Zd ZdZdn� fdd� Zdd� Zdodd �Zdpd d �Zdqd d �Zdrdd�Z dsdd�Z dtdd�Z dudd�Z dd� Z ddddejjddddddf dd�Zdeddfdd �Zddedfd d �Zdvd d �Zdwd!d"�Zdxd#d$�Zdyd%d&�Zdzd'd(�Zd{d)d*�Zd+d,� Zd|d-d.�Z d}d/d0�Z d~d1d2�Z dd3d4�Z d�d5d6�Z d�d7d8�Z!d9d:� Z"d;d<� Z#d=d>� Z$d?d@� Z%d�dAdB�Z&d�dCdD�Z'd�dEdF�Z(d�dGdH�Z)dIdJ� Z*dKdL� Z+dMdN� Z,dOdP� Z-dQdR� Z.d�dSdT�Z/d�dUdV�Z0d�dWdX�Z1d�dYdZ�Z2d�d\d]�Z3d^d_� Z4d`da� Z5dbdc� Z6ddde� Z7dfdg� Z8dhdi� Z9djdk� Z:dldm� Z;� Z<S )��BotoTranslationaA Boto-based XML translation implementation of gsutil Cloud API. This class takes gsutil Cloud API objects, translates them to XML service calls, and translates the results back into gsutil Cloud API objects for use by the caller. This class does not support encryption and ignores encryption and decryption parameters. Behavior when encountering encrypted objects is undefined. TODO: Implement support. This class does not support handling a Requester Pays user project for billing, and any given user project will be ignored. TODO: Support user_project. Nr c sz t t| �j|||||||| d� |} t�, t� ttt�r@dandt_W d � n1 sZ0 Y t j � ddd�| _ dS )a� Performs necessary setup for interacting with the cloud storage provider. Args: bucket_storage_uri_class: boto storage_uri class, used by APIs that provide boto translation or mocking. logger: logging.logger for outputting log messages. status_queue: Queue for relaying status to UI. provider: Provider prefix describing cloud storage provider to connect to. 'gs' and 's3' are supported. Function implementations ignore the provider argument and use this one instead. credentials: Unused. debug: Debug level for the API implementation (0..3). http_headers (dict|None): Arbitrary headers to be included in every request. trace_token: Unused in this subclass. perf_trace_token: Performance trace token to use when making API calls ('gs' provider only). user_project: Unused in this subclass )�provider�debug� http_headers� trace_token�perf_trace_tokenT� N�GSUtil�default_api_version�1) �superrh ra rV r' � isinstancerX �bool�value�botor � get_value� api_version) r^ �bucket_storage_uri_class�logger� status_queueri � credentialsrj rk rl rm � user_project�_�� __class__rQ rR ra � s$ � $ �zBotoTranslation.__init__c C s dS )z*Service account credentials unused for S3.NrQ �r^ rQ rQ rR �GetServiceAccountId� s z#BotoTranslation.GetServiceAccountIdc C sh |}| � |�}| �� }z| j|jd|d�|d�W S tyb } z| j||d� W Y d}~n d}~0 0 dS )�,See CloudApi class for function doc strings.T��validate�headers��fields�� bucket_nameN)�_StorageUriForBucket�_CreateBaseHeaders�_BotoBucketToBucket� get_bucket� TRANSLATABLE_BOTO_EXCEPTIONS�_TranslateExceptionAndRaise)r^ r� ri r� r~ � bucket_urir� �erQ rQ rR � GetBucket� s ��zBotoTranslation.GetBucketc c s� |}t |d�}| �� }| jdkr,t|�|t< zbtjd| j d| j| jd�}|j |d�}|D ]0} | jdkrz| j � � | j krzqZ| j | |d�V qZW n. t y� } z| �| � W Y d } ~ n d } ~ 0 0 d S ) r� �� list_fields�gsz%s://F��suppress_consec_slashesry rj �r� �s3r� N)r* r� ri r" r! rv � storage_uriry rj �get_all_buckets�name�lowerr� r� r� ) r^ � project_idri r� r~ � get_fieldsr� Z provider_uriZ buckets_iterrN r� rQ rQ rR � ListBuckets� s& � zBotoTranslation.ListBucketsc C sv |}| � |�} | �� } | �|| � �z|jrJt�|j�} | j| �� | d� |r|| �� } || vrjt d| ��| j || j | d� |r�| �� } || vr�t d| ��| j || j | d� |j r�|j tkr�g |_ t�|j �} | j| d| d� |j�rt�|j�} | j| �� | d� |j�rjt�|j�}z| �� j|| d� W n8 tjj�yh } z|j dk�rT� W Y d}~n d}~0 0 |j �r�t � |j �}| j |d| d� |j!�r�|j!j"�r�|j!j#�r�| j$|j!j"|j!j#d| d� n| j%d| d� |j&�r�| j'|j&| d� |j(�r| j)|j(j*| d� |j+�r.|j+j,}|j+j-}| j.||| d� | j/||d�W S t0�yp } z| j1||d� W Y d}~n d}~0 0 dS )r� r� �Invalid canned ACL "%s".F�� Nr� r� )2r� r� �_AddPreconditionsToHeaders�aclr7 �BotoAclFromMessage� set_xml_acl�to_xml� canned_aclsr �set_acl� object_name� set_def_acl�corsrA r9 ZBotoCorsFromMessage�set_cors�defaultObjectAcl�set_def_xml_acl�labelsr? ZBotoTagsFromMessager� �set_tagsrv � exception�GSResponseError�status� lifecycler@ ZBotoLifecycleFromMessage�configure_lifecycle�logging� logBucket�logObjectPrefix�enable_logging�disable_logging� storageClass�set_storage_class� versioning�configure_versioning�enabled�website�mainPageSuffix� notFoundPage�set_website_configr� r� r� )r^ r� �metadata� canned_acl�canned_def_acl� preconditionsri r� r~ r� r� �boto_aclr� � boto_cors� boto_tagsr� �boto_lifecycle�main_page_suffixZ error_pagerQ rQ rR � PatchBucket s� � � � ��zBotoTranslation.PatchBucketc C s |}| � |�}d}|r&|jr&|j�� }| �� } |jdkr�t|�| t< d} |rX|jrX|j} |rz|jrz|jj rzt |jj �| d< z|j | || d� W n2 t y� } z| j | |d� W Y d} ~ n d} ~ 0 0 nHz|j | |d� W n4 t �y } z| j | |d� W Y d} ~ n d} ~ 0 0 | j||d�S ) r� � r� z x-goog-bucket-retention-period)r� �location� storage_classr� N)r� r� r� )r� r� r� r� �schemer" r! r� �retentionPolicy�retentionPeriod�str� create_bucketr� r� r� ) r^ r� r� r� ri r� r~ r� r� r� r� r� rQ rQ rR � CreateBucketd s: ��� &$z BotoTranslation.CreateBucketc C s ||f}| � |�}| �� }z|j|d� W n� t�y } z�| j||d�}|r�d|jv r�zN|�� r�| jdkr~td| |j d��td| |j d��ntd| |j d��W n2 ty� } z| j | |d� W Y d } ~ n d } ~ 0 0 n*|r�|j d kr�t d | ��n| j ||d� W Y d }~n d }~0 0 d S ) r� r� r� �BucketNotEmptyr� z�VersionedBucketNotEmpty (%s). Currently, gsutil does not support listing or removing S3 DeleteMarkers, so you may need to delete these using another tool to successfully delete this bucket.�r� z VersionedBucketNotEmpty (%s)�BucketNotEmpty (%s)N� zBucket %s does not exist.) r� r� � delete_bucketr� �_TranslateBotoException�reason�get_versioning_configri r r� r� r ) r^ r� r� ri r~ r� r� r� �translated_exception�e2rQ rQ rR � DeleteBucket� sD �� ���� &z BotoTranslation.DeleteBucketc c s� |}t |d�}| �|�} | �� } |du p.d|v } |du pHtdd� |D ��} z | j|pVd|p\d|| d�} W n2 ty� } z| j||d� W Y d}~n d}~0 0 z�| D ]�}| r�t|t�r�t � |j t j j �V q�| r�|}tg d ��}|r�|�r2|�|��s2d}t|d d��r |j}t|d d��r |j}| j||j |d �}| j||d �}t � |t j j�V q�W n4 t�y� } z| j||d� W Y d}~n d}~0 0 dS )r� r� N�prefixesc s s | ]}|� d �V qdS )zitems/N�� startswith��.0�fieldrQ rQ rR � <genexpr>� s z.BotoTranslation.ListObjects.<locals>.<genexpr>r� )�prefix� delimiter� all_versionsr� r� )rN �etagr� �updated� generation�metageneration�sizer� � version_id�r� r� )r* r� r� �any� list_bucketr� r� rs r r �CsObjectOrPrefixr� �CsObjectOrPrefixType�PREFIX�set�issubset�getattrr� r� � _GetBotoKey�_BotoKeyToObject�OBJECT)r^ r� r� r� r� ri r� r~ r� r� r� Zyield_prefixesZ yield_objectsZ objects_iterr� rP Zkey_to_convertZkey_http_fieldsr� Z return_objectrQ rQ rR � ListObjects� sZ � � $� ���� zBotoTranslation.ListObjectsc C s\ |}z | j | j|||d�|d�W S tyV } z | j||||d� W Y d}~n d}~0 0 dS )r� r� r� �r� r� r� N)r� r� r� r� )r^ r� r� r� ri r� r~ r� rQ rQ rR �GetObjectMetadata� s ���z!BotoTranslation.GetObjectMetadatac s � fdd�S )aA Curries a digester object into a form consumable by boto. Key instantiates its own digesters by calling hash_algs[alg]() [note there are no arguments to this function]. So in order to pass in our caught-up digesters during a resumable download, we need to pass the digester object but don't get to look it up based on the algorithm name. Here we use a lambda to make lookup implicit. Args: digester_object: Input object to be returned by the created function. Returns: A function which when called will return the input object. c s � S r[ rQ rQ ��digester_objectrQ rR �<lambda> � z0BotoTranslation._CurryDigester.<locals>.<lambda>rQ )r^ r rQ r rR �_CurryDigester s z BotoTranslation._CurryDigesterFc C s$ | � � }t||d� | dur.d| | f |d< n*| dkrDd| |d< n| dk rXd| |d< | r�t�| �}t�t�|d ��}| jd kr�t|� n| j |||d �}| r�| jd kr�i }| D ]}| � | | �||< q�ni }|p�d}| r�t�| �d }|�rt t |�t t�}nt}z`|tjju �r6| j|| | ||| ||d � n2|tjju �r\| j||| |||d � n td| ��W n8 t�y� } z | j||||d� W Y d}~n d}~0 0 | jdk�r | �r G dd� dt�}| D ]T}|tjjk�r�| dk�s t|dd��r ||jv �s d| |< n||j| �| |< �q�dS )r� )�compressed_encodingNz bytes=%s-%s�ranger z bytes=%s-zbytes=%s�urlr� r� rb )r� r` � num_callbacks� hash_algs)�progress_callback�num_progress_callbacksr� r z Unsupported DownloadStrategy: %sr r� c @ s e Zd ZdZdd� Zdd� ZdS )z6BotoTranslation.GetObjectMedia.<locals>.HashToDigestera Wrapper class to expose hash digests. boto creates its own digesters in s3's get_file, returning on-the-fly hashes only by way of key.local_hashes. To propagate the digest back to the caller, this stub class implements the digest() function. c S s || _ d S r[ ��hash_val)r^ r rQ rQ rR ra r s z?BotoTranslation.GetObjectMedia.<locals>.HashToDigester.__init__c S s | j S r[ r r� rQ rQ rR �digestu s z=BotoTranslation.GetObjectMedia.<locals>.HashToDigester.digestN)rd re rf rg ra r rQ rQ rQ rR �HashToDigesterj s r � local_hashes)r� r4 �json�loads�pickle�binascii� a2b_base64ri rS r� r �max�intrD r1 r �DownloadStrategy� RESUMABLE�_PerformResumableDownload�ONE_SHOT�_PerformSimpleDownloadr r� r� �objectr� r )r^ r� r� �download_streamri r� � object_sizer �download_strategyr_ �end_byter �serialization_data� digesters�decryption_tupler� Zserialization_dictrP r �algrb r r� r Zalg_namerQ rQ rR �GetObjectMedia s� � ���� � � �� z BotoTranslation.GetObjectMediac C sB z|j |||||d� W n$ ty< |j ||||d� Y n0 d S )N)�cb�num_cbr� r )r* r+ r� )�get_contents_to_file� TypeError)r^ r! rP r r r� r rQ rQ rR r � s � �z&BotoTranslation._PerformSimpleDownloadc C sF t jttjtjf} |jjj} t � } d} |} z�t ||�}|� � }d|v rP|� d� d||f |d< z |j |||j|d|d� W n( ty� |j |||j|dd� Y n0 |�� W dS | �yH } z|| dkr�| j�d t|�� t|t��r4|jtjk�r4| jd k�r |j |||j|dd� n|j |||j|d|d� W Y d}~njd}~0 tjj�y� } zD|jtjj j k�rzt|j ��n| dk�r�| j�d |j � W Y d}~n d}~0 0 |� � }|| k�r�|} d} n| d7 } | | k�r�td ��z |� � W n t j!�y Y n0 t"t#�#� d | t$� �}| dk�r6| j�d| |� t%�&|� q*dS )a\ Downloads bytes from key to fp, resuming as needed. Args: fp: File pointer into which data should be downloaded. start_byte: Start byte of the download. end_byte: End byte of the download. key: Key object from which data is to be downloaded headers: Headers to send when retrieving the file callback: (optional) a callback function that will be called to report progress on the download. The callback should accept two integer parameters. The first integer represents the number of bytes that have been successfully transmitted from the service. The second represents the total number of bytes that need to be transmitted. num_callbacks: (optional) If a callback is specified with the callback parameter, this determines the granularity of the callback by defining the maximum number of times the callback will be called during the file transfer. hash_algs: Dict of hash algorithms to apply to downloaded bytes. Raises: ResumableDownloadException on error. r r z bytes=%d-%d�Range)�override_num_retriesr )r/ Nrn zCaught exception (%s)r� zBCaught boto.exception.ResumableDownloadException (%s) - will retryzeToo many resumable download attempts failed without progress. You might try this download again later� zWGot retryable failure (%d progress-less in a row). Sleeping %d seconds before re-trying)'r � HTTPException�IOError�socket�error�gaierrorrN rO rj r) rZ �copy�pop�get_filerc r- �flushrz �info�reprrs �errno�EPIPEri rv r� r � disposition� ResumableTransferDisposition�ABORT_CUR_PROCESS�message�tell�close�IncompleteRead�min�randomr( �time�sleep)r^ �fpr_ r$ rP r� r` r r Zretryable_exceptionsrj � num_retriesZprogress_less_iterationsZlast_progress_byteZ cb_handlerr� Zsleep_time_secsrQ rQ rR r � s� � � � ��� � � � �z)BotoTranslation._PerformResumableDownloadc C sv |} | j |||d�} | �� } t|| j�} i } t� }d}t�| �D ](\}}d}|du r`|�|� q@|| |< q@| �|| � |r�z| j | |d| d� W n6 t y� } z | j ||||d� W Y d}~n d}~0 0 |j �r.t �|j �}z| j|�� || d� W n8 t �y, } z | j ||||d� W Y d}~n d}~0 0 |�rd| �� }||v�rRtd| ��| j|| j| d� | j||||d �S ) r� r� FTNr� r )�key_namer� r� )r� r� )�_StorageUriForObjectr� r> ri r� �six� iteritems�addr� � set_metadatar� r� r� r7 r� r� r� r� r r� r� r )r^ r� r� r� r� r� r� ri r� r~ � object_urir� Z meta_headers� metadata_plus�metadata_minusZmetadata_changed�k�vr� r� r� rQ rQ rR �PatchObjectMetadata sh � � � � � �z#BotoTranslation.PatchObjectMetadatac C s |j |||||dd� d S )NT)�md5�policyr* r� �rewind)�set_contents_from_file)r^ �dst_uri� upload_streamrW r� r r� rQ rQ rR �_PerformSimpleUpload^ s �z$BotoTranslation._PerformSimpleUploadc C s� |� � �� r |j||||d� n�t�� \}}z�t|d��4}|�t�} | r^|�| � |�t�} qDW d � n1 sr0 Y t|d�� } |j | ||d� W d � n1 s�0 Y W t � |� t � |� nt � |� t � |� 0 d S )N)rX r* r� �wb�rb)rX r� ) � get_provider�supports_chunked_transfer�set_contents_from_stream�tempfile�mkstemp�open�readr, �writerZ �osrC �unlink) r^ r[ r\ r� r r� Ztemp_fhZ temp_pathZout_fpZ stream_bytesZin_fprQ rQ rR �_PerformStreamingUploadl s, � * �& � z'BotoTranslation._PerformStreamingUploadc C s* t || j|d�} | j||||||d� d S )N)Z resume_url)r� r* r� )r rz ZSendFile) r^ rP r\ Z upload_size�tracker_callbackr� r% r r� Zresumable_uploadrQ rQ rR �_PerformResumableUpload� s ��z'BotoTranslation._PerformResumableUploadc C s� t |� | �� }|�t|| j�� |jrbd|v rN|d d|j�d� 7 < nd|j�d� |d< |jr�d|v r�|d d|j�d� 7 < nd|j�d� |d< d|v r�|d s�d|d< | �||� | � |j |j �}||fS ) a Shared upload implementation. Args: object_metadata: Object metadata describing destination object. preconditions: Optional gsutil Cloud API preconditions. Returns: Headers dictionary, StorageUri for upload (based on inputs) z x-goog-hashz ,crc32c=%s� z crc32c=%sz,md5=%szmd5=%s� content-typezapplication/octet-stream) r+ r� �updater> ri �crc32c�rstrip�md5Hashr� rL rN r� )r^ �object_metadatar� r� r[ rQ rQ rR � _UploadSetup� s4 � � � � �z BotoTranslation._UploadSetupc C s� z,| � ||� | �� }| j|j|d�|d�W S tjjy� } zB|jrtt� |j� t ��rtt d� t�d|j ����n� W Y d}~n d}~0 0 dS )a� Set ACLs on an uploaded object and return its metadata. Args: dst_uri: Generation-specific StorageUri describing the object. object_metadata: Metadata for the object, including an ACL if applicable. fields: If present, return only these Object metadata fields. Returns: gsutil Cloud API Object metadata. Raises: CommandException if the object was overwritten / deleted concurrently. r� r� rm z�Uploaded object (%s) was deleted or overwritten immediately after it was uploaded. This can happen if you attempt to upload to the same object multiple times concurrently.N)� _SetObjectAclr� r� �get_keyrv r� �InvalidUriErrorrA �NON_EXISTENT_OBJECT_REGEX�match�encoder0 r �join�textwrap�wrap�uri)r^ r[ rs r� r� r� rQ rQ rR �_HandleSuccessfulUpload� s ����z'BotoTranslation._HandleSuccessfulUploadc C sV | � � }|jr.t�|j�}|j|�� |d� n$| jdkrRt|�}|rR|j||d� dS )zCSets the ACL (if present in object_metadata) on an uploaded object.r� r� N)r� r� r7 r� r� r� ri rB )r^ rs r[ r� r� �s3_aclrQ rQ rR ru � s z BotoTranslation._SetObjectAclc C s� | r t d��| jdkr,| j||||| |d�S | j||d�\} }|sNtd| ��z4| j|j| d�||||||| d� | j||| d�W S ty� } z0t | j|j �}| j ||j |j |d � W Y d }~n d }~0 0 d S ) r� z.XML API does not support gzip-encoded uploads.r� )r� r� r� r� �r� z;No tracker callback function set for resumable upload of %sr� )r� r% r r� r� �r� r� �not_found_exceptionN) �NotImplementedErrorri � UploadObjectrt r rl �new_keyr r� r; rN r� r� )r^ r\ rs r� r� r� r% rk r �encryption_tupleri r� � gzip_encodedr� r[ r� r� rQ rQ rR �UploadObjectResumable� sT � �� �����z%BotoTranslation.UploadObjectResumablec C s� | r t d��| j||d�\} } z&| j| |||| d� | j| ||d�W S ty� } z0t| j|j�} | j| |j|j | d� W Y d} ~ n d} ~ 0 0 dS )r� �-XML API does not suport gzip-encoded uploads.r� )r� r r� r� r� N) r� rt rj r r� r; ri rN r� r� )r^ r\ rs r� r r� r� ri r� r� r� r[ r� r� rQ rQ rR �UploadObjectStreaming! s2 � ����z%BotoTranslation.UploadObjectStreamingc C s� | r t d��| j||d�\} } zHd} |jr@t|j�|j�d�g} | j| || ||| d� | j| || d�W S ty� } z0t| j |j �}| j ||j |j |d� W Y d}~n d}~0 0 dS )r� r� r� Nz "')rW r� r r� r� r� ) r� rt rr r3 �stripr] r r� r; ri rN r� r� )r^ r\ rs r� r� r� r r� ri r� r� r� r[ rW r� r� rQ rQ rR r� B s> � �����z BotoTranslation.UploadObjectc C sv |}| � � }| �||� | j|||d�}z|jd|d� W n6 typ } z | j| |||d� W Y d} ~ n d} ~ 0 0 dS )r� r� Fr� r N)r� r� rL � delete_keyr� r� ) r^ r� r� r� r� ri r~ r� r~ r� rQ rQ rR � DeleteObjectm s ��z BotoTranslation.DeleteObjectc C s0 | } |durt d��| �|j|j�} d}| jdkr:|}d}| �� }|�t|| j�� | �||� |rr||| � � j < |j r|dnd}| jdkr�t |�}|r�d}|j p�d}z,| j|j|j|||||d�}| j|| d�W S t�y* } z@t| j|j| j|j|j|d�}| j||j|j|d � W Y d}~n d}~0 0 dS ) r� Nz*XML API does not suport max_bytes_per_callr� TF)� preserve_aclr� r� �src_version_id�src_generationr� )Z src_provider�src_bucket_nameZsrc_object_namer� r� )r� rL rN r� ri r� ro r> r� r` � acl_headerr� rB r� �copy_keyr� r� r; r� )r^ �src_obj_metadata�dst_obj_metadatar� r� r� r �max_bytes_per_callr� r' ri r� r~ r[ r� r� r� r� Zdst_storage_classr� r� r� rQ rQ rR � CopyObject� sZ � ���zBotoTranslation.CopyObjectc C s� |}t |� |j}d|_|j} d|_| �� } | �t|| j�� | �|| � |js`t |_|j| d< | � | |�} g } |D ]"} | j | | j| j d�}| � |� qtz | j | | d� | j| ||d�W S ty� } z | �||j|j� W Y d}~n d}~0 0 dS )r� Nrn r� r� r� )r+ r� rN r� ro r> ri r� � contentTyper= rL r� �append�composer r� r� )r^ �src_objs_metadatar� r� r� ri r� r~ Z dst_obj_nameZdst_bucket_namer� r[ Zsrc_componentsZsrc_objZsrc_urir� rQ rQ rR � ComposeObject� s< � � �z BotoTranslation.ComposeObjectc C sB |r>| j dkr>|jdur&t|j�|d< |jdur>t|j�|d< dS )z'Adds preconditions (if any) to headers.r� Nzx-goog-if-generation-matchz x-goog-if-metageneration-match)ri � gen_matchr� �meta_gen_match)r^ r� r� rQ rQ rR r� � s �z*BotoTranslation._AddPreconditionsToHeadersc C s` i }| j r|�| j � | jdkr*| j|d< | jdkrD| jrD| j|d< tj�t�}|r\||t < |S )z:Creates base headers used for all API calls in this class.r� zx-goog-api-version�cookie) rk ro ri rx rm rh �environrL r- r. )r^ Z base_headersZrequest_reasonrQ rQ rR r� � s z"BotoTranslation._CreateBaseHeadersc C s |r t �|�r |�d��� S dS )z�Returns an MD5 from the etag iff the etag is a valid MD5 hash. Args: src_etag: Object etag for which to return the MD5. Returns: MD5 in hex string format, or None. �"'N)� MD5_REGEX�searchr� r� )r^ Zsrc_etagrQ rQ rR �_GetMD5FromETag s z BotoTranslation._GetMD5FromETagc C s" t jd| j|f d| j| jdd�S )z�Returns a boto storage_uri for the given bucket name. Args: bucket: Bucket name (string). Returns: Boto storage_uri for the bucket. z%s://%sF)r� ry rj r� )rv r� ri ry rj )r^ rN rQ rQ rR r� s �z$BotoTranslation._StorageUriForBucketc C s6 d| j ||f }|r |d| 7 }tj|d| j| jd�S )a8 Returns a boto storage_uri for the given object. Args: bucket: Bucket name (string). object_name: Object name (string). generation: Generation or version_id of object. If None, live version of the object is used. Returns: Boto storage_uri for the object. z %s://%s/%sz#%sFr� )ri rv r� ry rj )r^ rN r� r� Z uri_stringrQ rQ rR rL ) s �z$BotoTranslation._StorageUriForObjectc C s~ | j |||d�}z2| �� }|j|d�}|s>td| j|||d��|W S tyx } z | j||||d� W Y d}~n d}~0 0 dS )z�Gets the boto key for an object. Args: bucket_name: Bucket containing the object. object_name: Object name. generation: Generation or version of the object to retrieve. Returns: Boto key for the object. r� r� Z404r N)rL r� rv r< ri r� r� )r^ r� r� r� rQ r� rP r� rQ rQ rR r� > s* � ��zBotoTranslation._GetBotoKeyc C sh | � |j�}tj|j|jd�}| �� }| jdk�rB|r>d|v rRt|d�rR|�� |_|r^d|v r�z(t � |j |d��D ]}|j � |� qrW nT ty� } z<| j||jd�}|r�t|t�r�n| j||jd� W Y d}~n d}~0 0 |r�d |v �r>z |j|d�} t�| �|_W n6 t�y< } z | j||jd� W Y d}~n d}~0 0 |�rNd |v �r�z*t �|j|d��D ]}|j� |� �qbW nZ t�y� } z@| j||jd�}|�r�t|t��r�n| j||jd� W Y d}~n d}~0 0 |�r�d |v �rHz,|j|d�j} | �rtj� � |_ | |j _ W n6 t�yF } z | j||jd� W Y d}~n d}~0 0 |�rXd |v �r�z |j |d�} t �!| �|_"W n6 t�y� } z | j||jd� W Y d}~n d}~0 0 |�r�d |v �rhzt|j#|d�} | �r.d| v �r.| d } d| v }d| v }|�s�|�r.tj�$� |_%|�r | d |j%_&|�r.| d |j%_'W n6 t�yf } z | j||jd� W Y d}~n d}~0 0 |�rxd|v �r$zt|j(|d�}|�r�d|v �r�|d }d|v }d|v }|�s�|�r�tj�)� |_*|�r�|d |j*_+|�r�|d |j*_,W n6 t�y" } z | j||jd� W Y d}~n d}~0 0 |�r4d|v �rB|j-|d�|_.|�rRd|v �r�zfz |�/� �0� }t1�2|�|_3W nD t4j5j6�y� } z&| jdk�r�|j7dk�s�� W Y d}~n d}~0 0 W n6 t�y� } z | j||jd� W Y d}~n d}~0 0 |�r d|v �rd|j8|d�}|�rd| jdk�rHd|v �rH|d dk�rHtjj9d d �|_:n | jdk�rdtjj9d d �|_:|S ) z�Constructs an apitools Bucket from a boto bucket. Args: bucket: Boto bucket. fields: If present, construct the apitools Bucket with only this set of metadata fields. Returns: apitools Bucket. )r� �idr� r� �get_storage_classr� r� r� Nr� r� � encryptionr� r� ZLoggingZLogObjectPrefixZ LogBucketr� �WebsiteConfigurationZMainPageSuffixZ NotFoundPager� r� r� r� r� � Versioning�EnabledT)r� );r� r� �apitools_messages�Bucketr� ri �hasattrr� r� r7 ZBotoBucketAclToMessage�get_aclr� r� r� r� rs r r� �get_corsr9 ZBotoCorsToMessager� �BotoObjectAclToMessage� get_def_aclr� �get_encryption_config�default_kms_key_name�EncryptionValuer� �defaultKmsKeyName�get_lifecycle_configr@ ZBotoLifecycleToMessager� �get_logging_config� LoggingValuer� r� r� �get_website_config� WebsiteValuer� r� r� � get_locationr� r� �get_tagsr? ZBotoTagsToMessager� rv r� �StorageResponseErrorr� r� �VersioningValuer� )r^ rN r� r� Zcloud_api_bucketr� r� r� r� r� Zkeynamer� Z boto_loggingZlogging_configZlog_object_prefix_presentZlog_bucket_presentZ boto_websiteZwebsite_configZmain_page_suffix_presentZnot_found_page_presentr� r� rQ rQ rR r� ] s� � ���& & ���&�� & �& �& �� & � & � � �z#BotoTranslation._BotoBucketToBucketc C s� d}|r&d|v s&t dd� |D ��dkr0| �|�}d}|r@d|v rLt|dd�}d}|r\d|v rht|d d�}d}|rxd |v r�t|d d�}| �|�}d}|r�d |v r�| �|�}d} |r�d |v r�| �|�} d} |r�d|v r�t|dd�} | r�| �d�} d} |r�d|v �r,t|d��r,d|jv �r,t � |jd �� d�} d} |�r@d|v �r�t|d��rpd|jv �rpt � |jd �� d�} nD| � t|dd���r�t | � |j��} n | jdk�r�| j�d|j|� d} |�r�d|v �r�t�t�|tj��} d}|�r�d|v �r�|j�p�d}d}|�rd|v �r t|dd�}tj�r\| �r@t| t��r@| � d �} | �r\t| t��r\| � d �} t j |j j |j ||j!|j"|j#||| | | |||| || |d �}| �$||� |�r�d |v �r�t%t&| j�|�}| j'|||d � |S ) a Constructs an apitools Object from a boto key. Args: key: Boto key to construct Object from. fields: If present, construct the apitools Object with only this set of metadata fields. Returns: apitools Object corresponding to key. Nr� c S s g | ]}|� d �r|�qS )z metadata/r� r� rQ rQ rR � <listcomp>� r z4BotoTranslation._BotoKeyToObject.<locals>.<listcomp>rn � cacheControl� cache_control�componentCount�component_count�contentDisposition�content_dispositionr� � timeCreatedr� r� rp � cloud_hashes� rr rW r� zMNon-MD5 etag (%s) present for key %s, data integrity checks are not possible.� mediaLinkr� r r� �_storage_class�ascii)rN r� r� �contentEncoding�contentLanguager� r� r� r� rp rr r� r� r� r� r� r� r� r� r� )(�len� _TranslateBotoKeyCustomMetadatar� �_TranslateBotoKeyGeneration� _TranslateBotoKeyMetageneration�_TranslateBotoKeyTimestampr� r� r� �base64� b64encoderq r� r2 r� ri rz �warnr � b2a_base64r �dumps�HIGHEST_PROTOCOLr� rM �PY3rs �bytes�decoder� �ObjectrN r� �content_encoding�content_language� content_type�_TranslateDeleteMarkerr# r$ �_TranslateBotoKeyAcl)r^ rP r� �custom_metadatar� r� r� r� r� Z time_createdr� rp Zmd5_hashZ media_linkr� r� �cloud_api_object�generation_strrQ rQ rR r� � s� �� � � � ��z BotoTranslation._BotoKeyToObjectc C s` d}t |dd�r\tjjg d�}t�|j�D ]0\}}|�� dkr@q*|j� tjjj ||d�� q*|S )zCPopulates an apitools message from custom metadata in the boto key.Nr� )�additionalPropertieszcontent-language�rP ru ) r� r� r� � MetadataValuerM rN r� r� r� r� �AdditionalProperty)r^ rP r� rT rU rQ rQ rR r� [ s � ��z/BotoTranslation._TranslateBotoKeyCustomMetadatac C sJ d}| j dkr&t|dd�rFt|j�}n | j dkrFt|dd�rFt|j�}|S )zFReturns the generation/version_id number from the boto key if present.Nr� r� r� r� )ri r� �longr� r6 r� )r^ rP r� rQ rQ rR r� j s z+BotoTranslation._TranslateBotoKeyGenerationc C s( d}| j dkr$t|dd�r$t|j�}|S )z?Returns the metageneration number from the boto key if present.Nr� r� )ri r� r� r� )r^ rP r� rQ rQ rR r� u s z/BotoTranslation._TranslateBotoKeyMetagenerationc C s� |j r�d|j v r"|j �d�d }n|j �d�d }d}ztj�||�W S ty� zd}tj�|j |�W Y S ty� Y Y dS 0 Y n0 dS )z�Parses the timestamp from the boto key into an datetime object. This avoids a dependency on dateutil. Args: key: Boto key to get timestamp from. Returns: datetime object if string is parsed successfully, None otherwise. �.�ZZ000Zz.000000Zz%Y-%m-%dT%H:%M:%S.%fZz%a, %d %b %Y %H:%M:%S %ZN)� last_modifiedrq �datetime�strptime� ValueError)r^ rP Zkey_us_timestamp�fmtrQ rQ rR r� } s z*BotoTranslation._TranslateBotoKeyTimestampc C sD t |t�r@|js$tj�� |_g |j_|jj�tjjjt dd�� dS )zCMarks deleted objects with a metadata value (for S3 compatibility).Tr� N) rs r r� r� r� r� r� r� r� r/ )r^ rP r� rQ rQ rR r� � s ��z&BotoTranslation._TranslateDeleteMarkerc C s� | j |jj|j|d�}| �� }zV| jdkrR|j|d�}t�|�D ]}|j� |� q@| jdkrr|j |d�}t ||� W n6 t j jy� } z|jdkr�n� W Y d}~n d}~0 0 dS )z8Updates cloud_api_object with the ACL from the boto key.r� r� r� r� � N)rL rN r� r� ri r� r7 r� r� r� � get_xml_aclr8 rv r� r� r� ) r^ rP r� r� �storage_uri_for_keyr� Zkey_aclr� r� rQ rQ rR r� � s" � z$BotoTranslation._TranslateBotoKeyAclc C s$ | j |||||d�}|r |�n� dS )a Translates a Boto exception and raises the translated or original value. Args: e: Any Exception. bucket_name: Optional bucket name in request that caused the exception. object_name: Optional object name in request that caused the exception. generation: Optional generation in request that caused the exception. not_found_exception: Optional exception to raise in the not-found case. Raises: Translated CloudApi exception, or the original exception if it was not translatable. )r� r� r� r� N)r� )r^ r� r� r� r� r� r� rQ rQ rR r� � s �z+BotoTranslation._TranslateExceptionAndRaisec C s t |tjj��r |jdkr.t|j|j|jd�S |jdksB|jdkrVt|j|j|jd�S |jdkr�|rvt |d|j� |S |r�|r�t |j| j |||d�S t |j| j |�S t |j|j|jd�S |jdkr�|jr�d |jv r�td | |j|jd�S |jd k� r�t|j�S |jd k�r t|j|j|jd�S t |tjj��r@td |j|jd�S t |tjj��rdt|j|j|jd�S t |tjj��r�|j�r�t�|j��r�t |jdd�S t|j�S t |tjj��r�|jtjjj k�r�t |j�S |jtjjj k�r�t|j�S t|j�S t |tjj ��rt |j�S dS )aX Translates boto exceptions into their gsutil Cloud API equivalents. Args: e: Any exception in TRANSLATABLE_BOTO_EXCEPTIONS. bucket_name: Optional bucket name in request that caused the exception. object_name: Optional object name in request that caused the exception. generation: Optional generation in request that caused the exception. not_found_exception: Optional exception to raise in the not-found case. Returns: ServiceException for translatable exceptions, None otherwise. Because we're using isinstance, check for subtypes first. i� )r� �bodyi� r� r� r� r� i� r� r� i� i� zBucket already exists.r� N) rs rv r� r� r� r �coder� r rM r< ri r: r rA r r r �StorageCreateErrorr �BotoServerErrorrw rx ry r r r> r? �ABORTr � START_OVERr )r^ r� r� r� r� r� rQ rQ rR r� � sh � �� � � z'BotoTranslation._TranslateBotoExceptionc C sh z4t j|jd| j| jd�}|r(|�� W S |�� W S W n. tyb } z| �|� W Y d}~n d}~0 0 dS ��5See CloudApiDelegator class for function doc strings.Fr� N) rv r� � url_stringry rj r� r� r� r� )r^ � storage_url� def_obj_aclr~ r� rQ rQ rR �XmlPassThroughGetAcl/ s � z$BotoTranslation.XmlPassThroughGetAclTc C s� | � � }z�tj|jd| j| jd�}|r�|rZ|�� }||vrFtd| ��|j||j |d� q�|�� }||vrvtd| ��|j ||j |d� n*|r�|j ||j |d� n|j ||j |d� W n. t y� } z| �|� W Y d}~n d}~0 0 dS )r� Fr� r� r� N)r� rv r� r� ry rj r� r r� r� r� r� r� r� r� ) r^ �acl_textr� �cannedr� r� r~ r� r� rQ rQ rR �XmlPassThroughSetAcl> s. � z$BotoTranslation.XmlPassThroughSetAclc C s� | � � }|jdkrt� }nt� }t�|d�}ztj�||� W nD t j y� } z*t d|� � |� � |�� f ��W Y d}~n d}~0 0 z,tj|jd| j| jd�}|j|d|d� W n. ty� } z| �|� W Y d}~n d}~0 0 dS )r� r� Nz3Requested CORS is invalid: %s at line %s, column %sFr� r� )r� r� �S3Corsr r � XmlHandler�xml�saxr � SaxExceptions�SAXParseExceptionr � getMessage� getLineNumber�getColumnNumberrv r� r� ry rj r� r� r� )r^ � cors_textr� r� Zcors_obj�hr� r~ rQ rQ rR �XmlPassThroughSetCors_ s. ���z%BotoTranslation.XmlPassThroughSetCorsc C s� | � � }tj|jd| j| jd�}z|jd|d�}W n. ty` } z| �|� W Y d}~n d}~0 0 t j j � |� � �t��}|jdd�S �r� Fr� r� Nz )�indent)r� rv r� r� ry rj r� r� r� r �dom�minidomr r� rz r0 � toprettyxml)r^ r� r� r~ r� r� � parsed_xmlrQ rQ rR �XmlPassThroughGetCorsz s � z%BotoTranslation.XmlPassThroughGetCorsc C s� | � � }z*tj|jd| j| jd�}|jd|d�}W n. ty` } z| �|� W Y d}~n d}~0 0 t j j � |� � �t��}|jdd�S r )r� rv r� r� ry rj r� r� r� r r r r r� rz r0 r )r^ r� r� r~ r� r� r rQ rQ rR �XmlPassThroughGetLifecycle� s � z*BotoTranslation.XmlPassThroughGetLifecyclec C s� | � � }|jdkrt� }nt� }t�|d�}ztj�||� W nD t j y� } z*t d|� � |� � |�� f ��W Y d}~n d}~0 0 z,tj|jd| j| jd�}|j|d|d� W n. ty� } z| �|� W Y d}~n d}~0 0 dS )r� r� Nz?Requested lifecycle config is invalid: %s at line %s, column %sFr� r� )r� r� � S3Lifecycler r r r r r r r r r r r rv r� r� ry rj r� r� r� )r^ �lifecycle_textr� r� Z lifecycle_objr r� r~ rQ rQ rR �XmlPassThroughSetLifecycle� s. ���z*BotoTranslation.XmlPassThroughSetLifecyclec C sp | � � }z,tj|jd| j| jd�}t|j|d��}W n. tyb } z| � |� W Y d}~n d}~0 0 t |�� � S �r� Fr� r� N) r� rv r� r� ry rj rC r� r� r� �XmlParseStringr )r^ r� r� r~ Zlogging_config_xmlr� rQ rQ rR �XmlPassThroughGetLogging� s � � z(BotoTranslation.XmlPassThroughGetLoggingc C sl | � � }z(tj|jd| j| jd�}|�� �� }W n. ty^ } z| � |� W Y d}~n d}~0 0 t |�� � S r� ) r� rv r� r� ry rj r� � get_xml_tagsr� r� r r )r^ r� r� r~ Ztagging_config_xmlr� rQ rQ rR �XmlPassThroughGetTagging� s � z(BotoTranslation.XmlPassThroughGetTaggingc C s� | � � }t� }t�|d�}ztj�||� W nD tjyp } z*t d|� � |� � |� � f ��W Y d}~n d}~0 0 z.t j|jd| j| jd�}|�� j||d� W n. ty� } z| �|� W Y d}~n d}~0 0 dS )r� NzDRequested labels/tagging config is invalid: %s at line %s, column %sFr� r� )r� r r r r r r r r r r r r rv r� r� ry rj r� r� r� r� )r^ Z tagging_textr� r� Ztags_objr r� r~ rQ rQ rR �XmlPassThroughSetTagging� s* ���z(BotoTranslation.XmlPassThroughSetTaggingc C sp | � � }z,tj|jd| j| jd�}t|j|d��}W n. tyb } z| � |� W Y d}~n d}~0 0 t |�� � S r ) r� rv r� r� ry rj rC r� r� r� r r )r^ r� r� r~ Zweb_config_xmlr� rQ rQ rR �XmlPassThroughGetWebsite� s � z(BotoTranslation.XmlPassThroughGetWebsite)NNr NNNN)NN)NNN)NNNNN)NNNN)NN)NNNNN)NNN)NNNNN)NNNN)NNN)NNNN)N)N) NNNNNNNNNF)NNNNNNF)NNNNNNNF)NNN) NNNNNNNNN)NNNN)N)N)N)N)N)NNNN)NNNN)F)TF)=rd re rf rg ra r� r� r� r� r� r� r r r r r r r) r1 r r rV r] rj rl rt r ru r� r� r� r� r� r� r� r� r� r� rL r� r� r� r� r� r� r� r� r� r� r� r r r r r r r r r r! � __classcell__rQ rQ r rR rh � sJ �1 � O � % ( � D � � v� � � C � � " � ' ! � 5 � $ � . � � I � + p � � S � !rh )�rg � __future__r r r r r� r r� r< r rh r rF �rer3 rc r| � threadingrG r Zxml.dom.minidomr r �xml.saxr r rM � six.movesr rv r r � boto.gs.corsr �boto.gs.lifecycler � boto.s3.corsr r �boto.s3.deletemarkerr �boto.s3.lifecycler r �boto.s3.prefixr �boto.s3.taggingr �boto.exception� boto.utilsZgslib.boto_resumable_uploadr �gslib.cloud_apir r r r r r r r r r r r Zgslib.devshell_auth_plugin�gslib�gslib.exceptionr r Zgslib.project_idr! r" �gslib.storage_urlr# r$ �"gslib.third_party.storage_apitoolsr% r� � gslib.utilsr& Zgslib.utils.boto_utilr' r( r) Z gslib.utils.cloud_api_helperr* r+ �gslib.utils.constantsr, r- r. r/ r0 r1 Zgslib.utils.hashing_helperr2 r3 �gslib.utils.metadata_utilr4 �&gslib.utils.parallelism_framework_utilr5 �gslib.utils.text_utilr6 Z gslib.utils.translation_helperr7 r8 r9 r: r; r< r= r> r? r@ rA rB rC �gslib.utils.unit_utilrD r� r r� r� r� rw r� r� r� �LockrV rX �compile�DOTALLrx r� rS rY r rZ rh rQ rQ rQ rR �<module> s� � �
SIMPAN
[ BATAL ]