CURRENT PATH:
/
opt
/
gsutil
/
gslib
/
__pycache__
/
[ ⬅ KEMBALI ]
|
[ HOME ]
Upload File:
Upload Ke Sini
Dir Baru
File Baru
Editing:
wildcard_iterator.cpython-39.pyc
a +(Wg� � @ s� d Z ddlmZ ddlmZ ddlmZ ddlmZ ddlZddlZddlZddl Z ddl Z ddl Z ddl Z ddl mZ ddl mZ dd l mZ dd lmZ dd lmZ dd lmZ dd lmZ ddlmZ ddlmZ ddlmZ ddlmZ ddlm Z ddl m Z ddl m!Z! ddl"m#Z# ddl"m$Z$ e j%�r8e&Z'e �(d�Z)dZ*G dd� de+�Z,G dd � d e,�Z-d d � Z.G d d � d e,�Z/G d!d"� d"e'�Z0d&d$d%�Z1dS )'z1Wildcard iterator class and supporting functions.� )�absolute_import)�print_function)�division)�unicode_literalsN)�BucketListingBucket)�BucketListingObject)�BucketListingPrefix)�AccessDeniedException)�CloudApi)�NotFoundException)�CommandException)�ContainsWildcard)�GenerationFromUrlAndString)�StorageUrlFromString)� StripOneSlash)�WILDCARD_REGEX)�storage_v1_messages)�UTF8)�FixWindowsEncodingIfNeeded)� PrintableStrz (?P<before>.*?)\*\*(?P<after>.*)a\ Invalid Unicode path encountered (%s). gsutil cannot proceed with such files present. Please remove or rename this file and try again. NOTE: the path printed above replaces the problematic characters with a hex-encoded printable representation. For more details (including how to convert to a gsutil-compatible encoding) see `gsutil help encoding`.c @ s e Zd ZdZdd� ZdS )�WildcardIteratora Class for iterating over Google Cloud Storage strings containing wildcards. The base class is abstract; you should instantiate using the wildcard_iterator() static factory method, which chooses the right implementation depending on the base string. c C s d| j j S )z2Returns string representation of WildcardIterator.zWildcardIterator(%s))� wildcard_url� url_string��self� r �&/opt/gsutil/gslib/wildcard_iterator.py�__repr__I s zWildcardIterator.__repr__N)�__name__� __module__� __qualname__�__doc__r r r r r r ? s r c @ s� e Zd ZdZd dd�Zd dd�Zdd � Zd d � Zd d � Zd dd�Z d dd�Z d dd�Z dd� Z d!dd�Z d"dd�Zd#dd�ZdS )$�CloudWildcardIteratoraV WildcardIterator subclass for buckets, bucket subdirs and objects. Iterates over BucketListingRef matching the Url string wildcard. It's much more efficient to first get metadata that's available in the Bucket (for example to get the name and size of each object), because that information is available in the object list results. FNc C s* || _ || _|| _|| _|p"t�� | _dS )a� Instantiates an iterator that matches the wildcard URL. Args: wildcard_url: CloudUrl that contains the wildcard to iterate. gsutil_api: Cloud storage interface. Passed in for thread safety, also settable for testing/mocking. all_versions: If true, the iterator yields all versions of objects matching the wildcard. If false, yields just the live object version. project_id: Project ID to use for bucket listings. logger: logging.Logger used for outputting debug messages during iteration. If None, the root logger will be used. N)r � all_versions� gsutil_api� project_id�logging� getLogger�logger)r r r$ r# r% r( r r r �__init__W s z CloudWildcardIterator.__init__c c sF | j �� }d}|rxt� }|D ]}|�|� q | j|d�}|�ddg� |�dg� |s\| jrx|�ddg� |�dd g� | jd gd �D �]�}|j}| j � � �r|�rt |�}| j j |j d | j| j j|d �D ]:} | jtjjkr�| j|| j| jd�V q�| �|| j�V q�n|V q�t| j j��s�| j �� �r�| j�s�zL| j j| j j | j j| j j| j j|d�} | j| j j| | j�pp|d�V W dS t t f�y� Y n0 |�r�d|| j j| j jf } nd|t | j j��p�d f } | g} | r�t | � d��}| � |j�\} }}}| �!|�}|�rtdg�n|}| j j |j | || j�p2|| j j|d�D ]�} |D ]�}| jtjjk�r�| j}|�"|j#��r4|�r�t |j#�|k�r�|�r�| �$|j��r�| j||| j�p�|d�V �q@nx| j} t| ��r�t%d|| f ��t | �}|�"|��rH|�r ||k�r d||d | f }| �&|� n| �|| �V �q@�qH�q@�q�q�dS )a! Iterator that gets called when iterating over the cloud wildcard. In the case where no wildcard is present, returns a single matching object, single matching prefix, or one of each if both exist. Args: bucket_listing_fields: Iterable fields to include in bucket listings. Ex. ['name', 'acl']. Iterator is responsible for converting these to list-style format ['items/name', 'items/acl'] as well as adding any fields necessary for listing such as prefixes. API implementation is responsible for adding pagination fields. If this is None, all fields are returned. expand_top_level_buckets: If true, yield no BUCKET references. Instead, expand buckets into top-level objects and prefixes. Yields: BucketListingRef of type BUCKET, OBJECT or PREFIX. N)� get_fieldsz items/name�prefixes�namezitems/generationzitems/metageneration� generation�metageneration�id�� bucket_fields�/)� delimiterr# �provider�fields)� with_version)r- r4 r5 �%s%s#%s�%s%sr )�prefixr3 r# r4 r5 znCloud folder %s%s contains a wildcard; gsutil does not currently support objects with wildcards in their name.)'r � HasGeneration�set�add�_GetToListFields�updater# �_ExpandBucketWildcardsr �IsBucketr r$ Z ListObjects� bucket_name�scheme�datatyper ZCsObjectOrPrefixTypeZOBJECT� _GetObjectRef�data� _GetPrefixRefr �IsObjectZGetObjectMetadata� object_namer- �bucket_url_stringr r r �pop�_BuildBucketFilterStrings�_GetRegexPatterns�matchr, �_SingleVersionMatchesr �append)r �bucket_listing_fields�expand_top_level_bucketsZsingle_version_requestr* �field�bucket_listing_refrI �urlZ obj_or_prefix� get_objectr Zurls_needing_expansionr9 r3 �prefix_wildcard�suffix_wildcardZregex_patternsZlisting_fields�pattern� gcs_objectZrstripped_prefixZurl_append_stringr r r �__iter__p s� � � � � ��� ��� � �� � � �� �� �� z CloudWildcardIterator.__iter__c C sb |g}d|v r&|� dd�}|�|� n|}||fD ] }|�d�r2|�|dd� � q2dd� |D �S )a� Returns list of regex patterns derived from the wildcard patterns. Args: wildcard_pattern (str): A wilcard_pattern to filter the resources. Returns: List of compiled regex patterns. This translates the wildcard_pattern and also creates some additional patterns so that we can treat ** in a/b/c/**/d.txt as zero or more folders. This means, a/b/c/d.txt will also be returned along with a/b/c/e/f/d.txt. z/**/r2 �**/� Nc S s g | ]}t �t�|���qS r )�re�compile�fnmatch� translate)�.0�pr r r � <listcomp>7 � z;CloudWildcardIterator._GetRegexPatterns.<locals>.<listcomp>)�replacerO � startswith)r Zwildcard_patternZwildcard_patternsZupdated_patternrX r r r rL s z'CloudWildcardIterator._GetRegexPatternsc C s( t �|�}|s |}d}|}d}n�|�� dkrN|d|�� � }||�� d� }nd}|}|�d�}|dkrx|d|d � }|p~d| }|�d�s�t|�}||�� d� }|�d�}|dkr�d}n||d d� }|�d�dkr�d}||7 }d}nd}| j�d t |�t |�t |�t |�t |�� ||||fS ) a� Builds strings needed for querying a bucket and filtering results. This implements wildcard object name matching. Args: wildcard: The wildcard string to match to objects. Returns: (prefix, delimiter, prefix_wildcard, suffix_wildcard) where: prefix is the prefix to be sent in bucket GET request. delimiter is the delimiter to be sent in bucket GET request. prefix_wildcard is the wildcard to be used to filter bucket GET results. suffix_wildcard is wildcard to be appended to filtered bucket GET results for next wildcard expansion iteration. For example, given the wildcard gs://bucket/abc/d*e/f*.txt we would build prefix= abc/d, delimiter=/, prefix_wildcard=d*e, and suffix_wildcard=f*.txt. Using this prefix and delimiter for a bucket listing request will then produce a listing result set that can be filtered using this prefix_wildcard; and we'd use this suffix_wildcard to feed into the next call(s) to _BuildBucketFilterStrings(), for the next iteration of listing/filtering. Raises: AssertionError if wildcard doesn't contain any wildcard chars. r2 � r N���� r[ z**zMwildcard=%s, prefix=%s, delimiter=%s, prefix_wildcard=%s, suffix_wildcard=%s ) r �search�start�find�endswithr �endr( �debugr ) r �wildcardrM r9 r3 rV rW Z wildcard_partrn r r r rK 9 sD �z/CloudWildcardIterator._BuildBucketFilterStringsc C s t | j|�}t| jj�t|�kS �N)r r �strr- )r Zlisted_generationZdecoded_generationr r r rN � s �z+CloudWildcardIterator._SingleVersionMatchesc c s� t | jj�}|r:t|�tdg�kr:t| jj�s:t|�V n�| j�� rvt| jj�svt|| jj | jj| jj |d�d�V n~t � | jj�}t �|�}| �|�}|r�|�d� | jj|| j| jj d�D ]4}|�|j�r�t d| jj |jf �}t||d�V q�dS )aG Expands bucket and provider wildcards. Builds a list of bucket url strings that can be iterated on. Args: bucket_fields: If present, populate only these metadata fields for buckets. Example value: ['acl', 'defaultObjectAcl'] Yields: BucketListingRefereneces of type BUCKET. r/ )r4 r5 �� root_objectzitems/id)r5 r% r4 z%s://%s/N)r r rI r; r rA r r@ r$ Z GetBucketrB r_ r` r] r^ r= r< Z ListBucketsr% rM r/ )r r1 Z bucket_url�regex�progr5 �bucketrT r r r r? � s< � ��� � �z,CloudWildcardIterator._ExpandBucketWildcardsc C s* |r&t � }|D ]}|�d| � q|S dS )a� Prepends 'items/' to the input fields and converts it to a set. This way field sets requested for GetBucket can be used in ListBucket calls. Note that the input set must contain only bucket or object fields; listing fields such as prefixes or nextPageToken should be added after calling this function. Args: get_fields: Iterable fields usable in GetBucket/GetObject calls. Returns: Set of fields usable in ListBuckets/ListObjects calls. zitems/N)r; r< )r r* Z list_fieldsrR r r r r= � s z&CloudWildcardIterator._GetToListFieldsc C sP |r.|j dur.t| j|j �}d||j|f }nd||jf }t|�}t||d�S )ah Creates a BucketListingRef of type OBJECT from the arguments. Args: bucket_url_string: Wildcardless string describing the containing bucket. gcs_object: gsutil_api root Object for populating the BucketListingRef. with_version: If true, return a reference with a versioned string. Returns: BucketListingRef of type OBJECT. Nr7 r8 rs )r- r r r, r r )r rI rY r6 Zgeneration_strZ object_stringZ object_urlr r r rD � s ��z#CloudWildcardIterator._GetObjectRefc C s t d||f �}t||d�S )a Creates a BucketListingRef of type PREFIX from the arguments. Args: bucket_url_string: Wildcardless string describing the containing bucket. prefix: gsutil_api Prefix for populating the BucketListingRef Returns: BucketListingRef of type PREFIX. r8 rs )r r )r rI r9 Z prefix_urlr r r rF � s z#CloudWildcardIterator._GetPrefixRefc c s | j |d�D ] }|V q dS )ah Iterates over the wildcard, returning refs for each expanded bucket. This ignores the object part of the URL entirely and expands only the the bucket portion. It will yield BucketListingRefs of type BUCKET only. Args: bucket_fields: Iterable fields to include in bucket listings. Ex. ['defaultObjectAcl', 'logging']. This function is responsible for converting these to listing-style format ['items/defaultObjectAcl', 'items/logging'], as well as adding any fields necessary for listing such as 'items/id'. API implemenation is responsible for adding pagination fields. If this is None, all fields are returned. Yields: BucketListingRef of type BUCKET, or empty iterator if no matches. r0 N)r? )r r1 �blrr r r � IterBuckets� s z!CloudWildcardIterator.IterBucketsc c s | j ||d�D ] }|V qdS )a} Iterates over the wildcard, yielding bucket, prefix or object refs. Args: bucket_listing_fields: If present, populate only these metadata fields for listed objects. expand_top_level_buckets: If true and the wildcard expands only to Bucket(s), yields the expansion of each bucket into a top-level listing of prefixes and objects in that bucket instead of a BucketListingRef to that bucket. Yields: BucketListingRef, or empty iterator if no matches. �rP rQ N�rZ )r rP rQ rx r r r �IterAll s � z CloudWildcardIterator.IterAllc c s& | j |dd�D ]}|�� r|V qdS )a& Iterates over the wildcard, yielding only object BucketListingRefs. Args: bucket_listing_fields: If present, populate only these metadata fields for listed objects. Yields: BucketListingRefs of type OBJECT or empty iterator if no matches. Trz N)rZ rG )r rP rx r r r � IterObjects s � z!CloudWildcardIterator.IterObjects)FNN)NF)N)N)F)N)NF)N)r r r r! r) rZ rL rK rN r? r= rD rF ry r| r} r r r r r" N s& � � )!U * r" c C s t jtj�| �d�S )a0 Returns an apitools Object class with supported file attributes. To provide size estimates for local to cloud file copies, we need to retrieve expose the local file's size. Args: filepath: Path to the file. Returns: apitools Object that with file name and size attributes filled-in. )�size)�apitools_messagesZObject�os�path�getsize)�filepathr r r �_GetFileObject. s r� c @ sR e Zd ZdZddd�Zddd�Zdd � Zd d � Zdd d �Zddd�Z ddd�Z dS )�FileWildcardIteratora� WildcardIterator subclass for files and directories. If you use recursive wildcards ('**') only a single such wildcard is supported. For example you could use the wildcard '**/*.txt' to list all .txt files in any subdirectory of the current directory, but you couldn't use a wildcard like '**/abc/**/*.txt' (which would, if supported, let you find .txt files in any subdirectory named 'abc'). NFc C s$ || _ || _|| _|p t�� | _dS )a' Instantiates an iterator over BucketListingRefs matching wildcard URL. Args: wildcard_url: FileUrl that contains the wildcard to iterate. exclude_tuple: (base_url, exclude_pattern), where base_url is top-level URL to list; exclude_pattern is a regex of paths to ignore during iteration. ignore_symlinks: If True, ignore symlinks during iteration. logger: logging.Logger used for outputting debug messages during iteration. If None, the root logger will be used. N)r � exclude_tuple�ignore_symlinksr& r' r( )r r r� r� r( r r r r) I s z FileWildcardIterator.__init__c c s0 |odt |�v }| jj}t�|�}|rz|�d�dd� }|�d�}|�d�rXtd| ��|s`d}|�t j �}| � ||�}n t � |�}|D ]�}t|�} zd| jr�t j�|�r�| jr�| j�d|� W q�t j�|�r�t| �V n |r�t|�nd} t| | d �V W q� t�y( td �t� t t |� ����Y q�0 q�dS ) a@ Iterator that gets called when iterating over the file wildcard. In the case where no wildcard is present, returns a single matching file or directory. Args: bucket_listing_fields: Iterable fields to include in listings. Ex. ['size']. Currently only 'size' is supported. If present, will populate yielded BucketListingObject.root_object with the file name and size. Raises: WildcardException: if invalid wildcard found. Yields: BucketListingRef of type OBJECT (for files) or PREFIX (for directories) r~ �beforeNrh Zafter�*z5Invalid wildcard with more than 2 consecutive *s (%s)z Skipping symbolic link %s...rs � ) r; r rH �FLAT_LIST_REGEXrM �grouprf �WildcardException�lstripr� �sep�_IterDir�glob�iglobr r� r� �islinkr( �info�isdirr r� r �UnicodeEncodeErrorr �join�textwrap�wrap�_UNICODE_EXCEPTION_TEXT�repr) r rP Z include_sizerp rM �base_dirZremaining_wildcardZ filepathsr� Z expanded_urlZ blr_objectr r r rZ ^ s@ � � �z FileWildcardIterator.__iter__c c s t j�|�d |kr |d7 }t jt�|�dd�D ]�\}}}g }|D ]J}t j�||�}| �|�sB|�|� nqB| j rBt j� |�rB| j � d|� qB||dd�< t � ||�D ]V} zt j�|t| ��V W q� ty� td�t�ttt j�|| �� ����Y q�0 q�q0dS )a� An iterator over the specified dir and wildcard. Args: directory (unicode): The path of the directory to iterate over. wildcard (str): The wildcard characters used for filename pattern matching. Yields: (str) A string containing the path to a file somewhere under the directory hierarchy of `directory`. Raises: ComandException: If this method encounters a file path that it cannot decode as UTF-8. r �\T)�topdownz Skipping symlink directory "%s"Nr� )r� r� � splitdrive�walk�six� ensure_textr� � _ExcludeDirrO r( r� r� r_ �filterr �UnicodeDecodeErrorr r� r� r� r� ) r � directoryrp �dirpath�dirnames� filenamesZfiltered_dirnames�dirnameZ full_dir_path�fr r r r� � s. � ��z FileWildcardIterator._IterDirc C sz | j du rdS | j \}}}|s"dS t|�jt|j�d� }|�| jj�rT|dd� }|�|�rv| jrr| j� d|� dS dS )z�Check a directory to see if it should be excluded from os.walk. Args: dir: String representing the directory to check. Returns: True if the directory should be excluded. NFri z!Skipping excluded directory %s...T) r� r r �lenrf r �delimrM r( r� )r �dir�base_urlZ exclude_dirsZexclude_patternZ str_to_checkr r r r� � s � � z FileWildcardIterator._ExcludeDirc c s$ | j |d�D ]}|�� r |V q dS )a� Iterates over the wildcard, yielding only object (file) refs. Args: bucket_listing_fields: Iterable fields to include in listings. Ex. ['size']. Currently only 'size' is supported. If present, will populate yielded BucketListingObject.root_object with the file name and size. Yields: BucketListingRefs of type OBJECT or empty iterator if no matches. �rP N)r| rG )r rP rS r r r r} s � z FileWildcardIterator.IterObjectsc c s | j |d�D ] }|V q dS )a� Iterates over the wildcard, yielding BucketListingRefs. Args: bucket_listing_fields: Iterable fields to include in listings. Ex. ['size']. Currently only 'size' is supported. If present, will populate yielded BucketListingObject.root_object with the file name and size. expand_top_level_buckets: Ignored; filesystems don't have buckets. Yields: BucketListingRefs of type OBJECT (file) or PREFIX (directory), or empty iterator if no matches. r� Nr{ )r rP rQ rS r r r r| s � z FileWildcardIterator.IterAllc C s t d��dS )z�Placeholder to allow polymorphic use of WildcardIterator. Args: unused_bucket_fields: Ignored; filesystems don't have buckets. Raises: WildcardException: in all cases. z6Iterating over Buckets not possible for file wildcardsN)r� )r Zunused_bucket_fieldsr r r ry * s �z FileWildcardIterator.IterBuckets)NFN)N)N)NF)N) r r r r! r) rZ r� r� r} r| ry r r r r r� ? s � <T r� c @ s( e Zd ZdZdd� Zdd� Zdd� ZdS ) r� z+Exception raised for invalid wildcard URLs.c C s t �| � || _d S rq )� StandardErrorr) �reason)r r� r r r r) : s zWildcardException.__init__c C s d| j S �NzWildcardException: %s�r� r r r r r > s zWildcardException.__repr__c C s d| j S r� r� r r r r �__str__A s zWildcardException.__str__N)r r r r! r) r r� r r r r r� 7 s r� Fc C s@ t | �}|pt�� }|�� r,t||||d�S t||||d�S dS )a� Instantiate a WildcardIterator for the given URL string. Args: url_str: URL string naming wildcard object(s) to iterate. gsutil_api: Cloud storage interface. Passed in for thread safety, also settable for testing/mocking. all_versions: If true, the iterator yields all versions of objects matching the wildcard. If false, yields just the live object version. project_id: Project id to use for bucket listings. exclude_tuple: (base_url, exclude_pattern), where base_url is top-level URL to list; exclude_pattern is a regex of paths to ignore during iteration. ignore_symlinks: For FileUrls, ignore symlinks during iteration if true. logger: logging.Logger used for outputting debug messages during iteration. If None, the root logger will be used. Returns: A WildcardIterator that handles the requested iteration. )r� r� r( )r# r% N)r r& r' � IsFileUrlr� r" )�url_strr$ r# r% r� r� r( rT r r r �CreateWildcardIteratorE s ��r� )FNNFN)2r! � __future__r r r r r_ r� r& r� r] r� r� Zgslib.bucket_listing_refr r r �gslib.cloud_apir r r �gslib.exceptionr �gslib.storage_urlr r r r r Z"gslib.third_party.storage_apitoolsr r �gslib.utils.constantsr Zgslib.utils.text_utilr r �PY3� Exceptionr� r^ r� r� �objectr r" r� r� r� r� r r r r �<module> s^ � c y �
SIMPAN
[ BATAL ]