CURRENT PATH:
/
opt
/
gsutil
/
gslib
/
__pycache__
/
[ ⬅ KEMBALI ]
|
[ HOME ]
Upload File:
Upload Ke Sini
Dir Baru
File Baru
Editing:
name_expansion.cpython-39.pyc
a +(Wg�w � @ s| d Z ddlmZ ddlmZ ddlmZ ddlmZ ddlZddlZddlZddl Z ddl Z ddl m Z ddl Z ddlmZ dd lmZ dd lmZ dd lmZ dd lmZ dd lmZ ddlZ ddlmZ G dd� de�Z G dd� de�Z G dd� de�Z d)dd�Z G dd� de�Z G dd � d e�Z!G d d � d e�Z"G d d � d e�Z#e�$d!d"d#g�Z%e�$d$d%d&g�Z&G d'd(� d(e j'�Z(dS )*aB Name expansion iterator and result classes. Name expansion support for the various ways gsutil lets users refer to collections of data (via explicit wildcarding as well as directory, bucket, and bucket subdir implicit wildcarding). This class encapsulates the various rules for determining how these expansions are done. � )�absolute_import)�print_function)�division)�unicode_literalsN)�encoding)�CommandException)�NO_URLS_MATCHED_GENERIC)�NO_URLS_MATCHED_TARGET)�PluralityCheckableIterator)�SeekAheadResult)�storage_v1_messages)�StorageUrlFromStringc @ s e Zd ZdZdd� Zdd� ZdS )�NameExpansionResulta* Holds one fully expanded result from iterating over NameExpansionIterator. The member data in this class need to be pickleable because NameExpansionResult instances are passed through Multiprocessing.Queue. In particular, don't include any boto state like StorageUri, since that pulls in a big tree of objects, some of which aren't pickleable (and even if they were, pickling/unpickling such a large object tree would result in significant overhead). The state held in this object is needed for handling the various naming cases (e.g., copying from a single source URL to a directory generates different dest URL names than copying multiple URLs to a directory, to be consistent with naming rules used by the Unix cp command). For more details see comments in _NameExpansionIterator. c C s6 || _ || _|| _|| _|| _|r,t�|�nd| _dS )a� Instantiates a result from name expansion. Args: source_storage_url: StorageUrl that was being expanded. is_multi_source_request: bool indicator whether multiple input URLs or src_url_str expanded to more than one BucketListingRef. is_multi_top_level_source_request: same as is_multi_source_request but measured before recursion. names_container: Bool indicator whether src_url names a container. expanded_storage_url: StorageUrl that was expanded. expanded_result: cloud object metadata in MessageToJson form (for pickleability), if any was iterated; None otherwise. Consumers must call JsonToMessage to get an apitools Object. N)�source_storage_url�is_multi_source_request�!is_multi_top_level_source_request�names_container�expanded_storage_urlr � MessageToJson�expanded_result)�selfr r r r r r � r �#/opt/gsutil/gslib/name_expansion.py�__init__@ s ���z NameExpansionResult.__init__c C s d| j S )Nz%s)r �r r r r �__repr__Y s z NameExpansionResult.__repr__N)�__name__� __module__� __qualname__�__doc__r r r r r r r / s r c @ s* e Zd ZdZd dd�Zdd� Zd d � ZdS ) �_NameExpansionIteratorzlClass that iterates over all source URLs passed to the iterator. See details in __iter__ function doc. FTNc C st || _ || _|| _|| _|| _|| _|| _| j�� | j_|| _ | | _ | | _ | | _ | s^t dg�n| | _ddd�| _dS )a� Creates a NameExpansionIterator. Args: command_name: name of command being run. debug: Debug level to pass to underlying iterators (range 0..3). logger: logging.Logger object. gsutil_api: Cloud storage interface. Settable for testing/mocking. url_strs: PluralityCheckableIterator of URL strings needing expansion. recursion_requested: True if -r specified on command-line. If so, listings will be flattened so mapped-to results contain objects spanning subdirectories. all_versions: Bool indicating whether to iterate over all object versions. cmd_supports_recursion: Bool indicating whether this command supports a '-r' flag. Useful for printing helpful error messages. project_id: Project id to use for bucket retrieval. ignore_symlinks: If True, ignore symlinks during iteration. continue_on_error: If true, yield no-match exceptions encountered during iteration instead of raising them. bucket_listing_fields: Iterable fields to include in expanded results. Ex. ['name', 'acl']. Underyling iterator is responsible for converting these to list-style format ['items/name', 'items/acl']. If this is None, only the object name is included in the result. Examples of _NameExpansionIterator with recursion_requested=True: - Calling with one of the url_strs being 'gs://bucket' will enumerate all top-level objects, as will 'gs://bucket/' and 'gs://bucket/*'. - 'gs://bucket/**' will enumerate all objects in the bucket. - 'gs://bucket/abc' will enumerate either the single object abc or, if abc is a subdirectory, all objects under abc and any of its subdirectories. - 'gs://bucket/abc/**' will enumerate all objects under abc or any of its subdirectories. - 'file:///tmp' will enumerate all files under /tmp, as will 'file:///tmp/*' - 'file:///tmp/**' will enumerate all files under /tmp or any of its subdirectories. Example if recursion_requested=False: calling with gs://bucket/abc/* lists matching objects or subdirs, but not sub-subdirs or objects beneath subdirs. Note: In step-by-step comments below we give examples assuming there's a gs://bucket with object paths: abcd/o1.txt abcd/o2.txt xyz/o1.txt xyz/o2.txt and a directory file://dir with file paths: dir/a.txt dir/b.txt dir/c/ �namez**�*)TFN)� command_name�debug�logger� gsutil_api�url_strs�recursion_requested� all_versions� HasPlurality� has_plurality�cmd_supports_recursion� project_id�ignore_symlinks�continue_on_error�set�bucket_listing_fields�_flatness_wildcard) r r# r$ r% r& r' r( r) r, r- r. r/ r1 r r r r c s A�z _NameExpansionIterator.__init__c c sl | j D �]^}t|�}|�� rT|�� s,|�� rT| j jr<td��t|ddd|dd�V qd}|�� r�|� � r�| j s�t | � |�j dgd��}n.t | � |�j| jdd��}|�� r�|� � r�d}|�� }| j jp�|}| j| j }| j r�t| ||| j�}nt|�}t |�}|�� �rh| j�r\ztt| ��W n8 t�yX } z | t�� d fV W Y d} ~ n d} ~ 0 0 n tt| ��t t|| j | j| j| j ��} | �� }| j j�p�|} | D ]�\} } |�p�| }| � � �r�t|| ||| j | j d�V n�t| j �}|�� �r�d | t!j"|f }n |j#|d �}t | � |�j$| jd ��}|�p,|�� }| j j�p:|} |D ] } t|| |d| j | j d�V �q@�q�qdS ) a9 Iterates over all source URLs passed to the iterator. For each src url, expands wildcards, object-less bucket names, subdir bucket names, and directory names, and generates a flat listing of all the matching objects/files. You should instantiate this object using the static factory function NameExpansionIterator, because consumers of this iterator need the PluralityCheckableIterator wrapper built by that function. Yields: gslib.name_expansion.NameExpansionResult. Raises: CommandException: if errors encountered. zPMultiple URL strings are not supported with streaming ("-") URLs or named pipes.FN)r r r r r r �id)� bucket_fieldsT)r1 �expand_top_level_buckets� z%s%s%s��wildcard_suffix�r1 )%r' r � IsFileUrl�IsStream�IsFifor+ r r � IsCloudUrl�IsBucketr( r �WildcardIterator� IterBuckets�IterAllr1 r* r2 � _ImplicitBucketSubdirIterator� _NonContainerTuplifyIterator�IsEmptyr/ r �sys�exc_info�_OmitNonRecursiveIteratorr# r, r% �IsObject� storage_url� root_object� url_string�os�sep�CreatePrefixUrl� IterObjects)r �url_strrI Zsrc_names_bucketZpost_step1_iterZsrc_url_expands_to_multir �subdir_exp_wildcardZpost_step2_iter�eZpost_step3_iterr r �blrZsrc_names_container� expanded_urlZurl_to_iterateZwc_iterr r r �__iter__� s� ����� ��� � * �� � � � ��� ��z _NameExpansionIterator.__iter__c C s" t jj|| j| j| j| j| jd�S )a2 Helper to instantiate gslib.WildcardIterator. Args are same as gslib.WildcardIterator interface, but this method fills in most of the values from instance state. Args: url_string: URL string naming wildcard objects to iterate. Returns: Wildcard iterator over URL string. )r) r- r. r% )�gslib�wildcard_iterator�CreateWildcardIteratorr& r) r- r. r% )r rK r r r r? Q s �z'_NameExpansionIterator.WildcardIterator)FTNFFN)r r r r r rU r? r r r r r ] s � V r c @ s" e Zd ZdZd dd�Zdd� ZdS ) � SeekAheadNameExpansionIteratorz�Creates and wraps a _NameExpansionIterator and yields SeekAheadResults. Unlike the NameExpansionIterator, which can make API calls upon __init__ to check for plurality, this iterator does no work until the first iteration occurs. FTNc C sP |dv o | | _ | j r dgnd} t||t�d�|t|�||||| d| d� | _dS )z5Initializes a _NameExpansionIterator with the inputs.)�cp�mv�rewrite�sizeN�dummyT�r) r, r- r. r/ r1 )�count_data_bytesr �logging� getLoggerr �name_expansion_iterator) r r# r$ r& r' r( r) r, r- r. Zfile_size_will_changer1 r r r r n s$ ��z'SeekAheadNameExpansionIterator.__init__c c sL | j D ]@}| jr>|jr>t�tj|j�}|jp.d}t|d�V qt� V qd S )Nr )Z data_bytes) rc r` r r � JsonToMessage�apitools_messages�Objectr] r )r �name_expansion_resultZiterated_metadataZ iterated_sizer r r rU � s � z'SeekAheadNameExpansionIterator.__iter__)FTNFF�r r r r r rU r r r r rY f s � %rY FTc C sD t |�}t| ||||||||| | | d� } t | �} | �� r@tt��| S )ar Static factory function for instantiating _NameExpansionIterator. This wraps the resulting iterator in a PluralityCheckableIterator and checks that it is non-empty. Also, allows url_strs to be either an array or an iterator. Args: command_name: name of command being run. debug: Debug level to pass to underlying iterators (range 0..3). logger: logging.Logger object. gsutil_api: Cloud storage interface. Settable for testing/mocking. url_strs: Iterable URL strings needing expansion. recursion_requested: True if -r specified on command-line. If so, listings will be flattened so mapped-to results contain objects spanning subdirectories. all_versions: Bool indicating whether to iterate over all object versions. cmd_supports_recursion: Bool indicating whether this command supports a '-r' flag. Useful for printing helpful error messages. project_id: Project id to use for the current command. ignore_symlinks: If True, ignore symlinks during iteration. continue_on_error: If true, yield no-match exceptions encountered during iteration instead of raising them. bucket_listing_fields: Iterable fields to include in expanded results. Ex. ['name', 'acl']. Underyling iterator is responsible for converting these to list-style format ['items/name', 'items/acl']. If this is None, only the object name is included in the result. Raises: CommandException if underlying iterator is empty. Returns: Name expansion iterator instance. For example semantics, see comments in NameExpansionIterator.__init__. r_ )r r rD r r ) r# r$ r% r& r' r( r) r, r- r. r/ r1 rc r r r �NameExpansionIterator� s& /� ri c @ s e Zd ZdZdd� Zdd� ZdS )rC z�Iterator that produces the tuple (False, blr) for each iterated value. Used for cases where blr_iter iterates over a set of BucketListingRefs known not to name containers. c C s || _ dS )zTInstantiates iterator. Args: blr_iter: iterator of BucketListingRef. N��blr_iter)r rk r r r r � s z%_NonContainerTuplifyIterator.__init__c c s | j D ]}d|fV qd S )NFrj )r rS r r r rU � s z%_NonContainerTuplifyIterator.__iter__Nrh r r r r rC � s rC c @ s e Zd ZdZdd� Zdd� ZdS )rG a� Iterator wrapper for that omits certain values for non-recursive requests. This iterates over tuples of (names_container, BucketListingReference) and omits directories, prefixes, and buckets from non-recurisve requests so that we can properly calculate whether the source URL expands to multiple URLs. For example, if we have a bucket containing two objects: bucket/foo and bucket/foo/bar and we do a non-recursive iteration, only bucket/foo will be yielded. c C s" || _ || _|| _|| _|| _dS )a� Instanties the iterator. Args: tuple_iter: Iterator over names_container, BucketListingReference from step 2 in the NameExpansionIterator recursion_requested: If false, omit buckets, dirs, and subdirs command_name: Command name for user messages cmd_supports_recursion: Command recursion support for user messages logger: Log object for user messages N)� tuple_iterr( r# r, r% )r rl r( r# r, r% r r r r s z"_OmitNonRecursiveIterator.__init__c c s| | j D ]p\}}| jsl|�� slt|j�}|�� r4d}n|j}| jrX| j� d||j| j � qv| j� d||j� q||fV qd S )N� directoryz-Omitting %s "%s". (Did you mean to do %s -r?)zOmitting %s "%s".) rl r( rH r rK r: � type_namer, r% �infor# )r r rS rT �descr r r rU s �z"_OmitNonRecursiveIterator.__iter__Nrh r r r r rG � s rG c @ s e Zd ZdZdd� Zdd� ZdS )rB a� Iterator wrapper that performs implicit bucket subdir expansion. Each iteration yields tuple (names_container, expanded BucketListingRefs) where names_container is true if URL names a directory, bucket, or bucket subdir. For example, iterating over [BucketListingRef("gs://abc")] would expand to: [BucketListingRef("gs://abc/o1"), BucketListingRef("gs://abc/o2")] if those subdir objects exist, and [BucketListingRef("gs://abc") otherwise. c C s || _ || _|| _|| _dS )a� Instantiates the iterator. Args: name_exp_instance: calling instance of NameExpansion class. blr_iter: iterator over BucketListingRef prefixes and objects. subdir_exp_wildcard: wildcard for expanding subdirectories; expected values are ** if the mapped-to results should contain objects spanning subdirectories, or * if only one level should be listed. bucket_listing_fields: Fields requested in enumerated results. N)rk �name_exp_instancerQ r1 )r rq rk rQ r1 r r r r 3 s z&_ImplicitBucketSubdirIterator.__init__c c s� | j D ]�}|�� rjt|j�j| jd�}t| j�|�j | j d��}|� � s^|D ]}d|fV qLq�d|fV q|� � r~d|fV qt d| ��qd S )Nr7 r9 TFz7_ImplicitBucketSubdirIterator got a bucket reference %s)rk �IsPrefixr rK rN rQ r rq r? rA r1 rD rH r )r rS � prefix_urlZimplicit_subdir_iteratorZexp_blrr r r rU E s& � �� �z&_ImplicitBucketSubdirIterator.__iter__Nrh r r r r rB ' s rB c @ s e Zd ZdZdd� ZdS )�CopyObjectInfozARepresents the information needed for copying a single object. c C s@ |j | _ |j| _|j| _|j| _|j| _|j| _|| _|| _dS )aE Instantiates the object info from name expansion result and destination. Args: name_expansion_result: StorageUrl that was being expanded. exp_dst_url: StorageUrl of the destination. have_existing_dst_container: Whether exp_url names an existing directory, bucket, or bucket subdirectory. N)r r r r r r � exp_dst_url�have_existing_dst_container)r rg ru rv r r r r ` s �zCopyObjectInfo.__init__N)r r r r r r r r r rt \ s rt �DestinationInforu rv �%NameExpansionIteratorDestinationTuple�name_expansion_iter� destinationc @ s( e Zd ZdZdd� Zdd� Zdd� ZdS ) �CopyObjectsIteratora� Iterator wrapper for copying objects and keeping track of source URL types. This is used in the cp command for copying from multiple source to multiple destinations. It takes a list of NameExpansionIteratorDestinationTuple. It wraps them and return CopyObjectInfo objects that wraps NameExpansionResult with the destination. It's used also for collecting analytics PerformanceSummary info, because there may be multiple source URLs and we want to know if any of them are file URLs, if any of them are cloud URLs, if any of them require daisy chain operations, and if any use different providers. The source URL type information will be aggregated at the end of _SequentialApply or _ParallelApply. c C s< || _ d| _d| _g | _|| _t| j�}|j| _|j| _ dS )a Instantiates the iterator. Args: name_expansion_dest_iter: NameExpansionIteratorDestinationTuple iterator. is_daisy_chain: The -D option in cp might have already been specified, in which case we do not need to check again for daisy chain operations. FN) �is_daisy_chain� has_file_src� has_cloud_src�provider_types�name_expansion_dest_iter�nextry �current_expansion_iterrz �current_destination)r r� r| �name_expansion_dest_tupler r r r � s z CopyObjectsIterator.__init__c C s | S )Nr r r r r rU � s z CopyObjectsIterator.__iter__c C s� zt | j�}W n4 tyB t | j�}|j| _|j| _| �� Y S 0 t|| jj | jj �}| j sn|j � � rnd| _ | js�|j �� r�d| _| jj �� r�| jj j}nd}| js�|dur�|j �� r�|j j|kr�d| _|j j| jvr�| j�|j j� |S )z@Keeps track of URL types as the command iterates over arguments.TN)r� r� � StopIterationr� ry rz r� �__next__rt ru rv r} r r: r~ r= �schemer| r �append)r rg r� �eltZdst_url_schemer r r r� � s8 �� � �z CopyObjectsIterator.__next__N)r r r r r rU r� r r r r r{ � s r{ )FTNFFN))r � __future__r r r r � collectionsra rL rE �six�apitools.base.pyr rV �gslib.exceptionr r r �"gslib.plurality_checkable_iteratorr �gslib.seek_ahead_threadr �"gslib.third_party.storage_apitoolsr re �gslib.wildcard_iteratorr �objectr r rY ri rC rG rB rt � namedtuplerw rx �Iteratorr{ r r r r �<module> s` . > � C25�� ��
SIMPAN
[ BATAL ]