CURRENT PATH:
/
opt
/
gsutil
/
gslib
/
commands
/
__pycache__
/
[ ⬅ KEMBALI ]
|
[ HOME ]
Upload File:
Upload Ke Sini
Dir Baru
File Baru
Editing:
rewrite.cpython-39.pyc
a +(Wg�f � @ s� d Z ddlmZ ddlmZ ddlmZ ddlmZ ddlZddlZddlZddl m Z ddl m Z dd l mZ dd lmZ dd lmZ dd lmZ dd lmZ ddlmZ ddlmZ ddlmZ ddl m Z ddl m Z ddl!m"Z" ddl#m$Z$ ddl%m&Z& ddl%m'Z' ddl(m)Z) ddl(m*Z* ddl(m+Z+ ddl(m,Z, ddl-m.Z. dd l-m/Z/ dd l0m1Z1 dd l2m3Z3 dd l2m4Z4 dd l5m6Z6 dd!l7m8Z8 d"Z9d#Z:d$e: d% Z;d&d'� Z<d2d(d)�Z=d*d+� Z>G d,d-� d-e?�Z@G d.d/� d/e�ZAd0d1� ZBdS )3zIImplementation of rewrite command (in-place cloud object transformation).� )�absolute_import)�print_function)�division)�unicode_literalsN)�encoding)�config)�EncryptionException)�Command)�CommandArgument)� ApiSelector)�CommandException)�NameExpansionIterator)� SeekAheadNameExpansionIterator)�FileProgressCallbackHandler)�StorageUrlFromString)�storage_v1_messages)� FileMessage)�GetCloudApiInstance)�NO_MAX)�UTF8)� CryptoKeyType)�CryptoKeyWrapperFromKey)�GetEncryptionKeyWrapper)�MAX_DECRYPTION_KEYS)�GcloudStorageFlag)�GcloudStorageMap)� StdinIterator)� ConvertRecursiveToFlatWildcard)�NormalizeStorageClass)� text_util)�PreconditionsFromHeaders�A z� gsutil rewrite -k [-O] [-f] [-r] [-s] url... gsutil rewrite -k [-O] [-f] [-r] [-s] -I gsutil rewrite -s [-k] [-O] [-f] [-r] url... gsutil rewrite -s [-k] [-O] [-f] [-r] -I z <B>SYNOPSIS</B> a� <B>DESCRIPTION</B> The gsutil rewrite command rewrites cloud objects, applying the specified transformations to them. The transformation(s) are atomic for each affected object and applied based on the input transformation flags. Object metadata values are preserved unless altered by a transformation. At least one transformation flag, -k or -s, must be included in the command. The -k flag is supported to add, rotate, or remove encryption keys on objects. For example, the command: gsutil rewrite -k -r gs://bucket updates all objects in gs://bucket with the current encryption key from your boto config file, which may either be a base64-encoded CSEK or the fully-qualified name of a Cloud KMS key. The rewrite command acts only on live object versions, so specifying a URL with a generation number fails. If you want to rewrite a noncurrent version, first copy it to the live version, then rewrite it, for example: gsutil cp gs://bucket/object#123 gs://bucket/object gsutil rewrite -k gs://bucket/object You can use the -s option to specify a new storage class for objects. For example, the command: gsutil rewrite -s nearline gs://bucket/foo rewrites the object, changing its storage class to nearline. If you specify the -k option and you have an encryption key set in your boto configuration file, the rewrite command skips objects that are already encrypted with the specified key. For example, if you run: gsutil rewrite -k -r gs://bucket and gs://bucket contains objects encrypted with the key specified in your boto configuration file, gsutil skips rewriting those objects and only rewrites objects that are not encrypted with the specified key. This avoids the cost of performing redundant rewrite operations. If you specify the -k option and you do not have an encryption key set in your boto configuration file, gsutil always rewrites each object, without explicitly specifying an encryption key. This results in rewritten objects being encrypted with either the bucket's default KMS key (if one is set) or Google-managed encryption (no CSEK or CMEK). Gsutil does not attempt to determine whether the operation is redundant (and thus skippable) because gsutil cannot be sure how the object is encrypted after the rewrite. Note that if your goal is to encrypt objects with a bucket's default KMS key, you can avoid redundant rewrite costs by specifying the bucket's default KMS key in your boto configuration file; this allows gsutil to perform an accurate comparison of the objects' current and desired encryption configurations and skip rewrites for objects already encrypted with that key. If have an encryption key set in your boto configuration file and specify multiple transformations, gsutil only skips those that would not change the object's state. For example, if you run: gsutil rewrite -s nearline -k -r gs://bucket and gs://bucket contains objects that already match the encryption configuration but have a storage class of standard, the only transformation applied to those objects would be the change in storage class. You can pass a list of URLs (one per line) to rewrite on stdin instead of as command line arguments by using the -I option. This allows you to use gsutil in a pipeline to rewrite objects identified by a program, such as: some_program | gsutil -m rewrite -k -I The contents of stdin can name cloud URLs and wildcards of cloud URLs. The rewrite command requires OWNER permissions on each object to preserve object ACLs. You can bypass this by using the -O flag, which causes gsutil not to read the object's ACL and instead apply the default object ACL to the rewritten object: gsutil rewrite -k -O -r gs://bucket <B>OPTIONS</B> -f Continues silently (without printing error messages) despite errors when rewriting multiple objects. If some of the objects could not be rewritten, gsutil's exit status is non-zero even if this flag is set. This option is implicitly set when running "gsutil -m rewrite ...". -I Causes gsutil to read the list of objects to rewrite from stdin. This allows you to run a program that generates the list of objects to rewrite. -k Rewrite objects with the current encryption key specified in your boto configuration file. The value for encryption_key may be either a base64-encoded CSEK or a fully-qualified KMS key name. If no value is specified for encryption_key, gsutil ignores this flag. Instead, rewritten objects are encrypted with the bucket's default KMS key, if one is set, or Google-managed encryption, if no default KMS key is set. -O When a bucket has uniform bucket-level access (UBLA) enabled, the -O flag is required and skips all ACL checks. When a bucket has UBLA disabled, the -O flag rewrites objects with the bucket's default object ACL instead of the existing object ACL. This is needed if you do not have OWNER permission on the object. -R, -r The -R and -r options are synonymous. Causes bucket or bucket subdirectory contents to be rewritten recursively. -s <class> Rewrite objects using the specified storage class. c C s( | j s| j�t|�� | jd7 _dS )z9Simple exception handler to allow post-completion status.� N)�continue_on_error�logger�error�str�op_failure_count)�cls�e� r* �%/opt/gsutil/gslib/commands/rewrite.py�_RewriteExceptionHandler� s r, c C s | j ||d� d S )N�� thread_state)� RewriteFunc)r( �name_expansion_resultr. r* r* r+ �_RewriteFuncWrapper� s r1 c c s. | D ]$}t |�jdur"td| ��|V qdS )zAGenerator function that ensures generation-less (live) arguments.Nz-"rewrite" called on URL with generation (%s).)r � generationr )�url_strs�url_strr* r* r+ �GenerationCheckGenerator� s �r5 c @ s e Zd ZdZdZdZdS )�_TransformTypesz Enum class for valid transforms.� crypto_key� storage_classN)�__name__� __module__� __qualname__�__doc__� CRYPTO_KEY� STORAGE_CLASSr* r* r* r+ r6 � s r6 c @ s� e Zd ZdZejdg ededdddej gej e � � gd� Z ej dddgd d ei d �Zeg d �ed �ed�e�ddd�rzdned�ed�ed�ed�ed�d�d�Zdd� Zdd� Zd d d �ZdS ) �RewriteCommandz)Implementation of gsutil rewrite command.�rewriter zfkIrROs:F) �command_name_aliases�usage_synopsis�min_args�max_args�supported_sub_args� file_url_ok�provider_url_ok�urls_start_arg�gs_api_support�gs_default_api�argparse_argumentsZrekey�rotate� command_helpzRewrite objects)� help_name�help_name_aliases� help_type�help_one_line_summary� help_text�subcommand_help_text)�storage�objects�update�-Iz--continue-on-error�GSUtil�encryption_keyNz--clear-encryption-keyz--no-preserve-acl�-r�-s)rW �-f�-k�-OrZ �-Rr[ )�gcloud_command�flag_mapc C s |j dkrtdt|� ��d S )N�gsz5"rewrite" called on URL with unsupported provider: %s)�schemer r& )�self�urlr* r* r+ � CheckProvider s �z RewriteCommand.CheckProviderc C s� | j | _i | _d| _d| _d| _ddg| _t� | _d| _ t t �| _ | j rP| j j nd| _| jr�| jD ]�\}}|dkrzd| _qb|dkr�| j�tj� qb|dkr�d| _qb|d kr�d| _qb|d ks�|d kr�d| _d| _qb|dkrb| j�tj� t|�| _qb| j�r| j�r td ��t� }n| j�s"td ��| j}| j�s>td| j ��t| j�pJi �| _ t |�}| j�rht |�}t | j | j!| j"| j#|| j| j$| j�p�| j ddgd� }d}| j�s�t |�}t%| j | j!| �&� || j| j| j$d�}t'dt(�D ]R}|d } t)t �*ddt+| � d��} | du �r �q2| j,t-j.k�r�| | j| j < �q�| jdu�rL| j | j| j< | j du �rzd�/t0�1d��} t2d| t3j4d� | j5t6|t7| j dg|d� | j �r�| j �r�d nd } td | j | | f ��dS ) z,Command entry point for the rewrite command.NFr] r[ r r\ TrW r^ rZ r_ z&No arguments allowed with the -I flag.z:The rewrite command (without -I) expects at least one URL.zgrewrite command requires at least one transformation flag. Currently supported transformation flags: %s�name�size)� project_idr# �bucket_listing_fields)� all_versionsri r" rX zdecryption_key%s� a> NOTE: No encryption_key was specified in the boto configuration file, so gsutil will not provide an encryption key in its rewrite API requests. This will decrypt the objects unless they are in buckets with a default KMS key set, in which case the service will automatically encrypt the rewritten objects with that key.z%s )�filer' )� fail_on_error� shared_attrs�seek_ahead_iterator�s� z*%d file%s/object%s could not be rewritten.)8�parallel_operationsr# �csek_hash_to_keywrapper�dest_storage_class�no_preserve_acl�read_args_from_stdinZ supported_transformation_flags�set�transform_typesr' r r � boto_file_encryption_keywrapper�crypto_key_sha256Zboto_file_encryption_sha256�sub_opts�addr6 r= �recursion_requestedrk r> r �argsr r r �headers� preconditionsr5 r r � command_name�debugr$ � gsutil_apiri r �GetSeekAheadGsutilApi�ranger r �getr& � crypto_typer �CSEK�join�textwrap�wrap�print�sys�stderr�Applyr1 r, ) rd �o�ar3 Zurl_strs_generator�name_expansion_iteratorrp Zseek_ahead_url_strs�i� key_number� keywrapper�msg� plural_strr* r* r+ � RunCommand s� � � �� � � � � ��� �zRewriteCommand.RunCommandc C s t | |d�}|j}| �|� |j|j|j|j|jd�}| jrDg |_ n|j sVt d| ��|j rb|j nd }d }|j r�|j j r�|j j }|�d�}|d up�|d u}d } | jd ur�| jjtjkr�| jj} d } | jd ur�| jjtjkr�| jj} | jd u} || ko�|| k} tj| jv�r | �s td||f ��g } tj| jv �rN| jt |j �k�rN| � d� tj| jv �rr| �rr| �rr| � d� t | �t | j�k�r�| j �!d|| f � d S t"�#t$j%t"�&|��}d |_d |_'d |_ d |_ tj| jv �r�| j|_ | d u�r�| |_ d }|d u�r&|| j(v �r| j(| }ntd ||f ��d }tj| jv �rr|�rP| �rP| �srd }n"|�rb| �sbd }n|�sr| �rrd }t)j*�+t,||j-�� t)j*�.� |j/�0t1|d t2�2� d|j3t1j4d�� t5|j/||d�j6}|j7|||j| j8||| j|jg d� |j/�0t1|d t2�2� d|j3t1j4d�� d S )Nr- )r2 �providera No OWNER permission found for object %s. If your bucket has uniform bucket-level access (UBLA) enabled, include the -O option in your command to avoid this error. If your bucket does not use UBLA, you can use the -O option to apply the bucket's default object ACL when rewriting.�asciiz�The "-k" flag was not passed to the rewrite command, but the encryption_key value in your boto config file did not match the key used to encrypt the object "%s" (hash: %s). To encrypt the object using a different key, you must specify the "-k" flag.z storage classzencryption keyz3Skipping %s, all transformations were redundant: %szOMissing decryption key with SHA256 hash %s. No decryption key matches object %sZ RewritingZRotatingZ DecryptingZ EncryptingF)�finishedrh � message_type)�src_url�operation_name)�src_generationr� �progress_callback�decryption_tuple�encryption_tupler� �fieldsT)9r �expanded_storage_urlrf �GetObjectMetadata� bucket_name� object_namer2 rc rv �aclr � kmsKeyName�customerEncryption� keySha256�encoderz r� r �CMEKr7 r� r{ r6 r= ry r r> ru r � storageClass�append�lenr$ �infor �PyValueToMessage�apitools_messages�Object�MessageToPyValue�idrt r� r� �write�_ConstructAnnounceText� url_string�flush� status_queue�putr �timerh � FILE_REWRITEr �call� CopyObjectr� )rd r0 r. r� Z transform_urlZ src_metadataZsrc_encryption_kms_keyZsrc_encryption_sha256Zsrc_was_encryptedZdest_encryption_kms_keyZdest_encryption_sha256Zshould_encrypt_destZencryption_unchangedZredundant_transformsZ dest_metadata�decryption_keywrapperr� r� r* r* r+ r/ � s ������ � � �� ���� �� �� � � ��� � ���� ��zRewriteCommand.RewriteFunc)N)r9 r: r; r< r �CreateCommandSpec� _SYNOPSISr r �JSONr � MakeZeroOrMoreCloudURLsArgument� command_spec�HelpSpec�_DETAILED_HELP_TEXT� help_specr r r r� �gcloud_storage_maprf r� r/ r* r* r* r+ r? � sN �� �� r? c C sz | dd� � d�}t|�}td�}|t|� | tkr\td�}d|t| | | d� }d||f }dt }|�|�S ) a� Constructs announce text for ongoing operations on url_string. This truncates the text to a maximum of MAX_PROGRESS_INDICATOR_COLUMNS, and informs the rewrite-related operation ('Encrypting', 'Rotating', or 'Decrypting'). Args: operation_name: String describing the operation, i.e. 'Rotating' or 'Encrypting'. url_string: String describing the file/object being processed. Returns: Formatted announce text for outputting operation progress. N� � z: z...z...%sz%s%s:z{0:%ds})�ljustr� � MAX_PROGRESS_INDICATOR_COLUMNS�format)r� r� Zjustified_op_stringZ start_lenZend_lenZ ellipsis_lenZbase_announce_text� format_strr* r* r+ r� E s ��� r� )N)Cr< � __future__r r r r r� r� r� �apitools.base.pyr �botor �gslib.cloud_apir � gslib.commandr �gslib.command_argumentr �gslib.cs_api_mapr �gslib.exceptionr �gslib.name_expansionr r �gslib.progress_callbackr �gslib.storage_urlr �"gslib.third_party.storage_apitoolsr r� �gslib.thread_messager � gslib.utils.cloud_api_helperr �gslib.utils.constantsr r � gslib.utils.encryption_helperr r r r �gslib.utils.shim_utilr r Zgslib.utils.system_utilr Zgslib.utils.text_utilr r Z gslib.utilsr � gslib.utils.translation_helperr r� r� r� r, r1 r5 �objectr6 r? r� r* r* r* r+ �<module> sb ��v w
SIMPAN
[ BATAL ]