CURRENT PATH:
/
opt
/
gsutil
/
gslib
/
commands
/
__pycache__
/
[ ⬅ KEMBALI ]
|
[ HOME ]
Upload File:
Upload Ke Sini
Dir Baru
File Baru
Editing:
test.cpython-39.pyc
a +(Wg q � @ s� d Z ddlmZ ddlmZ ddlmZ ddlmZ ddlmZ ddlZddl Z ddl Z ddl Z ddl Z ddl Z ddlZddlZddlZddlZddlmZ ddlZdd lmZ dd lmZ dd lmZ dd lmZ dd l m Z ddl m Z ddl m!Z! ddl m"Z" ddl m#Z# ddl$m%Z% ddl$m&Z& ddl'm(Z( z ddl)Z)W n e*�yX dZ)Y n0 ej+�rfe,Z-dZ.dZ/dZ0dZ1de1 de. Z2edd�Z3d d � Z4d d � Z5d d!� Z6d"d#� Z7d$d%� Z8d*d&d'�Z9G d(d)� d)e�Z:dS )+�&Implementation of gsutil test command.� )�absolute_import)�print_function)�division)�unicode_literals)� namedtupleN)�range)�ProjectIdException)�Command)�ResetFailureCount)�CommandException)�PopulateProjectId)� GetTestNames)�InvokedFromParFile)�unittest)�NO_MAX)�UTF8)� IS_WINDOWS� �2 �sequential_onlyz3 gsutil test [-l] [-u] [-f] [command command...] z <B>SYNOPSIS</B> a <B>DESCRIPTION</B> The gsutil test command runs the gsutil unit tests and integration tests. The unit tests use an in-memory mock storage service implementation, while the integration tests send requests to the production service using the `preferred API <https://cloud.google.com/storage/docs/request-endpoints#gsutil>`_ set in the boto configuration file. CAUTION: The ``test`` command creates test buckets and objects in your project. Force quitting the ``test`` command can leave behind stale buckets, objects, and HMAC keys in your project. To run both the unit tests and integration tests, run the command with no arguments: gsutil test To run the unit tests only (which run quickly): gsutil test -u Tests run in parallel regardless of whether the top-level -m flag is present. To limit the number of tests run in parallel to 10 at a time: gsutil test -p 10 To force tests to run sequentially: gsutil test -p 1 To have sequentially-run tests stop running immediately when an error occurs: gsutil test -f To run tests for one or more individual commands add those commands as arguments. For example, the following command will run the cp and mv command tests: gsutil test cp mv To list available tests, run the test command with the -l argument: gsutil test -l The tests are defined in the code under the gslib/tests module. Each test file is of the format test_[name].py where [name] is the test name you can pass to this command. For example, running "gsutil test ls" would run the tests in "gslib/tests/test_ls.py". You can also run an individual test class or function name by passing the test module followed by the class name and optionally a test name. For example, to run the an entire test class by name: gsutil test naming.GsutilNamingTests or an individual test function: gsutil test cp.TestCp.test_streaming You can list the available tests under a module or class by passing arguments with the -l option. For example, to list all available test functions in the cp module: gsutil test -l cp To output test coverage: gsutil test -c -p 500 coverage html This will output an HTML report to a directory named 'htmlcov'. Test coverage is compatible with v4.1 of the coverage module (https://pypi.python.org/pypi/coverage). <B>OPTIONS</B> -b Run tests against multi-regional US buckets. By default, tests run against regional buckets in us-central1. -c Output coverage information. -f Exit on first sequential test failure. -l List available tests. -p N Run at most N tests in parallel. The default value is %d. -s Run tests against S3 instead of GS. -u Only run unit tests. �TestProcessDataz name return_code stdout stderrc s G � �fdd�dt j�� � S )z�Creates a closure of CustomTestResult. Args: total_tests: The total number of tests being run. Returns: An instance of CustomTestResult. c s$ e Zd ZdZ�� �fdd�Z� ZS )z3MakeCustomTestResultClass.<locals>.CustomTestResultzDA subclass of unittest.TextTestResult that prints a progress report.c s� t � | ��|� | jr�d�|�� �d�dd � �}d| j�t| j�t| j �t| j �|f }|d d� }|� d�}| j � d| � d S )N�.���z( %d/%d finished - E[%d] F[%d] s[%d] - %s�I z%s - )�super� startTest�dots�join�id�splitZtestsRun�len�errorsZfailures�skipped�ljust�stream�write)�self�testZtest_id�message)�CustomTestResult� __class__� total_tests� �"/opt/gsutil/gslib/commands/test.pyr � s �� z=MakeCustomTestResultClass.<locals>.CustomTestResult.startTest)�__name__� __module__� __qualname__�__doc__r � __classcell__r- �r* r, )r+ r. r* � s r* )r ZTextTestResult)r, r- r4 r. �MakeCustomTestResultClass� s r5 c C sX | g}g }|rT|� � }|D ]6}t|tj�r6|�|� q|�|�� td�d� � qq |S )zGTakes a list of test suites and returns a list of contained test names.�gslib.tests.test_N)�pop� isinstancer � TestSuite�appendr r! )� test_suiteZsuites� test_names�suiter( r- r- r. �GetTestNamesFromSuites� s r> c C s t | j��d�d d | j S )z<Converts a python.unittest to its gsutil test-callable name.�'� r )�strr+ r �_testMethodName)� test_caser- r- r. �TestCaseToName� s �rD c C s ddl m} g }g }g }g }| g}g }|rp|�� }t|tjj�rX|jD ]} |�| � qFq&t|tj �r&|�|� q&|D ]�} t | | j d�} t | dd�r�|�t | �� qtt | dd�s�|�t | �� qtt | dd�s�|�t | �� qtt| |�r�|�t | �� qt|�t | �� qtt |�t |�t |�t |�fS )a� Splits a test suite into groups with different running properties. Args: test_suite: A python unittest test suite. Returns: 4-part tuple of lists of test names: (tests that must be run sequentially, tests that must be isolated in a separate process but can be run either sequentially or in parallel, unit tests that can be run in parallel, integration tests that can run in parallel) r )�GsUtilUnitTestCaseNZrequires_isolationFZis_parallelizableT)Z"gslib.tests.testcase.unit_testcaserE r7 r8 r r= r9 Z_testsr: ZTestCase�getattrrB rD �sorted) r; rE �isolated_tests�sequential_testsZ parallelizable_integration_testsZparallelizable_unit_testsZitems_to_evaluateZcases_to_evaluateZ suite_or_case�itemrC Z test_methodr- r- r. � SplitParallelizableTestSuite� s: �rK c C s d}| D ]}|s|d7 }q|S )z*Counts number of falses in the input list.r r@ r- )Z input_listZ num_falserJ r- r- r. �CountFalseInList s rL c C s� |}t � stjgng }tjjr$dgng }tjjr6dgng } g } zddt� g} W n tyb Y n0 t � � } | } t |�|k �r�|t | �k �r�t j �� } |r�|| d< t� }dd� t|tjg | dg | | d t g | | t d �d � g �D �}t�| �D ] \}}t�|�|t�|�< q�|�tj|tjtj|d �� |d 7 }|�d� t � � | dkrptd|| t |�t | �f � t � � } qp|t | �k�r�tdt |�t | �f � |S )ag Creates test processes to run tests in parallel. Args: parallel_tests: List of all parallel tests. test_index: List index of last created test before this function call. process_list: List of running subprocesses. Created processes are appended to this list. process_done: List of booleans indicating process completion. One 'False' will be added per process created. max_parallel_tests: Maximum number of tests to run in parallel. root_coverage_file: The root .coverage filename if coverage is requested. Returns: Index of last created test. �-s�-bz-oz GSUtil:default_project_id=%sZGSUTIL_COVERAGE_OUTPUT_FILEc S s g | ]}t �|��qS r- )�six� ensure_str)�.0�partr- r- r. � <listcomp>H s z'CreateTestProcesses.<locals>.<listcomp>r( �--r6 N)�stdout�stderr�envr@ Fr z.Created %d new processes (total %d/%d created)z.Test process creation finished (%d/%d created)) r �sys� executable�tests�util� RUN_S3_TESTS�USE_MULTIREGIONAL_BUCKETSr r �timerL r! �os�environ�copy�dict�list�gslib� GSUTIL_PATH�_SEQUENTIAL_ISOLATION_FLAGrO � iteritemsrP r: � subprocess�Popen�PIPE�print)Zparallel_tests� test_index� process_list� process_done�max_parallel_tests�root_coverage_fileZorig_test_indexZexecutable_prefixZ s3_argumentZmultiregional_bucketsZproject_id_argZprocess_create_start_time� last_log_timerW Zenvstr�cmd�k�vr- r- r. �CreateTestProcesses sz � � ��������� �� �� �ru c @ sp e Zd ZdZejdg ededdddegd� Z ej dg dd e i d �Z d d � Z d d� Ze� e� fdd�Zdd� ZdS )� TestCommandr r( r zbuflp:scTF) �command_name_aliases�usage_synopsis�min_args�max_args�supported_sub_args� file_url_ok�provider_url_ok�urls_start_arg�supported_private_args� command_helpz2Run gsutil unit/integration tests (for developers))� help_name�help_name_aliases� help_type�help_one_line_summary� help_text�subcommand_help_textc C s< g }g }g }d}d}d} t |�} t�� } } t|d||||d�} t |�| k �r�tt |��D ]�}|| sZ|| �� du rxqZd||< || �� \}}|| j�� || j�� || j dkr�|d7 }|� t || || j ||d�� qZt |�| k �rt|| ||||d�} t |�| k r@t�� | dk�r�t dt |�| |f � t |�| k�rR|d7 }n t |�} d}|d k�r�g }tt |��D ] }|| �sx|� || � �qxt�� | }t d j tt|��|d �� t�� } t�d� q@t�� }|�r0|D ]J}|jdk�r�|j�d �}t d |j � |D ]}t |�t��� � �q�q�||| fS )aZ Executes the parallel/isolated portion of the test suite. Args: parallel_integration_tests: List of tests to execute. max_parallel_tests: Maximum number of parallel tests to run at once. coverage_filename: If not None, filename for coverage output. Returns: (int number of test failures, float elapsed time) r )rp NTr@ )�name� return_coderU rV r z %d/%d finished - %d failures� zL{sec} seconds elapsed since beginning parallel tests. Still running: {procs})�secZprocs� zResults for failed test %s:)r! r^ ru r �poll� communicaterU �closerV � returncoder: r rk �formatrA �int�sleepr� r r� �decoder �strip)r' �parallel_integration_testsro Zcoverage_filenamerm rn Zprocess_results�num_parallel_failuresZ progress_less_logging_cyclesZcompleted_as_of_last_log�num_parallel_testsZparallel_start_timerq rl Zproc_numrU rV Z still_running�elapsedZprocess_run_finish_time�resultZ new_stderr�liner- r- r. �RunParallelTests� s� ���� �� � �z TestCommand.RunParallelTestsc C s~ t d|| |f � t d|| t|| �|t|�|t|�f � | �|� t � |sb|rbt d� n|rnt d� |szt d� dS )z6Prints test results for parallel and sequential tests.z-Parallel tests complete. Success: %s Fail: %szDRan %d tests in %.3fs (%d sequential in %.3fs, %d parallel in %.3fs)�OKzFAILED (parallel tests)zFAILED (sequential tests)N)rk �float�PrintSkippedTests)r' �num_sequential_tests�sequential_success�sequential_skipped�sequential_time_elapsedr� r� �parallel_time_elapsedr- r- r. �PrintTestResults� s( � ��� z TestCommand.PrintTestResultsc C sh t |�dkst |�dkrdt|�}t|�}|�|�}td� |D ]$\}}td|�� � td| � q>dS )a� Prints all skipped tests, and the reasons they were skipped. Takes the union of sequentual_skipped and parallel_skipped, and pretty-prints the resulting methods and reasons. Note that these two arguments are lists of tuples from TestResult.skipped as described here: https://docs.python.org/2/library/unittest.html#unittest.TestResult.skipped Args: sequentual_skipped: An instance of TestResult.skipped. parallel_skipped: An instance of TestResult.skipped. r zTests skipped:z z Reason: N)r! �set�unionrk r )r' r� Zparallel_skippedZ all_skipped�method�reasonr- r- r. r� � s z TestCommand.PrintSkippedTestsc ) C sn d}d}t }d}d}| jr�| jD ]�\}}|dkr:dtj_q |dkrHd}q |dkrVd}q |dkrdd}q |dt krvd}q |dkr�t|�}q |d kr�tjjs�td ��dtj_ q |d kr dtj_ q |r�t s�td ��tjj r�|t kr�| j �d t � t }tt� �}|�r.| j�s.tdt|� � tdd�t|��� dS | j�r�g } | jD ]>} | |v �sb| �d�d |v �rr| �d| � n | �| � �q@ndd� |D �} t�� t�� } | �rlt�� } | D ]�} z| �| �}| � |� W n� t t f�yf } zzd| t |�f }z t | � W nH t!�yH } z.t"�#� }t$�%dd|�}|d| 7 }W Y d}~n d}~0 0 t|��W Y d}~n d}~0 0 �q�|�r�t&| �}tdt|� � tdd�t|��� dS t'�(� �)� t'j*k�r�d}nd }t'�+t'j,� |�r t j d gd d t-�.� d gd!�}|�/� |�0� d}d}t1| �\}}}}tjj �s&g }t'�2d"|� t'�2d#|� t'�2d$|� t'�2d%|� |�pnt|�dk�on| }d&t3j4d'< |�r�| �5� } t6| �} tj7|| |d(�} | �8| �} | �9� }�nf|dk�r�||7 }g }t:�:� } t|�t|� �rXtd)t|�t|� � || }!| �;td*d� |!D ���} | �5� }"t6|"�} tj7|| |d(�} | �8| �} | �9� }| j<}#nd}"d}t:�:� | }$||7 }t|�}%|%�s�n�g }#t=||%�}&|%dk�r�|dk�r�d+}'|&t k�r�|'d,7 }'td-�t>�?|'|%|&f ��� n td.|% � | �@|||�r�|jAjBnd�\}}(| �C|"||#|$|%||(� |�rJ|�D� |�E� |�F� td/|jAjB � d0t3j4d'< |�rj|�sjtG� dS dS )1z)Command entry point for the test command.FrN Tz-cz-fz-lrT z-prM zbS3 tests require S3 credentials. Please add appropriate credentials to your .boto file and re-run.z-uzrCoverage has been requested but the coverage module was not found. You can install it with "pip install coverage".zCReducing parallel tests to %d due to S3 maximum bucket limitations.zFound %d test names:� z r r �gslib.tests.test_%sc S s g | ] }d | �qS )r� r- )rQ r� r- r- r. rS H � z*TestCommand.RunCommand.<locals>.<listcomp>zLFailed to import test code from file %s. TestLoader provided this error: %sz\nz z Additional traceback: %sNr@ � rd zgslib/third_party/*z gslib/tests/*�*)�sourceZomitzSequential tests to run: %szIsolated tests to run: %sz Parallel unit tests to run: %sz%Parallel integration tests to run: %s�1�GSUTIL_TEST_ANALYTICS)� verbosity� resultclass�failfastz Running %d tests sequentially.c S s g | ]}|�qS r- r- )rQ � test_namer- r- r. rS � r� z1Running %d tests in parallel mode (%d processes).z� Please be patient while your CPU is incinerated. If your machine becomes unresponsive, consider reducing the amount of parallel test processes by running 'gsutil test -p <num_processes>'.� z4Running %d tests sequentially in isolated processes.z%Coverage information was saved to: %s�0)H� _DEFAULT_TEST_PARALLEL_PROCESSES�sub_optsrZ r[ r] rf �longZ HAS_S3_CREDSr r\ ZRUN_INTEGRATION_TESTS�coverage�#_DEFAULT_S3_TEST_PARALLEL_PROCESSES�logger�warnrG r �argsrk r! r r r: r ZinstallHandlerZ TestLoaderr9 ZloadTestsFromNameZaddTests� ImportError�AttributeErrorrA � __import__� Exception� traceback� format_exc�re�subr> �logging� getLogger�getEffectiveLevel�INFO�disable�ERROR�tempfile� gettempdirZerase�startrK �debugr_ r` ZcountTestCasesr5 ZTextTestRunner�runZ wasSuccessfulr^ ZloadTestsFromNamesr# �min�textwrap�wrapr� Z data_files�filenamer� �stop�combine�saver ))r' r� Z list_testsro Zperform_coverager �o�ar< Zcommands_to_testr� �loaderr= � command_nameZsuite_for_current_command�e�msg� stack_trace�errr� Zcoverage_controllerr� r� rI rH Zparallel_unit_testsr� Zrun_tests_sequentiallyr, r� Zrunner�retZsequential_start_timeZsequential_tests_to_runr� r� r� r� � num_processesr) r� r- r- r. � RunCommand sd ��� � "" ��� �� � ��� ���� ��� zTestCommand.RunCommandN)r/ r0 r1 r2 r �CreateCommandSpec� _SYNOPSISr rf � command_spec�HelpSpec�_DETAILED_HELP_TEXT� help_specr� r� r� r� r� r- r- r- r. rv h s2 � � Zrv )N);r2 � __future__r r r r � collectionsr r� r_ rh r� rX r� r� r^ r� rO � six.movesr rd �gslib.cloud_apir � gslib.commandr r �gslib.exceptionr �gslib.project_idr Z gslib.testsrZ Zgslib.tests.utilr r r �gslib.utils.constantsr r Zgslib.utils.system_utilr r� r� �PY3r� r� r� r� rf r� r� r r5 r> rD rK rL ru rv r- r- r- r. �<module> sr �^��b�5 � N
SIMPAN
[ BATAL ]