model_doc = ModelCard()Model documentation
Functions to jumpstart and facilitate model documentation
Each user has a specific documentation need, ranging from simply logging the model training to a more complex description of the model pipeline with a discusson of the model outcomes. gingado addresses this variety of needs by offering a class of objects, “Documenters”, that facilitate model documentation. A base class facilitates the creation of generic ways to document models, and gingado includes two specific model documentation templates off-the-shelf as described below.
The model documentation is performed by Documenters, objects that subclass from the base class ggdModelDocumentation. This base class offers code that can be used by any Documenter to read the model in question, format the information according to a template and save the resulting documentation in a JSON format. Documenters save the underlying information using the JSON format. With the JSON documentation file at hand, the user can then deploy existing third-party libraries to transform the information stored in JSON into a variety of formats (eg, HTML, PDF) as needed.
One current area of development is the automatic filing of some fields related to the model. The objective is to automatise documentation of the information that can be fetched automatically from the model, leaving time for the analyst to concentrate on other tasks, such as considering the ethical implications of the machine learning model being trained.
Base class
gingado has a ggdModelDocumentation base class that contains the basic functionalities for Documenters. It is not meant to be used by itself, but only as a hyperclass for Documenters objects. gingado ships with two such objects that subclass ggdModelDocumentation: ModelCard and ForecastCard. They are both described below in their respective sections.
Users are encouraged to submit a PR with their own Documenter models subclassing ggdModelDocumentation; see Section 5 for more information.
ggdModelDocumentation
ggdModelDocumentation
()
Base class for gingado Documenters
setup_template
setup_template
(self)
Set up the template from the JSON documentation
show_template
show_template
(self, indent: 'bool' = True)
Show documentation template in JSON format.
Args:
indent (bool): Whether to print JSON documentation template with indentation for easier human reading.
documentation_path
documentation_path
(self)
Show path to documentation
save_json
save_json
(self, file_path: 'str')
Save the documentation in JSON format in the specified file.
Args:
file_path (str): Path to save JSON file.
read_json
read_json
(self, file_path: 'str')
Load documentation JSON from path.
Args:
file_path (str): Path to JSON file or path defined in `file_path` if None.
show_json
show_json
(self)
Show documentation in JSON format
read_model
read_model
(self, model)
Read automatically information from the model and add to documentation.
Args:
model: The model to be documented.
open_questions
open_questions
(self)
List open fields in the documentation
fill_model_info
fill_model_info
(self, model_info: 'str | dict', model_info_keyname: 'str' = 'model_details')
Called automatically, or by the user, to add model information to the documentation according to its template.
Args:
model_info (str | dict): Information about the model to be added in the documentation.
model_info_keyname (str): Dictionary key in the Documenter template to which this information should be linked.
fill_info
fill_info
(self, new_info: 'dict')
Include information in model documentation.
Args:
new_info (dict): Dictionary with information to be added to the model documentation.
Documenters
ModelCard
ModelCard - the model documentation template inspired by the work of Mitchell et al. (2018) already comes with gingado. Its template can be used by users as is, or tweaked according to each need. The ModelCard template can also serve as inspiration for any custom documentation needs. Users with documentation needs beyond the out-of-the-box solutions provided by gingado can create their own class of Documenters (more information on that below), and compatibility with these custom documentation routines with the rest of the code is ensured. Users are encouraged to submit a pull request with their own documentation models subclassing ggdModelDocumentation if these custom templates can also benefit other users.
Like all gingado Documenters, a ModelCard is can be easily created on a standalone basis as shown below, or as part of a gingado.ggdBenchmark object.
By default, it autofills the template with the current date and time. Users can add other information to be automatically added by a customised Documenter object.
model_doc_with_autofill = ModelCard(autofill=True)
model_doc_no_autofill = ModelCard(autofill=False)Below is a comparison of the model_details section of the model document, with and without the autofill.
model_doc_with_autofill.show_json()['model_details']{'developer': 'Person or organisation developing the model',
'datetime': '2024-02-27 08:47:49 ',
'version': 'Model version',
'type': 'Model type',
'info': 'Information about training algorithms, parameters, fairness constraints or other applied approaches, and features',
'paper': 'Paper or other resource for more information',
'citation': 'Citation details',
'license': 'License',
'contact': 'Where to send questions or comments about the model'}model_doc_no_autofill.show_json()['model_details']{'developer': 'Person or organisation developing the model',
'datetime': 'Model date',
'version': 'Model version',
'type': 'Model type',
'info': 'Information about training algorithms, parameters, fairness constraints or other applied approaches, and features',
'paper': 'Paper or other resource for more information',
'citation': 'Citation details',
'license': 'License',
'contact': 'Where to send questions or comments about the model'}ModelCard
ModelCard
(file_path: 'str' = '', autofill: 'bool' = True, indent_level: 'int | None' = 2)
A gingado Documenter based on @ModelCards
autofill_template
autofill_template
(self)
Create an empty model card template, then fills it with information that is automatically obtained from the system
ForecastCard
ForecastCard is a model documentation template inspired by Mitchell et al. (2018), but with fields that are more specifically targeted towards forecasting or nowcasting use cases.
Because a ForecastCard Documenter object is targeted to forecasting and nowcasting models, it contains some specialised fields, as illustrated below.
model_doc = ForecastCard()
model_doc.show_template(){
"model_details": {
"field_description": "Basic information about the model",
"variable": "Variable(s) being forecasted or nowcasted",
"jurisdiction": "Jurisdiction(s) of the variable being forecasted or nowcasted",
"developer": "Person or organisation developing the model",
"datetime": "Model date",
"version": "Model version",
"type": "Model type",
"pipeline": "Description of the pipeline steps being used",
"info": "Information about training algorithms, parameters, fairness constraints or other applied approaches, and features",
"econometric_model": "Information about the econometric model or technique",
"paper": "Paper or other resource for more information",
"citation": "Citation details",
"license": "License",
"contact": "Where to send questions or comments about the model"
},
"intended_use": {
"field_description": "Use cases that were envisioned during development",
"primary_uses": "Primary intended uses",
"primary_users": "Primary intended users",
"out_of_scope": "Out-of-scope use cases"
},
"factors": {
"field_description": "Factors could include demographic or phenotypic groups, environmental conditions, technical attributes, or others",
"relevant": "Relevant factors",
"evaluation": "Evaluation factors"
},
"metrics": {
"field_description": "Metrics should be chosen to reflect potential real world impacts of the model",
"performance_measures": "Model performance measures",
"estimation_approaches": "How are the evaluation metrics calculated? Include information on the cross-validation approach, if used"
},
"data": {
"field_description": "Details on the dataset(s) used for the training and evaluation of the model",
"datasets": "Datasets",
"preprocessing": "Preprocessing",
"cutoff_date": "Cut-off date that separates training from evaluation data"
},
"ethical_considerations": {
"field_description": "Ethical considerations that went into model development, surfacing ethical challenges and solutions to stakeholders. Ethical analysis does not always lead to precise solutions, but the process of ethical contemplation is worthwhile to inform on responsible practices and next steps in future work.",
"sensitive_data": "Does the model use any sensitive data (e.g., protected classes)?",
"risks_and_harms": "What risks may be present in model usage? Try to identify the potential recipients, likelihood, and magnitude of harms. If these cannot be determined, note that they were considered but remain unknown",
"use_cases": "Are there any known model use cases that are especially fraught?",
"additional_information": "If possible, this section should also include any additional ethical considerations that went into model development, for example, review by an external board, or testing with a specific community."
},
"caveats_recommendations": {
"field_description": "Additional concerns that were not covered in the previous sections",
"caveats": "For example, did the results suggest any further testing? Were there any relevant groups that were not represented in the evaluation dataset?",
"recommendations": "Are there additional recommendations for model use? What are the ideal characteristics of an evaluation dataset for this model?"
}
}model_doc.show_json(){'model_details': {'variable': 'Variable(s) being forecasted or nowcasted',
'jurisdiction': 'Jurisdiction(s) of the variable being forecasted or nowcasted',
'developer': 'Person or organisation developing the model',
'datetime': '2024-02-27 08:47:50 ',
'version': 'Model version',
'type': 'Model type',
'pipeline': 'Description of the pipeline steps being used',
'info': 'Information about training algorithms, parameters, fairness constraints or other applied approaches, and features',
'econometric_model': 'Information about the econometric model or technique',
'paper': 'Paper or other resource for more information',
'citation': 'Citation details',
'license': 'License',
'contact': 'Where to send questions or comments about the model'},
'intended_use': {'primary_uses': 'Primary intended uses',
'primary_users': 'Primary intended users',
'out_of_scope': 'Out-of-scope use cases'},
'factors': {'relevant': 'Relevant factors',
'evaluation': 'Evaluation factors'},
'metrics': {'performance_measures': 'Model performance measures',
'estimation_approaches': 'How are the evaluation metrics calculated? Include information on the cross-validation approach, if used'},
'data': {'datasets': 'Datasets',
'preprocessing': 'Preprocessing',
'cutoff_date': 'Cut-off date that separates training from evaluation data'},
'ethical_considerations': {'sensitive_data': 'Does the model use any sensitive data (e.g., protected classes)?',
'risks_and_harms': 'What risks may be present in model usage? Try to identify the potential recipients, likelihood, and magnitude of harms. If these cannot be determined, note that they were considered but remain unknown',
'use_cases': 'Are there any known model use cases that are especially fraught?',
'additional_information': 'If possible, this section should also include any additional ethical considerations that went into model development, for example, review by an external board, or testing with a specific community.'},
'caveats_recommendations': {'caveats': 'For example, did the results suggest any further testing? Were there any relevant groups that were not represented in the evaluation dataset?',
'recommendations': 'Are there additional recommendations for model use? What are the ideal characteristics of an evaluation dataset for this model?'}}ForecastCard
ForecastCard
(file_path: 'str' = '', autofill: 'bool' = True, indent_level: 'int | None' = 2)
A gingado Documenter for forecasting or nowcasting use cases
autofill_template
autofill_template
(self)
Create an empty model card template, then fills it with information that is automatically obtained from the system
Basic functioning of model documentation
After a Documenter object, such as ModelCard or ForecastCard is instanciated, the user can see the underlying template with the module show_template, as below:
model_doc = ModelCard(autofill=False)
assert model_doc.show_template(indent=False) == ModelCard.template
model_doc.show_template(){
"model_details": {
"field_description": "Basic information about the model",
"developer": "Person or organisation developing the model",
"datetime": "Model date",
"version": "Model version",
"type": "Model type",
"info": "Information about training algorithms, parameters, fairness constraints or other applied approaches, and features",
"paper": "Paper or other resource for more information",
"citation": "Citation details",
"license": "License",
"contact": "Where to send questions or comments about the model"
},
"intended_use": {
"field_description": "Use cases that were envisioned during development",
"primary_uses": "Primary intended uses",
"primary_users": "Primary intended users",
"out_of_scope": "Out-of-scope use cases"
},
"factors": {
"field_description": "Factors could include demographic or phenotypic groups, environmental conditions, technical attributes, or others",
"relevant": "Relevant factors",
"evaluation": "Evaluation factors"
},
"metrics": {
"field_description": "Metrics should be chosen to reflect potential real world impacts of the model",
"performance_measures": "Model performance measures",
"thresholds": "Decision thresholds",
"variation_approaches": "Variation approaches"
},
"evaluation_data": {
"field_description": "Details on the dataset(s) used for the quantitative analyses in the documentation",
"datasets": "Datasets",
"motivation": "Motivation",
"preprocessing": "Preprocessing"
},
"training_data": {
"field_description": "May not be possible to provide in practice. When possible, this section should mirror 'Evaluation Data'. If such detail is not possible, minimal allowable information should be provided here, such as details of the distribution over various factors in the training datasets.",
"training_data": "Information on training data"
},
"quant_analyses": {
"field_description": "Quantitative Analyses",
"unitary": "Unitary results",
"intersectional": "Intersectional results"
},
"ethical_considerations": {
"field_description": "Ethical considerations that went into model development, surfacing ethical challenges and solutions to stakeholders. Ethical analysis does not always lead to precise solutions, but the process of ethical contemplation is worthwhile to inform on responsible practices and next steps in future work.",
"sensitive_data": "Does the model use any sensitive data (e.g., protected classes)?",
"human_life": "Is the model intended to inform decisions about matters central to human life or flourishing - e.g., health or safety? Or could it be used in such a way?",
"mitigations": "What risk mitigation strategies were used during model development?",
"risks_and_harms": "What risks may be present in model usage? Try to identify the potential recipients,likelihood, and magnitude of harms. If these cannot be determined, note that they were considered but remain unknown",
"use_cases": "Are there any known model use cases that are especially fraught?",
"additional_information": "If possible, this section should also include any additional ethical considerations that went into model development, for example, review by an external board, or testing with a specific community."
},
"caveats_recommendations": {
"field_description": "Additional concerns that were not covered in the previous sections",
"caveats": "For example, did the results suggest any further testing? Were there any relevant groups that were not represented in the evaluation dataset?",
"recommendations": "Are there additional recommendations for model use? What are the ideal characteristics of an evaluation dataset for this model?"
}
}The method show_json prints the Documenter’s documentation template, where the unfilled information retains the descriptions from the original template:
model_doc = ModelCard(autofill=True)
model_doc.show_json(){'model_details': {'developer': 'Person or organisation developing the model',
'datetime': '2024-02-27 08:47:51 ',
'version': 'Model version',
'type': 'Model type',
'info': 'Information about training algorithms, parameters, fairness constraints or other applied approaches, and features',
'paper': 'Paper or other resource for more information',
'citation': 'Citation details',
'license': 'License',
'contact': 'Where to send questions or comments about the model'},
'intended_use': {'primary_uses': 'Primary intended uses',
'primary_users': 'Primary intended users',
'out_of_scope': 'Out-of-scope use cases'},
'factors': {'relevant': 'Relevant factors',
'evaluation': 'Evaluation factors'},
'metrics': {'performance_measures': 'Model performance measures',
'thresholds': 'Decision thresholds',
'variation_approaches': 'Variation approaches'},
'evaluation_data': {'datasets': 'Datasets',
'motivation': 'Motivation',
'preprocessing': 'Preprocessing'},
'training_data': {'training_data': 'Information on training data'},
'quant_analyses': {'unitary': 'Unitary results',
'intersectional': 'Intersectional results'},
'ethical_considerations': {'sensitive_data': 'Does the model use any sensitive data (e.g., protected classes)?',
'human_life': 'Is the model intended to inform decisions about matters central to human life or flourishing - e.g., health or safety? Or could it be used in such a way?',
'mitigations': 'What risk mitigation strategies were used during model development?',
'risks_and_harms': 'What risks may be present in model usage? Try to identify the potential recipients,likelihood, and magnitude of harms. If these cannot be determined, note that they were considered but remain unknown',
'use_cases': 'Are there any known model use cases that are especially fraught?',
'additional_information': 'If possible, this section should also include any additional ethical considerations that went into model development, for example, review by an external board, or testing with a specific community.'},
'caveats_recommendations': {'caveats': 'For example, did the results suggest any further testing? Were there any relevant groups that were not represented in the evaluation dataset?',
'recommendations': 'Are there additional recommendations for model use? What are the ideal characteristics of an evaluation dataset for this model?'}}The template is protected from editing once a Documenter has been created. This way, even if a user unwarrantedly changes the template, this does not interfere with the Documenter functionality.
model_doc.template = None
model_doc.show_template()
assert model_doc.show_template(indent=False) == ModelCard.template{
"model_details": {
"field_description": "Basic information about the model",
"developer": "Person or organisation developing the model",
"datetime": "Model date",
"version": "Model version",
"type": "Model type",
"info": "Information about training algorithms, parameters, fairness constraints or other applied approaches, and features",
"paper": "Paper or other resource for more information",
"citation": "Citation details",
"license": "License",
"contact": "Where to send questions or comments about the model"
},
"intended_use": {
"field_description": "Use cases that were envisioned during development",
"primary_uses": "Primary intended uses",
"primary_users": "Primary intended users",
"out_of_scope": "Out-of-scope use cases"
},
"factors": {
"field_description": "Factors could include demographic or phenotypic groups, environmental conditions, technical attributes, or others",
"relevant": "Relevant factors",
"evaluation": "Evaluation factors"
},
"metrics": {
"field_description": "Metrics should be chosen to reflect potential real world impacts of the model",
"performance_measures": "Model performance measures",
"thresholds": "Decision thresholds",
"variation_approaches": "Variation approaches"
},
"evaluation_data": {
"field_description": "Details on the dataset(s) used for the quantitative analyses in the documentation",
"datasets": "Datasets",
"motivation": "Motivation",
"preprocessing": "Preprocessing"
},
"training_data": {
"field_description": "May not be possible to provide in practice. When possible, this section should mirror 'Evaluation Data'. If such detail is not possible, minimal allowable information should be provided here, such as details of the distribution over various factors in the training datasets.",
"training_data": "Information on training data"
},
"quant_analyses": {
"field_description": "Quantitative Analyses",
"unitary": "Unitary results",
"intersectional": "Intersectional results"
},
"ethical_considerations": {
"field_description": "Ethical considerations that went into model development, surfacing ethical challenges and solutions to stakeholders. Ethical analysis does not always lead to precise solutions, but the process of ethical contemplation is worthwhile to inform on responsible practices and next steps in future work.",
"sensitive_data": "Does the model use any sensitive data (e.g., protected classes)?",
"human_life": "Is the model intended to inform decisions about matters central to human life or flourishing - e.g., health or safety? Or could it be used in such a way?",
"mitigations": "What risk mitigation strategies were used during model development?",
"risks_and_harms": "What risks may be present in model usage? Try to identify the potential recipients,likelihood, and magnitude of harms. If these cannot be determined, note that they were considered but remain unknown",
"use_cases": "Are there any known model use cases that are especially fraught?",
"additional_information": "If possible, this section should also include any additional ethical considerations that went into model development, for example, review by an external board, or testing with a specific community."
},
"caveats_recommendations": {
"field_description": "Additional concerns that were not covered in the previous sections",
"caveats": "For example, did the results suggest any further testing? Were there any relevant groups that were not represented in the evaluation dataset?",
"recommendations": "Are there additional recommendations for model use? What are the ideal characteristics of an evaluation dataset for this model?"
}
}Users can find which fields in their templates are still open by using the module open_questions. The levels of the template are reflected in the resulting dictionary, with double underscores separating the different dictionary levels in the underlying template.
Below we see that after inputting information for the item caveats in the section caveats_recommendations, this item does not appear in the results of the open_questions method.
model_doc.fill_info({'caveats_recommendations': {'caveats': 'This is another test'}})
assert model_doc.json_doc['caveats_recommendations']['caveats'] == "This is another test"
# note that caveats_recommendations__caveats is no longer considered an open question
# after being filled in through `fill_info`.
print([oq for oq in model_doc.open_questions() if oq.startswith('caveats')])['caveats_recommendations__recommendations']And now the complete result of the open_questions method:
model_doc.open_questions()['model_details__developer',
'model_details__version',
'model_details__type',
'model_details__info',
'model_details__paper',
'model_details__citation',
'model_details__license',
'model_details__contact',
'intended_use__primary_uses',
'intended_use__primary_users',
'intended_use__out_of_scope',
'factors__relevant',
'factors__evaluation',
'metrics__performance_measures',
'metrics__thresholds',
'metrics__variation_approaches',
'evaluation_data__datasets',
'evaluation_data__motivation',
'evaluation_data__preprocessing',
'training_data__training_data',
'quant_analyses__unitary',
'quant_analyses__intersectional',
'ethical_considerations__sensitive_data',
'ethical_considerations__human_life',
'ethical_considerations__mitigations',
'ethical_considerations__risks_and_harms',
'ethical_considerations__use_cases',
'ethical_considerations__additional_information',
'caveats_recommendations__recommendations']If the user wants to fill in an empty field such as the ones identified above by the method open_questions, the user simply needs to pass to the module fill_info a dictionary with the corresponding information. Depending on the template, the dictionary may be nested.
Note
it is technically possible to attribute the element directly to the attribute json_doc, but this should be avoided in favour of using the method fill_info. The latter tests whether the new information is valid according to the documentation template and also enables filling of more than one question at the same time. In addition, attributing information directly to json_doc is not logged, and may unwarrantedly create new entries that are not part of the template (eg, if a new dictionary key is created due to typos).
The template serves to provide specific instances of the Documenter object with a form-like structure, indicating which fields are open and thus require some answers or information. Consequently, the template does not change when the actual document object changes after information is added by fill_info.
new_info = {
'metrics': {'performance_measures': "This is a test"},
'caveats_recommendations': {'caveats': "This is another test"}
}
model_doc.fill_info(new_info)
print([model_doc.json_doc['metrics'], ModelCard.template['metrics']])
assert model_doc.show_template(indent=False) == ModelCard.template[{'performance_measures': 'This is a test', 'thresholds': 'Decision thresholds', 'variation_approaches': 'Variation approaches'}, {'field_description': 'Metrics should be chosen to reflect potential real world impacts of the model', 'performance_measures': 'Model performance measures', 'thresholds': 'Decision thresholds', 'variation_approaches': 'Variation approaches'}]Reading information from models
gingado’s ggdModelDocumentation base class is able to extract information from machine learning models from a number of widely used libraries and make it available to the Documenter objects. This is done through the method read_model, which recognises whether the model is a gingado object or any of scikit-learn, keras, or fastai models and read the model characteristics appropriately. For filing out information from other models (eg, pytorch or even models coded from scratch, machine learning or not), the user can benefit from the module fill_model_info that every Documenter should have, as demonstrated below.
In the case of ModelCard, these informations are included under model_details, item info. But the model information could be saved in another area of a custom Documenter.
Note
the model-specific information saved is different depending on the model’s original library.
Preliminaries
The mock dataset below is used to construct models using different libraries, to demonstrate how they are read by Documenters.
from sklearn.datasets import make_classification# some mock up data
X, y = make_classification()
X.shape, y.shape((100, 20), (100,))gingado Benchmark
from gingado.benchmark import ClassificationBenchmark# the gingado benchmark
gingado_clf = ClassificationBenchmark(verbose_grid=1).fit(X, y)Fitting 10 folds for each of 6 candidates, totalling 60 fits# a new instance of ModelCard is created and used to document the model
model_doc_gingado = ModelCard()
model_doc_gingado.read_model(gingado_clf.benchmark)
print(model_doc_gingado.show_json()['model_details']['info'])
# but given that gingado Benchmark objects already document the best model at every fit, we can check that they are equal:
assert model_doc_gingado.show_json()['model_details']['info'] == gingado_clf.model_documentation.show_json()['model_details']['info']{'_estimator_type': 'classifier', 'best_estimator_': RandomForestClassifier(max_features=None, n_estimators=250, oob_score=True), 'best_index_': 5, 'best_params_': {'max_features': None, 'n_estimators': 250}, 'best_score_': 0.93, 'classes_': array([0, 1]), 'cv_results_': {'mean_fit_time': array([0.17820907, 0.48922832, 0.19631364, 0.42252822, 0.24131639,
0.57173564]), 'std_fit_time': array([0.01348286, 0.03802951, 0.02532672, 0.01887258, 0.02557295,
0.06991078]), 'mean_score_time': array([0.0051146 , 0.01220322, 0.00839796, 0.01019723, 0.00599849,
0.01011045]), 'std_score_time': array([0.00056082, 0.00302682, 0.00915481, 0.00059421, 0.00319521,
0.00068275]), 'param_max_features': masked_array(data=['sqrt', 'sqrt', 'log2', 'log2', None, None],
mask=[False, False, False, False, False, False],
fill_value='?',
dtype=object), 'param_n_estimators': masked_array(data=[100, 250, 100, 250, 100, 250],
mask=[False, False, False, False, False, False],
fill_value='?',
dtype=object), 'params': [{'max_features': 'sqrt', 'n_estimators': 100}, {'max_features': 'sqrt', 'n_estimators': 250}, {'max_features': 'log2', 'n_estimators': 100}, {'max_features': 'log2', 'n_estimators': 250}, {'max_features': None, 'n_estimators': 100}, {'max_features': None, 'n_estimators': 250}], 'split0_test_score': array([0.8, 0.8, 0.9, 0.9, 0.9, 0.9]), 'split1_test_score': array([0.9, 0.9, 0.9, 0.9, 0.9, 1. ]), 'split2_test_score': array([0.8, 0.8, 0.9, 0.8, 0.9, 0.9]), 'split3_test_score': array([1. , 0.9, 1. , 1. , 1. , 1. ]), 'split4_test_score': array([0.9, 1. , 0.9, 1. , 1. , 1. ]), 'split5_test_score': array([0.8, 0.8, 0.9, 0.8, 0.8, 0.8]), 'split6_test_score': array([1., 1., 1., 1., 1., 1.]), 'split7_test_score': array([0.8, 0.8, 0.8, 0.7, 0.8, 0.8]), 'split8_test_score': array([1. , 1. , 1. , 1. , 0.9, 1. ]), 'split9_test_score': array([1. , 0.9, 0.9, 0.9, 0.9, 0.9]), 'mean_test_score': array([0.9 , 0.89, 0.92, 0.9 , 0.91, 0.93]), 'std_test_score': array([0.08944272, 0.08306624, 0.06 , 0.1 , 0.07 ,
0.0781025 ]), 'rank_test_score': array([4, 6, 2, 4, 3, 1])}, 'multimetric_': False, 'n_features_in_': 20, 'n_splits_': 10, 'refit_time_': 0.6470375061035156, 'scorer_': <sklearn.metrics._scorer._PassthroughScorer object at 0x000001D1015FB190>}scikit-learn
from sklearn.ensemble import RandomForestClassifiersklearn_clf = RandomForestClassifier().fit(X, y)model_doc_sklearn = ModelCard()
model_doc_sklearn.read_model(sklearn_clf)
print(model_doc_sklearn.show_json()['model_details']['info']){'_estimator_type': 'classifier', 'classes_': array([0, 1]), 'estimator_': DecisionTreeClassifier(), 'estimators_': [DecisionTreeClassifier(max_features='sqrt', random_state=984336080), DecisionTreeClassifier(max_features='sqrt', random_state=387935964), DecisionTreeClassifier(max_features='sqrt', random_state=1119873145), DecisionTreeClassifier(max_features='sqrt', random_state=1050144277), DecisionTreeClassifier(max_features='sqrt', random_state=941639386), DecisionTreeClassifier(max_features='sqrt', random_state=844869905), DecisionTreeClassifier(max_features='sqrt', random_state=1084376571), DecisionTreeClassifier(max_features='sqrt', random_state=838680530), DecisionTreeClassifier(max_features='sqrt', random_state=2131844637), DecisionTreeClassifier(max_features='sqrt', random_state=281664374), DecisionTreeClassifier(max_features='sqrt', random_state=387195694), DecisionTreeClassifier(max_features='sqrt', random_state=1900725751), DecisionTreeClassifier(max_features='sqrt', random_state=1398287329), DecisionTreeClassifier(max_features='sqrt', random_state=107007245), DecisionTreeClassifier(max_features='sqrt', random_state=851863750), DecisionTreeClassifier(max_features='sqrt', random_state=1383879282), DecisionTreeClassifier(max_features='sqrt', random_state=217952844), DecisionTreeClassifier(max_features='sqrt', random_state=320258304), DecisionTreeClassifier(max_features='sqrt', random_state=653524562), DecisionTreeClassifier(max_features='sqrt', random_state=547335341), DecisionTreeClassifier(max_features='sqrt', random_state=65298601), DecisionTreeClassifier(max_features='sqrt', random_state=90961819), DecisionTreeClassifier(max_features='sqrt', random_state=1684018311), DecisionTreeClassifier(max_features='sqrt', random_state=655454563), DecisionTreeClassifier(max_features='sqrt', random_state=1786516343), DecisionTreeClassifier(max_features='sqrt', random_state=994298205), DecisionTreeClassifier(max_features='sqrt', random_state=1293597554), DecisionTreeClassifier(max_features='sqrt', random_state=1133785409), DecisionTreeClassifier(max_features='sqrt', random_state=847212468), DecisionTreeClassifier(max_features='sqrt', random_state=1194070920), DecisionTreeClassifier(max_features='sqrt', random_state=1506960310), DecisionTreeClassifier(max_features='sqrt', random_state=705803780), DecisionTreeClassifier(max_features='sqrt', random_state=1531279072), DecisionTreeClassifier(max_features='sqrt', random_state=1780828084), DecisionTreeClassifier(max_features='sqrt', random_state=815253514), DecisionTreeClassifier(max_features='sqrt', random_state=935926288), DecisionTreeClassifier(max_features='sqrt', random_state=19607747), DecisionTreeClassifier(max_features='sqrt', random_state=2111856427), DecisionTreeClassifier(max_features='sqrt', random_state=784991697), DecisionTreeClassifier(max_features='sqrt', random_state=1241164616), DecisionTreeClassifier(max_features='sqrt', random_state=1387079999), DecisionTreeClassifier(max_features='sqrt', random_state=168074365), DecisionTreeClassifier(max_features='sqrt', random_state=1788266150), DecisionTreeClassifier(max_features='sqrt', random_state=1189226461), DecisionTreeClassifier(max_features='sqrt', random_state=991777677), DecisionTreeClassifier(max_features='sqrt', random_state=242884733), DecisionTreeClassifier(max_features='sqrt', random_state=157311037), DecisionTreeClassifier(max_features='sqrt', random_state=2055890918), DecisionTreeClassifier(max_features='sqrt', random_state=1727577809), DecisionTreeClassifier(max_features='sqrt', random_state=269422783), DecisionTreeClassifier(max_features='sqrt', random_state=1389820690), DecisionTreeClassifier(max_features='sqrt', random_state=1929534429), DecisionTreeClassifier(max_features='sqrt', random_state=1414737041), DecisionTreeClassifier(max_features='sqrt', random_state=1247576130), DecisionTreeClassifier(max_features='sqrt', random_state=1353547520), DecisionTreeClassifier(max_features='sqrt', random_state=894390039), DecisionTreeClassifier(max_features='sqrt', random_state=158653442), DecisionTreeClassifier(max_features='sqrt', random_state=243266945), DecisionTreeClassifier(max_features='sqrt', random_state=455577999), DecisionTreeClassifier(max_features='sqrt', random_state=1371394730), DecisionTreeClassifier(max_features='sqrt', random_state=1451808483), DecisionTreeClassifier(max_features='sqrt', random_state=1589801337), DecisionTreeClassifier(max_features='sqrt', random_state=1380089697), DecisionTreeClassifier(max_features='sqrt', random_state=1541728708), DecisionTreeClassifier(max_features='sqrt', random_state=1730746642), DecisionTreeClassifier(max_features='sqrt', random_state=1569100702), DecisionTreeClassifier(max_features='sqrt', random_state=87894264), DecisionTreeClassifier(max_features='sqrt', random_state=182251102), DecisionTreeClassifier(max_features='sqrt', random_state=1598824537), DecisionTreeClassifier(max_features='sqrt', random_state=1566880189), DecisionTreeClassifier(max_features='sqrt', random_state=2036645957), DecisionTreeClassifier(max_features='sqrt', random_state=69788070), DecisionTreeClassifier(max_features='sqrt', random_state=725498370), DecisionTreeClassifier(max_features='sqrt', random_state=886772744), DecisionTreeClassifier(max_features='sqrt', random_state=1619512775), DecisionTreeClassifier(max_features='sqrt', random_state=847881750), DecisionTreeClassifier(max_features='sqrt', random_state=505640702), DecisionTreeClassifier(max_features='sqrt', random_state=338379070), DecisionTreeClassifier(max_features='sqrt', random_state=1789388801), DecisionTreeClassifier(max_features='sqrt', random_state=184042237), DecisionTreeClassifier(max_features='sqrt', random_state=796977047), DecisionTreeClassifier(max_features='sqrt', random_state=1147247523), DecisionTreeClassifier(max_features='sqrt', random_state=2106557009), DecisionTreeClassifier(max_features='sqrt', random_state=1638844864), DecisionTreeClassifier(max_features='sqrt', random_state=1373057286), DecisionTreeClassifier(max_features='sqrt', random_state=1356063932), DecisionTreeClassifier(max_features='sqrt', random_state=1318119814), DecisionTreeClassifier(max_features='sqrt', random_state=1196983038), DecisionTreeClassifier(max_features='sqrt', random_state=132896044), DecisionTreeClassifier(max_features='sqrt', random_state=152120878), DecisionTreeClassifier(max_features='sqrt', random_state=2100501339), DecisionTreeClassifier(max_features='sqrt', random_state=197690499), DecisionTreeClassifier(max_features='sqrt', random_state=1765855662), DecisionTreeClassifier(max_features='sqrt', random_state=1579978128), DecisionTreeClassifier(max_features='sqrt', random_state=1581716817), DecisionTreeClassifier(max_features='sqrt', random_state=1439289735), DecisionTreeClassifier(max_features='sqrt', random_state=1808084503), DecisionTreeClassifier(max_features='sqrt', random_state=1398215665), DecisionTreeClassifier(max_features='sqrt', random_state=2058382938), DecisionTreeClassifier(max_features='sqrt', random_state=304859939)], 'estimators_samples_': [array([73, 68, 87, 11, 98, 49, 84, 67, 6, 53, 16, 59, 41, 13, 23, 79, 30,
1, 62, 40, 36, 25, 46, 66, 30, 71, 26, 72, 22, 69, 68, 10, 60, 72,
96, 59, 77, 67, 77, 89, 59, 81, 49, 84, 11, 44, 40, 29, 46, 33, 43,
78, 48, 82, 77, 60, 67, 24, 59, 3, 33, 28, 62, 36, 0, 16, 88, 85,
44, 88, 9, 67, 83, 27, 41, 41, 24, 0, 35, 59, 82, 99, 66, 72, 2,
27, 61, 54, 61, 55, 3, 97, 38, 12, 8, 48, 99, 35, 61, 8]), array([58, 87, 7, 15, 26, 50, 50, 23, 37, 50, 13, 53, 61, 56, 76, 42, 52,
64, 32, 1, 50, 9, 94, 56, 44, 93, 6, 47, 61, 87, 33, 50, 37, 5,
58, 68, 27, 33, 42, 86, 71, 11, 14, 60, 78, 11, 86, 97, 27, 95, 38,
85, 66, 43, 85, 58, 10, 66, 62, 17, 76, 42, 43, 8, 85, 8, 97, 41,
46, 68, 28, 71, 18, 21, 33, 66, 92, 24, 65, 91, 6, 21, 39, 99, 78,
68, 57, 28, 3, 56, 82, 82, 68, 21, 85, 39, 37, 19, 18, 4]), array([98, 52, 38, 15, 45, 45, 59, 63, 63, 5, 0, 95, 88, 22, 34, 9, 79,
49, 15, 8, 80, 44, 45, 20, 16, 92, 51, 94, 76, 20, 42, 65, 98, 52,
19, 19, 38, 75, 50, 24, 7, 34, 91, 5, 18, 15, 95, 60, 90, 94, 42,
16, 54, 44, 28, 80, 85, 3, 17, 3, 12, 29, 20, 29, 28, 87, 10, 21,
98, 17, 78, 90, 59, 60, 20, 0, 95, 13, 16, 3, 93, 14, 26, 38, 17,
58, 37, 81, 47, 92, 6, 32, 55, 1, 76, 3, 81, 33, 53, 63]), array([21, 55, 28, 35, 51, 79, 78, 70, 19, 66, 19, 20, 67, 21, 96, 18, 20,
79, 51, 87, 69, 54, 99, 3, 99, 35, 94, 42, 97, 37, 64, 91, 64, 75,
73, 2, 53, 10, 59, 97, 42, 21, 84, 61, 65, 29, 2, 6, 26, 46, 37,
66, 78, 17, 17, 83, 63, 27, 64, 53, 25, 10, 18, 97, 84, 7, 95, 59,
54, 55, 33, 56, 5, 5, 65, 1, 1, 91, 67, 99, 22, 14, 13, 14, 88,
84, 8, 74, 39, 26, 33, 88, 81, 56, 61, 49, 76, 40, 14, 73]), array([68, 98, 66, 11, 17, 79, 81, 54, 46, 92, 82, 9, 91, 98, 61, 77, 16,
23, 68, 58, 36, 98, 43, 74, 51, 69, 78, 41, 56, 57, 70, 84, 39, 52,
93, 0, 96, 9, 55, 54, 68, 9, 34, 94, 28, 13, 63, 28, 58, 18, 46,
56, 42, 74, 45, 77, 43, 68, 35, 55, 9, 5, 3, 49, 59, 27, 6, 42,
49, 23, 61, 13, 50, 19, 41, 30, 28, 23, 27, 20, 63, 57, 68, 82, 4,
72, 66, 70, 86, 95, 60, 78, 9, 37, 65, 13, 88, 0, 20, 73]), array([56, 30, 68, 59, 97, 18, 96, 63, 78, 42, 32, 51, 16, 51, 92, 85, 74,
26, 42, 43, 65, 34, 55, 40, 97, 48, 87, 38, 93, 79, 11, 94, 84, 11,
58, 10, 90, 79, 62, 3, 46, 22, 0, 53, 54, 45, 75, 71, 45, 15, 95,
13, 84, 58, 35, 48, 12, 48, 99, 71, 44, 98, 86, 28, 89, 51, 4, 0,
68, 28, 66, 81, 93, 64, 37, 50, 48, 90, 68, 77, 56, 73, 87, 60, 57,
88, 36, 87, 28, 71, 50, 25, 75, 27, 54, 5, 76, 6, 0, 70]), array([29, 87, 45, 48, 6, 96, 86, 96, 0, 83, 18, 61, 7, 70, 37, 29, 33,
50, 45, 85, 4, 57, 67, 19, 57, 84, 8, 58, 36, 11, 5, 60, 88, 75,
65, 79, 15, 67, 87, 34, 11, 83, 66, 55, 99, 7, 17, 53, 68, 23, 64,
90, 41, 43, 88, 71, 8, 30, 94, 74, 92, 48, 97, 76, 1, 27, 47, 43,
4, 41, 27, 74, 5, 4, 50, 43, 80, 75, 72, 30, 70, 74, 53, 64, 69,
40, 15, 81, 65, 61, 68, 8, 54, 43, 0, 63, 54, 74, 71, 93]), array([32, 45, 35, 17, 32, 88, 7, 56, 31, 10, 75, 58, 41, 29, 32, 10, 32,
22, 99, 17, 74, 55, 1, 3, 95, 82, 7, 79, 86, 90, 80, 33, 89, 30,
33, 60, 33, 20, 24, 54, 5, 3, 97, 77, 36, 53, 54, 22, 46, 88, 1,
98, 50, 54, 4, 58, 77, 57, 15, 38, 19, 59, 91, 90, 8, 68, 89, 20,
84, 77, 79, 42, 35, 72, 93, 5, 33, 52, 50, 63, 92, 73, 7, 72, 45,
76, 58, 78, 50, 31, 16, 8, 20, 73, 48, 4, 76, 61, 19, 41]), array([32, 21, 68, 10, 21, 53, 29, 64, 76, 40, 57, 36, 67, 44, 28, 96, 67,
37, 85, 17, 60, 78, 74, 96, 28, 70, 60, 75, 61, 71, 41, 14, 40, 61,
96, 67, 58, 19, 81, 14, 90, 52, 26, 47, 10, 77, 82, 0, 29, 27, 8,
87, 23, 65, 65, 66, 32, 97, 7, 65, 59, 91, 84, 11, 32, 53, 48, 93,
76, 94, 55, 68, 56, 98, 61, 89, 85, 86, 73, 71, 81, 0, 93, 53, 68,
83, 31, 9, 42, 63, 40, 22, 53, 18, 87, 56, 90, 46, 87, 81]), array([85, 15, 91, 50, 60, 63, 66, 80, 80, 94, 62, 39, 90, 86, 85, 62, 28,
12, 42, 78, 21, 32, 99, 92, 72, 24, 56, 93, 43, 61, 6, 23, 34, 28,
29, 13, 49, 96, 22, 65, 73, 81, 87, 46, 37, 21, 31, 55, 42, 20, 44,
84, 13, 52, 56, 7, 68, 53, 65, 40, 95, 40, 8, 28, 5, 50, 53, 71,
84, 51, 79, 74, 14, 57, 58, 81, 38, 60, 27, 32, 30, 35, 29, 70, 11,
96, 28, 46, 22, 14, 83, 14, 37, 73, 30, 70, 4, 54, 15, 43]), array([23, 56, 70, 4, 69, 20, 43, 4, 16, 16, 60, 70, 82, 78, 37, 68, 29,
47, 7, 25, 59, 16, 99, 50, 52, 90, 52, 3, 11, 71, 96, 14, 94, 48,
73, 49, 43, 70, 65, 99, 8, 66, 37, 61, 91, 32, 35, 74, 14, 82, 79,
68, 18, 75, 30, 83, 77, 10, 57, 74, 48, 53, 6, 65, 62, 57, 39, 46,
59, 76, 86, 95, 79, 4, 51, 67, 14, 40, 31, 75, 44, 61, 83, 9, 14,
94, 57, 44, 6, 74, 95, 3, 9, 12, 53, 87, 6, 23, 96, 27]), array([60, 0, 43, 16, 83, 67, 14, 51, 66, 89, 19, 69, 12, 33, 30, 45, 4,
55, 99, 65, 29, 82, 17, 9, 47, 54, 26, 94, 28, 12, 12, 76, 53, 10,
31, 63, 83, 47, 55, 34, 86, 91, 18, 72, 87, 4, 26, 87, 1, 97, 40,
98, 75, 71, 38, 53, 84, 61, 66, 5, 25, 67, 55, 39, 98, 11, 71, 89,
36, 64, 79, 80, 42, 37, 51, 64, 67, 97, 85, 61, 9, 84, 82, 60, 46,
4, 70, 51, 39, 90, 66, 90, 5, 66, 28, 59, 34, 79, 91, 95]), array([29, 0, 8, 93, 93, 85, 60, 63, 60, 94, 9, 28, 93, 14, 52, 96, 24,
28, 14, 86, 7, 95, 1, 48, 0, 58, 45, 82, 59, 37, 1, 70, 3, 17,
2, 88, 56, 68, 60, 59, 7, 28, 60, 49, 59, 78, 60, 32, 99, 50, 97,
93, 55, 90, 65, 30, 88, 11, 34, 76, 4, 85, 50, 77, 47, 15, 98, 28,
26, 97, 55, 84, 53, 88, 19, 28, 12, 44, 68, 99, 11, 94, 76, 18, 52,
76, 52, 56, 39, 74, 16, 38, 52, 1, 12, 59, 5, 16, 86, 74]), array([54, 40, 31, 33, 71, 43, 46, 71, 69, 23, 29, 77, 34, 18, 47, 62, 73,
97, 84, 84, 41, 32, 89, 2, 94, 20, 0, 65, 65, 8, 26, 95, 75, 77,
35, 92, 25, 23, 62, 44, 59, 18, 3, 32, 65, 43, 24, 19, 26, 0, 31,
63, 20, 66, 93, 24, 56, 13, 95, 70, 22, 96, 55, 41, 34, 30, 93, 92,
66, 93, 60, 39, 76, 36, 68, 93, 0, 15, 19, 96, 93, 65, 86, 69, 78,
39, 19, 76, 75, 96, 55, 41, 99, 26, 5, 69, 34, 6, 91, 30]), array([16, 2, 21, 98, 71, 0, 50, 62, 14, 36, 72, 56, 12, 52, 60, 26, 17,
51, 81, 33, 76, 20, 60, 45, 2, 72, 66, 10, 54, 16, 77, 7, 50, 3,
9, 68, 91, 34, 24, 38, 84, 10, 89, 14, 68, 49, 81, 64, 17, 72, 8,
35, 84, 80, 99, 57, 67, 30, 16, 73, 22, 71, 64, 4, 22, 56, 6, 38,
36, 70, 89, 81, 88, 88, 99, 91, 94, 60, 10, 82, 79, 94, 33, 34, 12,
52, 52, 64, 40, 73, 85, 85, 19, 26, 42, 69, 86, 0, 67, 4]), array([67, 92, 11, 45, 50, 97, 30, 11, 96, 96, 46, 0, 78, 43, 40, 16, 80,
73, 46, 4, 95, 40, 60, 17, 17, 6, 10, 96, 24, 7, 99, 71, 52, 35,
30, 82, 97, 49, 21, 58, 12, 77, 76, 98, 79, 20, 6, 2, 27, 25, 82,
24, 9, 19, 70, 82, 27, 61, 78, 23, 76, 44, 7, 23, 21, 85, 50, 92,
76, 40, 0, 73, 55, 51, 47, 65, 15, 64, 14, 29, 84, 82, 91, 97, 84,
78, 63, 34, 45, 97, 70, 10, 13, 69, 37, 49, 70, 48, 7, 5]), array([71, 80, 94, 34, 87, 33, 12, 88, 61, 92, 74, 88, 38, 11, 20, 17, 14,
2, 69, 43, 38, 24, 42, 81, 17, 75, 23, 49, 15, 46, 89, 70, 7, 3,
44, 28, 49, 35, 78, 22, 60, 30, 57, 97, 10, 60, 10, 65, 52, 14, 87,
91, 87, 44, 78, 22, 9, 21, 19, 63, 23, 41, 98, 70, 12, 66, 33, 1,
51, 95, 55, 10, 52, 2, 2, 98, 67, 82, 30, 43, 90, 46, 88, 78, 45,
82, 86, 49, 15, 66, 4, 32, 64, 53, 2, 94, 56, 19, 97, 53]), array([56, 88, 4, 4, 40, 43, 43, 75, 60, 37, 0, 8, 40, 24, 24, 15, 2,
79, 12, 81, 26, 70, 61, 6, 67, 88, 26, 23, 49, 86, 36, 98, 56, 3,
26, 40, 76, 50, 58, 55, 83, 88, 41, 51, 60, 86, 88, 48, 36, 19, 43,
69, 20, 80, 42, 74, 93, 94, 55, 82, 79, 12, 81, 37, 98, 33, 12, 44,
44, 86, 79, 63, 66, 32, 66, 5, 89, 27, 12, 53, 64, 29, 24, 78, 86,
82, 44, 59, 87, 59, 66, 49, 95, 44, 20, 78, 94, 24, 2, 18]), array([33, 44, 27, 12, 5, 62, 31, 17, 12, 74, 7, 37, 38, 40, 48, 50, 68,
80, 45, 77, 16, 5, 65, 0, 24, 88, 78, 57, 73, 46, 65, 59, 87, 5,
24, 80, 96, 25, 95, 95, 65, 60, 41, 57, 24, 98, 38, 63, 12, 10, 28,
20, 71, 17, 53, 94, 7, 11, 74, 39, 32, 14, 64, 9, 77, 47, 73, 4,
31, 73, 29, 6, 50, 26, 98, 51, 10, 53, 50, 37, 90, 50, 67, 12, 71,
79, 44, 29, 14, 88, 14, 27, 91, 99, 89, 52, 6, 28, 46, 73]), array([38, 85, 25, 18, 53, 18, 68, 8, 41, 75, 87, 38, 80, 43, 54, 92, 74,
10, 81, 32, 74, 98, 93, 7, 10, 69, 63, 0, 74, 32, 27, 28, 83, 5,
81, 55, 80, 56, 95, 86, 79, 63, 11, 53, 87, 87, 21, 1, 99, 95, 74,
69, 83, 65, 38, 1, 65, 49, 67, 98, 43, 19, 77, 42, 40, 43, 76, 19,
29, 26, 98, 88, 67, 5, 35, 67, 56, 89, 98, 45, 72, 51, 43, 95, 31,
26, 81, 77, 58, 19, 3, 30, 6, 15, 69, 72, 28, 28, 64, 67]), array([64, 26, 75, 91, 87, 78, 16, 14, 85, 95, 84, 94, 54, 62, 71, 51, 77,
17, 92, 84, 18, 56, 37, 34, 63, 13, 26, 39, 24, 83, 61, 49, 28, 19,
14, 3, 11, 15, 13, 28, 11, 69, 73, 36, 77, 91, 19, 3, 29, 32, 38,
24, 9, 73, 16, 48, 26, 36, 62, 27, 60, 57, 73, 64, 15, 19, 18, 47,
90, 71, 32, 55, 65, 47, 81, 10, 64, 82, 67, 59, 41, 59, 36, 92, 46,
87, 88, 54, 55, 3, 64, 22, 85, 18, 71, 80, 16, 58, 52, 96]), array([57, 85, 23, 40, 66, 75, 57, 77, 30, 58, 74, 24, 64, 22, 55, 34, 66,
5, 86, 15, 31, 50, 89, 83, 11, 88, 75, 52, 93, 78, 91, 39, 38, 23,
66, 36, 5, 95, 82, 69, 90, 93, 50, 16, 60, 44, 97, 59, 27, 11, 16,
9, 59, 18, 73, 12, 82, 34, 0, 3, 51, 78, 96, 40, 41, 23, 53, 80,
31, 12, 7, 31, 72, 0, 36, 78, 3, 81, 34, 48, 97, 0, 53, 22, 32,
56, 2, 37, 18, 27, 42, 46, 0, 68, 95, 43, 59, 46, 68, 6]), array([ 3, 35, 99, 10, 52, 57, 12, 70, 3, 18, 23, 34, 26, 76, 97, 55, 92,
54, 53, 50, 70, 85, 60, 66, 15, 61, 14, 69, 99, 93, 43, 55, 95, 16,
61, 14, 84, 65, 17, 63, 0, 44, 74, 27, 44, 75, 31, 68, 94, 32, 45,
88, 32, 39, 80, 21, 15, 63, 41, 41, 7, 21, 6, 76, 30, 92, 59, 64,
53, 24, 56, 33, 89, 48, 64, 79, 55, 37, 22, 85, 81, 19, 14, 53, 61,
74, 27, 12, 72, 52, 37, 68, 88, 63, 84, 3, 93, 29, 9, 90]), array([79, 4, 55, 6, 2, 48, 40, 56, 47, 17, 27, 85, 79, 92, 68, 0, 21,
2, 78, 65, 74, 12, 92, 67, 13, 86, 60, 90, 48, 20, 15, 59, 30, 2,
36, 81, 76, 87, 76, 64, 45, 43, 27, 48, 17, 15, 61, 27, 52, 69, 32,
44, 39, 76, 4, 23, 82, 12, 37, 15, 94, 82, 90, 5, 12, 55, 4, 21,
73, 94, 37, 21, 39, 25, 99, 93, 29, 93, 16, 3, 51, 4, 54, 45, 32,
23, 93, 25, 45, 85, 77, 5, 79, 98, 8, 57, 91, 47, 90, 59]), array([67, 73, 32, 90, 72, 19, 1, 64, 19, 49, 55, 84, 95, 56, 65, 63, 73,
23, 50, 22, 43, 17, 32, 19, 66, 65, 62, 3, 88, 40, 44, 80, 0, 55,
23, 1, 50, 68, 22, 7, 46, 58, 71, 71, 7, 96, 4, 29, 71, 3, 32,
49, 98, 69, 65, 6, 62, 43, 40, 74, 20, 25, 94, 25, 71, 12, 2, 41,
73, 70, 73, 62, 71, 73, 4, 34, 41, 4, 37, 5, 58, 63, 52, 14, 87,
65, 53, 90, 52, 97, 52, 95, 80, 19, 17, 9, 98, 40, 87, 29]), array([38, 31, 0, 89, 85, 86, 99, 45, 92, 35, 0, 68, 82, 85, 28, 85, 12,
3, 80, 34, 7, 39, 22, 62, 73, 12, 4, 13, 31, 90, 85, 88, 61, 54,
84, 80, 0, 64, 47, 99, 50, 93, 35, 92, 24, 4, 11, 22, 21, 1, 96,
85, 3, 59, 36, 39, 52, 99, 43, 69, 81, 19, 16, 43, 38, 30, 14, 77,
68, 79, 47, 51, 90, 67, 89, 51, 28, 95, 82, 3, 33, 7, 4, 67, 70,
57, 34, 64, 47, 21, 85, 97, 1, 36, 22, 36, 36, 66, 26, 59]), array([74, 93, 27, 12, 95, 28, 45, 93, 96, 11, 77, 43, 66, 31, 23, 11, 52,
6, 20, 53, 53, 95, 41, 35, 70, 68, 96, 41, 71, 86, 1, 8, 56, 47,
81, 65, 99, 7, 93, 95, 89, 68, 86, 68, 65, 37, 1, 16, 59, 2, 1,
58, 50, 98, 53, 58, 16, 62, 87, 56, 18, 38, 32, 86, 35, 50, 10, 33,
40, 3, 37, 68, 5, 92, 78, 84, 24, 68, 9, 89, 43, 21, 55, 4, 34,
12, 73, 43, 70, 55, 39, 61, 55, 69, 40, 15, 5, 52, 27, 53]), array([ 2, 59, 35, 11, 82, 39, 32, 81, 23, 33, 52, 27, 31, 3, 32, 7, 79,
9, 45, 69, 53, 77, 56, 14, 83, 50, 76, 41, 52, 10, 91, 74, 37, 33,
43, 49, 46, 38, 85, 24, 19, 47, 87, 80, 93, 61, 81, 41, 77, 8, 47,
89, 47, 45, 88, 31, 96, 19, 41, 36, 16, 47, 76, 73, 74, 84, 30, 40,
92, 27, 34, 68, 65, 99, 75, 30, 26, 66, 43, 58, 15, 19, 12, 16, 2,
20, 98, 74, 65, 45, 55, 8, 31, 84, 47, 8, 55, 78, 91, 42]), array([39, 14, 19, 95, 31, 92, 3, 22, 19, 59, 25, 78, 85, 44, 36, 82, 38,
1, 39, 67, 49, 63, 24, 67, 4, 10, 66, 76, 66, 87, 37, 14, 57, 39,
17, 69, 46, 3, 42, 74, 24, 74, 68, 7, 81, 81, 13, 6, 10, 51, 82,
61, 98, 60, 37, 27, 44, 4, 21, 13, 31, 91, 7, 36, 12, 84, 85, 11,
16, 17, 52, 23, 98, 18, 70, 47, 72, 45, 34, 56, 76, 80, 66, 1, 56,
89, 51, 56, 53, 38, 71, 10, 57, 88, 50, 60, 38, 28, 33, 78]), array([68, 31, 77, 50, 19, 49, 87, 59, 32, 10, 52, 40, 42, 9, 49, 32, 91,
91, 30, 92, 88, 45, 85, 17, 85, 95, 33, 60, 30, 76, 71, 6, 38, 30,
74, 83, 89, 39, 38, 30, 13, 42, 89, 87, 79, 77, 69, 53, 34, 3, 13,
32, 98, 80, 99, 50, 33, 38, 8, 62, 93, 79, 97, 46, 64, 72, 37, 52,
79, 3, 27, 72, 39, 39, 12, 32, 42, 28, 78, 39, 76, 11, 60, 57, 85,
1, 96, 69, 5, 5, 55, 14, 85, 89, 40, 51, 74, 57, 58, 50]), array([17, 42, 16, 57, 12, 17, 69, 84, 57, 74, 30, 47, 21, 44, 23, 50, 76,
86, 92, 84, 53, 49, 54, 81, 84, 23, 89, 13, 52, 65, 81, 77, 93, 25,
91, 81, 17, 42, 14, 32, 65, 83, 2, 86, 52, 64, 23, 82, 74, 82, 17,
97, 87, 72, 73, 77, 87, 6, 87, 6, 46, 76, 0, 30, 79, 32, 43, 98,
42, 51, 71, 69, 57, 45, 30, 12, 18, 54, 90, 11, 32, 92, 88, 56, 26,
43, 62, 57, 90, 96, 61, 76, 79, 55, 76, 58, 64, 1, 23, 59]), array([74, 54, 90, 19, 62, 90, 40, 28, 29, 68, 13, 80, 49, 65, 98, 20, 61,
71, 71, 55, 22, 74, 13, 17, 1, 58, 57, 66, 80, 70, 86, 42, 44, 88,
62, 36, 79, 4, 92, 44, 65, 11, 63, 47, 91, 58, 45, 44, 75, 27, 47,
4, 36, 43, 17, 20, 10, 8, 30, 67, 60, 74, 52, 48, 49, 76, 12, 31,
18, 43, 85, 53, 41, 46, 0, 31, 81, 75, 52, 77, 33, 52, 39, 77, 67,
6, 51, 90, 91, 63, 40, 69, 79, 14, 51, 35, 4, 89, 62, 47]), array([20, 35, 99, 91, 15, 45, 78, 27, 68, 25, 98, 1, 40, 16, 7, 52, 44,
5, 65, 43, 27, 25, 61, 5, 54, 5, 72, 79, 71, 8, 34, 45, 61, 82,
95, 18, 56, 84, 70, 7, 42, 35, 95, 65, 37, 63, 96, 73, 39, 44, 64,
99, 55, 69, 45, 94, 55, 83, 86, 64, 41, 67, 65, 64, 75, 80, 48, 87,
98, 5, 83, 42, 12, 51, 3, 56, 70, 32, 70, 90, 86, 19, 52, 22, 11,
9, 46, 61, 72, 75, 39, 8, 39, 72, 82, 56, 12, 60, 55, 33]), array([64, 97, 63, 50, 28, 93, 62, 97, 24, 57, 91, 19, 99, 81, 48, 40, 39,
46, 36, 87, 20, 66, 30, 71, 96, 54, 54, 41, 59, 19, 42, 89, 38, 42,
20, 69, 62, 46, 8, 93, 91, 92, 12, 72, 10, 86, 92, 30, 38, 73, 61,
99, 53, 56, 80, 88, 3, 79, 28, 95, 27, 7, 73, 83, 52, 67, 55, 59,
43, 61, 80, 73, 45, 57, 4, 96, 92, 26, 92, 29, 42, 63, 51, 74, 72,
86, 68, 17, 85, 13, 24, 82, 26, 96, 23, 41, 54, 50, 4, 68]), array([56, 32, 39, 8, 36, 49, 51, 68, 48, 50, 51, 48, 29, 94, 78, 95, 79,
18, 91, 17, 90, 95, 59, 70, 26, 68, 83, 88, 13, 61, 84, 98, 76, 87,
73, 35, 83, 39, 34, 14, 80, 6, 44, 58, 82, 25, 62, 69, 42, 72, 46,
97, 36, 46, 39, 33, 68, 48, 98, 4, 33, 85, 88, 29, 68, 95, 42, 11,
83, 3, 0, 54, 44, 36, 34, 48, 96, 37, 37, 38, 18, 71, 41, 67, 46,
93, 73, 74, 81, 93, 58, 99, 48, 51, 75, 7, 33, 45, 38, 61]), array([67, 91, 21, 53, 95, 93, 66, 77, 43, 34, 35, 5, 87, 58, 4, 12, 1,
40, 93, 26, 36, 37, 95, 34, 31, 44, 58, 16, 1, 99, 20, 83, 46, 16,
26, 71, 19, 7, 55, 57, 32, 57, 97, 48, 59, 0, 82, 15, 90, 27, 71,
99, 69, 59, 93, 99, 3, 64, 20, 4, 62, 17, 8, 23, 47, 8, 23, 45,
38, 35, 53, 90, 55, 47, 72, 67, 90, 23, 76, 67, 57, 97, 40, 28, 15,
26, 81, 5, 99, 15, 23, 35, 45, 24, 68, 71, 72, 20, 50, 34]), array([52, 43, 20, 4, 48, 89, 4, 19, 31, 40, 3, 3, 20, 42, 81, 27, 30,
4, 80, 8, 1, 24, 2, 99, 17, 20, 18, 28, 43, 96, 23, 78, 38, 88,
10, 87, 97, 77, 83, 68, 22, 5, 8, 2, 43, 25, 1, 65, 76, 79, 11,
2, 45, 33, 92, 6, 9, 54, 8, 29, 66, 11, 79, 79, 82, 8, 43, 22,
5, 92, 32, 71, 60, 44, 10, 66, 35, 36, 10, 10, 75, 2, 32, 0, 27,
69, 65, 93, 45, 71, 42, 73, 23, 91, 12, 15, 22, 60, 82, 31]), array([58, 51, 73, 14, 19, 61, 63, 96, 22, 51, 11, 43, 19, 7, 9, 36, 62,
11, 61, 60, 85, 30, 12, 63, 56, 56, 48, 52, 20, 22, 50, 78, 58, 82,
71, 96, 44, 76, 54, 64, 98, 26, 65, 22, 91, 35, 29, 9, 0, 67, 88,
77, 83, 6, 21, 24, 8, 51, 38, 10, 22, 71, 89, 90, 39, 10, 96, 51,
63, 30, 98, 50, 66, 74, 18, 52, 67, 98, 98, 56, 99, 82, 52, 79, 38,
58, 98, 96, 6, 76, 33, 27, 8, 79, 40, 29, 72, 61, 63, 26]), array([83, 47, 88, 61, 37, 12, 56, 40, 64, 48, 58, 98, 22, 21, 4, 19, 86,
66, 22, 25, 60, 35, 48, 54, 18, 43, 53, 42, 54, 48, 49, 44, 14, 38,
19, 86, 52, 85, 12, 17, 8, 51, 42, 47, 26, 99, 66, 77, 97, 21, 53,
25, 72, 87, 44, 4, 5, 22, 14, 46, 50, 67, 63, 28, 55, 38, 77, 61,
64, 7, 96, 68, 47, 40, 52, 27, 1, 39, 44, 70, 36, 71, 75, 31, 15,
66, 15, 46, 8, 94, 90, 75, 13, 87, 12, 75, 64, 43, 18, 25]), array([27, 53, 21, 7, 28, 95, 74, 6, 9, 72, 63, 41, 19, 71, 62, 29, 93,
14, 80, 21, 29, 67, 28, 0, 1, 10, 99, 27, 23, 17, 11, 99, 0, 65,
40, 68, 42, 47, 93, 34, 19, 3, 35, 43, 13, 44, 68, 30, 74, 94, 31,
77, 21, 41, 65, 9, 24, 77, 54, 54, 30, 12, 7, 91, 27, 24, 29, 13,
73, 88, 10, 93, 65, 59, 76, 22, 4, 6, 45, 16, 98, 70, 82, 64, 23,
18, 39, 8, 67, 19, 5, 34, 49, 26, 73, 28, 5, 67, 35, 14]), array([60, 72, 55, 77, 95, 65, 49, 87, 32, 83, 96, 1, 60, 42, 31, 26, 65,
47, 46, 15, 26, 42, 87, 48, 70, 61, 57, 74, 33, 16, 23, 41, 52, 46,
46, 19, 51, 51, 76, 57, 34, 2, 28, 91, 88, 1, 39, 36, 70, 99, 86,
33, 11, 90, 66, 38, 15, 50, 88, 43, 77, 70, 76, 75, 74, 24, 16, 39,
50, 46, 39, 68, 61, 45, 22, 50, 32, 93, 66, 62, 6, 43, 97, 39, 87,
89, 71, 75, 19, 98, 6, 69, 88, 23, 62, 28, 68, 11, 35, 30]), array([88, 60, 8, 59, 74, 64, 60, 27, 0, 50, 0, 33, 34, 48, 62, 39, 50,
23, 35, 19, 14, 33, 54, 95, 37, 83, 21, 89, 88, 2, 58, 7, 75, 91,
64, 79, 16, 84, 49, 91, 98, 76, 77, 38, 29, 21, 54, 5, 72, 41, 25,
45, 50, 35, 28, 3, 84, 84, 91, 49, 33, 68, 96, 27, 4, 13, 53, 31,
2, 90, 77, 48, 73, 25, 65, 80, 80, 79, 8, 50, 3, 18, 82, 8, 92,
58, 27, 24, 26, 95, 2, 15, 4, 33, 27, 78, 91, 76, 87, 29]), array([44, 65, 62, 18, 59, 85, 45, 27, 61, 70, 42, 1, 3, 34, 86, 6, 82,
68, 66, 77, 34, 11, 5, 2, 81, 93, 76, 44, 89, 76, 86, 82, 59, 50,
56, 15, 60, 94, 82, 63, 16, 93, 4, 67, 48, 67, 14, 35, 0, 99, 53,
62, 25, 85, 37, 87, 26, 3, 98, 26, 40, 39, 70, 63, 91, 32, 90, 26,
60, 44, 5, 55, 37, 61, 6, 98, 15, 7, 15, 47, 69, 72, 98, 27, 98,
65, 70, 27, 56, 66, 99, 28, 3, 27, 35, 20, 32, 43, 91, 37]), array([ 2, 50, 10, 16, 14, 99, 92, 35, 30, 47, 48, 4, 79, 49, 89, 29, 28,
11, 91, 34, 16, 2, 50, 96, 20, 76, 8, 27, 31, 58, 7, 84, 39, 24,
28, 66, 55, 18, 14, 24, 69, 11, 63, 47, 67, 99, 13, 94, 19, 46, 55,
58, 62, 97, 29, 47, 8, 38, 53, 51, 2, 38, 11, 70, 36, 40, 88, 3,
77, 1, 72, 67, 22, 51, 22, 41, 73, 18, 84, 96, 77, 62, 46, 55, 10,
95, 31, 32, 34, 49, 46, 46, 82, 13, 71, 32, 24, 55, 10, 34]), array([55, 13, 3, 80, 54, 41, 17, 86, 35, 56, 46, 44, 39, 79, 63, 62, 94,
26, 43, 77, 85, 74, 32, 67, 57, 0, 29, 74, 45, 20, 48, 99, 35, 91,
47, 3, 26, 52, 4, 89, 90, 7, 3, 32, 50, 33, 7, 97, 98, 91, 88,
28, 18, 41, 0, 34, 49, 40, 46, 53, 62, 68, 19, 77, 57, 2, 45, 14,
54, 5, 70, 33, 46, 47, 66, 60, 39, 39, 77, 99, 68, 33, 66, 14, 73,
66, 68, 40, 85, 24, 81, 66, 54, 68, 24, 63, 64, 5, 32, 35]), array([15, 59, 71, 53, 30, 15, 16, 75, 60, 86, 46, 76, 90, 21, 66, 33, 73,
54, 50, 99, 93, 51, 67, 82, 41, 5, 89, 51, 67, 23, 70, 97, 47, 33,
23, 46, 80, 44, 43, 52, 51, 83, 84, 33, 27, 98, 99, 91, 10, 61, 92,
39, 28, 71, 26, 3, 21, 14, 32, 53, 94, 86, 91, 38, 82, 39, 15, 18,
75, 35, 5, 84, 91, 7, 70, 42, 11, 61, 3, 35, 67, 44, 21, 94, 26,
29, 72, 52, 44, 97, 14, 24, 83, 65, 24, 2, 32, 35, 21, 32]), array([ 9, 68, 62, 70, 52, 89, 91, 35, 5, 70, 44, 47, 11, 15, 66, 7, 87,
58, 36, 39, 73, 85, 12, 1, 64, 46, 37, 95, 14, 75, 80, 64, 88, 75,
64, 2, 0, 16, 57, 5, 7, 80, 74, 55, 17, 43, 72, 10, 75, 58, 95,
38, 89, 18, 11, 12, 69, 65, 90, 61, 35, 36, 15, 45, 52, 22, 89, 28,
63, 28, 71, 58, 73, 78, 37, 90, 56, 69, 98, 4, 25, 91, 45, 86, 96,
74, 91, 40, 80, 12, 8, 38, 15, 64, 95, 62, 27, 66, 63, 89]), array([54, 82, 84, 80, 27, 9, 74, 67, 1, 25, 79, 50, 24, 45, 32, 89, 14,
98, 41, 72, 35, 56, 68, 46, 32, 64, 8, 41, 72, 16, 67, 13, 75, 27,
42, 5, 42, 60, 53, 22, 27, 48, 65, 6, 86, 39, 60, 74, 85, 23, 76,
17, 95, 20, 41, 60, 7, 27, 93, 33, 95, 26, 7, 98, 57, 92, 96, 38,
94, 7, 83, 98, 71, 27, 13, 96, 4, 55, 46, 77, 41, 91, 35, 23, 44,
13, 6, 61, 24, 36, 31, 68, 6, 37, 66, 97, 98, 29, 76, 92]), array([ 3, 65, 89, 14, 43, 68, 38, 22, 48, 97, 20, 38, 67, 84, 53, 22, 36,
77, 29, 50, 58, 48, 65, 29, 41, 45, 60, 49, 60, 61, 53, 5, 65, 76,
83, 23, 86, 22, 37, 3, 31, 12, 91, 28, 86, 84, 28, 49, 39, 96, 60,
87, 30, 51, 78, 5, 84, 97, 43, 10, 82, 49, 45, 12, 87, 25, 51, 46,
10, 5, 2, 15, 74, 10, 2, 12, 91, 19, 78, 57, 48, 60, 49, 94, 70,
31, 30, 3, 67, 67, 70, 22, 12, 90, 31, 60, 53, 30, 49, 99]), array([21, 12, 99, 58, 95, 21, 64, 36, 88, 3, 66, 87, 65, 74, 64, 70, 88,
26, 63, 34, 47, 2, 49, 7, 78, 67, 17, 61, 95, 85, 27, 50, 94, 10,
96, 1, 70, 93, 38, 56, 96, 93, 94, 87, 58, 22, 5, 11, 7, 45, 79,
21, 36, 90, 9, 43, 57, 69, 76, 2, 19, 29, 35, 92, 16, 49, 24, 88,
30, 20, 72, 95, 41, 43, 75, 28, 61, 72, 46, 25, 10, 10, 25, 5, 66,
79, 39, 25, 71, 35, 33, 87, 78, 72, 37, 29, 20, 90, 2, 92]), array([65, 99, 12, 95, 57, 89, 24, 7, 40, 99, 31, 49, 25, 37, 47, 38, 22,
95, 61, 58, 61, 4, 93, 40, 2, 61, 97, 53, 38, 12, 12, 66, 45, 71,
88, 36, 71, 55, 86, 53, 49, 0, 25, 1, 28, 10, 65, 2, 76, 15, 8,
65, 9, 82, 44, 30, 80, 84, 8, 85, 53, 95, 51, 4, 87, 66, 67, 31,
43, 98, 22, 0, 64, 78, 32, 13, 36, 5, 87, 41, 70, 89, 92, 77, 88,
11, 19, 80, 53, 14, 96, 64, 21, 30, 12, 79, 5, 85, 90, 73]), array([12, 77, 4, 60, 78, 44, 83, 77, 71, 70, 76, 49, 88, 30, 6, 48, 45,
26, 44, 23, 52, 47, 41, 72, 94, 16, 45, 93, 17, 42, 0, 11, 77, 23,
12, 67, 22, 37, 9, 48, 50, 74, 5, 54, 81, 75, 76, 14, 56, 70, 14,
39, 19, 88, 70, 83, 98, 71, 1, 23, 89, 52, 3, 78, 14, 69, 79, 42,
11, 29, 5, 47, 77, 99, 69, 43, 91, 11, 93, 29, 84, 20, 75, 22, 43,
75, 32, 97, 79, 57, 69, 2, 2, 42, 7, 47, 49, 15, 41, 43]), array([ 2, 88, 65, 34, 1, 85, 40, 22, 77, 96, 47, 12, 47, 37, 5, 7, 62,
42, 96, 95, 51, 12, 47, 78, 19, 66, 17, 33, 7, 75, 97, 53, 22, 53,
5, 36, 33, 33, 46, 24, 31, 46, 92, 32, 18, 68, 7, 10, 87, 35, 7,
51, 17, 78, 98, 39, 32, 21, 38, 22, 41, 74, 71, 64, 24, 17, 40, 22,
97, 57, 15, 12, 33, 32, 84, 32, 22, 34, 91, 93, 76, 88, 24, 34, 3,
66, 51, 4, 23, 22, 38, 81, 0, 51, 59, 21, 99, 47, 48, 48]), array([46, 77, 97, 66, 73, 71, 82, 54, 19, 28, 91, 38, 88, 94, 32, 23, 85,
52, 27, 17, 26, 66, 82, 9, 68, 78, 75, 31, 45, 96, 23, 30, 51, 76,
95, 22, 42, 53, 77, 13, 73, 19, 99, 57, 87, 29, 16, 72, 86, 84, 40,
37, 78, 76, 11, 67, 86, 88, 32, 82, 64, 12, 94, 95, 2, 33, 31, 33,
23, 33, 48, 40, 4, 76, 40, 65, 56, 15, 76, 36, 91, 41, 88, 69, 71,
86, 8, 58, 7, 94, 41, 80, 2, 64, 34, 72, 99, 29, 76, 56]), array([ 3, 56, 61, 92, 33, 22, 37, 88, 97, 1, 57, 98, 65, 16, 9, 14, 61,
82, 47, 25, 92, 65, 56, 57, 44, 83, 85, 58, 61, 56, 50, 76, 4, 51,
77, 12, 49, 33, 44, 48, 67, 39, 68, 49, 82, 69, 39, 84, 34, 66, 38,
30, 78, 75, 96, 63, 51, 10, 77, 63, 90, 75, 48, 18, 38, 27, 79, 72,
89, 27, 19, 24, 37, 97, 61, 86, 30, 81, 99, 40, 20, 96, 17, 13, 72,
70, 60, 75, 76, 50, 41, 0, 29, 49, 57, 96, 15, 92, 17, 93]), array([23, 39, 59, 91, 65, 34, 14, 82, 77, 60, 32, 63, 71, 88, 86, 89, 27,
19, 29, 86, 46, 99, 43, 29, 75, 17, 90, 74, 32, 81, 3, 82, 58, 63,
85, 80, 74, 4, 39, 94, 98, 74, 86, 18, 65, 66, 85, 36, 1, 98, 23,
70, 71, 86, 32, 37, 1, 73, 8, 87, 9, 51, 68, 22, 79, 44, 71, 18,
80, 48, 14, 7, 17, 9, 75, 79, 67, 22, 35, 99, 74, 65, 73, 1, 91,
87, 42, 85, 2, 91, 82, 6, 46, 33, 34, 76, 8, 0, 42, 70]), array([52, 69, 44, 75, 31, 1, 90, 27, 61, 20, 94, 65, 59, 59, 73, 92, 39,
34, 49, 8, 63, 94, 88, 95, 16, 61, 85, 57, 92, 73, 28, 90, 39, 41,
83, 15, 22, 62, 39, 37, 19, 67, 87, 47, 90, 25, 75, 38, 28, 65, 58,
81, 40, 18, 20, 66, 21, 90, 44, 90, 61, 26, 52, 44, 35, 31, 1, 86,
34, 36, 68, 19, 95, 50, 51, 84, 6, 19, 87, 54, 94, 44, 72, 89, 87,
51, 39, 27, 18, 64, 91, 88, 63, 33, 3, 75, 45, 25, 99, 18]), array([15, 27, 16, 71, 47, 86, 41, 2, 77, 38, 18, 27, 64, 80, 75, 53, 7,
20, 18, 98, 95, 43, 88, 20, 60, 30, 38, 74, 86, 77, 45, 13, 41, 89,
2, 72, 68, 92, 84, 99, 21, 55, 76, 55, 28, 70, 70, 33, 84, 0, 70,
96, 84, 31, 34, 11, 6, 90, 60, 36, 27, 85, 16, 8, 52, 55, 97, 37,
80, 16, 2, 66, 49, 68, 69, 69, 56, 4, 23, 92, 24, 0, 80, 15, 88,
59, 88, 22, 64, 77, 66, 35, 23, 99, 37, 37, 4, 59, 79, 98]), array([89, 28, 17, 57, 95, 77, 12, 98, 39, 86, 5, 26, 58, 81, 83, 35, 55,
22, 18, 4, 8, 19, 27, 47, 4, 16, 15, 44, 73, 27, 99, 34, 84, 71,
74, 94, 23, 80, 0, 35, 42, 65, 60, 31, 84, 55, 24, 91, 11, 7, 55,
93, 79, 66, 3, 1, 27, 21, 52, 29, 4, 35, 93, 57, 85, 27, 83, 86,
43, 92, 98, 80, 41, 11, 58, 99, 88, 14, 45, 81, 27, 89, 98, 63, 49,
8, 84, 21, 63, 62, 50, 9, 67, 1, 62, 19, 72, 21, 25, 78]), array([14, 70, 58, 39, 97, 67, 46, 75, 91, 74, 2, 91, 97, 61, 70, 12, 7,
6, 82, 97, 24, 90, 95, 58, 29, 34, 6, 77, 27, 88, 47, 37, 39, 38,
42, 6, 87, 30, 2, 86, 66, 73, 29, 44, 54, 95, 44, 22, 19, 55, 90,
6, 60, 46, 13, 31, 42, 34, 48, 22, 72, 57, 75, 85, 96, 88, 15, 4,
40, 9, 99, 62, 19, 59, 54, 3, 1, 26, 91, 66, 35, 36, 11, 51, 12,
93, 18, 77, 72, 70, 85, 95, 95, 68, 55, 90, 21, 11, 34, 54]), array([80, 0, 4, 1, 21, 37, 25, 31, 64, 84, 34, 91, 78, 47, 12, 40, 5,
23, 25, 95, 41, 50, 71, 89, 5, 85, 48, 31, 92, 98, 81, 7, 67, 37,
67, 27, 66, 86, 28, 47, 96, 47, 98, 31, 17, 89, 78, 13, 66, 61, 57,
8, 60, 31, 30, 72, 25, 69, 55, 78, 67, 18, 62, 54, 26, 32, 72, 75,
25, 64, 58, 51, 47, 41, 38, 52, 30, 32, 5, 13, 3, 95, 37, 64, 43,
71, 87, 2, 40, 95, 46, 34, 7, 20, 85, 87, 56, 7, 95, 9]), array([32, 12, 94, 7, 64, 17, 2, 69, 93, 3, 97, 15, 50, 55, 40, 90, 98,
50, 75, 80, 21, 1, 99, 70, 67, 44, 7, 99, 77, 4, 12, 99, 84, 59,
28, 70, 75, 42, 87, 8, 4, 74, 6, 55, 42, 84, 78, 51, 57, 55, 78,
10, 28, 96, 25, 61, 1, 7, 87, 55, 16, 9, 23, 42, 18, 59, 14, 80,
33, 28, 5, 21, 46, 65, 61, 97, 30, 50, 64, 78, 99, 26, 16, 45, 20,
24, 41, 52, 98, 78, 6, 73, 21, 43, 37, 45, 9, 49, 47, 23]), array([25, 58, 85, 56, 96, 68, 98, 30, 13, 65, 27, 23, 7, 48, 17, 82, 63,
98, 26, 19, 49, 6, 40, 88, 3, 17, 31, 99, 28, 51, 57, 9, 49, 26,
25, 63, 53, 40, 85, 54, 39, 50, 12, 68, 33, 93, 38, 98, 29, 64, 16,
97, 78, 16, 36, 51, 91, 11, 91, 71, 89, 29, 61, 26, 98, 92, 92, 85,
61, 39, 1, 1, 81, 9, 50, 27, 2, 84, 64, 76, 26, 55, 9, 52, 43,
49, 44, 62, 43, 25, 93, 34, 61, 60, 4, 47, 15, 88, 79, 92]), array([85, 96, 7, 68, 39, 50, 18, 60, 38, 50, 89, 84, 83, 75, 22, 39, 51,
66, 63, 0, 99, 56, 95, 29, 69, 9, 24, 81, 88, 89, 94, 8, 30, 94,
5, 37, 84, 43, 63, 33, 2, 30, 92, 94, 49, 8, 51, 99, 43, 57, 35,
23, 78, 90, 37, 71, 28, 70, 41, 20, 87, 5, 4, 23, 44, 89, 67, 32,
13, 90, 53, 43, 48, 30, 53, 57, 96, 41, 69, 58, 24, 32, 47, 92, 2,
24, 99, 52, 78, 47, 90, 5, 84, 70, 2, 37, 15, 64, 59, 55]), array([ 8, 58, 33, 29, 59, 91, 37, 70, 62, 50, 14, 76, 22, 5, 67, 90, 99,
10, 45, 27, 51, 15, 90, 65, 69, 50, 72, 97, 56, 75, 79, 36, 6, 33,
82, 77, 8, 26, 29, 80, 54, 3, 72, 8, 45, 45, 93, 62, 90, 86, 84,
81, 41, 8, 0, 15, 72, 24, 93, 59, 78, 78, 27, 7, 99, 46, 63, 53,
56, 42, 63, 70, 25, 51, 40, 42, 36, 80, 6, 44, 87, 27, 97, 30, 23,
13, 41, 48, 68, 39, 79, 22, 48, 30, 48, 29, 8, 0, 59, 6]), array([92, 40, 45, 14, 89, 27, 21, 14, 51, 67, 57, 55, 11, 66, 53, 39, 69,
44, 51, 79, 2, 41, 77, 82, 19, 57, 34, 31, 67, 76, 16, 81, 57, 49,
17, 92, 48, 33, 26, 3, 18, 17, 54, 10, 42, 56, 55, 66, 25, 76, 47,
53, 27, 33, 52, 29, 67, 98, 3, 62, 95, 76, 69, 91, 29, 57, 44, 2,
63, 57, 82, 25, 88, 6, 2, 61, 14, 41, 27, 24, 93, 46, 98, 99, 49,
43, 78, 1, 60, 54, 97, 74, 33, 67, 96, 13, 89, 50, 50, 33]), array([64, 76, 81, 89, 25, 36, 56, 20, 65, 43, 21, 88, 35, 11, 52, 95, 41,
41, 22, 4, 69, 20, 1, 45, 24, 6, 7, 49, 40, 20, 52, 20, 22, 95,
79, 7, 68, 43, 16, 50, 89, 41, 99, 52, 42, 70, 50, 73, 90, 72, 3,
24, 50, 7, 42, 20, 83, 27, 32, 58, 76, 99, 89, 45, 78, 43, 88, 26,
62, 23, 10, 12, 89, 73, 78, 3, 36, 43, 64, 41, 42, 94, 40, 57, 4,
76, 49, 21, 53, 17, 0, 51, 5, 27, 40, 98, 48, 92, 56, 48]), array([33, 29, 27, 77, 20, 1, 22, 16, 59, 83, 3, 7, 45, 84, 48, 80, 6,
18, 66, 92, 10, 40, 2, 96, 11, 17, 77, 73, 80, 58, 87, 4, 21, 16,
37, 38, 38, 8, 33, 88, 54, 18, 24, 8, 75, 94, 55, 24, 98, 25, 82,
30, 95, 55, 88, 71, 55, 83, 11, 5, 75, 83, 9, 33, 26, 39, 66, 20,
88, 60, 41, 84, 65, 47, 89, 87, 99, 41, 29, 34, 96, 49, 67, 14, 80,
15, 47, 31, 25, 50, 98, 73, 43, 86, 25, 54, 82, 2, 26, 97]), array([75, 83, 81, 80, 99, 5, 39, 45, 74, 67, 54, 10, 82, 82, 27, 29, 47,
25, 94, 54, 57, 42, 91, 79, 9, 82, 63, 58, 94, 28, 72, 93, 16, 73,
99, 35, 36, 3, 90, 56, 88, 99, 97, 68, 7, 27, 49, 85, 89, 86, 45,
21, 53, 33, 95, 70, 27, 26, 70, 79, 79, 49, 73, 28, 24, 70, 15, 44,
71, 20, 40, 91, 64, 3, 43, 71, 7, 43, 88, 16, 25, 85, 27, 62, 85,
25, 27, 31, 98, 62, 41, 44, 59, 73, 65, 18, 5, 91, 10, 90]), array([41, 19, 25, 39, 42, 6, 25, 53, 61, 48, 79, 64, 47, 38, 34, 1, 47,
4, 48, 19, 9, 88, 42, 68, 10, 63, 99, 9, 92, 44, 50, 75, 88, 29,
81, 51, 45, 29, 26, 71, 88, 30, 48, 58, 14, 32, 50, 83, 14, 67, 65,
18, 72, 1, 42, 5, 76, 10, 78, 77, 83, 29, 11, 84, 58, 51, 55, 25,
91, 37, 56, 38, 46, 9, 97, 93, 38, 0, 93, 26, 3, 86, 60, 62, 69,
26, 30, 95, 77, 48, 36, 23, 78, 18, 13, 98, 46, 19, 28, 59]), array([30, 56, 29, 3, 50, 40, 26, 16, 11, 40, 77, 17, 49, 64, 77, 81, 1,
82, 8, 7, 57, 74, 60, 17, 32, 88, 59, 75, 60, 45, 21, 27, 42, 42,
55, 65, 71, 68, 42, 3, 7, 54, 86, 97, 42, 24, 51, 56, 0, 23, 39,
2, 68, 89, 40, 92, 47, 90, 16, 13, 69, 2, 55, 28, 69, 93, 46, 61,
65, 64, 1, 19, 27, 91, 6, 88, 83, 21, 94, 47, 31, 18, 32, 90, 39,
53, 83, 35, 80, 3, 23, 63, 65, 97, 55, 67, 72, 30, 94, 66]), array([38, 65, 93, 91, 76, 76, 48, 42, 82, 83, 80, 4, 79, 28, 25, 10, 78,
86, 86, 3, 86, 62, 44, 95, 85, 16, 26, 47, 30, 44, 78, 85, 10, 67,
94, 44, 31, 38, 72, 30, 9, 54, 41, 31, 63, 95, 94, 42, 48, 8, 69,
1, 60, 78, 23, 26, 10, 17, 35, 84, 42, 50, 41, 53, 51, 83, 5, 68,
15, 36, 16, 91, 12, 36, 29, 58, 92, 83, 28, 41, 10, 3, 27, 22, 88,
28, 12, 78, 32, 50, 92, 65, 3, 82, 73, 7, 60, 32, 48, 38]), array([78, 5, 85, 82, 38, 95, 58, 71, 91, 91, 38, 5, 42, 38, 84, 44, 44,
67, 73, 98, 15, 78, 35, 43, 87, 19, 50, 90, 94, 12, 39, 2, 24, 55,
39, 2, 5, 22, 9, 95, 54, 20, 24, 67, 84, 37, 98, 80, 34, 84, 70,
2, 37, 37, 62, 40, 40, 26, 84, 20, 79, 48, 99, 95, 87, 45, 34, 82,
17, 52, 50, 36, 89, 26, 22, 18, 58, 73, 52, 25, 93, 28, 9, 32, 67,
7, 76, 35, 95, 82, 79, 36, 6, 13, 58, 97, 92, 91, 81, 29]), array([ 0, 67, 37, 47, 54, 64, 42, 73, 53, 75, 9, 5, 78, 64, 98, 55, 10,
84, 85, 9, 90, 63, 34, 49, 66, 70, 51, 55, 29, 59, 57, 92, 2, 79,
5, 8, 78, 22, 95, 10, 37, 73, 13, 77, 51, 53, 50, 50, 24, 24, 87,
2, 53, 42, 96, 11, 46, 85, 83, 25, 29, 17, 18, 79, 45, 28, 19, 13,
87, 7, 44, 98, 70, 65, 89, 15, 80, 71, 67, 40, 9, 64, 42, 65, 76,
4, 2, 37, 3, 82, 2, 9, 13, 48, 3, 40, 3, 34, 31, 56]), array([29, 15, 64, 97, 37, 25, 20, 2, 0, 41, 57, 1, 36, 91, 97, 98, 95,
37, 33, 40, 55, 14, 44, 72, 99, 82, 20, 16, 89, 40, 27, 26, 1, 26,
40, 95, 31, 82, 33, 6, 19, 98, 93, 27, 74, 23, 32, 7, 52, 95, 71,
59, 42, 67, 29, 13, 41, 90, 83, 47, 76, 0, 84, 8, 36, 32, 39, 35,
84, 42, 10, 95, 58, 11, 52, 75, 56, 63, 0, 60, 35, 78, 24, 79, 60,
62, 47, 18, 98, 22, 91, 71, 51, 18, 37, 53, 61, 1, 72, 34]), array([84, 12, 19, 7, 60, 60, 97, 68, 77, 10, 60, 13, 22, 32, 96, 13, 3,
25, 1, 32, 65, 79, 73, 78, 86, 85, 59, 87, 20, 59, 15, 27, 21, 14,
79, 87, 73, 27, 15, 20, 97, 38, 30, 61, 74, 77, 27, 28, 14, 68, 24,
2, 93, 60, 98, 17, 57, 52, 79, 32, 19, 79, 34, 77, 41, 49, 5, 14,
90, 29, 39, 92, 16, 45, 84, 99, 90, 11, 66, 43, 82, 88, 45, 82, 68,
96, 79, 76, 51, 85, 20, 9, 64, 30, 44, 58, 2, 88, 64, 52]), array([95, 23, 94, 45, 67, 51, 83, 38, 79, 73, 47, 47, 42, 48, 72, 36, 97,
17, 58, 75, 14, 87, 66, 12, 37, 43, 92, 18, 57, 76, 34, 91, 83, 27,
92, 48, 59, 66, 4, 55, 46, 30, 88, 19, 43, 91, 12, 62, 74, 20, 17,
30, 1, 35, 72, 19, 82, 13, 8, 72, 40, 81, 56, 71, 23, 9, 39, 82,
86, 75, 41, 4, 39, 83, 63, 58, 51, 20, 95, 83, 30, 82, 80, 74, 93,
10, 71, 78, 60, 33, 46, 22, 78, 71, 13, 41, 77, 55, 78, 51]), array([ 1, 8, 1, 16, 70, 5, 39, 31, 79, 12, 84, 51, 43, 47, 85, 73, 42,
11, 31, 33, 41, 82, 3, 72, 43, 25, 38, 20, 22, 40, 73, 40, 30, 23,
57, 71, 22, 40, 85, 96, 5, 24, 77, 57, 11, 63, 62, 32, 99, 44, 70,
11, 43, 60, 71, 65, 95, 62, 29, 42, 7, 23, 83, 6, 72, 50, 34, 22,
70, 40, 11, 51, 85, 57, 41, 20, 43, 27, 67, 95, 59, 9, 43, 97, 27,
39, 23, 90, 50, 43, 21, 68, 0, 17, 35, 41, 41, 31, 42, 2]), array([90, 78, 70, 32, 0, 53, 4, 60, 5, 73, 5, 69, 59, 55, 61, 34, 99,
92, 68, 54, 23, 91, 26, 28, 42, 70, 83, 58, 26, 40, 48, 52, 9, 3,
14, 53, 5, 75, 72, 79, 19, 23, 61, 21, 60, 87, 35, 67, 30, 36, 73,
50, 86, 15, 65, 48, 34, 50, 59, 53, 84, 59, 56, 25, 23, 91, 71, 29,
43, 4, 0, 14, 18, 65, 1, 6, 94, 8, 81, 85, 61, 77, 40, 54, 38,
92, 34, 92, 32, 38, 47, 91, 7, 4, 58, 0, 38, 78, 85, 40]), array([25, 24, 63, 35, 31, 79, 6, 47, 8, 76, 79, 51, 16, 88, 92, 16, 61,
73, 59, 34, 18, 53, 59, 30, 31, 60, 49, 52, 37, 61, 91, 30, 66, 70,
35, 7, 67, 50, 75, 21, 34, 67, 68, 85, 21, 39, 48, 44, 56, 77, 74,
43, 55, 61, 44, 44, 7, 56, 99, 91, 83, 88, 26, 0, 17, 70, 84, 72,
2, 89, 0, 44, 36, 4, 58, 16, 84, 20, 87, 57, 74, 39, 30, 49, 20,
3, 48, 76, 62, 52, 33, 1, 1, 64, 47, 39, 74, 3, 23, 13]), array([24, 16, 26, 27, 48, 11, 25, 22, 35, 62, 13, 41, 43, 74, 34, 15, 30,
70, 13, 87, 39, 66, 11, 37, 98, 12, 38, 85, 51, 28, 33, 58, 0, 56,
70, 28, 19, 88, 47, 17, 6, 96, 99, 72, 95, 78, 13, 18, 70, 53, 60,
62, 0, 35, 35, 34, 69, 91, 53, 41, 95, 24, 57, 21, 3, 91, 21, 74,
3, 9, 73, 30, 6, 65, 36, 38, 36, 30, 15, 0, 39, 64, 8, 21, 96,
82, 54, 45, 49, 26, 18, 67, 89, 27, 76, 51, 2, 77, 56, 48]), array([76, 2, 17, 83, 71, 15, 38, 2, 62, 50, 75, 47, 95, 55, 48, 88, 73,
53, 86, 78, 84, 86, 96, 46, 42, 84, 73, 52, 84, 59, 88, 11, 35, 6,
56, 66, 60, 53, 29, 28, 97, 37, 8, 35, 7, 90, 29, 80, 48, 62, 53,
27, 90, 63, 7, 53, 5, 68, 42, 31, 52, 42, 83, 63, 62, 41, 80, 6,
16, 36, 13, 49, 48, 66, 45, 25, 88, 52, 36, 30, 99, 16, 48, 64, 64,
65, 12, 12, 50, 67, 77, 53, 47, 18, 73, 55, 46, 30, 63, 24]), array([96, 70, 4, 46, 67, 29, 73, 28, 42, 82, 98, 20, 56, 71, 30, 53, 68,
74, 29, 47, 85, 4, 49, 57, 45, 16, 97, 16, 17, 30, 13, 77, 34, 17,
14, 29, 85, 22, 10, 82, 79, 77, 18, 32, 34, 94, 32, 41, 33, 71, 8,
55, 23, 55, 39, 80, 58, 75, 87, 48, 4, 18, 96, 45, 3, 81, 5, 92,
61, 23, 98, 9, 0, 28, 44, 85, 26, 17, 39, 72, 79, 2, 19, 60, 82,
14, 7, 41, 52, 17, 49, 71, 53, 51, 26, 77, 68, 20, 40, 57]), array([61, 9, 24, 28, 56, 63, 63, 17, 47, 0, 38, 67, 46, 55, 11, 57, 46,
70, 14, 35, 36, 84, 50, 21, 21, 48, 28, 47, 71, 7, 94, 25, 96, 37,
23, 98, 39, 0, 36, 2, 41, 95, 46, 44, 10, 67, 61, 92, 84, 49, 89,
22, 13, 53, 6, 56, 17, 6, 61, 52, 4, 41, 34, 50, 14, 73, 2, 36,
82, 38, 48, 62, 77, 83, 44, 26, 89, 31, 49, 4, 78, 45, 11, 12, 71,
23, 90, 97, 88, 29, 97, 12, 81, 99, 43, 85, 62, 76, 75, 41]), array([58, 94, 6, 7, 99, 65, 18, 17, 42, 99, 93, 48, 68, 6, 34, 50, 30,
42, 18, 40, 6, 6, 1, 17, 79, 99, 55, 65, 91, 32, 19, 16, 64, 93,
42, 7, 78, 29, 25, 64, 93, 35, 22, 98, 4, 80, 21, 96, 61, 48, 86,
31, 83, 59, 89, 42, 75, 95, 67, 40, 87, 12, 64, 0, 5, 17, 69, 99,
9, 72, 11, 65, 40, 86, 68, 71, 11, 86, 67, 64, 77, 53, 42, 57, 85,
92, 53, 98, 6, 59, 51, 99, 24, 71, 55, 64, 38, 52, 95, 79]), array([10, 95, 87, 17, 83, 9, 31, 47, 24, 50, 72, 73, 48, 45, 58, 24, 49,
91, 20, 32, 62, 15, 30, 80, 57, 33, 26, 54, 58, 35, 4, 26, 94, 27,
55, 96, 28, 24, 25, 57, 80, 5, 98, 6, 10, 59, 0, 97, 75, 69, 97,
20, 23, 49, 98, 93, 54, 17, 52, 41, 69, 44, 95, 27, 42, 18, 82, 17,
48, 70, 56, 84, 86, 87, 59, 68, 58, 2, 46, 41, 15, 51, 12, 94, 54,
18, 40, 50, 7, 97, 7, 36, 16, 94, 87, 77, 58, 98, 6, 66]), array([ 7, 12, 95, 55, 94, 4, 26, 3, 68, 74, 15, 25, 2, 23, 16, 80, 11,
18, 88, 0, 32, 34, 76, 11, 49, 96, 20, 75, 11, 37, 43, 74, 90, 78,
51, 42, 4, 77, 40, 51, 73, 11, 16, 82, 70, 56, 94, 54, 3, 17, 37,
95, 13, 13, 94, 65, 6, 10, 92, 16, 15, 42, 10, 11, 11, 74, 3, 33,
5, 83, 2, 26, 31, 67, 88, 90, 6, 57, 44, 34, 70, 20, 24, 27, 41,
65, 42, 94, 25, 78, 31, 48, 75, 53, 7, 11, 90, 11, 40, 99]), array([71, 99, 89, 27, 4, 97, 67, 75, 99, 7, 80, 28, 71, 67, 67, 27, 88,
16, 53, 95, 49, 53, 56, 89, 25, 1, 45, 86, 23, 15, 46, 28, 75, 8,
11, 57, 67, 42, 21, 78, 49, 51, 68, 0, 98, 36, 76, 87, 6, 45, 94,
52, 5, 29, 72, 92, 92, 41, 30, 33, 37, 53, 80, 16, 55, 45, 89, 62,
84, 12, 21, 92, 59, 6, 39, 54, 4, 22, 29, 44, 19, 92, 81, 52, 73,
86, 41, 76, 82, 22, 51, 54, 66, 99, 25, 32, 39, 77, 26, 13]), array([43, 11, 47, 44, 57, 40, 81, 0, 37, 73, 27, 62, 29, 5, 78, 39, 30,
13, 8, 12, 16, 57, 54, 21, 72, 0, 36, 67, 51, 66, 81, 99, 26, 41,
38, 39, 90, 92, 87, 52, 89, 40, 16, 96, 1, 3, 38, 86, 32, 35, 82,
57, 47, 7, 36, 73, 88, 48, 15, 19, 85, 30, 28, 76, 45, 93, 91, 83,
6, 85, 37, 38, 76, 21, 57, 93, 58, 63, 78, 16, 69, 16, 71, 51, 27,
20, 94, 85, 68, 71, 95, 38, 8, 29, 6, 55, 36, 59, 37, 87]), array([70, 16, 0, 88, 37, 76, 98, 16, 46, 20, 96, 76, 94, 46, 46, 73, 17,
29, 98, 76, 96, 83, 80, 17, 40, 69, 27, 66, 54, 85, 87, 37, 58, 82,
25, 76, 27, 99, 2, 60, 89, 80, 93, 7, 16, 27, 38, 77, 35, 91, 76,
64, 54, 46, 27, 51, 85, 31, 94, 18, 37, 79, 21, 10, 91, 28, 37, 59,
30, 46, 43, 2, 97, 5, 18, 43, 85, 77, 52, 7, 89, 46, 33, 87, 24,
6, 35, 60, 80, 97, 52, 11, 32, 47, 5, 6, 16, 85, 12, 52]), array([40, 13, 57, 35, 71, 49, 37, 60, 87, 22, 81, 39, 52, 92, 3, 34, 37,
18, 28, 36, 25, 40, 45, 22, 33, 88, 65, 30, 63, 45, 13, 0, 2, 62,
3, 43, 47, 54, 2, 24, 35, 91, 53, 56, 34, 30, 43, 11, 56, 27, 82,
19, 9, 70, 56, 99, 62, 32, 31, 26, 7, 54, 18, 98, 82, 22, 75, 62,
32, 5, 99, 75, 52, 84, 28, 94, 3, 53, 94, 45, 60, 61, 48, 55, 14,
76, 99, 46, 93, 42, 61, 44, 84, 16, 30, 19, 22, 93, 26, 23]), array([17, 85, 8, 10, 16, 30, 51, 61, 42, 70, 5, 36, 2, 55, 97, 77, 29,
13, 98, 64, 74, 21, 70, 58, 30, 83, 1, 27, 67, 25, 1, 48, 31, 90,
48, 40, 80, 45, 59, 98, 5, 39, 91, 28, 53, 33, 64, 40, 38, 51, 94,
65, 97, 32, 41, 78, 71, 28, 49, 54, 4, 86, 79, 78, 88, 96, 59, 74,
39, 75, 28, 30, 70, 2, 57, 1, 54, 96, 38, 62, 86, 66, 98, 87, 27,
52, 74, 73, 48, 77, 73, 24, 30, 75, 77, 52, 75, 30, 61, 0]), array([56, 11, 67, 2, 62, 52, 2, 73, 94, 65, 94, 88, 73, 30, 44, 44, 75,
17, 22, 69, 78, 25, 5, 73, 2, 91, 81, 52, 66, 97, 13, 88, 68, 33,
56, 29, 34, 59, 53, 15, 90, 87, 30, 59, 12, 54, 25, 86, 13, 56, 0,
1, 47, 60, 49, 86, 72, 49, 6, 1, 7, 43, 89, 77, 78, 57, 82, 42,
13, 84, 34, 4, 39, 22, 36, 31, 30, 43, 83, 24, 78, 63, 80, 96, 22,
83, 3, 28, 73, 91, 20, 3, 73, 26, 58, 66, 84, 82, 12, 89]), array([18, 85, 69, 91, 37, 6, 65, 82, 50, 3, 40, 84, 31, 30, 19, 73, 31,
1, 50, 51, 92, 91, 51, 14, 59, 57, 65, 32, 50, 65, 31, 98, 13, 74,
21, 39, 33, 63, 21, 88, 36, 37, 62, 35, 8, 86, 6, 97, 88, 0, 60,
41, 85, 72, 5, 98, 8, 77, 60, 9, 33, 85, 1, 86, 22, 89, 31, 88,
90, 28, 91, 67, 80, 57, 89, 95, 67, 17, 3, 44, 40, 70, 69, 63, 84,
7, 69, 44, 34, 26, 13, 52, 3, 58, 23, 98, 58, 10, 84, 21]), array([57, 18, 16, 80, 70, 8, 34, 40, 73, 52, 34, 25, 64, 24, 26, 59, 44,
70, 93, 57, 78, 16, 91, 65, 1, 91, 74, 71, 67, 67, 32, 21, 53, 10,
67, 31, 47, 73, 48, 22, 54, 9, 93, 53, 56, 63, 24, 98, 15, 12, 91,
29, 86, 5, 25, 21, 61, 20, 20, 95, 31, 38, 20, 13, 51, 47, 61, 26,
85, 46, 69, 60, 74, 64, 25, 92, 77, 35, 3, 64, 39, 56, 22, 86, 2,
91, 23, 43, 94, 38, 93, 29, 91, 61, 71, 56, 46, 92, 3, 52]), array([96, 22, 30, 91, 71, 26, 20, 94, 42, 72, 63, 43, 25, 87, 40, 44, 41,
10, 29, 54, 95, 97, 59, 63, 92, 49, 97, 68, 31, 35, 59, 20, 45, 64,
9, 69, 81, 93, 88, 32, 90, 76, 84, 85, 34, 48, 29, 36, 24, 51, 80,
89, 79, 15, 63, 53, 49, 20, 41, 0, 20, 39, 21, 99, 77, 9, 34, 96,
29, 89, 80, 98, 58, 98, 23, 83, 53, 27, 90, 92, 94, 95, 65, 42, 33,
35, 2, 5, 64, 79, 61, 4, 81, 72, 67, 43, 96, 59, 77, 16]), array([95, 98, 63, 57, 74, 47, 48, 74, 41, 88, 93, 59, 56, 4, 53, 24, 44,
17, 58, 45, 41, 43, 32, 26, 22, 92, 75, 49, 94, 1, 42, 84, 64, 77,
73, 43, 38, 5, 59, 38, 95, 52, 70, 55, 49, 74, 97, 95, 39, 19, 25,
31, 74, 87, 21, 43, 5, 10, 29, 44, 25, 39, 74, 57, 19, 58, 22, 22,
6, 36, 91, 69, 39, 41, 11, 20, 77, 62, 78, 38, 21, 54, 3, 6, 99,
12, 30, 33, 5, 57, 18, 36, 61, 66, 71, 16, 23, 7, 85, 9]), array([96, 30, 5, 84, 91, 64, 39, 9, 95, 36, 56, 25, 53, 75, 47, 54, 67,
61, 87, 87, 0, 71, 46, 64, 87, 88, 45, 83, 83, 98, 93, 75, 61, 74,
46, 77, 26, 91, 60, 23, 21, 61, 8, 1, 7, 23, 20, 6, 54, 33, 28,
56, 23, 54, 85, 93, 59, 75, 29, 40, 43, 18, 67, 94, 63, 32, 75, 17,
93, 3, 28, 72, 20, 47, 11, 92, 18, 70, 61, 22, 25, 20, 97, 18, 87,
2, 48, 53, 99, 24, 86, 85, 36, 50, 49, 52, 64, 41, 79, 45]), array([51, 11, 98, 23, 94, 94, 19, 22, 51, 2, 37, 15, 2, 46, 19, 31, 10,
72, 92, 26, 59, 37, 35, 33, 38, 99, 66, 80, 8, 71, 25, 62, 47, 53,
21, 64, 98, 64, 39, 15, 19, 1, 52, 16, 49, 92, 10, 40, 43, 93, 5,
70, 33, 54, 86, 17, 42, 33, 32, 47, 51, 14, 79, 35, 20, 75, 88, 62,
99, 86, 71, 27, 8, 47, 23, 65, 86, 52, 72, 97, 49, 94, 13, 75, 32,
68, 15, 27, 13, 15, 59, 78, 83, 32, 68, 22, 31, 92, 0, 80]), array([48, 73, 51, 54, 69, 54, 1, 54, 66, 42, 65, 87, 18, 35, 15, 29, 77,
96, 42, 96, 10, 63, 93, 50, 83, 58, 96, 71, 10, 37, 66, 55, 17, 3,
50, 59, 32, 17, 71, 16, 49, 10, 52, 21, 61, 12, 61, 73, 60, 55, 80,
87, 16, 75, 59, 26, 13, 22, 1, 79, 73, 59, 73, 36, 97, 17, 12, 95,
28, 65, 97, 7, 81, 91, 92, 97, 59, 59, 34, 12, 29, 25, 27, 19, 31,
19, 26, 29, 60, 19, 48, 73, 44, 8, 37, 34, 13, 51, 23, 68])], 'feature_importances_': array([0.02255645, 0.02748645, 0.01537776, 0.03050232, 0.0208906 ,
0.02155879, 0.02768038, 0.03659055, 0.03612084, 0.09769932,
0.07244548, 0.01221973, 0.28028859, 0.0344751 , 0.01857089,
0.14272386, 0.01528442, 0.01722933, 0.0509372 , 0.01936194]), 'n_classes_': 2, 'n_features_in_': 20, 'n_outputs_': 1}Keras
from tensorflow import kerasWARNING:tensorflow:From C:\Users\st004186\Envs\gingado\lib\site-packages\keras\src\losses.py:2976: The name tf.losses.sparse_softmax_cross_entropy is deprecated. Please use tf.compat.v1.losses.sparse_softmax_cross_entropy instead.
keras_clf = keras.Sequential()
keras_clf.add(keras.layers.Dense(16, activation='relu', input_shape=(20,)))
keras_clf.add(keras.layers.Dense(8, activation='relu'))
keras_clf.add(keras.layers.Dense(1, activation='sigmoid'))
keras_clf.compile(optimizer='sgd', loss='binary_crossentropy')
keras_clf.fit(X, y, batch_size=10, epochs=10)WARNING:tensorflow:From C:\Users\st004186\Envs\gingado\lib\site-packages\keras\src\backend.py:873: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.
WARNING:tensorflow:From C:\Users\st004186\Envs\gingado\lib\site-packages\keras\src\optimizers\__init__.py:309: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.
Epoch 1/10
WARNING:tensorflow:From C:\Users\st004186\Envs\gingado\lib\site-packages\keras\src\utils\tf_utils.py:492: The name tf.ragged.RaggedTensorValue is deprecated. Please use tf.compat.v1.ragged.RaggedTensorValue instead.
1/10 [==>...........................] - ETA: 8s - loss: 0.721610/10 [==============================] - 1s 3ms/step - loss: 0.7106
Epoch 2/10
1/10 [==>...........................] - ETA: 0s - loss: 0.654510/10 [==============================] - 0s 4ms/step - loss: 0.6968
Epoch 3/10
1/10 [==>...........................] - ETA: 0s - loss: 0.687010/10 [==============================] - 0s 2ms/step - loss: 0.6850
Epoch 4/10
1/10 [==>...........................] - ETA: 0s - loss: 0.712610/10 [==============================] - 0s 2ms/step - loss: 0.6756
Epoch 5/10
1/10 [==>...........................] - ETA: 0s - loss: 0.600810/10 [==============================] - 0s 2ms/step - loss: 0.6654
Epoch 6/10
1/10 [==>...........................] - ETA: 0s - loss: 0.658510/10 [==============================] - 0s 2ms/step - loss: 0.6569
Epoch 7/10
1/10 [==>...........................] - ETA: 0s - loss: 0.624210/10 [==============================] - 0s 2ms/step - loss: 0.6481
Epoch 8/10
1/10 [==>...........................] - ETA: 0s - loss: 0.590610/10 [==============================] - 0s 2ms/step - loss: 0.6399
Epoch 9/10
1/10 [==>...........................] - ETA: 0s - loss: 0.648210/10 [==============================] - 0s 2ms/step - loss: 0.6335
Epoch 10/10
1/10 [==>...........................] - ETA: 0s - loss: 0.489010/10 [==============================] - 0s 2ms/step - loss: 0.6267<keras.src.callbacks.History at 0x1d10aea88b0>model_doc_keras = ModelCard()
model_doc_keras.read_model(keras_clf)
model_doc_keras.show_json()['model_details']['info']'{"class_name": "Sequential", "config": {"name": "sequential", "layers": [{"module": "keras.layers", "class_name": "InputLayer", "config": {"batch_input_shape": [null, 20], "dtype": "float32", "sparse": false, "ragged": false, "name": "dense_input"}, "registered_name": null}, {"module": "keras.layers", "class_name": "Dense", "config": {"name": "dense", "trainable": true, "dtype": "float32", "batch_input_shape": [null, 20], "units": 16, "activation": "relu", "use_bias": true, "kernel_initializer": {"module": "keras.initializers", "class_name": "GlorotUniform", "config": {"seed": null}, "registered_name": null}, "bias_initializer": {"module": "keras.initializers", "class_name": "Zeros", "config": {}, "registered_name": null}, "kernel_regularizer": null, "bias_regularizer": null, "activity_regularizer": null, "kernel_constraint": null, "bias_constraint": null}, "registered_name": null, "build_config": {"input_shape": [null, 20]}}, {"module": "keras.layers", "class_name": "Dense", "config": {"name": "dense_1", "trainable": true, "dtype": "float32", "units": 8, "activation": "relu", "use_bias": true, "kernel_initializer": {"module": "keras.initializers", "class_name": "GlorotUniform", "config": {"seed": null}, "registered_name": null}, "bias_initializer": {"module": "keras.initializers", "class_name": "Zeros", "config": {}, "registered_name": null}, "kernel_regularizer": null, "bias_regularizer": null, "activity_regularizer": null, "kernel_constraint": null, "bias_constraint": null}, "registered_name": null, "build_config": {"input_shape": [null, 16]}}, {"module": "keras.layers", "class_name": "Dense", "config": {"name": "dense_2", "trainable": true, "dtype": "float32", "units": 1, "activation": "sigmoid", "use_bias": true, "kernel_initializer": {"module": "keras.initializers", "class_name": "GlorotUniform", "config": {"seed": null}, "registered_name": null}, "bias_initializer": {"module": "keras.initializers", "class_name": "Zeros", "config": {}, "registered_name": null}, "kernel_regularizer": null, "bias_regularizer": null, "activity_regularizer": null, "kernel_constraint": null, "bias_constraint": null}, "registered_name": null, "build_config": {"input_shape": [null, 8]}}]}, "keras_version": "2.15.0", "backend": "tensorflow"}'Other models
Native support for automatic documentation of other model types, such as from fastai, pytorch is expected to be available in future versions. Until then, any models coded form scratch by the user as well as any other model can be documented by passing the information as an argument to the Documenter’s fill_model_info method. This can be done with a string or dictionary. For example:
import numpy as np
import torch
import torch.nn.functional as Fclass MockDataset(torch.utils.data.Dataset):
def __init__(self, X, y):
self.X = torch.from_numpy(X.astype(np.float32))
self.y = torch.from_numpy(y.astype(np.float32))
self.len = self.X.shape[0]
def __len__(self):
return self.len
def __getitem__(self, idx):
return self.X[idx], self.y[idx]
class PytorchNet(torch.nn.Module):
def __init__(self):
super(PytorchNet, self).__init__()
self.layer1 = torch.nn.Linear(20, 16)
self.layer2 = torch.nn.Linear(16, 8)
self.layer3 = torch.nn.Linear(8, 1)
def forward(self, x):
x = torch.relu(self.layer1(x))
x = torch.relu(self.layer2(x))
x = torch.sigmoid(self.layer3(x))
return x
pytorch_clf = PytorchNet()
dataloader = MockDataset(X, y)
loss_func = torch.nn.BCELoss()
optimizer = torch.optim.SGD(pytorch_clf.parameters(), lr=0.001, momentum=0.9)
for epoch in range(10):
running_loss = 0.0
for i, data in enumerate(dataloader, 0):
_X, _y = data
optimizer.zero_grad()
y_pred_epoch = pytorch_clf(_X)
loss = loss_func(y_pred_epoch, _y.reshape(1))
loss.backward()
optimizer.step()model_doc_pytorch = ModelCard()
model_doc_pytorch.fill_model_info("This model is a neural network consisting of two fully connected layers and ending in a linear layer with a sigmoid activation")
model_doc_pytorch.show_json()['model_details']['info']'This model is a neural network consisting of two fully connected layers and ending in a linear layer with a sigmoid activation'Creating a custom Documenter
gingado users can easily transform their model documentation needs into a Documenter object. The main advantages of doing this are:
- the documentation template becomes a “recyclable” object that can be saved, loaded, and used in other models or code routines; and
- model documentation can be more closely aligned with model creation and training, thus decreasing the probability that the model and its documentation diverge during the process of model development.
A gingado Documenter must:
- subclass
ggdModelDocumentation(or implement all its methods if the user does not want to keep a dependency togingado), - include the actual template for the documentation as a dictionary (with at most two levels of keys) in a class attribute called
template, - ensure that
templatecomplies with JSON specifications, - have
file_path,autofillandindent_levelas arguments in__init__, - follow the
scikit-learnconvention of storing the__init__parameters inselfattributes with the same name, and - implement the
autofill_templatemethod using thefill_infomethod to set the automatically filled information fields.
References
Mitchell, Margaret, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2018. “Model Cards for Model Reporting.” CoRR abs/1810.03993. http://arxiv.org/abs/1810.03993.