Utils

Functions to support the use of gingado

Support for model documentation

get_datetime

get_datetime ()

Returns the time now

d = get_datetime()
assert isinstance(d, str)
assert len(d) > 0

read_attr

read_attr (obj)

Reads and yields the type and values of fitted attributes from an object.

Args:
    obj: Object from which attributes will be read.

Function read_attr helps gingado Documenters to read the object behind the scenes.

It collects the type of estimator, and any attributes resulting from fitting an object (in ie, those that end in “_” without being double underscores).

For example, the attributes of an untrained and a trained random forest are, in sequence:

from sklearn.ensemble import RandomForestRegressor

rf_unfit = RandomForestRegressor(n_estimators=3)
rf_fit = RandomForestRegressor(n_estimators=3)\
    .fit([[1, 0], [0, 1]], [[0.5], [0.5]]) # random numbers
list(read_attr(rf_unfit)), list(read_attr(rf_fit))

([{'_estimator_type': 'regressor'}],
 [{'_estimator_type': 'regressor'},
  {'estimator_': DecisionTreeRegressor()},
  {'estimators_': [DecisionTreeRegressor(max_features=1.0, random_state=1283971169),
    DecisionTreeRegressor(max_features=1.0, random_state=926731869),
    DecisionTreeRegressor(max_features=1.0, random_state=2118836830)]},
  {'estimators_samples_': [array([1, 1]), array([0, 0]), array([0, 1])]},
  {'feature_importances_': array([0., 0.])},
  {'n_features_in_': 2},
  {'n_outputs_': 1}])

Support for time series

Objects of the class Lag are similar to scikit-learn’s transformers.

Lag

Lag (lags=1, jump=0, keep_contemporaneous_X=False)

A transformer for lagging variables.

Args:
    lags (int): The number of lags to apply.
    jump (int): The number of initial observations to skip before applying the lag.
    keep_contemporaneous_X (bool): Whether to keep the contemporaneous values of X in the output.

fit

fit (self, X: numpy.ndarray, y=None)

Fits the Lag transformer.

Args:
    X (np.ndarray): Array-like data of shape (n_samples, n_features).
    y: Array-like data of shape (n_samples,) or (n_samples, n_targets) or None.
    
Returns:
    self: A fitted version of the `Lag` instance.

transform

transform (self, X: numpy.ndarray)

Applies the lag transformation to the dataset `X`.

Args:
    X (np.ndarray): Array-like data of shape (n_samples, n_features).
    
Returns:
    A lagged version of `X`.

fit_transform

fit_transform (self, X, y=None, **fit_params)

Fit to data, then transform it.

Fits transformer to `X` and `y` with optional parameters `fit_params`
and returns a transformed version of `X`.

Parameters
----------
X : array-like of shape (n_samples, n_features)
    Input samples.

y :  array-like of shape (n_samples,) or (n_samples, n_outputs),                 default=None
    Target values (None for unsupervised transformations).

**fit_params : dict
    Additional fit parameters.

Returns
-------
X_new : ndarray array of shape (n_samples, n_features_new)
    Transformed array.

The code below demonstrates how Lag works in practice. Note in particular that, because Lag is a transformer, it can be used as part of a scikit-learn’s Pipeline.

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

randomX = np.random.rand(15, 2)
randomY = np.random.rand(15)

lags = 3
jump = 2

pipe = Pipeline([
    ('scaler', StandardScaler()),
    ('lagger', Lag(lags=lags, jump=jump, keep_contemporaneous_X=False))
]).fit_transform(randomX, randomY)

Below we confirm that the lagger removes the correct number of rows corresponding to the lagged observations:

assert randomX.shape[0] - lags - jump == pipe.shape[0]

And because Lag is a transformer, its parameters (lags and jump) can be calibrated using hyperparameter tuning to achieve the best performance for a model.

Support for data augmentation with SDMX

Note

please note that working with SDMX may take some minutes depending on the amount of information you are downloading.

list_SDMX_sources

list_SDMX_sources ()

Fetches the list of SDMX sources.

Returns:
    The list of codes representing the SDMX sources available for data download.

sources = list_SDMX_sources()
print(sources)

assert len(sources) > 0
# all elements are of type 'str'
assert sum([isinstance(src, str) for src in sources]) == len(sources)

['ABS', 'ABS_XML', 'BBK', 'BIS', 'CD2030', 'ECB', 'EC_COMP', 'EC_EMPL', 'EC_GROW', 'ESTAT', 'ILO', 'IMF', 'INEGI', 'INSEE', 'ISTAT', 'LSD', 'NB', 'NBB', 'OECD', 'SGR', 'SPC', 'STAT_EE', 'UNICEF', 'UNSD', 'WB', 'WB_WDI']

list_all_dataflows

list_all_dataflows (codes_only: bool = False, return_pandas: bool = True)

Lists all SDMX dataflows. Note: When using as a parameter to an `AugmentSDMX` object
or to the `load_SDMX_data` function, set `codes_only=True`"

Args:
    codes_only (bool): Whether to return only the dataflow codes.
    return_pandas (bool): Whether to return the result in a pandas DataFrame format.
    
Returns:
    All available dataflows for all SDMX sources.

dflows = list_all_dataflows(return_pandas=False)

assert isinstance(dflows, dict)
all_sources = list_SDMX_sources()
assert len([s for s in dflows.keys() if s in all_sources]) == len(dflows.keys())

list_all_dataflows returns by default a pandas Series, facilitating data discovery by users like so:

dflows = list_all_dataflows(return_pandas=True)
assert type(dflows) == pd.core.series.Series

dflows

ABS_XML  ABORIGINAL_POP_PROJ                 Projected population, Aboriginal and Torres St...
         ABORIGINAL_POP_PROJ_REMOTE          Projected population, Aboriginal and Torres St...
         ABS_ABORIGINAL_POPPROJ_INDREGION    Projected population, Aboriginal and Torres St...
         ABS_ACLD_LFSTATUS                   Australian Census Longitudinal Dataset (ACLD):...
         ABS_ACLD_TENURE                     Australian Census Longitudinal Dataset (ACLD):...
                                                                   ...                        
UNSD     DF_UNData_UNFCC                                                       SDMX_GHG_UNDATA
WB       DF_WITS_Tariff_TRAINS                                WITS - UNCTAD TRAINS Tariff Data
         DF_WITS_TradeStats_Development                             WITS TradeStats Devlopment
         DF_WITS_TradeStats_Tariff                                      WITS TradeStats Tariff
         DF_WITS_TradeStats_Trade                                        WITS TradeStats Trade
Name: dataflow, Length: 11082, dtype: object

This format allows for more easily searching dflows by source:

list_all_dataflows(codes_only=True, return_pandas=True)

ABS_XML  0                 ABORIGINAL_POP_PROJ
         1          ABORIGINAL_POP_PROJ_REMOTE
         2    ABS_ABORIGINAL_POPPROJ_INDREGION
         3                   ABS_ACLD_LFSTATUS
         4                     ABS_ACLD_TENURE
                            ...               
UNSD     3                     DF_UNData_UNFCC
WB       0               DF_WITS_Tariff_TRAINS
         1      DF_WITS_TradeStats_Development
         2           DF_WITS_TradeStats_Tariff
         3            DF_WITS_TradeStats_Trade
Name: dataflow, Length: 11082, dtype: object

dflows['BIS']

WS_CBPOL_D                                    Policy rates daily
WS_CBPOL_M                                  Policy rates monthly
WS_CBS_PUB                              BIS consolidated banking
WS_CPMI_CASHLESS                   CPMI cashless payments (T5-6)
WS_CPMI_CT1                       CPMI comparative tables type 1
WS_CPMI_CT2                       CPMI comparative tables type 2
WS_CPMI_DEVICES                             CPMI payment devices
WS_CPMI_INSTITUTIONS                           CPMI institutions
WS_CPMI_MACRO                                         CPMI Macro
WS_CPMI_PARTICIPANTS                           CPMI participants
WS_CPMI_SYSTEMS         CPMI systems (T8-9-11-13-14-16-17-18-19)
WS_CPP                                Commercial property prices
WS_CREDIT_GAP                             BIS credit-to-GDP gaps
WS_DEBT_SEC2_PUB                             BIS debt securities
WS_DER_OTC_TOV                          OTC derivatives turnover
WS_DPP                      Detailed residential property prices
WS_DSR                                    BIS debt service ratio
WS_EER_D                      BIS effective exchange rates daily
WS_EER_M                    BIS effective exchange rates monthly
WS_GLI                               Global liquidity indicators
WS_LBS_D_PUB                              BIS locational banking
WS_LONG_CPI                             BIS long consumer prices
WS_NA_SEC_DSS                         Debt securities statistics
WS_OTC_DERIV2                        OTC derivatives outstanding
WS_SPP                      BIS property prices: selected series
WS_TC                            BIS long series on total credit
WS_XRU                           US dollar exchange rates, m,q,a
WS_XRU_D                         US dollar exchange rates, daily
WS_XTD_DERIV                         Exchange traded derivatives
Name: dataflow, dtype: object

Or the user can search dataflows by their human-readable name instead of their code. For example, this is one way to see if any dataflow has information on interest rates:

dflows[dflows.str.contains('Interest rates', case=False)]

BBK    BBSDI          Discount interest rates pursuant to section 25...
       BBSEI          Expectation of inflation rate and expected rea...
ECB    RIR                                        Retail Interest Rates
ESTAT  EI_MFIR_M                          Interest rates - monthly data
       ENPE_IRT_ST                          Money market interest rates
       IRT_ST_A               Money market interest rates - annual data
       IRT_ST_M              Money market interest rates - monthly data
       IRT_ST_Q            Money market interest rates - quarterly data
       TEIMF100                  Day-to-day money market interest rates
Name: dataflow, dtype: object

The function load_SDMX_data is a convenience function that downloads data from SDMX sources (and any specific dataflows passed as arguments) if they match the key and parameters set by the user.

load_SDMX_data

load_SDMX_data (sources: dict, keys: dict, params: dict, verbose: bool = True)

Loads datasets from SDMX.

Args:
    sources (dict): A dictionary with the sources and dataflows per source.
    keys (dict): The keys to be used in the SDMX query.
    params (dict): The parameters to be used in the SDMX query.
    verbose (bool): Whether to communicate download steps to the user.
    
Returns:
    A pandas DataFrame with data from SDMX or None if no data matches the sources, keys, and parameters.

df = load_SDMX_data(sources={'ECB': 'CISS', 'BIS': 'WS_CBPOL_D'}, keys={'FREQ': 'D'}, params={'startPeriod': 2003})

assert type(df) == pd.DataFrame
assert df.shape[0] > 0
assert df.shape[1] > 0

Querying data from ECB's dataflow 'CISS' - Composite Indicator of Systemic Stress...
Querying data from BIS's dataflow 'WS_CBPOL_D' - Policy rates daily...