Dataset Tools - Restore Dataset from an Algorithm-Ready File

analysis_engine.restore_dataset.restore_dataset will load a dataset from a file, s3 or redis and merge any missing records back in to redis. Use this to restore missing dataset values after a host goes offline or on a fresh install or redis server restart or redis flush.

Restore an algorithm dataset from file, s3 or redis to redis for ensuring all datasets are ready for Algorithmic backtesting

Supported Datasets:

  • SA_DATASET_TYPE_ALGO_READY - Algorithm-ready datasets
analysis_engine.restore_dataset.restore_dataset(show_summary=True, force_restore=False, algo_dataset=None, dataset_type=20000, serialize_datasets=['daily', 'minute', 'quote', 'stats', 'peers', 'news1', 'financials', 'earnings', 'dividends', 'company', 'news', 'calls', 'puts', 'pricing', 'tdcalls', 'tdputs'], path_to_file=None, compress=False, encoding='utf-8', redis_enabled=True, redis_key=None, redis_address=None, redis_db=None, redis_password=None, redis_expire=None, redis_serializer='json', redis_encoding='utf-8', redis_output_db=None, s3_enabled=True, s3_key=None, s3_address=None, s3_bucket=None, s3_access_key=None, s3_secret_key=None, s3_region_name=None, s3_secure=False, slack_enabled=False, slack_code_block=False, slack_full_width=False, datasets_compressed=True, verbose=False)[source]

Restore missing dataset nodes in redis from an algorithm-ready dataset file on disk. Use this to restore redis from scratch.

Parameters:
  • show_summary – optional - show a summary of the algorithm-ready dataset using analysis_engine.show_dataset.show_dataset (default is True)
  • force_restore – optional - boolean - publish whatever is in the algorithm-ready dataset into redis. If False this will ensure that datasets are only set in redis if they are not already set
  • algo_dataset – optional - already loaded algorithm-ready dataset
  • dataset_type – optional - dataset type (default is SA_DATASET_TYPE_ALGO_READY)
  • serialize_datasets – optional - list of dataset names to deserialize in the dataset
  • path_to_file – optional - path to an algorithm-ready dataset in a file
  • compress – optional - boolean flag for decompressing the contents of the path_to_file if necessary (default is False and algorithms use zlib for compression)
  • encoding – optional - string for data encoding

(Optional) Redis connectivity arguments

Parameters:
  • redis_enabled – bool - toggle for auto-caching all datasets in Redis (default is True)
  • redis_key – string - key to save the data in redis (default is None)
  • redis_address – Redis connection string format: host:port (default is localhost:6379)
  • redis_db – Redis db to use (default is 0)
  • redis_password – optional - Redis password (default is None)
  • redis_expire – optional - Redis expire value (default is None)
  • redis_serializer – not used yet - support for future pickle objects in redis
  • redis_encoding – format of the encoded key in redis
  • redis_output_db – optional - integer publish to a separate redis database

(Optional) Minio (S3) connectivity arguments

Parameters:
  • s3_enabled – bool - toggle for auto-archiving on Minio (S3) (default is True)
  • s3_key – string - key to save the data in redis (default is None)
  • s3_address – Minio S3 connection string format: host:port (default is localhost:9000)
  • s3_bucket – S3 Bucket for storing the artifacts (default is dev) which should be viewable on a browser: http://localhost:9000/minio/dev/
  • s3_access_key – S3 Access key (default is trexaccesskey)
  • s3_secret_key – S3 Secret key (default is trex123321)
  • s3_region_name – S3 region name (default is us-east-1)
  • s3_secure – Transmit using tls encryption (default is False)

(Optional) Slack arguments

Parameters:
  • slack_enabled – optional - boolean for publishing to slack
  • slack_code_block – optional - boolean for publishing as a code black in slack
  • slack_full_width – optional - boolean for publishing as a to slack using the full width allowed

Additonal arguments

Parameters:
  • datasets_compressed – optional - boolean for publishing as compressed strings default is True
  • verbose – optional - bool for increasing logging