Publish Data to S3 and Redis and Slack and a File

Dataset Publishing API

analysis_engine.publish.publish(data, label=None, convert_to_json=False, is_df=False, output_file=None, df_compress=False, compress=False, redis_enabled=True, redis_key=None, redis_address=None, redis_db=None, redis_password=None, redis_expire=None, redis_serializer='json', redis_encoding='utf-8', s3_enabled=True, s3_key=None, s3_address=None, s3_bucket=None, s3_access_key=None, s3_secret_key=None, s3_region_name=None, s3_secure=False, slack_enabled=False, slack_code_block=False, slack_full_width=False, verbose=False, silent=False, **kwargs)[source]

Publish data to multiple optional endpoints: - a local file path (output_file) - minio (s3_bucket and s3_key) - redis (redis_key) - slack

Returns:

status value
Parameters:
  • data – data to publish
  • convert_to_json – convert data to a json-serialized string. this function will throw if json.dumps(data) fails
  • is_df – convert pd.DataFrame using pd.DataFrame.to_json() to a json-serialized string. this function will throw if to_json() fails
  • label – log tracking label
  • output_file – path to save the data to a file
  • df_compress – optional - compress data that is a pandas.DataFrame before publishing
  • compress – optional - compress before publishing (default is False)
  • verbose – optional - boolean to log output (default is False)
  • silent – optional - boolean no log output (default is False)
  • kwargs – optional - future argument support

(Optional) Redis connectivity arguments

Parameters:
  • redis_enabled – bool - toggle for auto-caching all datasets in Redis (default is True)
  • redis_key – string - key to save the data in redis (default is None)
  • redis_address – Redis connection string format: host:port (default is localhost:6379)
  • redis_db – Redis db to use (default is 0)
  • redis_password – optional - Redis password (default is None)
  • redis_expire – optional - Redis expire value (default is None)
  • redis_serializer – not used yet - support for future pickle objects in redis
  • redis_encoding – format of the encoded key in redis

(Optional) Minio (S3) connectivity arguments

Parameters:
  • s3_enabled – bool - toggle for auto-archiving on Minio (S3) (default is True)
  • s3_key – string - key to save the data in redis (default is None)
  • s3_address – Minio S3 connection string format: host:port (default is localhost:9000)
  • s3_bucket – S3 Bucket for storing the artifacts (default is dev) which should be viewable on a browser: http://localhost:9000/minio/dev/
  • s3_access_key – S3 Access key (default is trexaccesskey)
  • s3_secret_key – S3 Secret key (default is trex123321)
  • s3_region_name – S3 region name (default is us-east-1)
  • s3_secure – Transmit using tls encryption (default is False)

(Optional) Slack arguments

Parameters:
  • slack_enabled – optional - boolean for publishing to slack
  • slack_code_block – optional - boolean for publishing as a code black in slack
  • slack_full_width – optional - boolean for publishing as a to slack using the full width allowed