Whilst the number of multilingual programming teams for clinical study work has been growing, so has the issue of cross-language data compatibility. If care is not taken we risk antiquated or inadequate formats becoming the de facto standard, or worse yet a return to complete interoperability from the use of language specific data formats (Rds, Pickle, etc).
This presentation will discuss a number of open data formats, with a comparative analysis of their strengths and weaknesses. In particular, this presentation will introduce and recommend Apache Parquet, a high performance binary data format that comes with client libraries for most popular analytical programming languages as well as integration into many popular tools and databases. Finally, the talk will touch on our experiences and challenges of trying to integrate Apache Parquet within our data analysis workflows at Roche.
Break
Whilst the number of multilingual programming teams for clinical study work has been growing, so has the issue of cross-language data compatibility. If care is not taken we risk antiquated or inadequate formats becoming the de facto standard, or worse yet a return to complete interoperability from the use of language specific data formats (Rds, Pickle, etc). This presentation will discuss a number of open data formats, with a comparative analysis of their strengths and weaknesses. In particular, this presentation will introduce and recommend Apache Parquet, a high performance binary data format that comes with client libraries for most popular analytical programming languages as well as integration into many popular tools and databases. Finally, the talk will touch on our experiences and challenges of trying to integrate Apache Parquet within our data analysis workflows at Roche.