All notable changes to this project will be documented in this file.
and this project adheres to Semantic Versioning.
[0.19.8] - 2023-10-02#
Upgrade deps, better build times due to latest duckdb
[0.19.8] - 2023-06-21#
arrays_as_tableoption added to convert all arrays to their own table.
[0.19.6] - 2023-06-21#
Errors get raised for postgresql conversion.
[0.19.5] - 2023-06-07#
Parquet naming of headers where incorrect for dates.
[0.19.4] - 2023-05-26#
Allow multiple files while downloading from s3
Stop detecting floats where precision is too low.
[0.19.3] - 2023-05-11#
CSV output to S3 broken in some cases.
Stop csv directory being made when using S3
[0.19.1] - 2023-05-10#
JSON Input sources from STDIN, HTTP, S3 and allow all inputs to be GZIPed if have
Command line now accepts multiple files from any source.
S3 output for CSV and PARQUET that is streamed from the input.
json-pathquery to filter JSON objects.
statsproduce statistics on ouput files.
[0.18.0] - 2023-03-17#
Better type guessing for database inserts.
no_linkoption that removes
_linkfields in the output.
[0.17.1] - 2023-01-07#
Truncate cell that is larger than xlsx allows.
Allow more rows in xlsx in non threaded mode.
[0.17.0] - 2022-12-03#
Web Assembly version of libflatterer. Available to use here https://lite.flatterer.dev/.
Upgrade to vue 3 and vite for web frontend.
Ignore blank lines in json lines files
Better errors when too many files are open
Better errors when file could be json-stream
[0.16.2] - 2022-10-25#
Support python 3.11
[0.16.2] - 2022-10-16#
Error not writing larger XLSX files
[0.16.1] - 2022-10-16#
Cors for web api
[0.16.0] - 2022-07-27#
Local web interface for exploring flatterer features
[0.15.0] - 2022-07-27#
evolveoption for sqlite and postgres. Can add data to existing tables and will alter tables if new fields are needed.
No need for output directory when using
postgresoutputs . Will use temp space if not supplied.
sqlite-pathoption will switch on
[0.14.2] - 2022-07-27#
Postgres connection from environment variable
[0.14.1] - 2022-07-23#
sql_scriptoption to export scripts that for sqlite and postgres to make output backward compatable with earlier versions.
[0.14.0] - 2022-07-22#
pushdownoption. Copy data from top level objects down to child (one-to-many) tables. This is useful if the data has its own keys (such as
idfields) that you want to exist in the related tables. Also useful for denormalizing the data so querying on a common field, requires less joining.
postgresoption. Export to postgres database by supplying a connection string.
postgres-schemaoption. Choose a postgres schema to insert data into.
dropoption. If table already exists.
sqlite and postgres scripts in output directory. No longer required as the actual database tables can be created by
[0.13.2] - 2022-06-24#
filesoption, so multiple files can be supplied at once.
[0.13.1] - 2022-05-27#
Threads option now can output xlsx
[0.13.0] - 2022-04-28#
Threads option, so that can be run on all cores. Works best with ndjson input.
Parquet export option.
json-linesoption renamed to
json-streamoption that works in same way as the old json-lines option and accepts concatonated json.
[0.12.12] - 2022-04-15#
Datapackage output uses correct date type
Lists of strings are now escaped the same way as optional quoted CSVs
[0.12.11] - 2022-04-09#
Clearer errors when error happens in rust. BREAKING CHANGE, if catching certain error types in python these may have changed.
datapackage output now has forign keys.
[0.12.10] - 2022-03-21#
Python decimal converted to float not string.
[0.12.9] - 2022-03-07#
SQLite export lower memory use
[0.12.8] - 2022-03-01#
SQLite export has indexes and foreign key contraints.
[0.12.7] - 2022-02-27#
main_table_name was number caused exception
[0.12.6] - 2022-02-26#
list of JSON strings supplied to
datapackage.json named correctly.
[0.12.4] - 2022-02-18#
flattenpython function now accepts iterator
iterator_flatten now deprecated as it is just a subset of `flatten’
[0.12.3] - 2022-02-15#
More lenient if tmp directory can not be deleted.
[0.12.1] - 2022-02-15#
Preview option in python CLI and library.
[0.12] - 2022-02-02#
SQLite export option
[0.11] - 2022-01-25#
Support top level object. All list of objects are streamed and top level object data saved in main table.
New yaglish parser for both json stream and arrays.
Library has schema_guess function to tell if data is a JSON Stream or has an array of object.
Empty objects do not make a line in output.
[0.10.1] - 2021-12-22#
ctrlc support added.
Logging output improved.
Traceback not shown for CLI use.
Occurences where output folder not being deleted.
[0.10] - 2021-12-22#
tables.csvinput in order to control tab names. Tables File option
Beginning to use logging.
Better handling of long excel sheet names names. See https://github.com/kindly/flatterer/issues/12
field_typeno longer required in fields.csv.
_linkfields have are 0 indexed.
Removal of redundant
_link_<main_table>from main table.
More human readable error messages.
Bad characters in XLSX stripped and raise warning.
Check limits on XLSX files and raise error if found.
Lots of edge cases handled better. See https://github.com/kindly/flatterer/issues/17
All errors will remove output directory instead of leaving unusable directory or partial data.
[0.9] - 2021-12-12#
Removed unwrap on channel send, to remove possible panic.
Table ordering of output in JSON input order. Making
fields.csvtable order reflect the input data.
Lib has new
FlatFiles::new_with_dafualts()to make using the library less verbose.
Use insta for more tests.
Lib has preview option, meaning CSV output will optionally only show specified number of lines.
[0.8] - 2021-12-01#
Paths to data in sqlite and postgres start at root of output.
Clippy for linting and insta for tests.
[0.7.1] - 2021-12-01#
Do less work when just exporting Metadata.
Minor speedup due to not using
[0.7] - 2021-11-24#
Ability to add
fields.csvso you can rename column headings.
schema-titlesoption to get field names out of JSONSchema.
Speed improvments using
[0.6.2] - 2021-11-24#
Change to pypi metadata
Tests run in action
Changelog in docs
[0.6.1] - 2021-11-24#
Regression in speed due to new error handling
[0.6] - 2021-11-23#
New error handling using
anyhow, giving errors more context.
[0.5] - 2021-11-22#
[0.4] - 2021-11-20#
Postgresql and sqlite scripts to load CSV data into databases.
Wheel builds for Windows and MacOS, automatically published using github actions.
[0.3] - 2021-10-25#
Inline One to One option to mean that if an array only has one item in for all the data then treat it as sub-object.