v0.8.42

Highlights

User Experience

• Improved Search Experience - preview cards now display usage and freshness information
• Update to Schema History - incorporated Community feedback to remove “Blame” terminology
• Improved UI-Based Ingestion - easily configure metadata ingestion from Snowflake, BigQuery, Looker, and Tableau with an easy-to-follow form; YAML is still supported!

Developer/Community Experience

• Python 3.6 is no longer supported for ingestion – we expect this to impact fewer than 1% of DataHub users (based on PyPi download stats). Please upgrade to Python 3.7 or newer
• Update to GitHub Issue management - issues will be marked as “Inactive” after 30 days of no activity and will be automatically closed following an additional 30 days of inactivity
• We’ve updated our Slack Guidelines! Read them here

Metadata Ingestion

• You can now test your Snowflake connection via the CLI and UI-based Ingestion to ensure you have proper access levels required for general ingestion, profiling, and usage. We will be expanding this functionality to other cloud-based ingestion sources in upcoming cycles.
• Hard delete will now discover and remove soft deleted entities
• Resolved issue of assertion error with dbt stateful ingestion

Full Commit Log

What's Changed

• feat(quickstart,docs): updates for v0.8.41 by @anshbansal in https://github.com/datahub-project/datahub/pull/5409
• fix(ingest): ensure upgrade checks run async by @shirshanka in https://github.com/datahub-project/datahub/pull/5383
• fix(ingest): pass transport options to usage history looker api calls by @mayurinehate in https://github.com/datahub-project/datahub/pull/5417
• feat(quickstart): moving to official confluent images for m1 by @shirshanka in https://github.com/datahub-project/datahub/pull/5416
• fix(documentation) Fix erratic cursor in documentation editor bug by @chriscollins3456 in https://github.com/datahub-project/datahub/pull/5411
• feat(ui): Supporting enriched search preview + misc improvements by @jjoyce0510 in https://github.com/datahub-project/datahub/pull/5419
• chore: remove unnecessary modules from codebase by @shirshanka in https://github.com/datahub-project/datahub/pull/5420
• fix(ingest): extract usage for dashboards allowed by pattern by @mayurinehate in https://github.com/datahub-project/datahub/pull/5424
• fix(docker): fix kafka-setup command to support same capabilities as … by @shirshanka in https://github.com/datahub-project/datahub/pull/5428
• fix(protobuf): ownership fixes by @leifker in https://github.com/datahub-project/datahub/pull/5425
• fix(ui): add dataset qualifiedName parameter to lineage query by @alexey-kravtsov in https://github.com/datahub-project/datahub/pull/5427
• fix(glossary) Fix dropdown where disabled buttons are still clickable by @chriscollins3456 in https://github.com/datahub-project/datahub/pull/5430
• docs(bigquery): add changelog and unittest for profiling limits by @MugdhaHardikar-GSLab in https://github.com/datahub-project/datahub/pull/5407
• fix(siblings): fixing lineage fetching for siblings & sources by @gabe-lyons in https://github.com/datahub-project/datahub/pull/5415
• fix(ui): Fixing unreleased search preview bugs by @jjoyce0510 in https://github.com/datahub-project/datahub/pull/5432
• feat(ui): Adding Statistics Summary to Dataset + Dashboard Profiles by @jjoyce0510 in https://github.com/datahub-project/datahub/pull/5440
• feat(ingest): add test source connection feature, structured report file by @shirshanka in https://github.com/datahub-project/datahub/pull/5442
• fix(ingest/glue): handle error when generating s3 tags for virtual view tables by @timcosta in https://github.com/datahub-project/datahub/pull/5398
• feat(ingest): model - adding a small extension to support communicati… by @shirshanka in https://github.com/datahub-project/datahub/pull/5429
• fix(bigquery-usage): fix dataset name for sharded table by @MugdhaHardikar-GSLab in https://github.com/datahub-project/datahub/pull/5412
• feat(ingestion) Add new endpoint to test an ingestion connection by @chriscollins3456 in https://github.com/datahub-project/datahub/pull/5438
• feat(cli,build): remove deprecated variables GMS_HOST/_PORT by @anshbansal in https://github.com/datahub-project/datahub/pull/5451
• fix(search): make filters by default an empty list if null by @aditya-radhakrishnan in https://github.com/datahub-project/datahub/pull/5454
• fix(hive): add column comment as a column description by @MugdhaHardikar-GSLab in https://github.com/datahub-project/datahub/pull/5449
• feat(groups): add native groups concept to DataHub by @aditya-radhakrishnan in https://github.com/datahub-project/datahub/pull/5443
• fix(ingest): fix serialization of report to handle nesting by @shirshanka in https://github.com/datahub-project/datahub/pull/5455
• fix(tableau): fix tableau db error, add more logs by @mayurinehate in https://github.com/datahub-project/datahub/pull/5423
• build(deps): bump terser from 5.9.0 to 5.14.2 in /docs-website by @dependabot in https://github.com/datahub-project/datahub/pull/5448
• feat(doc): spark-lineage - Adding spark lineage configuration doc for Amazon EMR by @treff7es in https://github.com/datahub-project/datahub/pull/5459
• feat(schema-history): remove blame language for the schema history feature by @aditya-radhakrishnan in https://github.com/datahub-project/datahub/pull/5457
• Search header: Menu icon alignment by @Ankit-Keshari-Vituity in https://github.com/datahub-project/datahub/pull/5458
• build(deps): bump terser from 4.8.0 to 4.8.1 in /datahub-web-react by @dependabot in https://github.com/datahub-project/datahub/pull/5446
• feat(ingest): snowflake - basic test connection capability by @shirshanka in https://github.com/datahub-project/datahub/pull/5464
• fix(ingest/trino): Avoid exception if $properties table empty or not readable by @glinmac in https://github.com/datahub-project/datahub/pull/5447
• feat(ingest): preflight - Add way to check/upgrade brew package version in preflight if needed by @treff7es in https://github.com/datahub-project/datahub/pull/5435
• fix(build): add base image with gradle wrapper cached by @anshbansal in https://github.com/datahub-project/datahub/pull/5467
• doc(bigquery): groups grants by requirements by @sgomezvillamor in https://github.com/datahub-project/datahub/pull/5468
• fix(docs,build): remove base image not needed, cleanup docs by @anshbansal in https://github.com/datahub-project/datahub/pull/5469
• feat(ui): Partial support for Chart usage by @jjoyce0510 in https://github.com/datahub-project/datahub/pull/5473
• fix(ingest): bigquery: multiproject profiling fix by @treff7es in https://github.com/datahub-project/datahub/pull/5474
• fix(ingest): kafka - revert deps back to < 1.9.0 by @shirshanka in https://github.com/datahub-project/datahub/pull/5476
• feat(docker): support multiplatform image for datahub-upgrade by @shirshanka in https://github.com/datahub-project/datahub/pull/5477
• feat(quickstart): experimental support for backup restore for quickstart by @shirshanka in https://github.com/datahub-project/datahub/pull/5418
• feat(dbt): updating source lineage logic by @gabe-lyons in https://github.com/datahub-project/datahub/pull/5414
• Ingestion: Added form in Big Query type to edit the queries. by @Ankit-Keshari-Vituity in https://github.com/datahub-project/datahub/pull/5431
• docs: fix docsearch config by @hsheth2 in https://github.com/datahub-project/datahub/pull/5479
• Search Results: Added checkbox option to select multiple results at once. by @Ankit-Keshari-Vituity in https://github.com/datahub-project/datahub/pull/5422
• feat(delete): hard delete deletes soft deleted entities by @anshbansal in https://github.com/datahub-project/datahub/pull/5478
• fix(docs): add missing closing marker for note section by @shirshanka in https://github.com/datahub-project/datahub/pull/5480
• fix(build): intermittent failure in github actions by @anshbansal in https://github.com/datahub-project/datahub/pull/5452
• feat(model, ingest): add user email in dashboard user usage counts by @mayurinehate in https://github.com/datahub-project/datahub/pull/5471
• feat(ingest): add support for capability report in snowflake test connection by @mayurinehate in https://github.com/datahub-project/datahub/pull/5472
• feat(build): automatically mark issues as stale to close inactive issues by @anshbansal in https://github.com/datahub-project/datahub/pull/5482
• fix(ingest): loosen confluent-kafka dep requirement by @hsheth2 in https://github.com/datahub-project/datahub/pull/5489
• refactor(ingest): cleanup importlib.import_module calls by @hsheth2 in https://github.com/datahub-project/datahub/pull/5490
• build(ingest): make gradle build less chatty by @hsheth2 in https://github.com/datahub-project/datahub/pull/5491
• fix(ingest): Fixing dbt trino datatypes by @aezomz in https://github.com/datahub-project/datahub/pull/5379
• refactor(ci): use custom action for checking codegen status by @hsheth2 in https://github.com/datahub-project/datahub/pull/5493
• feat(spark-lineage): Support ssl cert disable functionality by @MugdhaHardikar-GSLab in https://github.com/datahub-project/datahub/pull/5488
• docs(auth): fix link to point to new doc by @anshbansal in https://github.com/datahub-project/datahub/pull/5501
• docs(updating-datahub): add note for breaking change in looker usage … by @mayurinehate in https://github.com/datahub-project/datahub/pull/5499
• fix(ingest): cleanup unused flake8 noqa statements by @hsheth2 in https://github.com/datahub-project/datahub/pull/5492
• refactor(ci): refactor Docker build-and-push workflows by @hsheth2 in https://github.com/datahub-project/datahub/pull/5494
• docs(slack) Update to Slack guidelines by @maggiehays in https://github.com/datahub-project/datahub/pull/5504
• feat(cli): delete - make delete easier to use by @anshbansal in https://github.com/datahub-project/datahub/pull/5485
• fix(ingest): use temp dir for file generated during test by @hsheth2 in https://github.com/datahub-project/datahub/pull/5505
• feat(ui) Show Glossary and Domains header links to everyone by @chriscollins3456 in https://github.com/datahub-project/datahub/pull/5506
• Home Page: Flickering Issue on search input field by @Ankit-Keshari-Vituity in https://github.com/datahub-project/datahub/pull/5503
• fix(ingest): respect rest emitter timeout setting by @hsheth2 in https://github.com/datahub-project/datahub/pull/5508
• Search Page: Flickering Issue on search input field by @Ankit-Keshari-Vituity in https://github.com/datahub-project/datahub/pull/5515
• Ingestion: Added form to Looker and Tableau by @Ankit-Keshari-Vituity in https://github.com/datahub-project/datahub/pull/5487
• feat(identity): update azure and okta connectors to emit Origin aspects by @aditya-radhakrishnan in https://github.com/datahub-project/datahub/pull/5495
• feat(ui): Adding Search Select feature(frontend only) by @jjoyce0510 in https://github.com/datahub-project/datahub/pull/5507
• test(ingest): limit GMS retries in test by @hsheth2 in https://github.com/datahub-project/datahub/pull/5509
• fix(ingest)- airflow: update subdag check for compatibility with older Airflow versions by @vcs9 in https://github.com/datahub-project/datahub/pull/5523
• fix(ingest): fix unbound variable bug in cli ingest list-runs by @hsheth2 in https://github.com/datahub-project/datahub/pull/5527
• fix(ui) Display Term Group name properly in Recently Viewed by @chriscollins3456 in https://github.com/datahub-project/datahub/pull/5528
• feat(ingestion) Add frontend connection test for Snowflake by @chriscollins3456 in https://github.com/datahub-project/datahub/pull/5520
• fix(glossary) Fix Glossary success messages and sort Glossary by @chriscollins3456 in https://github.com/datahub-project/datahub/pull/5533
• feat(apache-ranger): Apache Ranger Authorizer support in datahub-gms by @mohdsiddique in https://github.com/datahub-project/datahub/pull/4999
• feat(ingest): add deprecation warning for Python 3.6 by @hsheth2 in https://github.com/datahub-project/datahub/pull/5519
• docs(townhall) add past townhall agendas by @maggiehays in https://github.com/datahub-project/datahub/pull/5536
• feat(ingestion): add groups to ldap users by @alexey-kravtsov in https://github.com/datahub-project/datahub/pull/5470
• chore(issues): reduce time for issues to be marked stale and then closed by @aditya-radhakrishnan in https://github.com/datahub-project/datahub/pull/5537
• fix(ingestion) Set pipeline_name on UI recipes with forms by @chriscollins3456 in https://github.com/datahub-project/datahub/pull/5535
• fix(ui): Correctly hit /logOut on logout by @jjoyce0510 in https://github.com/datahub-project/datahub/pull/5538
• [analytics-tab] fix analytics tab config variable for gms by @salihcaan in https://github.com/datahub-project/datahub/pull/5529
• feat(ui): Support batch adding / remove tags from search lists. (Batch Actions part 2/7) by @jjoyce0510 in https://github.com/datahub-project/datahub/pull/5534
• fix(ingestionSource): improve error experience when ingestion source is in an inconsistent state by @RyanHolstien in https://github.com/datahub-project/datahub/pull/5522
• docs(schema history): Correct typo in schema history documentation by @skylersinclair in https://github.com/datahub-project/datahub/pull/5545
• fix(docker): Fixing dev docker and quickstart by @jjoyce0510 in https://github.com/datahub-project/datahub/pull/5550
• feat(ui): Support Batch adding and removing Glossary Terms (Batch Actions 3/7) by @jjoyce0510 in https://github.com/datahub-project/datahub/pull/5544
• feat(ci): test quickstart works by @anshbansal in https://github.com/datahub-project/datahub/pull/5518
• test(ingest): mark trino/hana tests as xfail due to flakes by @hsheth2 in https://github.com/datahub-project/datahub/pull/5549
• feat(ingestion/superset): add display_uri to config by @milimetric in https://github.com/datahub-project/datahub/pull/5408
• fix(quickstart): failure on a path not being present by @anshbansal in https://github.com/datahub-project/datahub/pull/5554
• fix(dbt): fix issue of assertion error when stateful ingestion is used with dbt tests by @mohdsiddique in https://github.com/datahub-project/datahub/pull/5540

New Contributors

• @timcosta made their first contribution in https://github.com/datahub-project/datahub/pull/5398
• @glinmac made their first contribution in https://github.com/datahub-project/datahub/pull/5447
• @skylersinclair made their first contribution in https://github.com/datahub-project/datahub/pull/5545
• @milimetric made their first contribution in https://github.com/datahub-project/datahub/pull/5408

Full Changelog: https://github.com/datahub-project/datahub/compare/v0.8.41...v0.8.42