Dear Google Cloud Community,
As part of my recent deep dive into Dataform, I sought to understand how it could uplift our data-driven team members and refine their day-to-day data operations. I must say, I'm genuinely enthralled by its capabilities. But as with any robust tool, there's always room for growth. Below, I've outlined a series of enhancements that could potentially elevate the Dataform experience:
- Simplified GitHub Integration: The need for a Personal Access Token or SSH key to synchronize with GitHub makes the process a tad convoluted. A more streamlined approach would be welcome.
- Unit Test Improvements:
- Introduce dataform test functionality directly in the UI, ensuring parity with the CLI.
- Ensure the compatibility of unit tests with incremental updates.
- Documentation Consistency: I've noticed exclusive content on Dataform's main website. Bridging this gap by integrating such resources into Google's documentation would greatly aid accessibility.
- UI Multi-tab Support: The ability to concurrently open multiple SQLX files, along with the freedom to toggle or split views, can make for a more fluid navigation experience.
- Error Mapping in SQLX: Pinpointing errors would be more efficient if they directly matched the corresponding lines in SQLX files.
- Workflow Enhancements: Streamlining the transition from source table to source table test and then to subsequent tables, akin to solutions like DBT, would be beneficial.
- User-friendly Access Control: Rethinking the current access control, especially concerning incremental updates, to be more intuitive would be a step in the right direction.
- Versatile Source Declarations: A provision to declare multiple external sources within a singular SQLX file would add another layer of flexibility.
- Expanding Compiled Graph Accessibility: Beyond the development workspace, visualizing the COMPILED GRAPH on the default branch would be a valued addition.
- Tag Utility for Data Sources: Incorporating tags for data source declarations and ensuring their accurate rendering in the Compiled Graph would heighten clarity.
- Search Capabilities: Implementing search functions, much like on GitHub, would be a game-changer.
- Enhanced GitHub Synchronization: Augmenting this aspect, especially around rebases and conflict resolution, would offer a smoother experience, particularly for those on the learning curve.
Your acknowledgment of these suggestions would be appreciated. If there are existing workarounds to any of these points that I may have overlooked, kindly enlighten me.
I firmly believe that by focusing on these facets, we can unleash the full potential of Dataform, catering to an even wider user base.
Warm regards,
Yong-Jin Lee
LinkedIn
Twitter
