Hey HN, I’m Etai, together with Andrew a co-founder of Secoda (https://www.secoda.co). Secoda is a collaborative workspace for data teams that makes it easy to share metadata, queries, charts and documentation with any employee.
Companies store a growing amount of knowledge in BI tools, data warehouses, data pipelines, queries and documentation. Because these tools are not connected, it has become more difficult to manage all of this. Even with great practices, organizations still struggle to get value out of their data - up to 73% of all enterprise data goes unused. One of the big contributors to this problem is that organizations create data silos by not documenting and centralizing their data knowledge in a single place where every employee can access information about data.
Today, most data teams end up documenting all this data with Google Sheets or Confluence, which get outdated quickly. Because data documentation is outdated and hard to find, employees struggle to discover, understand and use it. This overwhelms data teams with repetitive questions about how to use and where to find company data.
In our last roles, Andrew and I had a hard time understanding context around different data resources. It was difficult to understand which table to use, what dashboard to trust, who to talk to about a particular metric or why we changed our pricing model. All of this data knowledge was in our data teams head and it made it really difficult to try to work with data. It would take around 2 weeks to get an answer to any data request because the data team was so backed up with questions. This sucked.
Secoda is unique because it's focused on helping the data team curate knowledge for the less technical employee. Data teams can use the tool to curate knowledge for specific departments or roles so that only the right people are able to see the data knowledge that they should see. We currently integrate into data warehouses, BI tools, dbt as well as Airflow and once teams connect their data to Secoda, they can get a comprehensive view of all their data knowledge in one place.
We’d love to hear your feedback or experience with the problem that we're solving and would be thrilled if you would sign up at https://secoda.co to let us know what you think!
This addresses key usability issues with most data initiatives. Don’t be dismayed that people who haven’t tried to share reports with teams don’t get it… I get it and I presume end users will love it. There is a particular need for metadata support in Google data studio that would be great if you could help solve: there are no folders or any real way to curate data outside of report sharing and leaving text boxes on the reports themselves. It’s kind of a nightmare for an otherwise powerful and popular tool.
I’d also like to see comment threads over data discoveries. Like a snapshot of a report with in-context exploration would be super helpful. In my experience the default behavior is a report screenshot dropped into a slack thread, and I know there can be better. QlikView does a decent job at this but it has the trappings of enterprise software. I think that would unlock a lot of value out of reports and give a place for teams to understand opportunities and celebrate wins. Congrats on the launch!
Thanks for the positive message. We appreciate the support from people who understand the problem that we're trying to solve with Secoda. We haven't had someone request Google Data Studio as an integration, but if you were interested in discussing what that would look like please reach out to andrew@secoda.co. It looks like there's an API for accessing assets in Google Data Studio, so it's definitely possible.
Our most requested feature is discussion threads attached to each data resource (table, dashboard, etc) to build context around a resource. So that will be coming in the near future, and we are happy to hear you also think it would be super helpful!
Long term, do you see yourselves in the MDM space [0]? Informally, I think of MDM as "enterprise into analytics", and my first impression of Secoda is "analytics out to enterprise". Schemas and data dictionaries seem central to both. An MDM solution like Tamr looks a lot heavier weight than I think the experience you target.
You've hit the nail on the head here. Long term we want to bridge the gap between the data team and rest of the organization through a central repository of data knowledge. Using MDM's terms the data team = IT, the rest of the organization = business, and repository of data knowledge = master data. So it's almost a one-to-one comparison, and we've never heard of MDM so this is great! We are also trying to make the experience lightweight and user friendly so that everyone is interested in exploring their company's data.
An example of what lineage would look like is the following: You have source tables A and B that are joined to create a model C and then C is used in dashboard D. We are able to infer the lineage A->C->D and B->C->D.
We extract lineage in a couple of different ways. The main way is by parsing SQL queries in your data warehouse to determine which tables and dashboards are upstream/downstream. The other way we extract lineage information provided directly from dbt and BigQuery who have nice APIs for this information.
We are working releasing an API in Q4 that supports pushing information from say an Airflow DAG to Secoda to give us more lineage context. Hopefully this answers your questions.
This looks really cool, but not being able to self-host without an enterprise plan is a deal breaker for us :( We use Metabase right now not because we like it but because we can host it on our own servers, with our own firewalls, behind our own Cloudflare Access rules, etc etc. Understand why you might have chosen to go this route, though. Best of luck!
Hi there, I'm the CTO of Secoda so happy to answer your question. Our dbt integration works with dbt cloud, and are working on making it compatible with dbt core as well. What we'll pull from dbt via their API is the metadata associated with the models, docs, and, jobs. We have a free version of the platform, so you can sign up and test it out if you'd like to see what that information looks like in Secoda.
The catalog portion of the Secoda product is similar to Amundsen, but we also have a Data Dictionary for defining metrics, Analysis Documents for queries and charts, and Requests for handling data questions. We take these different pieces of functionality and make everything interconnected, so that it's one unified repository for your data knowledge.
Additionally, we try and make Secoda easy to use for both technical and non-technical users, whereas a tool like Amundsen is more focused on the technical user.
How does it compare to traditional reporting tools. Lot of modern day BI tools like looker for instance, do have features like this. Where do you position yours company as compared to traditional tools?
The main way that Secoda differs from a traditional reporting tool is that we offer a more complete view into the data knowledge/context of an organization. A traditional reporting tool like Looker does great for reports, but it misses context about how certain models are created, where data is coming from downstream, and more general knowledge that is stored inside of a wiki. Secoda takes all of the context across a data stack and puts it in one central place. So we actually see ourselves as complementary to these reporting tools as we are a layer on top of them.
You can add tags to any data resource in Secoda (table, column, dashboard, dictionary term, etc). When you tag resources, you can search for them by tag. Our customers have found it very useful for keeping everything organized.
Such an important problem - I really hope you become the new standard for companies' data. The number of times I've seen complex, undocumented queries...
Thanks, we hope so too! At Secoda, we really want to make data less intimidating for anyone in the organization to explore and use. When the whole company can use data to help inform their decisions it really does make a big difference.
We're happy to help you or whoever would set it up and show you how other teams have been using the tool. Feel free to shot me an email at etai@secoda.co if you'd like any help along the way
We have pre-built integrations with many popular data tools (Snowflake, dbt, Tableau, etc.) that can be setup in less than 5 minutes. If those don't fit your needs then we can definitely setup some time to discuss the API.
Companies store a growing amount of knowledge in BI tools, data warehouses, data pipelines, queries and documentation. Because these tools are not connected, it has become more difficult to manage all of this. Even with great practices, organizations still struggle to get value out of their data - up to 73% of all enterprise data goes unused. One of the big contributors to this problem is that organizations create data silos by not documenting and centralizing their data knowledge in a single place where every employee can access information about data.
Today, most data teams end up documenting all this data with Google Sheets or Confluence, which get outdated quickly. Because data documentation is outdated and hard to find, employees struggle to discover, understand and use it. This overwhelms data teams with repetitive questions about how to use and where to find company data.
In our last roles, Andrew and I had a hard time understanding context around different data resources. It was difficult to understand which table to use, what dashboard to trust, who to talk to about a particular metric or why we changed our pricing model. All of this data knowledge was in our data teams head and it made it really difficult to try to work with data. It would take around 2 weeks to get an answer to any data request because the data team was so backed up with questions. This sucked.
Secoda is unique because it's focused on helping the data team curate knowledge for the less technical employee. Data teams can use the tool to curate knowledge for specific departments or roles so that only the right people are able to see the data knowledge that they should see. We currently integrate into data warehouses, BI tools, dbt as well as Airflow and once teams connect their data to Secoda, they can get a comprehensive view of all their data knowledge in one place.
We’d love to hear your feedback or experience with the problem that we're solving and would be thrilled if you would sign up at https://secoda.co to let us know what you think!