Menu
Home
People
Places
Arts
History
Plants & Animals
Science
Life & Culture
Technology
Reference.org
DataOps
open-in-new
<h2 id="history">History</h2><p> DataOps was first introduced by Lenny Liebmann, Contributing Editor, <a href="/facts/InformationWeek/dfQvdt6b">InformationWeek</a>, in a blog post on the IBM Big Data & Analytics Hub titled "3 reasons why DataOps is essential for <a href="/facts/Big_data/XGtrVg4z">big data</a> success" on June 19, 2014.<a class="footnote-ref" id="fnref:9" href="#fn:9"><sup>9</sup></a> The term DataOps was later popularized by Andy Palmer of Tamr and Steph Locke.<a class="footnote-ref" id="fnref:10" href="#fn:10"><sup>10</sup></a><a class="footnote-ref" id="fnref:11" href="#fn:11"><sup>11</sup></a> DataOps is a moniker for "Data Operations."<a class="footnote-ref" id="fnref:12" href="#fn:12"><sup>12</sup></a> 2017 was a significant year for DataOps with significant ecosystem development, analyst coverage, increased keyword searches, surveys, publications, and open source projects.<a class="footnote-ref" id="fnref:13" href="#fn:13"><sup>13</sup></a> Gartner named DataOps on the Hype Cycle for Data Management in 2018.<a class="footnote-ref" id="fnref:14" href="#fn:14"><sup>14</sup></a> </p> <h2 id="goals-and-philosophy">Goals and philosophy</h2> <p>The volume of data is forecast to grow at a rate of 32% CAGR to 180 Zettabytes by the year 2025 (Source: IDC).<a class="footnote-ref" id="fnref:15" href="#fn:15"><sup>15</sup></a> DataOps seeks to provide the tools, processes, and organizational structures to cope with this significant increase in data.<a class="footnote-ref" id="fnref:16" href="#fn:16"><sup>16</sup></a> Automation streamlines data preboarding, ingestion, and the management of large integrated databases, freeing the data team to develop new analytics in a more efficient and effective way.<a class="footnote-ref" id="fnref:17" href="#fn:17"><sup>17</sup></a><a class="footnote-ref" id="fnref:18" href="#fn:18"><sup>18</sup></a> DataOps seeks to increase velocity, reliability, and quality of data analytics.<a class="footnote-ref" id="fnref:19" href="#fn:19"><sup>19</sup></a> It emphasizes communication, collaboration, integration, automation, measurement and cooperation between <a href="/facts/Data_scientists/AHYUo0Rn">data scientists</a>, analysts, data/ETL (<a href="/facts/Extract,_transform,_load/WRaRgJL5">extract, transform, load</a>) engineers, <a href="/facts/Information_technology/glMzunwe">information technology (IT)</a>, and quality assurance/governance. </p> <h2 id="implementation">Implementation</h2> <p>Toph Whitmore at Blue Hill Research offers these DataOps leadership principles for the <a href="/facts/Information_technology/glMzunwe">information technology</a> department:<a class="footnote-ref" id="fnref:20" href="#fn:20"><sup>20</sup></a> </p> <ul><li><i>“Establish progress and performance measurements at every stage of the data flow. Where possible, benchmark data-flow cycle times.</i></li> <li><i>Define rules for an abstracted semantic layer. Ensure everyone is “speaking the same language” and agrees upon what the data (and metadata) is and is not.</i></li> <li><i>Validate with the “eyeball test”: Include continuous-improvement -oriented human feedback loops. Consumers must be able to trust the data, and that can only come with incremental validation.</i></li> <li>Automate as many stages of the data flow as possible including BI, data science, and analytics.</li> <li><i>Using benchmarked performance information, identify bottlenecks and then optimize for them. This may require investment in commodity hardware, or automation of a formerly-human-delivered data-science step in the process.</i></li> <li><i>Establish governance discipline, with a particular focus on two-way data control, data ownership, transparency, and comprehensive <a href="/facts/Data_lineage/cKtPUrde">data lineage</a> tracking through the entire workflow.</i></li> <li><i>Design process for growth and extensibility. The data flow model must be designed to accommodate volume and variety of data. Ensure enabling technologies are priced affordably to scale with that enterprise data growth.”</i></li></ul> <h2 id="events">Events</h2> <ul><li>Data Opticon<a class="footnote-ref" id="fnref:21" href="#fn:21"><sup>21</sup></a></li> <li>Data Ops Summit<a class="footnote-ref" id="fnref:22" href="#fn:22"><sup>22</sup></a></li> <li>Data Ops Online Champion<a class="footnote-ref" id="fnref:23" href="#fn:23"><sup>23</sup></a></li></ul> <h2 id="references">References</h2> <ol> <li id="fn:1"><p>Ereth, Julian (2018). "DataOps-Towards a Definition" (PDF). Proceedings of LWDA 2018: 109. <a href="http://ceur-ws.org/Vol-2191/paper13.pdf" target="_blank">http://ceur-ws.org/Vol-2191/paper13.pdf</a> <a href="#fnref:1" class="footnote-back-ref">↩</a></p></li> <li id="fn:2"><p>"DataOps – It's a Secret". www.datasciencecentral.com. Retrieved 2017-04-05. <a href="https://www.datasciencecentral.com/profiles/blogs/dataops-it-s-a-secret" target="_blank">https://www.datasciencecentral.com/profiles/blogs/dataops-it-s-a-secret</a> <a href="#fnref:2" class="footnote-back-ref">↩</a></p></li> <li id="fn:3"><p>"What is DataOps (data operations)? - Definition from WhatIs.com". SearchDataManagement. Retrieved 2017-04-05. <a href="http://searchdatamanagement.techtarget.com/definition/DataOps" target="_blank">http://searchdatamanagement.techtarget.com/definition/DataOps</a> <a href="#fnref:3" class="footnote-back-ref">↩</a></p></li> <li id="fn:4"><p>"From DevOps to DataOps, By Andy Palmer - Tamr Inc". Tamr Inc. 2015-05-07. Archived from the original on 2018-07-12. Retrieved 2017-03-21. <a href="https://web.archive.org/web/20180712103947/https://www.tamr.com/from-devops-to-dataops-by-andy-palmer/" target="_blank">https://web.archive.org/web/20180712103947/https://www.tamr.com/from-devops-to-dataops-by-andy-palmer/</a> <a href="#fnref:4" class="footnote-back-ref">↩</a></p></li> <li id="fn:5"><p>"What is DataOps (data operations)? - Definition from WhatIs.com". SearchDataManagement. Retrieved 2017-04-05. <a href="http://searchdatamanagement.techtarget.com/definition/DataOps" target="_blank">http://searchdatamanagement.techtarget.com/definition/DataOps</a> <a href="#fnref:5" class="footnote-back-ref">↩</a></p></li> <li id="fn:6"><p>"From DevOps to DataOps, By Andy Palmer - Tamr Inc". Tamr Inc. 2015-05-07. Archived from the original on 2018-07-12. Retrieved 2017-03-21. <a href="https://web.archive.org/web/20180712103947/https://www.tamr.com/from-devops-to-dataops-by-andy-palmer/" target="_blank">https://web.archive.org/web/20180712103947/https://www.tamr.com/from-devops-to-dataops-by-andy-palmer/</a> <a href="#fnref:6" class="footnote-back-ref">↩</a></p></li> <li id="fn:7"><p>DataKitchen (2017-03-07). "Lean Manufacturing Secrets that You Can Apply to Data Analytics". Medium. Retrieved 2017-08-24. <a href="https://medium.com/data-ops/lean-manufacturing-secrets-that-you-can-apply-to-data-analytics-31d1a319cbf0" target="_blank">https://medium.com/data-ops/lean-manufacturing-secrets-that-you-can-apply-to-data-analytics-31d1a319cbf0</a> <a href="#fnref:7" class="footnote-back-ref">↩</a></p></li> <li id="fn:8"><p>"What is DataOps? | Nexla: Scalable Data Operations Platform for the Machine Learning Age". www.nexla.com. Retrieved 2017-09-07. <a href="https://www.nexla.com/define-dataops/" target="_blank">https://www.nexla.com/define-dataops/</a> <a href="#fnref:8" class="footnote-back-ref">↩</a></p></li> <li id="fn:9"><p>"3 reasons why DataOps is essential for big data success". IBM Big Data & Analytics Hub. Retrieved 2018-08-10. <a href="http://www.ibmbigdatahub.com/blog/3-reasons-why-dataops-essential-big-data-success" target="_blank">http://www.ibmbigdatahub.com/blog/3-reasons-why-dataops-essential-big-data-success</a> <a href="#fnref:9" class="footnote-back-ref">↩</a></p></li> <li id="fn:10"><p>Mango Solutions: #DataOps - it's a thing (honest), retrieved 2021-06-28 <a href="https://www.youtube.com/watch?v=64PIa9gcuh0" target="_blank">https://www.youtube.com/watch?v=64PIa9gcuh0</a> <a href="#fnref:10" class="footnote-back-ref">↩</a></p></li> <li id="fn:11"><p>"From DevOps to DataOps, By Andy Palmer - Tamr Inc". Tamr Inc. 2015-05-07. Archived from the original on 2018-07-12. Retrieved 2017-03-21. <a href="https://web.archive.org/web/20180712103947/https://www.tamr.com/from-devops-to-dataops-by-andy-palmer/" target="_blank">https://web.archive.org/web/20180712103947/https://www.tamr.com/from-devops-to-dataops-by-andy-palmer/</a> <a href="#fnref:11" class="footnote-back-ref">↩</a></p></li> <li id="fn:12"><p>"What is DataOps (data operations)? - Definition from WhatIs.com". SearchDataManagement. Retrieved 2017-04-05. <a href="http://searchdatamanagement.techtarget.com/definition/DataOps" target="_blank">http://searchdatamanagement.techtarget.com/definition/DataOps</a> <a href="#fnref:12" class="footnote-back-ref">↩</a></p></li> <li id="fn:13"><p>DataKitchen (2017-12-19). "2017: The Year of DataOps". data-ops. Retrieved 2018-01-24. <a href="https://medium.com/data-ops/2017-the-year-of-dataops-b2023c17d2af" target="_blank">https://medium.com/data-ops/2017-the-year-of-dataops-b2023c17d2af</a> <a href="#fnref:13" class="footnote-back-ref">↩</a></p></li> <li id="fn:14"><p>"Gartner Hype Cycle for Data Management Positions Three Technologies in the Innovation Trigger Phase in 2018". Gartner. Retrieved 2019-07-19. <a href="https://www.gartner.com/en/newsroom/press-releases/2018-09-11-gartner-hype-cycle-for-data-management-positions-three-technologies-in-the-innovation-trigger-phase-in-2018" target="_blank">https://www.gartner.com/en/newsroom/press-releases/2018-09-11-gartner-hype-cycle-for-data-management-positions-three-technologies-in-the-innovation-trigger-phase-in-2018</a> <a href="#fnref:14" class="footnote-back-ref">↩</a></p></li> <li id="fn:15"><p>"What is DataOps? | Nexla: Scalable Data Operations Platform for the Machine Learning Age". www.nexla.com. Retrieved 2017-09-07. <a href="https://www.nexla.com/define-dataops/" target="_blank">https://www.nexla.com/define-dataops/</a> <a href="#fnref:15" class="footnote-back-ref">↩</a></p></li> <li id="fn:16"><p>"What is DataOps? | Nexla: Scalable Data Operations Platform for the Machine Learning Age". www.nexla.com. Retrieved 2017-09-07. <a href="https://www.nexla.com/define-dataops/" target="_blank">https://www.nexla.com/define-dataops/</a> <a href="#fnref:16" class="footnote-back-ref">↩</a></p></li> <li id="fn:17"><p>"5 trends driving Big Data in 2017". CIO Dive. Retrieved 2017-09-07. <a href="http://www.ciodive.com/news/5-trends-driving-big-data-in-2017/445239/" target="_blank">http://www.ciodive.com/news/5-trends-driving-big-data-in-2017/445239/</a> <a href="#fnref:17" class="footnote-back-ref">↩</a></p></li> <li id="fn:18"><p>"From DevOps to DataOps, By Andy Palmer - Tamr Inc". Tamr Inc. 2015-05-07. Archived from the original on 2018-07-12. Retrieved 2017-03-21. <a href="https://web.archive.org/web/20180712103947/https://www.tamr.com/from-devops-to-dataops-by-andy-palmer/" target="_blank">https://web.archive.org/web/20180712103947/https://www.tamr.com/from-devops-to-dataops-by-andy-palmer/</a> <a href="#fnref:18" class="footnote-back-ref">↩</a></p></li> <li id="fn:19"><p>"Unravel Data Advances Application Performance Management for Big Data". Database Trends and Applications. 2017-03-10. Retrieved 2017-09-07. <a href="http://www.dbta.com/Editorial/News-Flashes/Unravel-Data-Advances-Application-Performance-Management-for-Big-Data-116893.aspx" target="_blank">http://www.dbta.com/Editorial/News-Flashes/Unravel-Data-Advances-Application-Performance-Management-for-Big-Data-116893.aspx</a> <a href="#fnref:19" class="footnote-back-ref">↩</a></p></li> <li id="fn:20"><p>"DataOps – It's a Secret". www.datasciencecentral.com. Retrieved 2017-04-05. <a href="https://www.datasciencecentral.com/profiles/blogs/dataops-it-s-a-secret" target="_blank">https://www.datasciencecentral.com/profiles/blogs/dataops-it-s-a-secret</a> <a href="#fnref:20" class="footnote-back-ref">↩</a></p></li> <li id="fn:21"><p>"DataOpticon - YouTube". www.youtube.com. Retrieved 2021-06-28. <a href="https://www.youtube.com/channel/UCAnJYf1L59gVp2SAeHMce1A/featured" target="_blank">https://www.youtube.com/channel/UCAnJYf1L59gVp2SAeHMce1A/featured</a> <a href="#fnref:21" class="footnote-back-ref">↩</a></p></li> <li id="fn:22"><p>"DataOps Summit". www.dataopssummit-sf.com. Archived from the original on 2021-07-02. Retrieved 2021-06-28. <a href="https://web.archive.org/web/20210702063917/https://www.dataopssummit-sf.com/about/" target="_blank">https://web.archive.org/web/20210702063917/https://www.dataopssummit-sf.com/about/</a> <a href="#fnref:22" class="footnote-back-ref">↩</a></p></li> <li id="fn:23"><p>Intelligence, Corinium Global. "DataOps Champions Online 2021 | Corinium". dco-dataops.coriniumintelligence.com. Retrieved 2021-06-28. <a href="https://dco-dataops.coriniumintelligence.com/" target="_blank">https://dco-dataops.coriniumintelligence.com/</a> <a href="#fnref:23" class="footnote-back-ref">↩</a></p></li> </ol>