Data for GenAI, a Different Ball Game
Data governance is a crucial aspect of any project involving the handling of data, especially in the field of artificial intelligence. In GenAI projects, which blend genetics and AI, the importance of data governance is even more pronounced. Proper data governance ensures that data is accurate, secure, and accessible, ultimately leading to better decision-making and more reliable outcomes. The following text is an extract from a series we are publishing on our BA4BI Blog: https://ba4bi.blogspot.com/2025/02/data-governance-for-genai.html
Data quality, data consistency and data security enforcement
In old school data governance policies, data quality (DQ) is first about complying with specs and only then does “fit for purpose” comes in as the deciding criterion as I described in Business Analysis for Business Intelligence(1):
Data quality for BI purpose is defined and gauged with reference to fitness for purpose as defined by the analytical use of the data and complying with three levels of data quality as defined by:
[Level 1] database administrators
[Level 2] data warehouse architects
[Level 3] business intelligence analysts
On level 1, data quality is narrowed down to data integrity or the degree to which the attributes of an instance describe the instance accurately and whether the attributes are valid, i.e. comply with defined ranges or definitions managed by the business users. This definition remains very close to the transaction view.
On level 2, data quality is expressed as the percentage completeness and correctness of the analytical perspectives. In other words, to what degree is each dimension, each fact table complete enough to produce significant information for analytical purpose? Issues like sparsity and spreads in the data values are harder to tackle. Timeliness and consistency need to be controlled and managed on the data warehouse level.
On level 3, data quality is the measure in which the available data are capable of adequately answering the business questions. Some use the criterion of accessibility with regards to the usability and clarity of the data. Although this seems a somewhat vague definition, it is most relevant to anyone with some analytical mileage on his odometer. I remember a vast data mining project in a mail order company producing the following astonishing result: 99.9% of all dresses sold were bought by women!
In GenAI, we can pay few attention to the aforementioned level 1 while emphasizing the higher level aspects of data quality. And there, the true challenge lies in testing the validity of three interacting aspects of GenAI data: quality, quantity and density. As mentioned above: quality in the sense of “fit-for-use-case” reducing bias and detecting trustworthy sources, quantity by guaranteeing sufficient data to include all -expected and non-expected- patterns and finally density: to make sure the language model can deliver meaningful proximity measures between the concepts in the data set.
For the complete article on data governance in GenAI projects, visit https://ba4bi.blogspot.com/2025/02/data-governance-for-genai.html.
For more information visit:
Enterprise Data Architects | Lingua Franca Consulting | The Netherlands
https://www.linguafrancaconsulting.eu/
+31(0)114 700 210
Terneuzen, Netherlands
Lingua Franca proves the disconnect between business and ICT can be remedied. Enterprise architecture as well as business analysis and data science cases prove it. Just ask our clients!
Lingua Franca is the Benelux and North France’s partner of KNIME Analytics Platform and KNIME Business Hub; The free local and enterprise low code analytics and data science tool for both Fortune 500 as well as small business since the KNIME Analytics Platform can be downloaded and used for free! Check out the use cases Lingua Franca has worked on.
