Environmental, social, and governance (ESG) data. It has become a standard input for institutional investors, asset managers, and private equity firms, shaping investment decisions from first screening to portfolio monitoring and regulatory reporting. Yet the kind of ESG data most investors rely on is often self-reported, slow to update, and hard to compare from one provider to the next.
If you're reading this, you likely use ESG data regularly. You may be comparing ESG data providers, or you may have noticed that a company's ESG rating and its headlines don't always tell the same story. In this guide, we cover where ESG data comes from, why traditional ESG metrics fall short, how AI and natural language processing fill the gaps, and how to choose an ESG data provider.
Who created ESG?
Kofi Annan, former United Nations Secretary-General, invited a group of financial institutions to develop guidance on how to better integrate environmental, social, and governance issues into asset management, securities brokerage, and research. In 2004, they published "Who Cares Wins: Connecting Financial Markets to a Changing World." The work fed into the UN Principles for Responsible Investment, launched in 2006, and helped make ESG criteria a standard part of how investors evaluate companies.
Stronger, more resilient, and sustainable
According to the "Who Cares Wins" report, the contributors were convinced that "in a more globalized, interconnected, and competitive world, the way that environmental, social, and corporate governance issues are managed is part of companies' overall management quality needed to compete successfully."
The endorsing institutions concluded that "a better consideration of environmental, social, and governance factors will ultimately contribute to stronger and more resilient investment markets, as well as contribute to the sustainable development of societies."
Two decades later, that idea supports a large market for ESG data and ESG ratings, and a growing body of regulation, from SFDR and CSRD in Europe to the EU ESG Rating Regulation.
Where does ESG data come from?
ESG data covers a wide range of issues: carbon emissions and other environmental impacts, climate change exposure, labor practices and human rights, product safety, corporate governance, and business ethics. It reaches investors through three main channels.
- Company disclosures and sustainability reporting. Most ESG data starts with what companies publish: sustainability reports, annual filings, and policy documents. ESG frameworks such as the Global Reporting Initiative (GRI) and the Task Force on Climate-related Financial Disclosures (TCFD) aim to make ESG reporting more standardized, and regulations like the EU's CSRD are making parts of it mandatory. Standardization helps with benchmarking, but it doesn't change a basic fact: ESG data mainly comes from the same companies it is meant to evaluate.
- ESG ratings and ESG rating agencies. ESG rating agencies, including some data vendors, such as MSCI, Sustainalytics, and Bloomberg, collect disclosures, add their own research, and turn them into ESG scores. These scores make it easier to compare a company's ESG performance against peers, and they feed into ESG funds, indices, and screening rules.
- Outside-in data. The third channel is what others report about a company: news coverage, NGO investigations, regulatory actions, court records, and trade press. This is where a company's actual conduct shows up, often well before it reaches a sustainability report or a rating. It is also the hardest source to use, because it is spread across millions of documents in dozens of languages.
Today's ESG data challenges
At their core, ESG metrics capture a company's performance on a given ESG issue. Measuring that consistently has proven difficult. In the Journal of Applied Corporate Finance, Sakis Kotsantonis and George Serafeim identified four recurring problems:
- ESG measurement, data collection, and company reporting practices remain inconsistent.
- A lack of transparency in benchmarking makes peer rankings hard to rely on.
- ESG data providers fill data gaps in different ways, which can produce large discrepancies.
- Differences in how providers interpret the data are considerable, and they grow as more data becomes available.
ows up clearly in the ratings themselves. Researchers at MIT Sloan's Aggregate Confusion Project found that ESG ratings from six prominent ESG rating agencies correlate at just 0.54 on average. By comparison, credit ratings from Moody's and Standard & Poor's correlate at 0.92.
Beyond inconsistency, traditional ESG data has four structural blind spots:
- It's self-reported. Companies decide what to disclose, and few volunteer their own controversies, which is what makes greenwashing possible.
- It's slow. Sustainability reports are annual, and ratings often refresh on a similar cycle.
- It dilutes severe issues. Blending many ESG factors into one score lets strong governance metrics offset a serious environmental or human rights breach.
- It misses private companies. Coverage is thin in low-disclosure markets, where much of private equity invests.
Artificial intelligence (AI) to meet rising ESG data demands
Even as the debate around ESG investing has become more polarized, demand for ESG data keeps growing. In BNP Paribas' 2025 ESG survey of 420 institutional investors, 85% said they integrate sustainability criteria into their investment decisions.
"Nearly half of investors (48%) anticipate allocating more budget to their sustainable investment strategy on ESG data acquisition." (BNP Paribas ESG Survey 2025)
Spending more on the same disclosure-based data doesn't close the gaps above. However, advances in machine learning, natural language processing, and large language models make it possible to extract ESG information from unstructured sources such as news, NGO reports, regulatory filings, and court records. This outside-in data surfaces ESG controversies and events as they are reported, often long before they appear in a rating, and it covers companies that publish little or nothing about themselves.
How to get ESG data using natural language processing (NLP)
Natural language processing (NLP) algorithms can read billions of documents across languages, detect which ones describe an ESG issue, and separate a genuine allegation from a passing mention. Large language models add the ability to summarize what happened, map it to ESG frameworks such as SFDR, CSRD, or the UN Global Compact, and assess its severity.
Two steps make the difference between noise and reliable ESG data:
- Attributing each mention to the right company. Entity resolution and a regularly updated knowledge graph link every article to a single, verified entity, including private companies, subsidiaries, and brands. Without this step, a scandal at a namesake ends up in the wrong portfolio.
- Keeping every signal traceable. Each flag should link back to its source documents. As regulations like the EU ESG Rating Regulation raise the bar for transparency, investors need to show where an assessment came from, not just what it says.
Several ESG rating agencies now integrate NLP-derived datasets, and asset managers and private equity firms use them in their ESG risk management. The same approach powers adverse media screening and adverse media monitoring, which banks, insurers, and investors use to check clients and counterparties for reputational and KYC risk.
Why does ESG risk show up in controversies first
An ESG controversy is an adverse event or conduct attributed to a company in public sources: a chemical spill, a corruption probe, a data breach, a forced labor allegation, a misleading climate claim. Controversies are where stated policy meets actual practice. That is why they are often the earliest visible sign of ESG risk, and why investors increasingly track them alongside ESG ratings.
Controversies also capture risks that disclosures tend to understate. Social factors like labor conditions in a supply chain rarely appear in a sustainability report until something goes wrong. Neither does the climate risk behind a delayed decarbonization plan, or the environmental impact of a single project.
Not every controversy matters equally, so severity is the key question. Two factors drive it: whether the harm can be reversed, and how far it reaches. SESAMm's Controversy Exposure Score is built around that idea. It rates a company's exposure to reported ESG controversies on a 0 to 100 scale over a rolling 24-month window. Rather than averaging, it applies "worst-of" logic, so a company's most severe controversy drives the score, whichever pillar it falls under. It measures impact on people and the environment and reflects what has already been reported, which makes it a complement to ESG ratings rather than a prediction.
For ESG due diligence and portfolio monitoring, that shift from what a company says to what is reported about it changes what investors can see, especially for private companies with little or no disclosure.
How to choose an ESG data provider
ESG data providers fall into roughly three groups. Traditional ESG rating agencies score companies mainly from disclosures. ESG reporting and compliance platforms help companies collect and publish their own data. Risk intelligence providers monitor what is reported about companies from the outside in. Many investors combine more than one. Whichever you choose, five questions separate a strong ESG data provider from the rest:
- Does it cover the companies you actually hold? If your portfolio is mostly private or in emerging markets, a provider built on listed-company disclosures will leave much of it uncovered.
- Can you trace every data point to its source? You should be able to click from an ESG score or alert to the documents behind it.
- Is the methodology public? You need to know how ESG metrics are weighted, how severity is judged, and what a score doesn't measure.
- How fast does it update? Annual refreshes suit reporting. Due diligence and monitoring need data that updates as events happen.
- Is it ready for regulation? Ask how the provider supports SFDR, CSRD, and the EU ESG Rating Regulation, and whether it is seeking authorization with ESMA.
For a deeper checklist, see our 7-question buyer's guide to choosing an ESG data provider.
.avif)
.png)